From 3f99d243fb761f1e840c8560638b67a8dd3775c3 Mon Sep 17 00:00:00 2001 From: Rafael Vuijk Date: Sat, 5 Sep 2026 23:28:45 +0000 Subject: [PATCH 1/2] The growth census is held by a test, not quoted in a remark `Saturation.RulesUpTo` argues in prose from the split of rules across RewriteRuleGrowth, and the figures it quoted were stale by a factor of four: "26 collect, 17 rearrange, 9 expand and 270 are unjudged", and "84% of the library", all measured before the growth declarations went in. The unjudged count went 261 to 152 over that work; the true split is 97 / 45 / 30 / 152. The argument survives the correction -- a ceiling that refuses Unknown still refuses 47% of the rules, so the dial is still whether unjudged rules may fire at all. The arithmetic in it did not, which is the reason to hold a number somewhere that fails when it moves rather than to reason from it in a comment. RuleAuthoringGuideTest already pinned the 152. It pins all four now, so the next time they move this remark fails with them. Full suite 9564 passed, 0 failed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012sonx8iAspMiwRwokT1Ura --- .../Core/Transformations/Saturation.cs | 23 +++++++++++++------ .../Transformations/RuleAuthoringGuideTest.cs | 12 ++++++++++ 2 files changed, 28 insertions(+), 7 deletions(-) diff --git a/Sources/AngouriMath/Core/Transformations/Saturation.cs b/Sources/AngouriMath/Core/Transformations/Saturation.cs index a749af5fe..2d47d69bc 100644 --- a/Sources/AngouriMath/Core/Transformations/Saturation.cs +++ b/Sources/AngouriMath/Core/Transformations/Saturation.cs @@ -43,13 +43,22 @@ internal static class Saturation /// one, and that is a measurement rather than a preference. It reads like the value a /// ceiling should refuse — it means the rule's growth was not judged, because its /// replacement is code rather than a written pattern, so admitting it accepts a rewrite - /// nobody measured. But it is where the rules are: of the sound rules in - /// , 26 collect, 17 rearrange, 9 expand and 270 are - /// unjudged. A ceiling that refuses the fourth value refuses 84% of the library, and - /// what is left cannot do much — over six expression pairs equal only through a larger - /// intermediate form, the ceiling proved two, - /// and — all nine expanding rules added — proved - /// the same two. The third was proved only with the unjudged rules admitted. + /// nobody measured. But it is where the rules are: of the 324 rules in + /// , 97 collect, 45 rearrange, 30 expand and 152 are + /// unjudged. A ceiling that refuses the fourth value still refuses 47% of the library, + /// and what is left could not do much when this was written — over six expression pairs + /// equal only through a larger intermediate form, the + /// ceiling proved two, and + /// proved the same two. The third was proved + /// only with the unjudged rules admitted. + /// + /// + /// Those figures moved and this paragraph did not. It read "26 collect, 17 + /// rearrange, 9 expand and 270 are unjudged", and 84% — measured before the growth + /// declarations went in, which took the unjudged count from 261 to 152. The argument + /// survives the correction and the arithmetic in it did not, which is the reason to state + /// counts rather than a conclusion drawn from them. RuleAuthoringGuideTest holds all + /// four of these now, so the next time they move this remark fails with them. /// /// /// So the dial that matters is not how far a rule may enlarge; it is whether rules nobody diff --git a/Sources/Tests/UnitTests/Core/Transformations/RuleAuthoringGuideTest.cs b/Sources/Tests/UnitTests/Core/Transformations/RuleAuthoringGuideTest.cs index 4ee656e05..f8d34ea28 100644 --- a/Sources/Tests/UnitTests/Core/Transformations/RuleAuthoringGuideTest.cs +++ b/Sources/Tests/UnitTests/Core/Transformations/RuleAuthoringGuideTest.cs @@ -87,6 +87,18 @@ public void ReplacementAPatternOrCode() Stated(152, rules.Count(rule => rule.Growth is RewriteRuleGrowth.Unknown), "how many rules sit at Unknown growth"); + // The rest of the census, because `Saturation.RulesUpTo` argues from it in prose and + // the figures it quoted had gone stale by a factor of four -- "26 collect, 17 + // rearrange, 9 expand and 270 unjudged", measured before the growth declarations + // went in. The argument survived; the arithmetic did not. A number a remark reasons + // from belongs somewhere that fails when it moves. + Stated(97, rules.Count(rule => rule.Growth is RewriteRuleGrowth.Collects), + "how many rules are declared Collects"); + Stated(45, rules.Count(rule => rule.Growth is RewriteRuleGrowth.Rearranges), + "how many rules are declared Rearranges"); + Stated(30, rules.Count(rule => rule.Growth is RewriteRuleGrowth.Expands), + "how many rules are declared Expands"); + // And the two the document names. By their own names rather than by what the prose // calls them: the first is `squared-sine-and-cosine-of-one-argument-sum-to-one` and // the prose calls it the Pythagorean identity, which is right and is not what to From d4003d3831db8d295cdbf8d657b8a4864ffb394d Mon Sep 17 00:00:00 2001 From: Rafael Vuijk Date: Sun, 6 Sep 2026 00:29:55 +0000 Subject: [PATCH 2/2] The 1915th, and every release beside it on one machine The performance column the release checklist asks for, and the first one where all five entries in key-commits.txt were measured -- one benchmark, five kernels, one machine, one hour. The four release-pair sections below it had to approximate that by measuring two commits at a time. v2.1.0 and v2.2.0 appear for the first time. They did not build until #1178. Since 2.4.0 the Simplify and Solve family is up 0.15% to 0.65% -- SimplifyHard +0.65%, SolveMediumHard +0.31%, SolveHard +0.23%. Small, consistent in sign, and above the noise on those rows by three orders of magnitude. Well inside PerformanceGate's 3%, so not a reason to hold a release; recorded because the alternative is claiming the period was free. The likeliest cause is the definedness conditions of #1176 and #1177, which give every cancelled quotient a larger condition tree. CompileEasy and CompileHard are flat against 2.4.0 and about half what they were at 2.3.0. And a rider on this file's own central claim, which the run happened to be able to test. It says allocation is deterministic -- "the same commit measured twice gives the same bytes". The five columns were run twice, for an unrelated reason, and that is true for the large benchmarks and false for the small ones: SolveHard @ v2.1.0 1,356,524,520 -> 1,356,525,608 0.00008% SimplifyHard @ v2.1.0 3,733,177,488 -> 3,733,136,616 0.001% CompileEasy @ v2.1.0 16,280 -> 16,519 1.47% CompileEasy @ v2.3.0 16,510 -> 16,206 1.84% The compile benchmarks have an allocation floor of about 2%. That matters because PerformanceGate fails at 3%: for CompileEasy that is one and a half noise-widths of margin rather than the comfortable threshold it reads as. A CompileEasy allocation move under 2% is not evidence; the same move on SolveHard is evidence a thousand times over. The script also gained a warning it needed. A branch name resolves to whatever this checkout's copy of it points at, and in a worktree the local `master` belongs to whichever checkout last moved it -- the first run of this table measured a `master` five days behind its remote and labelled the column `master`. Only the SHA printed beside the row gave it away. It now says so when a name resolves behind its remote-tracking counterpart. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012sonx8iAspMiwRwokT1Ura --- .../WhatsNew/version_performance_control.md | 64 +++++++++++++++++++ Sources/Utils/benchmark_key_commits.sh | 11 ++++ 2 files changed, 75 insertions(+) diff --git a/Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md b/Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md index d12f9b97e..fae34c8bf 100644 --- a/Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md +++ b/Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md @@ -180,6 +180,70 @@ honest reason -- a compiled delegate over `Complex` has nothing to put on the he --- +## The 1915th, and every release beside it on one machine + +The first column measured by `Sources/Utils/benchmark_key_commits.sh` reaching all five entries in +`key-commits.txt`. Every figure below is the same benchmark against a different kernel, run on one +machine within one hour — which is what the four release-pair sections further down had to +approximate by measuring two commits at a time. + +`v2.1.0` and `v2.2.0` are in it for the first time. They did not build before #1178: the benchmark +project carried cases reaching internals those kernels had not exposed, so the whole project failed +to compile and took the row with it, while the case that runs uses nothing but the public API. + +### Allocation + +| benchmark | v2.1.0 | v2.2.0 | v2.3.0 | v2.4.0 | 1915th | +|---|--:|--:|--:|--:|--:| +| `CompileEasy` | 16,519 | 16,309 | 16,206 | 11,003 | 10,970 | +| `CompileHard` | 38,542 | 38,089 | 38,355 | 20,396 | 20,434 | +| `Derivate` | 52,823 | 52,823 | 52,823 | 52,823 | 52,823 | +| `EvalTrig` | 1,341,377 | 1,341,377 | 1,341,377 | 1,341,377 | 1,341,377 | +| `EvalTrigPrecise` | 12,742,205 | 12,742,205 | 12,742,205 | 12,742,205 | 12,742,205 | +| `ParseEasy` | 18,061 | 18,061 | 18,061 | 18,109 | 18,104 | +| `ParseHard` | 3,498,137 | 3,498,137 | 3,522,409 | 3,522,457 | 3,531,201 | +| `SimplifyEasy` | 128,564 | 128,564 | 128,100 | 128,100 | 128,100 | +| `SimplifyHard` | 3,733,136,616 | 3,749,449,104 | 3,620,748,288 | 3,632,015,744 | 3,655,695,192 | +| `SolveEasy` | 20,220,106 | 20,234,998 | 8,852,269 | 8,853,671 | 8,853,834 | +| `SolveEasyMedium` | 80,184 | 95,873 | 95,791 | 97,193 | 97,335 | +| `SolveHard` | 1,356,525,608 | 1,486,580,536 | 1,431,859,752 | 1,446,799,616 | 1,450,058,720 | +| `SolveMedium` | 554,368 | 658,259 | 658,099 | 661,963 | 662,923 | +| `SolveMediumHard` | 155,191,480 | 171,395,096 | 162,305,664 | 164,549,808 | 165,055,344 | + +Bytes allocated. `EvalEasy`, `RunEasy`, `RunMedium` and `RunHard` allocate nothing in every column +and are left out. + +**Since 2.4.0, the Simplify and Solve family is up between 0.15% and 0.65%** — `SimplifyHard` ++0.65%, `SolveMediumHard` +0.31%, `SolveHard` +0.23%. Small, consistent in sign, and real: see the +next section for why 0.65% is far above the noise on those particular rows. It is well inside +`PerformanceGate`'s 3% and is not a reason to hold a release; it is a reason not to claim the period +was free. The likeliest cause is the definedness conditions added in #1176 and #1177, which give +every cancelled quotient a larger condition tree to carry. + +`CompileEasy` and `CompileHard` are flat against 2.4.0 and roughly **half** what they were at 2.3.0. + +### The determinism claim, measured + +This file says allocation is deterministic — "the same commit measured twice gives the same bytes". +The five columns above were run **twice**, and that is true for the large benchmarks and false for +the small compile ones: + +| same commit, two runs | run 1 | run 2 | apart | +|---|--:|--:|--:| +| `SolveHard` @ v2.1.0 | 1,356,524,520 | 1,356,525,608 | **0.00008%** | +| `SimplifyHard` @ v2.1.0 | 3,733,177,488 | 3,733,136,616 | **0.001%** | +| `CompileEasy` @ v2.1.0 | 16,280 | 16,519 | **1.47%** | +| `CompileEasy` @ v2.3.0 | 16,510 | 16,206 | **1.84%** | + +So the rule this file states holds where it matters most and needs a rider: **the compile +benchmarks have an allocation floor of about 2%.** That is worth knowing precisely because +`PerformanceGate` fails at 3% — for `CompileEasy` that is one and a half noise-widths of margin, not +the comfortable threshold the number suggests. A `CompileEasy` allocation move under 2% is not +evidence; the same move on `SolveHard` is evidence a thousand times over. + +Timings from the same run are in the generated table and are deliberately not reproduced here, for +the reason the next section gives. + ## The 1844th: what this file's timings can and cannot support The column for 2.4.0. Seventy-five commits from the 1769th, carrying the rule-set exchange -- diff --git a/Sources/Utils/benchmark_key_commits.sh b/Sources/Utils/benchmark_key_commits.sh index 31fb8daa6..014c5053a 100755 --- a/Sources/Utils/benchmark_key_commits.sh +++ b/Sources/Utils/benchmark_key_commits.sh @@ -50,6 +50,17 @@ echo "measuring:"; echo "$commits" | sed 's/^/ /' for commit in $commits; do resolved=$(git -C "$repository" rev-parse --short "$commit" 2>/dev/null) + + # A branch name resolves to whatever this checkout's copy of it points at, which is not + # necessarily the tip. In a worktree the local `master` belongs to whichever checkout last + # moved it: this measured a `master` five days behind its remote and reported it as `master`, + # and only the SHA printed beside the row gave it away. Say so rather than quietly measure + # the wrong commit. + upstream=$(git -C "$repository" rev-parse --short "origin/$commit" 2>/dev/null) + if [ -n "$upstream" ] && [ "$upstream" != "$resolved" ]; then + echo "=== $commit: local is $resolved, origin/$commit is $upstream -- measuring the local one" >&2 + echo " put the SHA you mean in key-commits.txt if that is not what you want" >&2 + fi if [ -z "$resolved" ]; then echo "=== $commit: not a commit in this repository" echo "unresolved" > "$results/$commit.status"