diff --git a/Sources/AngouriMath/Core/Transformations/Saturation.cs b/Sources/AngouriMath/Core/Transformations/Saturation.cs index a749af5fe..2d47d69bc 100644 --- a/Sources/AngouriMath/Core/Transformations/Saturation.cs +++ b/Sources/AngouriMath/Core/Transformations/Saturation.cs @@ -43,13 +43,22 @@ internal static class Saturation /// one, and that is a measurement rather than a preference. It reads like the value a /// ceiling should refuse — it means the rule's growth was not judged, because its /// replacement is code rather than a written pattern, so admitting it accepts a rewrite - /// nobody measured. But it is where the rules are: of the sound rules in - /// , 26 collect, 17 rearrange, 9 expand and 270 are - /// unjudged. A ceiling that refuses the fourth value refuses 84% of the library, and - /// what is left cannot do much — over six expression pairs equal only through a larger - /// intermediate form, the ceiling proved two, - /// and — all nine expanding rules added — proved - /// the same two. The third was proved only with the unjudged rules admitted. + /// nobody measured. But it is where the rules are: of the 324 rules in + /// , 97 collect, 45 rearrange, 30 expand and 152 are + /// unjudged. A ceiling that refuses the fourth value still refuses 47% of the library, + /// and what is left could not do much when this was written — over six expression pairs + /// equal only through a larger intermediate form, the + /// ceiling proved two, and + /// proved the same two. The third was proved + /// only with the unjudged rules admitted. + /// + /// + /// Those figures moved and this paragraph did not. It read "26 collect, 17 + /// rearrange, 9 expand and 270 are unjudged", and 84% — measured before the growth + /// declarations went in, which took the unjudged count from 261 to 152. The argument + /// survives the correction and the arithmetic in it did not, which is the reason to state + /// counts rather than a conclusion drawn from them. RuleAuthoringGuideTest holds all + /// four of these now, so the next time they move this remark fails with them. /// /// /// So the dial that matters is not how far a rule may enlarge; it is whether rules nobody diff --git a/Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md b/Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md index d12f9b97e..fae34c8bf 100644 --- a/Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md +++ b/Sources/AngouriMath/Docs/WhatsNew/version_performance_control.md @@ -180,6 +180,70 @@ honest reason -- a compiled delegate over `Complex` has nothing to put on the he --- +## The 1915th, and every release beside it on one machine + +The first column measured by `Sources/Utils/benchmark_key_commits.sh` reaching all five entries in +`key-commits.txt`. Every figure below is the same benchmark against a different kernel, run on one +machine within one hour — which is what the four release-pair sections further down had to +approximate by measuring two commits at a time. + +`v2.1.0` and `v2.2.0` are in it for the first time. They did not build before #1178: the benchmark +project carried cases reaching internals those kernels had not exposed, so the whole project failed +to compile and took the row with it, while the case that runs uses nothing but the public API. + +### Allocation + +| benchmark | v2.1.0 | v2.2.0 | v2.3.0 | v2.4.0 | 1915th | +|---|--:|--:|--:|--:|--:| +| `CompileEasy` | 16,519 | 16,309 | 16,206 | 11,003 | 10,970 | +| `CompileHard` | 38,542 | 38,089 | 38,355 | 20,396 | 20,434 | +| `Derivate` | 52,823 | 52,823 | 52,823 | 52,823 | 52,823 | +| `EvalTrig` | 1,341,377 | 1,341,377 | 1,341,377 | 1,341,377 | 1,341,377 | +| `EvalTrigPrecise` | 12,742,205 | 12,742,205 | 12,742,205 | 12,742,205 | 12,742,205 | +| `ParseEasy` | 18,061 | 18,061 | 18,061 | 18,109 | 18,104 | +| `ParseHard` | 3,498,137 | 3,498,137 | 3,522,409 | 3,522,457 | 3,531,201 | +| `SimplifyEasy` | 128,564 | 128,564 | 128,100 | 128,100 | 128,100 | +| `SimplifyHard` | 3,733,136,616 | 3,749,449,104 | 3,620,748,288 | 3,632,015,744 | 3,655,695,192 | +| `SolveEasy` | 20,220,106 | 20,234,998 | 8,852,269 | 8,853,671 | 8,853,834 | +| `SolveEasyMedium` | 80,184 | 95,873 | 95,791 | 97,193 | 97,335 | +| `SolveHard` | 1,356,525,608 | 1,486,580,536 | 1,431,859,752 | 1,446,799,616 | 1,450,058,720 | +| `SolveMedium` | 554,368 | 658,259 | 658,099 | 661,963 | 662,923 | +| `SolveMediumHard` | 155,191,480 | 171,395,096 | 162,305,664 | 164,549,808 | 165,055,344 | + +Bytes allocated. `EvalEasy`, `RunEasy`, `RunMedium` and `RunHard` allocate nothing in every column +and are left out. + +**Since 2.4.0, the Simplify and Solve family is up between 0.15% and 0.65%** — `SimplifyHard` ++0.65%, `SolveMediumHard` +0.31%, `SolveHard` +0.23%. Small, consistent in sign, and real: see the +next section for why 0.65% is far above the noise on those particular rows. It is well inside +`PerformanceGate`'s 3% and is not a reason to hold a release; it is a reason not to claim the period +was free. The likeliest cause is the definedness conditions added in #1176 and #1177, which give +every cancelled quotient a larger condition tree to carry. + +`CompileEasy` and `CompileHard` are flat against 2.4.0 and roughly **half** what they were at 2.3.0. + +### The determinism claim, measured + +This file says allocation is deterministic — "the same commit measured twice gives the same bytes". +The five columns above were run **twice**, and that is true for the large benchmarks and false for +the small compile ones: + +| same commit, two runs | run 1 | run 2 | apart | +|---|--:|--:|--:| +| `SolveHard` @ v2.1.0 | 1,356,524,520 | 1,356,525,608 | **0.00008%** | +| `SimplifyHard` @ v2.1.0 | 3,733,177,488 | 3,733,136,616 | **0.001%** | +| `CompileEasy` @ v2.1.0 | 16,280 | 16,519 | **1.47%** | +| `CompileEasy` @ v2.3.0 | 16,510 | 16,206 | **1.84%** | + +So the rule this file states holds where it matters most and needs a rider: **the compile +benchmarks have an allocation floor of about 2%.** That is worth knowing precisely because +`PerformanceGate` fails at 3% — for `CompileEasy` that is one and a half noise-widths of margin, not +the comfortable threshold the number suggests. A `CompileEasy` allocation move under 2% is not +evidence; the same move on `SolveHard` is evidence a thousand times over. + +Timings from the same run are in the generated table and are deliberately not reproduced here, for +the reason the next section gives. + ## The 1844th: what this file's timings can and cannot support The column for 2.4.0. Seventy-five commits from the 1769th, carrying the rule-set exchange -- diff --git a/Sources/Tests/UnitTests/Core/Transformations/RuleAuthoringGuideTest.cs b/Sources/Tests/UnitTests/Core/Transformations/RuleAuthoringGuideTest.cs index 4ee656e05..f8d34ea28 100644 --- a/Sources/Tests/UnitTests/Core/Transformations/RuleAuthoringGuideTest.cs +++ b/Sources/Tests/UnitTests/Core/Transformations/RuleAuthoringGuideTest.cs @@ -87,6 +87,18 @@ public void ReplacementAPatternOrCode() Stated(152, rules.Count(rule => rule.Growth is RewriteRuleGrowth.Unknown), "how many rules sit at Unknown growth"); + // The rest of the census, because `Saturation.RulesUpTo` argues from it in prose and + // the figures it quoted had gone stale by a factor of four -- "26 collect, 17 + // rearrange, 9 expand and 270 unjudged", measured before the growth declarations + // went in. The argument survived; the arithmetic did not. A number a remark reasons + // from belongs somewhere that fails when it moves. + Stated(97, rules.Count(rule => rule.Growth is RewriteRuleGrowth.Collects), + "how many rules are declared Collects"); + Stated(45, rules.Count(rule => rule.Growth is RewriteRuleGrowth.Rearranges), + "how many rules are declared Rearranges"); + Stated(30, rules.Count(rule => rule.Growth is RewriteRuleGrowth.Expands), + "how many rules are declared Expands"); + // And the two the document names. By their own names rather than by what the prose // calls them: the first is `squared-sine-and-cosine-of-one-argument-sum-to-one` and // the prose calls it the Pythagorean identity, which is right and is not what to diff --git a/Sources/Utils/benchmark_key_commits.sh b/Sources/Utils/benchmark_key_commits.sh index 31fb8daa6..014c5053a 100755 --- a/Sources/Utils/benchmark_key_commits.sh +++ b/Sources/Utils/benchmark_key_commits.sh @@ -50,6 +50,17 @@ echo "measuring:"; echo "$commits" | sed 's/^/ /' for commit in $commits; do resolved=$(git -C "$repository" rev-parse --short "$commit" 2>/dev/null) + + # A branch name resolves to whatever this checkout's copy of it points at, which is not + # necessarily the tip. In a worktree the local `master` belongs to whichever checkout last + # moved it: this measured a `master` five days behind its remote and reported it as `master`, + # and only the SHA printed beside the row gave it away. Say so rather than quietly measure + # the wrong commit. + upstream=$(git -C "$repository" rev-parse --short "origin/$commit" 2>/dev/null) + if [ -n "$upstream" ] && [ "$upstream" != "$resolved" ]; then + echo "=== $commit: local is $resolved, origin/$commit is $upstream -- measuring the local one" >&2 + echo " put the SHA you mean in key-commits.txt if that is not what you want" >&2 + fi if [ -z "$resolved" ]; then echo "=== $commit: not a commit in this repository" echo "unresolved" > "$results/$commit.status"