Skip to content

HermitCrab optimization round record: what was tried, and a faster package that is not mergeable - #490

Open
johnml1135 wants to merge 34 commits into
masterfrom
docs/hc-optimization-ledger
Open

johnml1135 wants to merge 34 commits into
masterfrom
docs/hc-optimization-ledger

Conversation

@johnml1135

@johnml1135 johnml1135 commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

Quick summary

Preserves the HermitCrab optimization round's experiments, measurements, and disposition so future
work can reuse the evidence instead of repeating it. The package measured faster (1.76x Amharic,
1.83x Mbugwe) but was archived: unattributable per mechanism, and its parity gate compared morph
counts only. Documentation only: no parser behavior changes, and the archived mechanisms
(including the edge prefilter) stay off master.

Where to look

  • Parity claims -- docs/hermitcrab-perf-2026-09-counts.md:59: the legacy WordAnalysisSignature compared
    morph counts only on the FLEx exports (Morpheme.Id is empty).
  • Archived code -- docs/hermitcrab-perf-2026-09-disposition.md points to branch perf/hc-optimization-archive,
    which still holds the edge prefilter (git grep EdgePrefilterEnabled).
  • The register -- docs/hermitcrab-optimization-ledger.md: one row per attempt, plus the rejected-optimization records.

Deliberately not included

  • No source changes, no re-measurement: reviving any archived mechanism needs a complete
    parse-identity signature first, not this PR.

Validation

  • git diff --check origin/master...HEAD -- clean, no output.
  • pwsh -NoProfile -File scripts/comment-hygiene.ps1 -BaseRef origin/master -- no in-scope files
    (docs-only diff).
  • Skipped: dotnet build, dotnet test, dotnet csharpier check ., ./local_check.sh -- docs-only
    change, no .cs/.csproj touched.

Reading this a year from now

This record covers the September 2026 HermitCrab optimization round and the October 2026 search-cost
round: the ledger of attempts, nine retained experimental mechanisms, and five rejected or closed experiments,
plus measurement methods and the final disposition. "Retained" does not mean merged or
production-ready -- the package itself was rejected for merge.

The package's branch-wide sample was faster on all five languages measured, "stable" on two of them
per the probe rules (1.76x Amharic, 1.83x Mbugwe), but the package figure could not be attributed to
any one mechanism -- only the deferred template clone was ever timed alone, at 1.054x. That
non-attribution, not an absence of speedup, is why the package was archived rather than merged.

Correctness limit: the branch-wide "parity" gate used WordAnalysisSignature, which joins
Morpheme.Id -- empty for every FLEx export used, so on Sena, Mbugwe, and Amharic the signature
degenerates to a morph count per analysis. It verified counts, not analyses. Any renewed work needs
a gloss + allomorph-index + syntactic-feature-structure signature, or better, a complete
parse-identity multiset including multiplicities.

Evidence, scope, and provenance

Durable records: ledger,
disposition,
counts, probe design,
nine mechanism records under docs/optimizations/, and five rejection records under
docs/rejected-optimizations/ (the fifth, state-position-traversal-dedup.md, is PR #511's
traversal dedupe, added after the original round closed).

The unmerged package is archived on branch
perf/hc-optimization-archive. The separately isolated perf/hc-edge-prefilter branch and
its own archive tag were deleted, but the edge-prefilter mechanism (source and tests) still lives in
the whole-round archive above -- confirmed directly with git grep EdgePrefilterEnabled against
both refs.

Historical verification (prior measurement, not rerun for this description update): 603 of 604
non-explicit NUnit tests passed, one skipped for Windows symlink privileges.

Original work generated with Claude Code.

🤖 Generated with Claude Code


This change is Reviewable

@codecov-commenter

codecov-commenter commented Aug 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.57%. Comparing base (74ee09b) to head (2eba2f5).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #490   +/-   ##
=======================================
  Coverage   74.57%   74.57%           
=======================================
  Files         453      453           
  Lines       38314    38314           
  Branches     5280     5280           
=======================================
  Hits        28573    28573           
  Misses       8576     8576           
  Partials     1165     1165           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@johnml1135
johnml1135 force-pushed the docs/hc-optimization-ledger branch from 004e199 to 3252bbc Compare September 15, 2026 13:11
@johnml1135 johnml1135 changed the title HermitCrab optimization ledger: 22 attempts, no speedup, one located target HermitCrab optimization round record: 22 attempts, 9 retained mechanisms, no net speedup as a package Sep 15, 2026
@johnml1135

Copy link
Copy Markdown
Collaborator Author

Two further optimization ideas were investigated against this ledger, on a branch based on master (a4b2974). Both are closed. Recording them here because the ledger's purpose is to stop the work being redone, and one of them looks irresistible on paper.

Neither appears in the existing 22 rows. Row 5 covers deterministic synthesis fold sharing and the rejected two-pass experiment covers nondeterministic traversal; neither is the idea below. Row 6 is the synthesis-side sibling of the second idea.

Idea A — a traversed dedup set for the deterministic FSA traversal

NondeterministicFsaTraversalMethod keeps a traversed set keyed on (State, AnnotationIndex, Registers); DeterministicFsaTraversalMethod does not. The motivating observation was that a deterministic FST can still be traversed nondeterministically when the input is nondeterministic — Annotation.Optional makes Advance fork skip-vs-consume, and the forks reconverge.

The mechanism is real and the key is complete on that path (no output operations, no priorities, and Matcher.Compile only determinizes when there are no variables, so VariableBindings is always null). On a synthetic input with 8 consecutive optional annotations it takes instance pushes from 510 to 44 with identical matches.

It does not pay on real grammars. Five FieldWorks exports, tracing on, each word parsed twice (dedup off then on), one word per process, no parse timeout:

grammar words rule attempts off → on Σ det arc checks off → on suppressed Σ nondet arc checks det share of FST work
Indonesian 121 15,040 → 15,040 34,386 → 34,386 0 990,111 3.4 %
Mbugwe 37 4,351,113 → 4,351,113 760,833 → 760,829 4 21,334,695 3.4 %
Sena 35 22,416,707 → 22,416,707 194,014 → 192,515 192 124,891,129 0.16 %
Amharic 37 71,315 → 71,315 9,979 → 9,979 0 44,559,287 0.02 %
Aweti 22 24,274,820 → 24,274,820 192,827 → 192,779 8 97,502,953 0.20 %

Parse-signature multisets identical on every completed word. The reason it is a no-op is structural: the deterministic traversal only runs on the synthesis side (analysis affix matchers set AllSubmatches, analysis rewrite/metathesis set Nondeterministic), and synthesis shapes carry almost no optional annotations, so the skip-vs-consume fork it would collapse rarely occurs. Deterministic traversal is 0.02–3.4 % of all CheckInputMatch calls, and dedup removes ≤ 0.8 % of that.

Where the duplicate explosion the idea was aiming at actually lives is the nondeterministic set, which already exists: on Amharic it suppressed 30.1 M candidate instances against 15.2 M pushed (66 % duplicates), and on one heavy word 82.7 M against 38.5 M.

Two latent issues noticed while auditing the existing keys, neither reachable from HermitCrab today:

  • NondeterministicFstTraversalMethod's key carries the output action list but not which annotations the actions applied to. Two paths through optional annotations can share state, index, registers and action sequence while having modified different output annotations. Unreachable because no HermitCrab FST carries IFstOperations.
  • Both nondeterministic keys omit VariableBindings, which CheckInputMatch reads. Whether two equal (State, AnnotationIndex, Registers) keys can carry different bindings on a real pattern is untested.
  • Both store the mutable Register[,] directly in the key while ExecuteCommands later mutates it. This can only cause a missed dedup, never a wrong one.

Idea B — canonicalize the order of independent rules in an Unordered analysis stratum

The analysis-side counterpart of row 6. In an Unordered stratum the cascade explores both [A,B] and [B,A]; if A and B commute, one order is redundant work. The prune skips a dispatch when the candidate is orthogonal to the trail's last rule and precedes it in source order — adjacent-transposition normalization, with non-orthogonal rules and templates acting as barriers.

Orthogonality was implemented as three cumulative levels — L0 shape (pure prefix vs pure suffix, single application, edge-blind stem pattern, no ModifyFromInput/variables), L1 + syntactic-feature commutation decided by enumerating the finite feature lattice the two rules mention in both the analysis (IsUnifiable gate then Add/Clear) and synthesis (Unify then PriorityUnion) directions, L2 + MPR/stem-name/partial/blocking. A second, independently written census over the XML agreed with the object-model implementation on every unordered stratum: 176 L0 pairs, 60 L1, 36 L2 across the five grammars.

Measured: 0.00–0.05 % of rule-unapplication attempts, parity clean on every word.

grammar words attempts off → on reduction dispatches skipped
Indonesian 121 15,040 → 15,040 0.00 % 21,030 0
Mbugwe 37 4,351,113 → 4,351,113 0.00 % 1,631,322 858
Sena 35 22,416,707 → 22,404,555 0.05 % 6,603,760 12,104
Amharic 29 36,774 → 36,758 0.04 % 27,264 16
Aweti 12 3,014,439 → 3,014,439 0.00 % 1,213,430 0
Sena at L0 (unsound control, same 35 words) 35 25,823,709 → 24,179,995 6.37 % 7,442,610 67,090

The L0 control is what makes this a finding rather than a null result: widening Sena from 2 qualifying pairs to 52 moves the reduction from 0.05 % to 6.37 %, so the limit is the predicate, not the hook. But each narrowing step is load-bearing — of Sena's 52 L0 pairs, 30 fail L1 because their syntactic features genuinely do not commute, and 20 fail L2 on the partial-rule clause, which is necessary there because that stratum holds 24 affix templates so IsLastAppliedRuleFinal takes a value and both final-template gates in SynthesisAffixProcessRule can fire. 6.37 % is what an incorrect prune wins.

A trail-depth census explains why widening cannot reach further. Instrumenting every dispatch with the length of the trail it extends:

  • Mbugwe, 155,672 dispatches: 68 % sit on trails 9+ rules deep, but at only 0.8 % does the trail's last rule have any orthogonal partner even at L0.
  • A heavy Sena word, 76,777 dispatches: 67 % at depth 9+, 16.6 % partner-available, 2.3 % actually skipped → the 6.37 %.

So the permutation blow-up is real and deep, but the rules filling those trails are ones the criterion excludes on shape (Mbugwe rejects 57 pairs MultiInsert, 17 AllomorphOutput, 21 RuleKind). The largest excluded class is same-side pairs (Sena: 84 rejections, the C(13,2) suffix pairs), and those can never commute — stem+A+B ≠ stem+B+A is arithmetic, not conservatism.

One consequence worth separating from performance. The prune is not signature-preserving and cannot be: a parse signature carries MorphemesInApplicationOrder, so unapplying a prefix before a suffix and the reverse are two distinct parses of one word even when the rules commute. What it preserves is the morph sequence and the root; what it drops is derivational history. Those two rows are indistinguishable to a reader, so this is an ambiguity reduction as much as an optimization — and it would need the correctness treatment rather than being presented as invisible.

This is now the fifth instance of the pattern in this ledger's "lesson that generalises": apparent redundancy that evaporates once the key is complete.

On rows 10 and 11 — reachability

The first reading of our data looked like it reopened the lexical reachability gate: the heavy Sena word made 272,152 unapplication attempts and the Aweti word 505,600, each finding zero roots. It does not reopen it. Those lexical-lookup counts come from runs truncated by a step budget, so they show the search never reached lookup within budget, not that lookup would have failed. Row 11's finding — that the expensive words fail on checks running after lexical lookup succeeds — is untouched by anything measured here.

On the complexity-cap step budget

b3fd2b55 was cherry-picked to bound the pathological words. It works where the cost is many rule applications: an Aweti word that previously crashed a test host at 12.1 GB after 1,717 s completes in 50 s at 200,000 steps, and whole-corpus runs go from 22/40 words to 40/40 at 757 MB peak instead of 12 GB.

It does not bound a word whose cost is one expensive traversal. An Amharic word ran 2,280 s under a 200,000-step budget with the nondeterministic counters frozen at 217,493 traversals and 53.9 M arc checks across successive samples — stuck inside a single matcher call. The budget is checked between leaf-rule applications and cannot interrupt one. Anyone relying on it as a safety bound should know that.

Also worth flagging for consumers: the budget ships on by default (2,000,000 steps, 10 s). A 10-second default silently truncates parses.

Reproduction

Branch based on master a4b2974. Measurements used a census harness that parses each word twice in one process with MaxDegreeOfParallelism: 1 and tracing on (so the analysis memo and MergeEquivalentAnalyses are off and counts are the unmemoized search), compares complete parse-signature multisets, and reports MorphologicalRuleAnalysis trace nodes as attempts and CheckInputMatch calls as arc checks. Grammars are FieldWorks GenerateHCConfig exports of five projects; word lists are the first 40 entries except Indonesian (all 121) and Sena (entries 1001–1040, since the head of that list is punctuation). Machine-safety aborts and test-host crashes are recorded as such and excluded from ratios, never counted as agreement.

@johnml1135

Copy link
Copy Markdown
Collaborator Author

Two observations from the same investigation that belong in the record but are not optimizations, so they did not fit the rows above.

A repeatable memory wall at 12.2–12.6 GB, independent of any cap

Across the five-grammar runs, 13 words (Sena and Aweti) ended as test-host crashes rather than completions, and they did so at a strikingly consistent managed heap: 12,225 / 12,259 / 12,270 / 12,278 / 12,282 / 12,312 / 12,323 / 12,324 / 12,336 / 12,340 / 12,359 / 12,365 / 12,563 MB. The consistency holds regardless of the per-process limit in force — including runs whose limit was 20 GB, so these are not the harness stopping them.

The clustering at ~12.3 GB rather than at whatever ceiling was configured is the interesting part; it is the signature one would expect from a single collection outgrowing the 2 GB per-object limit rather than from address-space or RAM exhaustion. A crash dump on one of these words would name the collection. Reported as an observation only — no diagnosis attempted.

For scale, one Aweti word (line 183 of its list) reached 13.2 GB and 30.1 M nondeterministic traversals / 139.3 M arc checks in 53 minutes without leaving analysis, and never reported a lexical lookup. Under a 200,000-step budget the same word completes in 231 s at 676 MB with 505,600 rule-unapplication attempts and 0 deterministic traversals.

Where analysis time goes, by traversal family

The counters added for this work split every CheckInputMatch call between the two FSA traversal families. Deterministic traversal — which is the synthesis side only, since analysis affix matchers set AllSubmatches and analysis rewrite/metathesis set Nondeterministic — accounts for:

grammar deterministic share of arc checks
Indonesian 3.36 %
Mbugwe 3.44 %
Aweti 0.20 %
Sena 0.16 %
Amharic 0.02 %

This is offered as calibration for anyone sizing a synthesis-side optimization: on the two grammars with the heaviest analysis, 99.8 % or more of FST work is on the nondeterministic side. It is consistent with the ledger's Amharic breakdown (analysis cascade ~95 % of analysis, all synthesis 0.3 %), measured independently and at the arc-check level rather than by wall-time bucket.

@johnml1135

Copy link
Copy Markdown
Collaborator Author

Added one record from a September round that post-dates this set: docs/rejected-optimizations/state-position-traversal-dedup.md, covering PR #511's (State, AnnotationIndex) traversal dedupe.

Three things it contributes to this ledger:

  1. A reproduction of the motivating pathology, with the mechanism named. It is not the state count — TraversalMethodBase.Advance forks an instance per Optional annotation, so instances are exponential in optional count and independent of |States|. Measured 2,097,150 instances popped on a 2-state FSA at 20 optional annotations.
  2. A fourth row for "the lesson that generalises." Apparent collapse 93.95%, sound collapse 0.28%. Same shape as rows 5, 6 and the fold-entry census: a key that omits what makes instances distinct shows large shareable work that is not there. Here the omitted state is variable bindings (which filter later arcs) and registers (which carry the captures AllSubmatches callers consume).
  3. A narrowed form that does pay, unlike most of this ledger: restricted to the deterministic method with no capture groups, the differential fuzz goes 112 divergences → 0 while the group-free microbenchmark stays cell-for-cell identical to the unnarrowed change.

Also closes the two extensions before anyone builds them — environment matchers are 0.20%/1.18% of traversal instances on Amharic/Mbugwe (this is #515), and the analysis rules' 93.95% is 99.7% alternate captures because AnalysisAffixProcessRule and AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries deliberately.

docs/rejected-optimizations/two-pass-nondeterministic-traversal.md now cross-references it: that experiment kept bindings in the key and rebuilt captures and was too expensive; this one drops both and is unsound. Opposite errors, same boundary.

🤖 Generated with Claude Code

@johnml1135

Copy link
Copy Markdown
Collaborator Author

2026-09-23 round: the missing Aweti speedup was reduplication, not traversal

One new row closes the gap #511 opened. Aweti index 182 had never completed here, on any stack we tried (#491, #494, #511, FieldWorks #1148, forced final templates, FLEx's parallel/no-memo mode). A copy-agreement prune in reduplication unapplication makes it finish in 9 s, and the whole Aweti list runs 4.95× faster with 0 divergences across five grammars. Proposed as #519 (on by default) and shipped in PanGloss v0.4.0.

New row

# Attempt Mechanism Result Verdict
23 Copy-agreement prune Reject a reduplication match whose copies can't unify segment by segment; keep copies with optional nodes or modifications Aweti 182: OOM → 9 s. Aweti list: 172 → 192 complete, 4.95× on the shared 172. Mbugwe ~0.2% fewer candidates; Sena/Amharic untouched Retained, on by default (#519); PanGloss v0.4.0

Why the ledger missed it: every prior row measured Sena, Mbugwe and Amharic, where full-copy rules are absent or rare. The 19 docs mention reduplication once (the probe-design classifier, marked "Unknown").

What else this round settled

  • The Aweti blow-up is 12 full-copy rules (mrule25-mrule36), not traversal. Analysis captures each copy independently and uses only the first, so every split becomes an analysis. Across the list, 99.7% of the matches that can be judged disagree.
  • Add allMatches to Traverse function to improve performance #511 as reviewed was unsound and gave no gain on our grammars. The old key produced 112/20k fuzz divergences; a narrowed form was fuzz-clean but no faster. See "Open since this round" for the redesign.
  • #1148 changes 1 entry / 1 allomorph on Aweti and Sena, and no speed. It can also drop a co-occurrence prohibition that references a removed duplicate MSA (reproduced).
  • FLEx never runs the Memoize HermitCrab's sequential analysis cascade #456 memo. FieldWorks (HCParser.cs:178) and the HC tool construct Morpher with the default parallelism, and the memo only engages at degree 1. On 21 completing Aweti words, the default mode uses 2.93× the CPU of the memo mode, with identical analyses. That makes it a separate FieldWorks change worth measuring.
  • Master is not worse than Maxwell's build. FLEx's own mode (parallel, no memo) also OOMs on index 182. His reported 83 s → 3 s and 4 → 1 min match this prune's effect; how his Add allMatches to Traverse function to improve performance #511/#1148 stack got there is still unexplained.
  • Closed, don't revisit:
  • Open, research-only:
    • Lazy composed FST for the phonology (unmeasured).
    • Two-way FSTs for productive copying: literature only, no HC mapping yet.

PanGloss v0.4.0 (same prune, Rust)

Release binaries, same machine, v0.3.3 vs v0.4.0: identical analyses on all 7,455 words both complete across the five grammars. Aweti: 172 -> 191 words complete within 20 s, 5.3x on the 172 shared words, median word 188 -> 26 ms, word-list index 182 60 s timeout -> 2.7 s. Sena, Amharic and Mbugwe are within noise to about 10% slower; a per-call allocation on rules that copy nothing is being removed.

Open since this round

Method notes worth keeping

  • Parity used gloss/allomorph/index signatures. Morpheme.Id is empty in FLEx exports.
  • The fixture edge-cases-zero-width-morpheme-identity-stability is nondeterministic on master (off/off hashes differ); exclude it from any off/on diff.
  • A seeded fuzz that compares "baseline minus classified" against "pruned" only tests the mechanism, not soundness. Soundness came from conformance, the parameterized ReduplicationRules/ModifyFromInputRules tests, three mutants, and the real-grammar sweep.

🤖 Generated with Claude Code

@johnml1135
johnml1135 force-pushed the docs/hc-optimization-ledger branch from 47318eb to 5bbfa01 Compare September 25, 2026 22:57
@johnml1135 johnml1135 changed the title HermitCrab optimization round record: 22 attempts, 9 retained mechanisms, no net speedup as a package HermitCrab optimization round record: 23 attempts; a faster package that is not mergeable Sep 25, 2026
Comment thread docs/hermitcrab-search-asymptotics-research.md Outdated
Comment thread docs/optimizations/edge-segment-prefilter.md Outdated
Comment thread docs/hermitcrab-optimization-ledger.md Outdated
Comment thread docs/hermitcrab-perf-2026-09-disposition.md Outdated
Comment thread docs/hermitcrab-perf-2026-09-disposition.md Outdated
Comment thread docs/hermitcrab-probe-design.md Outdated
Comment thread docs/hermitcrab-optimization-ledger.md Outdated
Comment thread docs/hermitcrab-optimization-ledger.md Outdated
Comment thread docs/hermitcrab-perf-2026-09-disposition.md Outdated
@claude

This comment has been minimized.

Comment thread docs/hermitcrab-2026-10-21-search-cost-round.md Outdated
Comment thread docs/hermitcrab-2026-10-21-search-cost-round.md Outdated
@claude

This comment has been minimized.

R1, R2, R5 and T1 are kept with their net-of-own-cost numbers in steps, R4 is a recorded dead end, R6 is unbuilt, R3 and R1 on #546 are in progress. Adds ledger rows 38-46.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Not-unapplied attempts are 70-91% of events. Skipping one with a cheaper test removes no step; it makes a step
cheaper. The plan therefore scored prefilters 0 in steps (R5). That scoring turned out to be the lesson of R5.
- The root-substring predicate at rule outputs had 13.6-38.5% ceilings with 0 violations on the five real
grammars, but 29 success-path nodes on conformance, so it needed static soundness flags (R2).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: F33. The two root censuses contradict each other. This line says the substring predicate had 0 violations on all five real grammars, but R4 (lines 314-317) gives the weaker subsequence predicate 334,508 violation events on Mbugwe and 50,187 on Aweti. A root that is a substring of a shape is also a subsequence of it, so every subsequence prune is also a substring prune. Its violations and ceilings (Indonesian 55.2% against 29.6%) therefore cannot exceed the substring ones. Say which census is wrong, or what differed between them (root set, nodes counted). R2's KEEP and R4's DEAD END both rest on these figures.

fixed a real Rust divergence in alpha-environment selection (PanGloss PR #13). Porting a change to the second
engine is a cheap second oracle.
6. **A ceiling measured with the flags ignored can be mostly unsound.** R4's census ceilings (up to 55.2%) were
mostly restoration violations; the sound share on the four target grammars was zero.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F34. "The sound share on the four target grammars was zero" contradicts line 322. On Indonesian the sound order-free tier prunes 30.8% of events with 0 shadow violations, and line 372 keeps it as an open question. The share that was zero or negative is the net gain. Ledger lesson 6 (hermitcrab-optimization-ledger.md:193) makes the same claim.

**Why it is not row 27.** Row 27 closed root reachability summaries (the prunable bucket was 0-5% on 6 of 8 lists,
and nodes may still compound). A substring test passes a compound, because the compound contains its head root;
it runs at every rule output, including template-slot outputs; and the flags turn it off where stem material can
be restored. Row 27's reopening condition, a shadow summary deciding a meaningful share, is met on Sena (17.7%) and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F35. This misquotes row 27's reopening condition. The ledger (line 132) asks for "a shadow summary on Aweti/v1 deciding a meaningful share", and R2's flag is off on Aweti. Sena and the stress fixture therefore do not meet the condition as written. Quote it in full, or rest R2 on the "why it is not row 27" argument alone. Ledger row 40 repeats the shortened quote.

**Claim tested.** R2 is off on four of five real grammars because their rules restore, reorder or duplicate stem
material. R4 asked whether a weaker sound test still prunes: "some root is a subsequence of the shape" for
order-preserving removal, with the segment classes that a dropped part can restore treated as wildcard gaps.
Ceilings from the census, minus any R1 overlap: Mbugwe 38.5%, Indonesian 29.6%, Aweti 14.8%, Amharic 13.6%.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F36. These ceilings are labelled "minus any R1 overlap", but they are the same four numbers as R2's raw census ceilings at line 151 (38.5, 29.6, 14.8, 13.6). R1 removes 6.6% of Mbugwe's events, so subtracting its overlap should change at least Mbugwe's figure. Drop the qualifier, or give the reduced figures.

| Indonesian | off (REDUP-meN copies parts out of LHS order) | 7,468 to 7,468 | 0.0 |
| Amharic | off (a rule modifies a part) | 8,560 to 8,560 | 0.0 |
| Aweti | off (a rule copies parts out of LHS order) | 1,596,848 to 1,596,848 | 0.0 |
| Mbugwe | off (every root is made of restorable segments) | 3,500,941 to 3,500,941 | 0.0 |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F37. Mbugwe's off-arm total is 3,500,941 events here, but R1's table (line 89) gives 5,289,909 for the same stride-40 list. Every other row matches R1 exactly (7,468, 8,560, 1,596,848, 110,986). Say why Mbugwe differs (word set, abort budget), as the method at line 22 promises.

| stress deep-derivation | 10 | 200.36 | 4.99 | 97.5% | 343.0 to 7.8 |

yɔɔna: 1,035.0 to 22.6 MiB (97.8% less); peak falls from 1,038 to 89 MiB. The kill criterion (under 50% on
yɔɔna) is not met. What still holds the lean memory is the node objects and two strings per node, about 190 bytes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F38. The per-node estimate is about half what the doc's own totals give. 22.6 MiB is 23.7 MB, and 21,200 events plus 41,130 steps (line 185) is 62,330 nodes. That works out to about 380 bytes a node, not 190.


## What the census said to aim at

From the [analysis-cost census](hermitcrab-optimization-ledger.md) the plan took these facts:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F39. The census figures behind every candidate (63.8%, 69-98%, 70-91%, the 29 success-path nodes) are cited to "the analysis-cost census", but the link goes to the ledger. git grep under docs/ finds none of these figures outside this file. Link the census result file, or name it along with the harness commit (research/analysis-cost-census @ 38e9622e).

@claude

This comment has been minimized.

Rows 43 and 44 move from in progress to done. R1 on #546 is draft PR #577
(commit bbb81a0, shadow 0 violations); R3 ordered look-ahead is draft
PR #576 (shadow 0 violations). Adds the round-doc section for both and
the PanGloss port issues (sillsdev/PanGloss #65, #66).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| 40 | **R2: root-substring prune at rule outputs, gated by static restoration flags** (issue #562; **PR #572**; reopens row 27) | Sena 8-17%, the other four real grammars 0 (flags off), stress fixture at least 40% | **KEEP, with a probe skip added.** Row 27's reopening condition (a shadow summary deciding a meaningful share) is met on Sena and the stress fixture, 0 violations. The prune is off on four of five real grammars (dropped part that can hold segments, reduplication, deeper phonology). A static gate on root count was **rejected as overfitting toy grammars** | Net of probes (1 probe = 1 event), less work: Sena **15.3%** (gross 17.7%), stress **66.5%** (gross 67.7%), Aweti, Amharic, Indonesian, Mbugwe **0 (flag off)**, conformance **15.3% MORE** (7,237 probes buy 92 events). The probe skip (keep when the output has at least as many segments as its input) removed 18 of 7,255 probes |
| 41 | **T1: lean trace, nodes keep shape text and no `Word`** (issue #563; **PR #571**; scored on memory) | Retained trace memory down 90-97% on Mbugwe yɔɔna (1,032 MiB) | **KEEP; lands after PR #534** (it rewrites the same callbacks). Opt-in `TraceManager.RetainWords = false`; default behaviour unchanged. Steps identical by construction and verified | Retained trace memory, lean vs full, sum over words: Mbugwe **93.8% less**, Aweti 93.6%, Sena 92.0%, Amharic 98.8%, Indonesian 98.7%, conformance 97.4%, stress 97.5%. yɔɔna 1,035.0 to 22.6 MiB. Parse counts, node counts, census rows and conformance verdicts identical |
| 42 | **R5: edge-segment prefilter port** (row 22's archive; issue #564; **PR #574**) | Scored 0 by the plan: a prefilter removes no step, it makes each step cheaper | **KEEP.** The plan's 0 was the wrong score: census steps do not move, and the work per step falls. The first write-up was gross and left out the prefilter's own unifications; the net table charges each one as one arc check | **Net matcher work** (traversals plus arc checks plus prefilter unifications), census, less work: Aweti **47.5%**, Mbugwe **19.6%**, Sena **19.6%**, Amharic **12.8%**, Indonesian **6.5%** (gross 6.9%), stress 77.7%, conformance 23.1%. Analysis events and synthesis steps **unchanged** on all 1,796 completed pairs per mode |
| 43 | **R1 on PR #546's shared-slot walk** (issue #566) | R1 and #546 compose; the residual after both is a ceiling only | **KEEP, stacked on #546 and #575.** Draft PR #577 (commit `bbb81a00`, Closes #566). Reviewed LAND. Shadow: 0 violations. The two compose in events, but on Mbugwe and Sena the check's own unifications cost more than the walk they save, so #546 alone keeps fewer unification calls there. | Events plus synthesis steps vs master, less work: Mbugwe **32.15%**, Sena **55.64%**, Aweti **20.06%**, Amharic **9.10%**. Unification calls vs master: Mbugwe **15.19% less**, Sena **4.81% less**, Aweti **12.18% less**, Amharic **0.63% MORE**. Against #546 alone: Mbugwe **1.07% MORE** and Sena **1.40% MORE** unification calls. Reopen condition for dropping one of the two (unchanged): R1 saves under 3% of slot events on the #546 walk where it saved over 5% on master. PanGloss port: sillsdev/PanGloss #66 (for #546 and #566) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: F40. KEEP for R1 on #546 skips the overlap rule set in advance, which this cell restates as "unchanged": the share of slot events R1 saves on the #546 walk is never given, and every events figure is against master. The only figures against #546 alone are unification calls, and those show R1 adding work on Mbugwe (1.07% more) and Sena (1.40% more). By lesson 1 that is a net loss on two of four grammars. Report the slot-event saving over #546 alone and the outcome of the rule, or say why KEEP holds anyway.

under tracing lookups are 1.67x (Mbugwe) to 2.19x (Aweti) more frequent. Whether Try-a-Word shows folded
duplicates is an owner product decision, and it is low in steps. It interacts with #534 and #563.
- **R3, ordered and edge-anchored look-ahead (#567): KEEP, PR #576.** It targets the 90% of Mbugwe's no-lookup
ceiling that R1 leaves, where required slots insert a single segment. Results are in the section above.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F41. "Results are in the section above" (here and on line 341) points the wrong way: the results are in the section below, at line 346.


## Open questions

- R3 and R1-on-#546 results (issues #567, #566).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F42. Still listed as an open question, but this commit records both results (lines 346-362, ledger rows 43-44). Drop the item, or replace it with what is still open, such as the R1 purity extension at line 354.

walk, so a template whose required slot cannot unapply is dropped for its whole subtree. Reviewed LAND. Shadow: 0
violations. Events plus synthesis steps, less work than master: Sena **55.64% less**, Mbugwe **32.15% less**, Aweti
**20.06% less**, Amharic **9.10% less**. Unification calls, less than master: Sena **4.81% less**, Mbugwe **15.19%
less**, Aweti **12.18% less**; Amharic **0.63% MORE**. Against #546 alone, Mbugwe **1.07% MORE** and Sena **1.40%

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F43. Unverified: for R1 on #546, Amharic is 0.63% MORE unification calls, the same to two decimals as R1 alone on master (line 105). Its events differ, though (9.10% less work here, 7.73% for R1 alone). If #546 changes nothing on Amharic, the events should match too. Check this figure against the result file.

@claude

This comment has been minimized.

Adds the round-doc section "Combined stack and R2's cost": the full stack
against master (work, unification, matcher per grammar), additivity
(near-additive on Mbugwe, overlapping on Sena, super-additive on Aweti),
R2's root cause (its own probe, low prune rate) and the witness-reuse fix
folded into #572 @ 01bdf5a, the 1-4 point residual, and the landing
decision: #572 is a draft and lands only on top of #577 and #574. Stress
grammars are PR #578 (opt-in, [Explicit], not run in CI).

Ledger: row 40 (R2) Outcome records the fold and the landing decision;
new row 47 records the combined stack measurement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

**R2's cost: root cause.** The extra unification and matcher work is R2's own probe, not a change in merging or order. With R2 on in the stack, every non-probe counter is equal or lower (events, lookups, nodes, look-ahead checks, edge-prefilter unifications). The added unifications track the added trie arcs almost one for one (conformance +37,582 unifications against +37,518 arcs). The prune rate is too low to pay for the probe: conformance prunes 13 of 7,214 probes, and Sena prunes 72 of 1,225 probes, where each prune saves about 575 unifications and a probe inspects about 70 arcs. Most probes pass, because they find a root, and a pass still walks the arcs up to the first accepting path. Bantu-tam prunes late, after R3 has cut nearly all the work, so each prune saves about 50 unifications against about 47 spent per probe.

**Witness-reuse fix (folded into #572 @ `01bdf5ae`).** Most probes re-find a root the input already had. Under R2's flag conditions an analysis rule's output is a run of the input's segments, so a root run found in the input that still lies inside that run is also a root run of the output. The fix records the witness and its window on the word, and skips the probe when the witness lies in the window. It is a two-field per-word value: no cache, and no size or prune-rate gate. A witness can only skip a probe, and a skipped probe can only keep a word, so it cannot drop a parse. Prune decisions match #572 whenever the witness is valid; per-word signatures equal master on 1,916 completed words, and event counts equal #572's on every grammar. In the stack the fix removes about 90% of R2's unification overhead against no R2: conformance 3.18% less becomes 14.63% less (15.66% less without R2), Sena 15.43% less becomes 17.11% less (18.63% less without R2), bantu-tam 64.03% less becomes 69.39% less (73.10% less without R2). Early-lexicon stays ahead of no R2 (84.04% less against 71.31% less).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F44. "Removes about 90% of R2's unification overhead" holds only on conformance. By this paragraph's own figures, R2's overhead against no R2 goes from 12.48 to 1.03 points on conformance (92% removed), but from 3.20 to 1.52 on Sena (53%) and from 9.07 to 3.71 on bantu-tam (59%). Say "about 90% on conformance, about half on Sena and bantu-tam".

@claude

This comment has been minimized.

R1 called a slot impure when a copied part may match nothing. An
unbounded run (max=-1) restores no segment in Untruncate, so Mbugwe's
vowel-fusion slots were treated as impure and the look-ahead stopped
testing behind them. PR #579 (stacked on #576) makes the purity test
exact against Untruncate. Row 48 of the ledger and a section in the
2026-10-27 round record the result: Mbugwe events plus synthesis steps
8.19% less -> 40.85% less than master, unification 1.83% less ->
22.55% less, matcher 7.65% less -> 30.47% less. Parses are equal to
master; shadow shows 0 violations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
## Impure slots: unbounded runs restore nothing (PR #579, stacked on #576)

- **R1 purity extension (PR #579, stacked on #576).** R1 called a slot impure when a copied part "may match nothing". Mbugwe's vowel-fusion allomorphs copy `OptionalSegmentSequence min=0 max=-1` runs, so every object and subject slot there was impure and the walk stopped testing behind them. `AnalysisMorphologicalTransform.Untruncate` writes nothing back for an unbounded quantifier (`max=-1`), so those allomorphs restore no segment and are pure. `RequiredSlotMaterial.RestoredSegments` now mirrors `Untruncate`: one segment for a bare segment constraint, `max(0, MaxOccur)` times the children for a quantifier, and the sum of the children for anything else. An allomorph whose parts restore 0 segments is pure. Bounded quantifiers, bare segments, optional alternations and any `ModifyFromInput` stay impure as before. The count is computed once per allomorph at construction, with no cache and no `Interlocked`. The Presence argument is unchanged. A kō-style copy-twice allomorph is still unordered, so R1's independent presence test runs behind it. The relation rests on `Untruncate` emitting nothing for `MaxOccur == -1`; the tests pin it.
- **Figures, #576 → #579 (less work than master).** Events plus synthesis steps: Mbugwe **8.19% less → 40.85% less**; the synthetic fusion stress grammar **2.11% less → 74.59% less**; other grammars unchanged. Unification calls: Mbugwe **1.83% less → 22.55% less**. Matcher work: Mbugwe **7.65% less → 30.47% less**. The look-ahead's own check unifications on Mbugwe are **13.88% more** (995,434 → 1,133,568), and they are included in those totals.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F45. The #576 baseline does not match the R3 record: here Mbugwe goes from 8.19% less, but the R3 section (line 353) and ledger row 44 give #576 on Mbugwe as 7.3% less. Lesson 8's "32.66 points" and row 48 rest on the 8.19. Say which run each figure comes from, or use one.

@@ -0,0 +1,439 @@
# HermitCrab analysis-side optimization, 2026-10-27 round

**Disposition: seven candidates are PRs (R1 #575, R2 #572, T1 #571, R5 #574, R3 #576, R1 on #546 #577, R1 purity extension #579), one is a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F46. The disposition now counts seven PRs, including #579, but the Decisions table (lines 31-40) still lists six and has no row for the R1 purity extension. Add a row: issue, #579, KEEP, stacks on #576.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor
  1. Verdict: approve with fixes. This PR changes documentation only. The new commit records the R1 purity extension (PR perf(hermitcrab): keep the required-slot look-ahead testing behind slots whose unmatched parts are unbounded runs #579, ledger row 48). Its description of Untruncate matches master, and its arithmetic checks out. Its baseline for Require slot literals in order and at reachable edges in the template look-ahead #576 does not match the R3 record, though. F20-F44 have had no reply, and their lines are unchanged.
  2. Most important: F40 (docs/hermitcrab-optimization-ledger.md:175) is still open: the R1-on-perf(hermitcrab): share trailing analysis template slot batches #546 KEEP never evaluates its own overlap rule. F33 (docs/hermitcrab-2026-10-27-analysis-round.md:54) is next: the two root censuses still contradict each other. New: F45 (docs/hermitcrab-2026-10-27-analysis-round.md:402). Require slot literals in order and at reachable edges in the template look-ahead #576 on Mbugwe is 8.19% less here, but the R3 section and row 44 give 7.3% less.
  3. Counts: Critical 0. Important: 5 open (F20, F21, F26, F33, F40). Low: 22 open (F22-F25, F27-F32, F34-F39, F41-F44) and 2 new (F45, F46).
  4. What I ran:
    • The compare 4b2f7557...2eba2f50 reports ahead: 1 commit, which adds the perf(hermitcrab): keep the required-slot look-ahead testing behind slots whose unmatched parts are unbounded runs #579 section, lesson 8 and ledger row 48. The diff was small, so I reviewed it by hand and did not run /code-review.
    • I checked AnalysisMorphologicalTransform.Untruncate (AnalysisMorphologicalTransform.cs:228-256). It adds one node for a segment constraint and loops MaxOccur times for a quantifier, so max=-1 restores nothing, as the record says.
    • I recomputed 40.85 - 8.19 = 32.66 points and 1,133,568 / 995,434 = 13.88% more. Both match.
    • gh pr checks 490: changes, Build on ubuntu-22.04, and both comment-hygiene checks pass. Build on windows-latest and claude-review are pending.
  5. Not verified: the measured figures themselves, PRs Prune analysis rule outputs that contain no root when no later rule can restore stem material #572 and Skip impossible analysis affix matches using edge segments #574-perf(hermitcrab): keep the required-slot look-ahead testing behind slots whose unmatched parts are unbounded runs #579, PanGloss Unable to configure translation engines properly #67, and the 16% Aweti limit. They rest on branches and grammars I could not access.

Public API, target framework, package, and machine.py parity: None verified (docs only).

Findings: F1-F19 addressed; F20, F21, F22, F23, F24, F25, F26, F27, F28, F29, F30, F31, F32, F33, F34, F35, F36, F37, F38, F39, F40, F41, F42, F44 open; F43 open, unverified; F45, F46 new.

Reviewed at 2eba2f5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants