HermitCrab analysis: keep equivalent-analysis merging on while tracing #565

HermitCrab turns off equivalent-analysis merging while tracing, so a traced parse repeats lookups that an untraced parse folds. The trigger is any traced parse (Try-a-Word style): analyses reaching lexical lookup rise by 1.67x on Mbugwe and 2.19x on Aweti. In steps the effect is small, so expected value is low.

Problem (census, steps)

AnalysisStratumRule skips merging when tracing ("Don't merge if tracing because it messes up the tracing"). Census, traced unmerged versus merged: analyses reaching lookup Mbugwe 9,334 to 15,617 (x1.67), Aweti x2.19. Analysis event totals move by only -0.2% to 2.4%, because most of the extra work is lookup-side and lookups are 0.7% of Mbugwe events. The synthesis-side effect is unverified (the census says synthesis steps are unchanged, which is hard to reconcile with lookup-driven synthesis; it needs rechecking before any claim).

Proposed change

Keep merging on while tracing and record folded duplicates as one node with a count. This is a product decision on how Try-a-Word shows folded duplicates, not only an optimization.

Soundness

Merging is already used untraced and is parse-identical there (ledger rows 4, 5, 16 family; PR #493). The open soundness question is the trace: callbacks for a merged duplicate are skipped, so LexicalLookupCompleted counts and trace-node counts change meaning, which affects PR #534's reported counts.

Memory bound

One count field per folded trace node. No cache.

Expected value (low, possibly negative)

Analysis events change by only 0.2 to 2.4% (the plan lists it as -0.2..2.4%, so the sign is uncertain); lookups divided by 1.67 (Mbugwe) to 2.19 (Aweti) under tracing. Memory gain probably modest (lookups are 0.7% of Mbugwe events), unmeasured.

Measurement plan

Census harness: censusmerge mode versus census mode on all grammars, report events, lookups and synthesis steps separately; first reconcile the synthesis-steps claim.

Kill criterion

Close if the synthesis-step reconciliation shows no reduction, or if the product owner of Try-a-Word does not want folded duplicates in the trace.

Dependencies

None. Interacts with PR #534 (trace nodes) and T1 (trace memory).

Related

• PR HermitCrab optimization round record: what was tried, and a faster package that is not mergeable #490: HermitCrab optimization round record (ledger rows cited above are in it).
• PR Add a conformance suite for HermitCrab, with its adequacy argument #480: HermitCrab conformance suite. Conformance per-fixture verdicts must be identical with and without the change.
• Units: deterministic analysis events and synthesis steps, never ms. "% less work" = (off - on) / off per grammar, never pooled. Census figures are ceilings unless stated. Ranges marked "est." are estimates that the shadow phase replaces with measurements.
• Owner constraint: no memo or cache; any new structure must be grammar-sized and at most a few MB.

🤖 Generated with Claude Code