Repository navigation
HermitCrab analysis: an affix-rule index cut port work 11.8% on Cebuano; C# gain unmeasured #599
Description
Claude-drafted.
An affix-rule index, built once at grammar load, cut analysis work by 11.8% on a 7,206-type Cebuano held-out set in the Rust port, and by 0 to 4.9% on five sample grammars. The C# gain is unmeasured. C# still visits every affix rule on every word and rejects allomorph by allomorph (#574).
Status: built and merged in the port (sillsdev/PanGloss PR #95); not built in C#.
What we know (port, work steps; parses identical on every completed word):
- Cebuano 7,206 held-out types: 5,664,386 to 4,996,598 (-11.8%). Aweti -4.9%, Sena -3.2%, Indonesian -0.4%, Amharic 0 (measured before the final syntactic-gate change, which cannot raise them).
- 141 conformance fixture grammars: 1,501,384 to 1,489,476 (-0.8%). No grammar or word went up.
- Memory: 0 to 4 KB per grammar.
- The index maps each rule to the mandatory first and last edge segments of its allomorphs. A rule is skipped before its visit only when Skip impossible analysis affix matches using edge segments #574's edge predicate rejects every allomorph, so results cannot change. Optional, unknown, modified, deleted and featural edges take the old path.
- It is consulted at every cascade recursion and template slot batch, against the current intermediate shape, and only after the syntactic-feature gate.
Not known: the C# gain. #574 leaves C# census steps unchanged (ledger row 42), so a C# gain would show as matcher work, not steps.
Proposed change: port the index to C# after #574 merges, behind the same identity gate (conformance suite plus the five sample grammars).
Related:
- PanGloss: PR perf(analysis): skip affix rules whose edges cannot match the current shape (#90) PanGloss#95 (merged); umbrella issue Restricted affixes and clitics are tried on every word: environment and inflection-class restrictions don't reduce parse work PanGloss#90 (open).
- Skip impossible analysis affix matches using edge segments #574 (PR, open) and HermitCrab analysis: port the edge-segment prefilter (time-only, 0 steps) #564 (issue, open): the per-allomorph prefilter this extends.
- HermitCrab: a restricted clitic adds 17.5% work steps, mostly host search that repeats per MSA (measured on the port) #610 (open): the PanGloss#90 clitic measurement that led here.
Next: after #574 merges, measure matcher work on Sena with and without the index on hc.dll.
Evidence -- numbers, method, commits
Work steps from PanGloss PR #95 (squash-merged 2026-10-30) and its lane report. Counts via --stats --work-columns --threads 1 --step-cap 200000 for Cebuano.
| Grammar | Work before | Work after | Change |
|---|---|---|---|
| Cebuano held-out, 7,206 types, round 2 | 5,664,386 | 4,996,598 | -11.8% |
Cebuano with enclitic g, held-out |
6,623,154 | 5,840,844 | -11.8% (g's added cost 958,768 to 844,246) |
| Sena (round 1) | 182,297,249 | 176,401,068 | -3.2% |
| Indonesian (round 1) | 1,617,711 | 1,611,898 | -0.4% |
| Amharic (round 1) | 32,238,244 | 32,238,244 | 0 |
| Aweti, 194 valid words and 12 invalid shapes, 30M-attempt cap (round 1) | 1,237,499,929 | 1,176,371,185 | -4.9% |
| Mbugwe, 200-word check (round 1) | 11,449,056 | 11,047,179 | -3.5% |
| 141 fixture grammars, round 2 | 1,501,384 | 1,489,476 | -0.8% |
Round 1 charged one step per visit that rejected a rule; round 2 charged only rules that pass the syntactic gate, which cannot raise the round 1 totals. Sample grammars were not re-run in round 2. Eleven Mbugwe words never complete on either build and are unverified.
Why the index must see the current shape: on Cebuano, earlier rules expose intermediate forms that end in a clitic's segment, so a check against the surface word alone misses them. Why it comes after the syntactic gate: a rule that gate rejects for free would otherwise pay for the lookup.
Paths not taken -- the environment pre-check
An input-word pre-check of the allomorph environment (E5 in the #490 round) is a dead end (see #601). On Cebuano it could save at most about 4% of an added enclitic's cost: 83% of the g clitic's added steps fall on words ending vowel + g, where its environment [V] _ is satisfied and a host search runs. The index is the general lever. Source: PanGloss#90 comments.
Sources -- pinned
- Prefilter record: docs/optimizations/edge-segment-prefilter.md; ledger row 22 and row 42; R5 write-up.
- Port: perf(analysis): skip affix rules whose edges cannot match the current shape (#90) PanGloss#95 and Restricted affixes and clitics are tried on every word: environment and inflection-class restrictions don't reduce parse work PanGloss#90.
Activity
johnml1135 commentedon Oct 31, 2026
Claude-drafted. Related: #610 records the PanGloss#90 measurement that led to this index. In that measurement, a restricted clitic's cost was mostly host search, and an environment pre-check would save at most about 4%.
[-]HermitCrab analysis: index affix rules by mandatory edge segments (skip rules before visiting)[/-][+]HermitCrab analysis: an affix-rule index cut port work 11.8% on Cebuano; C# gain unmeasured[/+]