Skip to content

Correctness review and refactor (0.3.0) - #22

Merged
Autoplectic merged 9 commits into
mainfrom
sofic-refactor
Oct 7, 2026
Merged

Correctness review and refactor (0.3.0)#22
Autoplectic merged 9 commits into
mainfrom
sofic-refactor

Conversation

@Autoplectic

@Autoplectic Autoplectic commented Oct 6, 2026

Copy link
Copy Markdown
Member

Full correctness review and breaking refactor of sofic, as a single PR (the same 9 commits as the stacked PRs #17–#21). Old names are not kept as aliases; CHANGELOG.md has the full old → new migration table.

Correctness fixes

  • All information quantities are in bits, including log-likelihoods, topological entropy, and collision entropy. AIC/BIC/WAIC keep the standard deviance scale; standard errors are unchanged.
  • TMC edge multiplicities now count in entropy and the Parry measure.
  • SFT construction raises instead of silently truncating at max_states.
  • Exact Fischer covers, following Lind & Marcus (right = right-resolving; the names were backwards). New exact Krieger covers.
  • SlidingBlockCode.apply keeps constraints longer than the window.
  • Stack CSSR no longer drops histories during determinization, which used to produce zero-mass states.
  • The canonical VPA no longer accepts words with pending calls.
  • prime_residuals was wrong (4 instead of 3 for Σ*aΣ), and atoms returned quotients; both are now exact.
  • Smaller fixes: exact crypticity, Markov start laws, zero-mass sampling, equivalent() alphabet, Baum-Welch on impossible data, Büchi empty loops, TikZ LaTeX output, and narrowed broad except Exception fallbacks.

New constructions

  • Canonical RFSA (Denis et al. 2002) and the maximized prime átomaton (Maarand & Tamm 2022). The átomaton need not be atomic, so it now subclasses NFA.
  • Faithful NL* (Bollig et al. 2009) and an exact equivalence oracle.
  • VPA: Alur–Madhusudan determinization; concrete union, intersection, complement, difference, concatenation, and star; emptiness, universality, inclusion, and equivalence; conversion to single- and multiple-entry form. The same operations are available on NWAs.

Restructure and renames

  • Removed duplicated code: shared transition matrices, start vectors, CSSR counters and significance tests, NFA simulation, label formatting, and example processes.
  • sofic.inference.hmm and sofic.inference.cssr replace generators/*_inference.py.
  • sofic.automata is split into the learning, vpa, enumeration, and canonical subpackages.
  • Learners are named learn_<target>_<method>, the history parameter is max_history, automata operations use full words, and example factories are snake_case only.

Testing

  • 1133 tests pass, including new property tests that check every VPA and canonical-automaton construction against brute-force reference semantics.
  • ruff is clean, and a full Sphinx build passes with -W.

After merging, tag v0.3.0.

Ryan James and others added 9 commits October 6, 2026 14:26
Ryan James @cursoragent
Fix confirmed correctness bugs: bits everywhere, TMC multiplicity, ex…
bf637c7 
…act Fischer/Krieger covers

- Report topological entropy, collision entropy, and HMM log-likelihoods in bits;
  model-selection criteria convert to natural logs to keep their standard scale.
- Count ATTR_MULTIPLICITY in TMC adjacency matrices (entropy and Parry measure).
- Raise instead of silently truncating SFT presentations at max_states.
- Markov words of length 0 and sample_path respect the initial distribution.
- Raise a clear error when sampling from an initial law with no mass.
- Exact crypticity C_mu - E in block_entropy_estimates(use_exact=True).
- Narrow broad except-Exception fallbacks that silently changed results.
- equivalent() compares over the union of both automata's alphabets.
- Exact right/left Fischer covers (Lind-Marcus convention: right = right-resolving)
  and new exact right/left Krieger covers.

Co-authored-by: Cursor <cursoragent@cursor.com>
Ryan James @cursoragent
Resolve suspected issues: sliding block images, Baum-Welch zero-proba…
9248860 
…bility data, Buchi empty loops

- SlidingBlockCode.apply tracks the source presentation so constraints longer
  than the window survive, and rejects block maps that miss an allowed block.
- Baum-Welch raises when every sequence is impossible and warns when some are.
- Buchi lasso acceptance rejects an empty loop (not an omega-word).
- suggest_lmax encodes symbols injectively instead of via repr.
- Tests: Viterbi on impossible data, YAML round trips for every cover class and
  SymbolicModel. ALERGIA's Hoeffding test, SlidingBlockCode.compose, and
  TextileSystem.induced_code were checked and are correct as written.

Co-authored-by: Cursor <cursoragent@cursor.com>
Ryan James @cursoragent
Update topological-entropy doctests and prose for bits
8687e1c 
Co-authored-by: Cursor <cursoragent@cursor.com>
Ryan James @cursoragent
Implement canonical RFSA, maximized prime atomaton, exact residuals/a…
59039db 
…toms, and NL*

- ResidualTable decides residual inclusion and union coverage exactly on the
  minimal DFA; canonical_rfsa_from_language builds the Denis-Lemay-Terlutte
  canonical RFSA (was a relabeled minimal DFA).
- Maximized prime atomaton = reverse of the canonical RFSA of the reversed
  language (Maarand & Tamm 2022). It need not be atomic (Tamm 2015), so it now
  subclasses NFA rather than AtomicAutomaton; CanonicalRFSA.dual() and
  MaximizedPrimeAtomaton.dual() reverse between them.
- ResidualFiniteStateAutomaton.validate checks every state accepts a residual.
- prime_residuals, atoms, prime_atoms are exact (were bounded-length checks that
  tested equality instead of union, and returned quotients as atoms).
- Faithful NL* (Bollig et al. 2009) learning the canonical RFSA, a reversed
  learner for the prime atomaton, NL*-style table extraction, and an exact
  AutomatonEquivalenceOracle returning shortest counterexamples.

Co-authored-by: Cursor <cursoragent@cursor.com>
Ryan James @cursoragent
Concrete VPA constructions, decision procedures, modular conversion, …
d109643 
…and a correct canonical VPA

- Normalized VPA form (explicit bottom, guarded returns); documented wildcard
  returns as firing on every stack symbol and on the empty stack with a bottom.
- Alur-Madhusudan determinization; DeterministicVisiblyPushdownAutomaton.from_vpa
  determinizes nondeterministic input instead of raising.
- Concrete union, intersection, complement, difference, concatenation, and
  Kleene star (boundary-bit construction for per-factor empty stacks); the lazy
  CompositeVisiblyPushdownAutomaton and the *_vpa free functions are removed.
- Emptiness via well-matched summary saturation, accepted_word witnesses,
  is_universal, includes, equivalent, and has_unmatched_word.
- to_single_entry / to_multiple_entry convert well-matched VPAs into modular
  form; modular minimize converts automatically when no modules are given.
- CanonicalVisiblyPushdownAutomaton raises NonWellMatchedLanguageError on
  languages with pending calls/returns, and no longer accepts pending calls or
  depends on class representatives (joint top-level/nested congruence).
- NestedWordAutomaton operations delegate through the tagged VPA encoding.
- Property tests check every operation against brute-force reference semantics.

Co-authored-by: Cursor <cursoragent@cursor.com>
Ryan James @cursoragent
Document NestedWordAutomaton operations delegated through VPAs
7ea187a 
Co-authored-by: Cursor <cursoragent@cursor.com>
Ryan James @cursoragent
Remove duplicated code across generators, CSSR, automata, viz, and ex…
1db9609 
…amples; fix stack CSSR history loss

- generators/matrices.py is the one home for symbol-labeled joint matrices,
  stationary/limit laws, and start vectors (model vs stationary policy); shared
  word enumeration and entropy helpers; StochasticModel.reverse flattened.
- CSSR family: shared suffix scan (_suffix_counts.py) and significance tests
  (_morph_tests.py) for process CSSR, transCSSR, and stack CSSR; dead helpers
  (_cssr_homogenize and others) removed. The two G-tests were already identical
  (scipy applies the same Yates correction); now documented.
- Automata: one run_nfa, one bounded equivalence-oracle base, shared word
  iteration, removed intersect/concatenate/star aliases, VPA membership simulates
  the normalized form used by the constructions.
- Viz: TikZ labels reuse the shared edge spec and probability formatter; fixes
  uncompilable \midcall / \uparrowA, escaped \times, and escaped sympy LaTeX.
- Examples: nine duplicate process pairs now share one implementation; one
  stationary helper; same-name/different-process pairs cross-referenced.
- Stack CSSR: determinization replaced a split state with its largest bucket,
  dropping histories that never emitted the splitting symbol (stale mappings and
  zero-mass states that crashed validation); they now stay, and states with no
  edge into the kept set are pruned.

Co-authored-by: Cursor <cursoragent@cursor.com>
Ryan James @cursoragent
Restructure: inference moves to sofic.inference; automata split into …
5b7023b 
…learning, vpa, enumeration, canonical packages

- sofic.inference.hmm (filtering, em, information) replaces
  generators/hmm_inference.py; sampling moves to generators/sampling.py.
- sofic.inference.cssr (counts, significance, process, subtree, transducer,
  stack) replaces generators/epsilon_inference.py, epsilon_transducer_inference.py,
  stack_inference.py and their private helpers; the spectral wrapper joins
  inference/spectral.py.
- sofic.automata.learning (active, rpni, edsm, dfasat, alergia, papni, nlstar,
  observation), sofic.automata.vpa (base, operations, deterministic, modular,
  canonical, simulation), sofic.automata.enumeration (icdfa, idfa, words), and
  sofic.automata.canonical (residual, rfsa, atomaton, dual).
- Old modules deleted without shims; docs pages and toctrees moved with them.
- Doctests and prose updated for topological entropy in bits.

Co-authored-by: Cursor <cursoragent@cursor.com>
Ryan James @cursoragent
Rename public API for clarity and add CHANGELOG for 0.3.0
2590b39 
- Learners follow learn_<target>_<method> (learn_epsilon_machine_cssr, ...);
  Lmax/L -> max_history; suggest_max_history; forward/backward normalize=;
  joint_block_distribution(block_length=); quasi symbol_matrices; word helpers
  private behind model methods. sofic.inference.cssr/spectral are modules again.
- Automata: generator_product, transducer_product, compose_transducers,
  compose_transducer_generator, determinize_wheeler, minimize_wheeler,
  encode_dyck_word(s), icdfa_/idfa_ prefixed helpers,
  ModularVisiblyPushdownAutomaton.
- Shifts: ProductAlphabetShift, covers from_presentation, *_cover functions;
  top-level from_yaml alias removed.
- Examples are snake_case only; cmpy duplicates of curated examples removed and
  look-alike processes given distinct names.
- CHANGELOG.md lists behavior changes, new features, module moves, and the full
  old -> new migration table.

Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced Oct 6, 2026
Closed
Closed
Closed
Closed
Closed
@Autoplectic
Autoplectic merged commit 496818f into main Oct 7, 2026
60 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Autoplectic