Repository navigation
CHS: integrate out missing observed factors; post-#92 CHS follow-ups - #103
Open
hmgaudecker wants to merge 18 commits into
Open
hmgaudecker wants to merge 18 commits into
hmgaudecker wants to merge 18 commits into
Conversation
Commits the uncommitted state of the gpu-01 checkout that produced the health-cognition July 2026 results (translog warm-start, 2-mixture linear): QR-JVP diagonal floor for rank-deficient covariances (chs/qr.py), the criterion sanitizer (chs/maximization_inputs.py), AF posterior-state / transition-period and constraints changes, and accompanying docs and tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Masking a non-finite loglikelihood as -1e300 with zeroed gradient entries produced fake convergence (health-cognition 2-mixture translog, one iteration at -1e300) and hides the degenerate steps that shrink the trust region. Restore the release behaviour: non-finite values reach optimagic and abort. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…pdate A component whose weight underflows the smallest normal float contributes nothing to the likelihood, but its state was still propagated. Under a translog transition it exploded to ~1e281 and then inf over later periods, so the gradient through its zero weight became 0 * inf = NaN for almost every parameter (health-cognition 2-mixture translog). Overwrite such components' state and covariance with the dominant component's after each update: the likelihood is unchanged and the gradient exact up to the underflowed weight. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
loglike_and_gradient now raises when the value or gradient is non-finite instead of handing NaN to the optimiser. With the new CHSEstimationOptions.error_handling (default "raise") set to "continue", optimagic replaces such a point by a penalty worse than the start, so a trust-region step into an explosive region is rejected and the radius shrinks -- what the removed sanitizer did for the value, without its zeroed gradients being accepted at finite points. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
…aussian Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
lower_bounds now replaces the template bound (so -inf drops it, needed for parameters in a LinearConstraint), and the start point is projected onto all LinearConstraint equalities. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
The counterpart of lower_bounds, e.g. 0 to restrict a loading's sign. Start values above a bound are lowered onto it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
`ModelSpec.missing_observed_factor_moments` maps an observed factor to the `(mean, variance)` data columns of its distribution where it is missing. In such cells the CHS filter treats the factor as N(m, v), independent of the latent states, and integrates it out in the predict step instead of raising. - process_data: validate that missing cells have a finite fill mean and a non-negative fill variance (errors name variable and period); substitute the mean into the observed-factor array and build a per-cell variance array for the declared factors that are actually missing somewhere. Undeclared factors and non-estimation purposes keep the "Variable has missings" check. - linear_kalman_predict: exact; the square-root stack gains rows sqrt(v_j) * B[:, j] / s_out, adding B diag(v) B' (with cross-equation terms). - kalman_predict: augment the sigma-point input by the possibly-missing factors (variance 0 where observed); sigma weights use n_latent + n_missing. The augmented sigma values are written into a promoted copy of the float32 observed-factor array so they are not rounded. - Models without declared moments, or whose declared factors are complete, take the unchanged code path (bit-identical likelihood). Tests compare the filter likelihood against a Monte-Carlo integral over the missing cells (1 and 2 mixtures), the predict steps against the analytic moments, the unscented path against the linear path for a linear translog, and cover the validation errors. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
`get_maximization_inputs` no longer falls back to `CHSEstimationOptions()`. That default seeds the template with a full AMN estimation, so callers that only wanted the likelihood functions silently ran AMN; the CHS state filter spent hours in AMN mixture EM to compute filtered states. - `get_maximization_inputs(..., *, chs_options)` is now required. - `get_individual_states_from_params(..., *, chs_options)` is required and forwarded with `start_params_strategy="none"`, so the filter uses the estimation's likelihood settings (e.g. the sigma-point scale) and never seeds start values. - `get_individual_states` takes `chs_options` and requires it for a `CHSEstimationResult` (unused for AF and AMN results). - Tests and docs notebooks pass options explicitly; new tests check that the filter never seeds, forwards the sigma-point scale, and rejects a CHS result without options. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
estimate_chs never forwarded the gradient split, so every estimation computed the full-sample gradient at once. Models with many possibly-missing observed factors (augmented sigma points) need several hundred GB for it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
…lumns Repairs from the round-0 review of integrating out missing observed factors: - Endogenous CHS models reject positive-variance missing inputs in periods with an outgoing transition: both half-steps would need the input's joint posterior. Zero-variance fills and complete inputs stay supported. - Promote fill variances to the covariance dtype before the square root, so float32-stored moments meet float64 precision on both predict paths. - Select each physical data column once when moment columns are shared across factors or alias another selected column. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
Callers pass state ranges of the latent factors, but observed factors are plotted as inputs too, which raised a KeyError. Missing ranges now come from the data. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
…al factors Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
`_calculate_kde_for_3d` returns `z[i, j]` at `(x[i], y[j])` (np.mgrid), while `go.Contour` reads rows as y. With equal grid sizes the transposed density plotted without error, so every pairwise contour showed the two factors' roles swapped against the axis labels and ranges. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
A missing observed factor in one of the 50 draws made the mean NaN at every grid point, so the plots were empty. Draws now come from persons with all other factors observed, with a fixed seed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
This was referenced Oct 18, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #103 +/- ##
==========================================
+ Coverage 95.83% 95.98% +0.15%
==========================================
Files 142 144 +2
Lines 14126 14729 +603
==========================================
+ Hits 13537 14137 +600
- Misses 589 592 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`from_dict` silently ignored any key it did not read, so a whole `estimation_options` block (`n_mixtures`, `robust_bounds`, `bounds_distance`) was dropped and the model was built with the defaults. Check the top-level keys against the ones `from_dict` reads, including `measurement_models` and `missing_observed_factor_moments`, and raise `ModelSpecInitializationError` on anything else. Re-derived from e27a16a (branch modelspec-reject-unknown-keys, cut from an old main whose key set was narrower). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrate out missing observed factors in the CHS filter, plus the CHS follow-ups that
af-calendar-adaptercollected after #92 was squash-merged.AF-branch follow-ups (8 Oct–12 Oct, not in #92)
error_handlingto optimagic.estimate_chs: tighten lower bounds on chosen parameters, replace bounds and start values on linear equality constraints,upper_bounds.Missing observed factors
chs_optionsis required inget_maximization_inputsand the CHS state filters;split_datasetmoves ontoCHSEstimationOptions.ModelSpec
ModelSpec.from_dictraisesModelSpecInitializationErroron unrecognized top-level keys instead of dropping them silently.Figures
The Pro closure audit of the missing-observed-factors change (round 1) is not closed yet.
Rebased onto main (c6e47c5); this also removes the stale
.readthedocs.ymlthat broke the docs builds of the stacked PRs.Tests:
pixi run -e tests-cpu pytest tests -m 'not long_running' -n 2on 9863a78 (#102, which sits on this branch): 859 passed.prek run --all-files(incl. ty) passes.https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq