Skip to content

Start Rt sampling after infection initialization - #924

Open
cdc-mitzimorris wants to merge 36 commits into
mainfrom
mem_923_start_Rt_process_post_init
Open

cdc-mitzimorris wants to merge 36 commits into
mainfrom
mem_923_start_Rt_process_post_init

Conversation

@cdc-mitzimorris

Copy link
Copy Markdown
Collaborator

This implements issue #923

  • Start Rt temporal processes after infection initialization across population, subpopulation, and mobility models; pad initialization records with starting Rt values.
  • Align weekly Rt changes with the post-initialization calendar.
  • Use supplied AR1 initial values as exact starting states, while preserving stationary initialization for ascertainment deviations.
  • Fully standardize random draws in innovation parameterizations and fix single-timepoint handling.
  • Add a reproducible weekly-Rt synthetic dataset and update integration tests to assess recovery against matching weekly truth.
  • Expand regression tests and update tutorials and API documentation for initialization and temporal-process behavior.

cdc-mitzimorris and others added 16 commits October 26, 2026 20:59
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Thank you for your contribution @cdc-mitzimorris 🚀! Your github-pages is ready for download 👉 here 👈!
(The artifact expires on 2026-11-07T13:18:27Z. You can re-generate it by re-running the workflow here.)

@codecov

codecov Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.95%. Comparing base (2441cb3) to head (68bf7f2).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #924      +/-   ##
==========================================
+ Coverage   98.89%   98.95%   +0.06%     
==========================================
  Files          59       59              
  Lines        2344     2396      +52     
==========================================
+ Hits         2318     2371      +53     
+ Misses         26       25       -1     
Flag Coverage Δ
unittests 98.95% <100.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Integer initial states cause both AR1 parameterizations to fail for multi-point trajectories because the scan carry changes from integer to floating-point dtype.

2 open issues
What changed in this PR

Implements #923 by starting Rt processes when renewal begins, rather than during infection initialization, while preserving full-length daily infection and Rt records.

Changes:

  • Sample Rt only after initialization, shift weekly calendar alignment, and pad daily records with starting values.
  • Use exact supplied AR1 starts and standardized innovation draws while preserving stationary ascertainment initialization.
  • Add reproducible weekly-Rt data, expand regression and recovery tests, and update documentation.
File Description
test/​test_temporal_processes.py Tests exact starts, standardized draws, covariance, and forecast replay.
test/​test_subpopulation_infections.py Verifies supplied AR1 starting Rt values are preserved.
test/​test_rt_initialization.py Adds shared initialization, calendar, and forecast regression tests.
test/​test_pyrenew_builder.py Checks distinct Rt and ascertainment timeline contexts.
test/​test_population_infections.py Tests exact initial Rt and shifted weekly alignment.
test/​test_datagen_weekly_rt.py Checks dataset consistency, reproducibility, and observation means.
test/​test_ascertainment.py Verifies stationary AR1 ascertainment starts.
test/​integration/​test_population_infections_he_weekly_rt.py Uses matching weekly truth for innovation-form recovery tests.
test/​integration/​test_population_infections_he_weekly_rt_state_centered.py Uses matching weekly truth for state-centered recovery tests.
test/​integration/​conftest.py Adds weekly-data fixtures and matching model configuration.
pyrenew/​latent/​temporal_processes.py Updates initialization, innovation scaling, and single-point handling.
pyrenew/​latent/​subpopulation_infections.py Starts shared subpopulation Rt processes after initialization.
pyrenew/​latent/​population_infections.py Samples post-initialization Rt and pads daily records.
pyrenew/​latent/​mobility_subpopulation_infections.py Documents the revised initialization behavior.
pyrenew/​datasets/​synthetic_CA_126_weekly_rt/​weekly_rt.csv Stores the weekly Rt truth.
pyrenew/​datasets/​synthetic_CA_126_weekly_rt/​weekly_hospital_admissions.csv Adds weekly hospital observations and expected counts.
pyrenew/​datasets/​synthetic_CA_126_weekly_rt/​true_parameters.json Records generating parameters and calendar metadata.
pyrenew/​datasets/​synthetic_CA_126_weekly_rt/​initial_infections.csv Stores the exponential initialization history.
pyrenew/​datasets/​synthetic_CA_126_weekly_rt/​expected_observations.csv Stores daily hospital and ED observation means.
pyrenew/​datasets/​synthetic_CA_126_weekly_rt/​daily_infections.csv Stores daily infections and repeated weekly Rt truth.
pyrenew/​datasets/​synthetic_CA_126_weekly_rt/​daily_hospital_admissions.csv Adds synthetic daily hospital observations.
pyrenew/​datasets/​synthetic_CA_126_weekly_rt/​daily_ed_visits.csv Adds synthetic daily ED observations.
pyrenew/​datasets/​datagen_he_CA_126.py Adds configurable, reproducible weekly-Rt generation.
pyrenew/​ascertainment/​base.py Leaves temporal initialization unset to preserve stationary AR1 starts.
pyrenew/​arrayutils.py Adds a helper for padding records with their first row.
pyproject.toml Includes the weekly-Rt dataset in wheel packaging.
docs/​tutorials/​latent_subpopulation_infections.qmd Explains exact starts and revised deviation behavior.
docs/​tutorials/​latent_mobility_infections.qmd Clarifies regional Rt timing and initialization growth.
docs/​tutorials/​latent_infections.qmd Explains post-initialization Rt sampling and process parameterizations.
docs/​tutorials/​ascertainment.qmd Clarifies stationary ascertainment starts versus exact latent Rt starts.

Review effort: Max

Comment thread pyrenew/latent/temporal_processes.py Resolved
Comment thread pyrenew/latent/population_infections.py Resolved
@cdc-mitzimorris

Copy link
Copy Markdown
Collaborator Author

@dylanhmorris ready for review

Comment thread docs/tutorials/latent_infections.qmd Resolved
Comment on lines +214 to +216
Daily baseline, deviation, and Rt records retain the full infection axis by prepending copies of their first values.
Initialization therefore adds no sampled states or innovations, and weekly records cover only the post-initialization period.
The first recorded Rt week is the calendar week containing the first renewal day; its start date may precede that day, but no initialization-only weeks are sampled.

@dylanhmorris dylanhmorris Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me check my understanding:

  • Recorded daily $\mathcal{R}(t)$ prepends (deterministic, not sampled) starting values for the days before $\mathcal{R}(t)$. The exponential growth initialization at an $r$ derived from $\mathcal{R}(0)$ makes this true in model world, so I think it's okay as long as its documented. That said, it wouldn't be my preference unless it is necessary, since other forms of initialization are possible.
  • Recorded weekly $\mathcal{R}(t)$ values do not prepend unsampled $\mathcal{R}(t)$ values.

Is there a key reason to do this? I'd prefer that the two behaviors be more consistent if that's possible. My preference (I think) is no prepending in either case, but I am persuadable.

Comment thread pyrenew/latent/base.py Resolved
Comment thread pyrenew/latent/base.py Resolved
Comment thread pyrenew/latent/base.py Resolved
Comment thread pyrenew/latent/base.py Resolved
Comment thread pyrenew/latent/base.py Resolved
Comment thread pyrenew/latent/base.py Resolved
Comment on lines +134 to +136
Rt determines its exponential initialization growth. Temporal
processes cover only post-initialization days; daily Rt records
prepend copies of the starting values for initialization.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See question about this above.

@dylanhmorris dylanhmorris left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cdc-mitzimorris question about prepending behaviors before I review further.

cdc-mitzimorris and others added 15 commits October 30, 2026 12:51
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
…ov/PyRenew into mem_923_start_Rt_process_post_init
@cdc-mitzimorris

Copy link
Copy Markdown
Collaborator Author

See question about this above.

@dylanhmorris - thanks - good catch!

We don’t need the prepending. The design included it to preserve existing output shapes: “Array shapes on the shared axis are unchanged.” Preserving those shapes isn’t a requirement here, and it leaves the tutorials and plotting code trimming off values we just added.

Instead, we should record daily Rt from the first renewal day, consistent with weekly. This would change the recorded arrays and their indexing, but leave the initialization and renewal calculations unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants