Add JOSS paper for submission by vahid-ahmadi · Pull Request #201 · PolicyEngine/microimpute

vahid-ahmadi · 2026-09-14T17:31:12Z

Summary

Adds paper.md and paper.bib for submission to the Journal of Open Source Software
Adds CITATION.cff, CODE_OF_CONDUCT.md and .github/workflows/draft-pdf.yml, none of which the repository had
Paper is ~1,365 words, inside JOSS's 750–1,750 range, and carries an architecture figure in the style of Add JOSS paper for submission policyengine.py#264

Follows PolicyEngine/policyengine.py#264, which was accepted and published.

This is a different paper from `paper/`

paper/main.tex is a 39-page empirical study — SCF-to-CPS wealth imputation, an SSI policy simulation, cross-dataset benchmarking. JOSS wants a short software paper and will desk-reject a research article, so this is a new, software-focused paper.md that cites the working paper as the validation study rather than replacing it.

The framing leads on the finding that justifies the package existing: benchmarking across six datasets alongside the wealth application shows no method dominating — QRF wins on nonlinear relationships, while matching achieves the lowest mean rank overall because it better preserves marginals. If performance is dataset-specific, the useful contribution is a harness that measures it, which is a stronger claim than "five imputation methods".

JOSS requirements

Blockers before submitting

Must fix — correctness

A pre-submission audit of the models turned up a set of silent correctness bugs. A reviewer will run these methods, and they produce plausible wrong numbers rather than errors. A second round of review, with every finding independently reproduced by an adversarial verifier before filing, added #207-#213.

Normalization preprocessing re-fits on the receiver, erasing the level difference between datasets #202 — normalization preprocessing re-fits on the receiver, erasing the level difference between datasets. Imputed mean 29.83 against a true 90.05 in the reproduction
Zero-inflated quantiles ignore the zero mass and are non-monotone #203 — zero-inflated quantiles ignore the zero mass; 87% of rows have non-monotone quantiles. Zero-inflated targets are the motivating case for this package
Default QRF grows trees to purity, giving miscalibrated quantiles and inert sample weights #205 — default QRF grows trees to purity, so nominal 10th/90th percentiles cover 30%/70%, and sample weights become inert
Model API and robustness defects found in a pre-submission audit #206 — model API and robustness defects: QuantReg.fit only fits the median, Matching returns identical values for every quantile, intercepts dropped on constant predictors, weights dropped in two places, train_size and random_state inert
Test suite fails with 8 errors on a clean checkout without rpy2 #204 — fixed and closed by Fix tests that pass for the wrong reason, and export ZeroInflatedImputer #217. The suite ran 8 failed, 4 passed without rpy2 because Matching was named unconditionally; it now builds the model list from what imported
Per-variable QRF models share one seed, making jointly imputed variables comonotonic #207 — fixed and closed by Give each per-variable QRF model its own seed #214. Per-variable seeds are derived and bounded, an unknown variable now raises rather than silently reusing the first target's draws, and QRF(seed=) validates at construction
Log loss is computed from class labels, and integer counts are silently switched to classification #209 — log loss is computed from class labels via fabricated 0.99/0.01 probabilities, so it is an affine function of accuracy rather than a proper score, and it disagrees with cross_validate_model on the same model and data (2.14 against 0.79, truth 0.69). Integer targets with fewer than ten levels are also silently switched to classification
Matching silently substitutes the training mean on failure, and that score is fed to Optuna #210 — fixed and closed by Prune failed Matching trials instead of scoring the training mean #215. Failed tuning trials are pruned rather than scored as a mean-predictor, the all-pruned error reports the underlying cause, and n_failed_records is reset per call and mirrored onto result.attrs
OLS random quantile sampling draws only ten residual values, shared across calls and variables #208 — OLS random quantile sampling draws ten residual values regardless of row count, shared across calls and variables. Lower priority: the path is opt-in and autoimpute never reaches it

Must fix — compliance

microimpute has no licence, so all rights are reserved #197 — no LICENSE file, no licence in packaging metadata. This is the first item on the JOSS checklist, and it also means the five downstream repositories currently depend on all-rights-reserved code
Read the package version from installed metadata #200 — __version__ reported 1.1.2 while the package was 3.1.1 (PR open, rebased and green; fallback is now PEP 440 parseable and tested)
Wheel ships subpackages as package data, sweeping in stale __pycache__ and working-tree junk #211 — fixed and closed by Declare subpackages instead of shipping the source tree as package data #216. Subpackages are declared rather than swept in as package data, with py.typed, the full author list, per-version classifiers and project URLs. The licence metadata half stays with microimpute has no licence, so all rights are reserved #197
Create a GitHub Release from a tag and archive to Zenodo for a DOI — there are six tags but zero releases
Expand .github/CONTRIBUTING.md beyond its current 12 lines to cover dev setup, running tests, reporting issues, and where to seek support

Should fix — reviewers will raise these

Three competing docs configs, and no installation page or API reference #198 — three competing docs configs, no installation page in the docs, no API reference
Python version matrix does not cover the declared supported range #199 — the Python matrix does not cover the declared >=3.12,<3.15 range; 3.13 never runs on PRs, 3.14 never on main, and the full suite effectively runs on one version
Add a usage code example to the README — the API section is prose only
Resolve or close draft PR Make microimpute.Imputer the canonical regime-gated sequential imputer #196, which questions which class is the canonical imputer; an open question about the central API undercuts the documentation a reviewer is asked to assess
Add CI to prevent breaking changes in policyengine-us-data #134 / PR Add downstream CI to prevent breaking policyengine-us-data #135 — downstream CI against policyengine-us-data, open since October
Sequential imputation runs out of memory with many variables #96 — sequential imputation runs out of memory with many variables, a known limit in a headline feature
"Normalized" mutual information divides nats by bits and drifts with sample size #212 — compute_predictor_correlations documents a normalized mutual information matrix, but divides MI in nats by an entropy in bits whose continuous-column value is log2(n_distinct). Two identical variables score 0.55 rather than 1.0, and the value drifts with sample size
Pre-submission cleanup: API exports, validation, CI gaps, vacuous tests, dead code, stale docs #213 — grouped cleanup: ZeroInflatedImputer is undocumented and unexported although the paper describes it; autoimpute() cannot reach four Imputer.fit parameters; validation.py has no test file; MDN tests are gated out of CI by a changed-filename grep; five tests cannot fail or assert nothing; dead config; and four stale docstrings, including a compare_distributions example that is wrong on both printed values

Worth considering

Move microimputation-dashboard/ to its own repository — a Next.js app roughly doubles the review surface for something that is not the Python package
Add a README in paper/ clarifying that the manuscript is an unpublished working paper distinct from this submission

An honest note on impact

Every dependent is inside the PolicyEngine organisation: policyengine-uk-data, uk-trade-shock-study, uk-public-services-imputation, nics-exemption-inactive-employees, imputation-paper. There is no identifiable external user, and the repository has 0 stars. JOSS does not require external adoption but reviewers weigh it. The strongest honest framing is production dependency — a microsimulation data pipeline depends on this — rather than community uptake, and that is how the paper puts it.

Authorship to confirm

Currently María (corresponding), Max, Nikhil, Vahid — by commit share that is 354 / 101 / 45 / small. @baogorek contributed 15 commits and is acknowledged rather than listed; worth confirming that is right with him.

vercel · 2026-09-14T17:31:19Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
microimpute-dashboard	Ready	Preview	Sep 21, 2026 2:17pm UTC

vahid-ahmadi · 2026-09-21T09:20:00Z

Reviewed as a JOSS reviewer would, checking the paper's assertions against the code rather than reading them. The mechanics are in good shape — 1,324 words, all required sections present, all 14 bib keys cited with no orphans, every DOI present resolving 200, CITATION.cff validating against schema 1.2.0, and the functions named in the paper all existing at the lines claimed. The optional-extras caveat and the "3 without optional extras" table cell are honest.

But there is a hard blocker and several claims the current code contradicts. Hold this until they are resolved.

Blocker

No LICENSE file. git ls-files | grep -i licen is empty, pyproject.toml has no license field, and the README has no licence section. JOSS requires an OSI-approved licence in a plain LICENSE file at the repository root; the submission is rejected at pre-review without it. Add MIT license #220 adds it — that needs to land first, along with license in pyproject.toml and in CITATION.cff.

Claims the code contradicts

The uniform fit/predict claim is false today. (Fixed in e8efaf3: reworded to "expressed as predictive distributions… scored on the same footing across a common grid", which is true on main. The underlying QuantReg defect stays open as Model API and robustness defects found in a pre-submission audit #206.) Software Design says "Every model implements fit(...) and predict(X_test, quantiles)… That uniformity is what makes the comparison possible." But Model API and robustness defects found in a pre-submission audit #206 documents quantreg.py:397 hardcoding q = 0.5, so predict(te, quantiles=[0.1, 0.5, 0.9]) raises RuntimeError: Failed to access 0.1 quantile for prediction for QuantReg while working for OLS and QRF. This is the paper's central design claim. Fix pre-submission imputation and evaluation correctness #219 fixes it.
The cross-frame standardisation claim is false today. (Fixed in e8efaf3. Confirmed the mechanism directly: autoimpute_helpers.py:147 captures the donor's transform params and :163 discards them with , _. The sentence now claims only consistent categorical encoding and describes numeric transformations as optional.) The paper says predictors are "standardised consistently across the two frames, so a model fitted on the donor can be applied to the receiver". Normalization preprocessing re-fits on the receiver, erasing the level difference between datasets #202 shows autoimpute_helpers.py:162-173 standardises donor and receiver independently, discarding the donor's transform — which erases exactly the level difference imputation exists to carry. Reproduction in the issue: imputed mean 29.83 against a true ~90.
The zero-inflated wrapper the paper advertises is broken. Zero-inflated quantiles ignore the zero mass and are non-monotone #203: zero_inflated.py:566-579 gives non-monotone quantiles ignoring the zero mass — at q=0.9, true 3.81 against predicted 0.00, with ~60% zeros at every quantile. Either gate on Fix pre-submission imputation and evaluation correctness #219 or drop the sentence.
The sample-weights sentence under-claims, and the package's own error text contradicts it. (Fixed in e8efaf3 — and it was wrong in both directions. QRF is now named; matching is removed, because statmatch_hotdeck.py passes weight_don to R's NND.hotdeck, which has no such parameter — that belongs to RANDwNND.hotdeck. Unverified without rpy2, so matching stays off the list until someone with R settles it.) The paper says "Ordinary least squares and statistical matching accept survey weights directly… quantile regression and mixture density networks raise an explicit error." QRF supports weights too — qrf.py:101-102 passes sample_weight to RandomForestQuantileRegressor.fit, and it bites end-to-end (bimodal target, weights 1 vs 99 on the upper mode: unweighted median 0.0, weighted median 100.0 — the raw quantile_forest library returns 0.0 either way, so this is microimpute's own behaviour). Meanwhile quantreg.py:322 tells users "QuantReg does not support sample weights. Use QRF or OLS for weighted imputation." Rewrite to name QRF, and reconcile with Default QRF grows trees to purity, giving miscalibrated quantiles and inert sample weights #205's claim that the default min_samples_leaf=1 makes those weights inert in practice.
The benchmarking rankings are about to be invalidated. The Summary asserts "quantile regression forests win where relationships are nonlinear, and matching better preserves marginal distributions". Fix pre-submission imputation and evaluation correctness #219's own description says these corrections "change imputed values and model rankings. Regenerate paper benchmarks… before submission", and that Matching's unsupported quantiles are now "excluded from distributional comparison" — which would remove the method that sentence ranks. Default QRF grows trees to purity, giving miscalibrated quantiles and inert sample weights #205 also means the published rankings were computed with a miscalibrated QRF. Merge Fix pre-submission imputation and evaluation correctness #219, regenerate, then rewrite this paragraph.

Repository requirements a reviewer checks

No CONTRIBUTING.md. The README's contributing section is two sentences with nothing on running tests, make format, or the changelog-fragment requirement CI enforces. JOSS requires community guidelines for contributing, reporting issues and seeking support. (CODE_OF_CONDUCT.md, added here, is good and has a real contact.)
Installation instructions do not reach the five methods the paper claims. The README says pip install microimpute and lists all five, but never mentions the [matching] extra (rpy2 plus R's StatMatch, a system dependency) or [mdn] (pytorch-tabular). A reviewer following it gets three methods and no explanation.

Smaller

The working paper is an unarchived PDF in this repository. juaristi2026microimpute is an @unpublished pointing at paper/main.pdf on main — yet the 167% SSI figure, the six-dataset benchmark and every ranking claim rest on it alone. A reviewer cannot verify a moving target inside the repo under review. Deposit on Zenodo or arXiv and cite the DOI.
Internal inconsistency in the dataset count: "Benchmarking across six further datasets, alongside the wealth application" then "With six benchmark datasets, the rank differences are not robust". Six plus wealth, or six including it?
Add 10.32614/CRAN.package.StatMatch to dorazio2022statmatch. The other missing DOIs (meinshausen2006qrf, bishop1994mdn, pedregosa2011scikit) legitimately have none.
CITATION.cff lacks version, date-released, license and doi — all expected at the archived-release step, and version will need to track pyproject.toml (see Read the package version from installed metadata #200).
Pin openjournals/openjournals-draft-action@master to a tag or SHA.
A sentence in the repo README distinguishing the root paper.md (this submission) from the paper/ directory (the research manuscript), which will otherwise confuse a reviewer.
paper.bib:claude2026 is dated 2026 but its note says "Opus 4".

Suggested order

#218 → #200 → #217 → rebase and land #219 → regenerate benchmarks → revise and merge this. The Lint failure here is inherited from main, not this PR.

@juaristi22

The DEFAULT_MODEL_PARAMS test asserted the whole mapping as a literal against itself, which froze values nothing in the package reads - it has no callers inside microimpute and the real defaults live in each model. It now checks the keys and shapes, which is what a downstream caller relies on. The constant itself stays, per @juaristi22's compatibility fix; VALID_YEARS keeps its exact assertion because two notebooks depend on those years. Drops test_published_notebook_config_imports: it parsed an 8 MB notebook to assert what the two tests above it already assert, and would fail as a confusing KeyError if either notebook were renamed. available_models() now returns None. autoimpute's own default is already dependency-aware, so the helper was duplicating production logic and, as written, only exercised Matching and MDN when MDN happened to be installed. The Imputer.fit weight docstring said weights go to the learner's weighted-fit interface without noting that QuantReg and MDN raise NotImplementedError. The paper in #201 makes claims about exactly this. Also drops the docs reformatting, which belongs to #218.

@juaristi22

The DEFAULT_MODEL_PARAMS test asserted the whole mapping as a literal against itself, which froze values nothing in the package reads - it has no callers inside microimpute and the real defaults live in each model. It now checks the keys and shapes, which is what a downstream caller relies on. The constant itself stays, per @juaristi22's compatibility fix; VALID_YEARS keeps its exact assertion because two notebooks depend on those years. Drops test_published_notebook_config_imports: it parsed an 8 MB notebook to assert what the two tests above it already assert, and would fail as a confusing KeyError if either notebook were renamed. available_models() now returns None. autoimpute's own default is already dependency-aware, so the helper was duplicating production logic and, as written, only exercised Matching and MDN when MDN happened to be installed. The Imputer.fit weight docstring said weights go to the learner's weighted-fit interface without noting that QuantReg and MDN raise NotImplementedError. The paper in #201 makes claims about exactly this. Also drops the docs reformatting, which belongs to #218.

@juaristi22

…ter (#217) * Fix tests that pass for the wrong reason, and export ZeroInflatedImputer tests/test_autoimpute.py named Matching unconditionally whenever MDN was absent, though HAS_MATCHING was already computed and unused. Without rpy2 that raised NameError before the call under test ran, which is the whole of #204: eight errors that looked like failures of the code under test. An available_models() helper builds the list from what actually imported. That NameError was also masking a real problem. Four pytest.raises calls had no match=, so they passed on any exception - including that NameError. test_autoimpute_missing_predictors was passing on it rather than on the missing-column error it claims to test, which is visible now that each raises names the message it expects. A fifth test asserted inside an except branch, so it would have passed silently if predict ever stopped raising, and a sixth skipped its assertion when both losses were NaN, which is exactly the case #210 produces. ZeroInflatedImputer was reachable only by full module path despite being a documented feature of the paper. DEFAULT_MODEL_PARAMS and VALID_YEARS had no references anywhere in the package, and the Imputer.fit docstring still described the bootstrap resampling scheme that was replaced by native sample_weight support. Fixes #204 * Allow Matching's own message in the missing-predictor test Matching surfaces a missing predictor from R rather than from pandas, so the message differs from the other models'. Only visible where rpy2 and StatMatch are installed. * Fix issues from review: preserve public config compatibility * Act on review: loosen frozen tests, correct the weight docstring The DEFAULT_MODEL_PARAMS test asserted the whole mapping as a literal against itself, which froze values nothing in the package reads - it has no callers inside microimpute and the real defaults live in each model. It now checks the keys and shapes, which is what a downstream caller relies on. The constant itself stays, per @juaristi22's compatibility fix; VALID_YEARS keeps its exact assertion because two notebooks depend on those years. Drops test_published_notebook_config_imports: it parsed an 8 MB notebook to assert what the two tests above it already assert, and would fail as a confusing KeyError if either notebook were renamed. available_models() now returns None. autoimpute's own default is already dependency-aware, so the helper was duplicating production logic and, as written, only exercised Matching and MDN when MDN happened to be installed. The Imputer.fit weight docstring said weights go to the learner's weighted-fit interface without noting that QuantReg and MDN raise NotImplementedError. The paper in #201 makes claims about exactly this. Also drops the docs reformatting, which belongs to #218. --------- Co-authored-by: María Juaristi <127882282+juaristi22@users.noreply.github.com>

The paper is a short software paper, distinct from the working paper in paper/, which it cites as the validation study. It is framed around the finding that motivates the package: benchmarking across seven domains shows no imputation method dominates, so the useful tool is one that measures which performs best on a user's own data. Also adds a citation file, a code of conduct, and the workflow that builds a draft PDF, none of which the repository had. The author list and ORCIDs still need confirming, and the missing licence in issue #197 blocks submission. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

- fit/predict signatures matched to the actual API - weights claim narrowed to OLS and matching - six benchmark datasets, not seven domains, with the source's hedge - optional extras (rpy2/StatMatch, PyTorch) disclosed - statsmodels and R mice weight columns corrected to No - dropped the uk-public-services-imputation claim (declares microimpute but never imports it)

Adds the donor-receiver framing, the autoimpute workflow end to end, and the predictor-analysis utilities. Cites the policyengine JOSS paper in the research impact statement and links the PolicyEngine website.

compute_predictor_correlations reports association, not contribution; only leave_one_out_analysis and progressive_predictor_inclusion measure contribution by loss.

…claims A line-by-line check against the code found three statements the package does not support. The standardisation claim was false. autoimpute_helpers.py:147 captures the donor's transform parameters and :163 discards them with ', _', re-fitting on the receiver - issue #202. A reproduction with a donor mean of 100 and a receiver shifted to a true mean of 149 returned 99.73, the donor mean exactly, so the level difference imputation exists to carry was annihilated. The sentence now claims only consistent categorical encoding and describes the numeric transformations as optional, which is what the code does. The survey-weights roster was wrong in both directions: it omitted QRF, which does accept a weight column, and credited matching with weighted donor selection, which is unverified - statmatch_hotdeck.py passes weight_don to R's NND.hotdeck, which has no such parameter. Matching is dropped from the list until that is settled with rpy2 installed. 'Every method returns quantiles of the conditional distribution' is the paper's central premise and is false as shipped: QuantReg.predict with an explicit grid raises RuntimeError (#206). Reworded to the weaker claim the code supports. Also: StatMatch's survey-weights cell goes Yes to Partly for the same NND.hotdeck reason, with the distinction explained in the prose; the log-loss claim is softened because #209 means it is not a proper scoring rule; 'distributed separately' dropped, since the dashboard lives in this repository; and the benchmarking sentence now records that matching achieves the lowest mean rank overall, which the working paper reports and the previous wording omitted. The comparison table is transposed so criteria are rows and tools are columns, with citations moved into the paragraph below - the tool names and their citations were forcing five-line-tall rows. The figure follows the policyengine.py JOSS paper's house style: a hand-written SVG in the same palette, committed alongside its PNG, in the Software Design section.

At full width the portrait diagram dominated the Software Design page.

vercel Bot deployed to Preview September 14, 2026 17:31 View deployment

vahid-ahmadi mentioned this pull request Sep 14, 2026

Model API and robustness defects found in a pre-submission audit #206

Open

vercel Bot deployed to Preview September 14, 2026 17:40 View deployment

vercel Bot deployed to Preview September 16, 2026 09:41 View deployment

vercel Bot deployed to Preview September 16, 2026 10:57 View deployment

vercel Bot deployed to Preview September 16, 2026 11:23 View deployment

vercel Bot deployed to Preview September 16, 2026 11:28 View deployment

vahid-ahmadi requested review from MaxGhenis and juaristi22 September 16, 2026 11:57

vahid-ahmadi assigned vahid-ahmadi and juaristi22 Sep 16, 2026

juaristi22 mentioned this pull request Sep 16, 2026

Fix pre-submission imputation and evaluation correctness #219

Draft

This was referenced Sep 21, 2026

Add MIT license #220

Open

Fix the Lint job, which is failing on main #218

Merged

Fix tests that pass for the wrong reason, and export ZeroInflatedImputer #217

Merged

vahid-ahmadi and others added 3 commits September 21, 2026 15:12

Retitle the paper and use British spelling

f680e57

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Use title case in the paper title

1d1f1d6

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

vahid-ahmadi and others added 5 commits September 21, 2026 15:12

Add María Juaristi's ORCID

b2e527e

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Expand software design section and cite policyengine

145b74c

Adds the donor-receiver framing, the autoimpute workflow end to end, and the predictor-analysis utilities. Cites the policyengine JOSS paper in the research impact statement and links the PolicyEngine website.

Describe the predictor-analysis utilities precisely

0316407

compute_predictor_correlations reports association, not contribution; only leave_one_out_analysis and progressive_predictor_inclusion measure contribution by loss.

vahid-ahmadi force-pushed the joss-paper branch from e8efaf3 to 556c117 Compare September 21, 2026 14:12

vercel Bot deployed to Preview September 21, 2026 14:13 View deployment

Reduce the figure to 62% width

6dc7b9b

At full width the portrait diagram dominated the Software Design page.

vercel Bot deployed to Preview September 21, 2026 14:17 View deployment

vahid-ahmadi marked this pull request as draft September 23, 2026 11:23

Navigation Menu

Add JOSS paper for submission - #201

Add JOSS paper for submission#201
vahid-ahmadi wants to merge 9 commits into
mainfrom
joss-paper

vahid-ahmadi commented Sep 14, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

vahid-ahmadi commented Sep 21, 2026 •

edited

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vahid-ahmadi commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

This is a different paper from paper/

JOSS requirements

Blockers before submitting

An honest note on impact

Authorship to confirm

Uh oh!

vercel Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vahid-ahmadi commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Blocker

Claims the code contradicts

Repository requirements a reviewer checks

Smaller

Suggested order

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vahid-ahmadi commented Sep 14, 2026 •

edited

Loading

This is a different paper from `paper/`

vercel Bot commented Sep 14, 2026 •

edited

Loading

vahid-ahmadi commented Sep 21, 2026 •

edited

Loading