smfeval: score the belief, not just the mean

A SLAM filter reports a pose and a covariance. APE/RPE check the pose. smfeval checks whether the covariance is honest.

Illustration, not smfeval output. smfeval emits the text verdict below; the figure shows what that verdict means geometrically. FAST-LIO2 on Oxford Spires keble-college-02. The estimate (blue) tracks the black track to 6 cm APE, which is an excellent APE, but the filter's reported 90% region is millimetres wide, so the error is about 39x the reported bound. The black track the figure labels "ground truth" is the reference track smfeval scores against. The heading rose is context only; the score is over translation. The belief is wrong where the mean is right, and that per-pose gap is what smfeval scores. (Data: Oxford Spires, CC BY-NC-SA 4.0.)

Try it now

notebooks/figure1_verdict.ipynb reproduces the headline verdict on one Oxford Spires sequence end to end (install, fetch data, verdict, NEES-vs-reference plot).

Install

pip install smfeval

The only dependencies are NumPy and SciPy (Python 3.10+).

Score a filter

$ smfeval nees estimate.SQUARE reference.tum --ref-body-frame lidar
median NEES 1.04e3   (calibrated: 2.37)
covariance scale gap k = 441, ~21x too tight per axis
90% coverage: 0.000  (calibrated: 0.900)

FAST-LIO2 on Oxford Spires christ-church-03. See exporters/fast_lio2/VALIDATION.md for the full reproduction. The hero figure above shows a different Spires scene, keble-college-02.

No .SQUARE file? smfeval needs a covariance for every pose, not just the poses, but it does not need the SQUARE format. If your filter outputs covariances, pass a plain TUM file plus a --cov sidecar. If it does not, Your filter doesn't write SQUARE yet? shows how to get them.

Under a calibrated belief the per-pose translation NEES has a known reference median of 2.37 (NEES is the error measured in standard deviations, squared). The scale gap k = median NEES / 2.37 is the factor by which the published covariance is too tight, so each axis is off by about a factor 21. Here the filter's 90% credible ellipsoid never contains the reference. smfeval score goes further. It localizes the regime that is wrong (bulk vs tail) and emits structured diagnoses with recommendations.

No reference? Run two filters and score them against each other

$ smfeval pair a.SQUARE b.SQUARE
matched 3101 pose pairs, scored 3101  (join 1.00, median gap 0.0 ms)
propriety caveat: pairwise scores are strictly proper only under a
honest reference sigma and independent errors; both violations push
conservative, so NEES_pair lower-bounds miscalibration.

pairwise median NEES 56.1   (calibrated: 2.37)
pairwise scale gap k >= 23.7, >=4.87x too tight per axis  (lower bound)
verdict: optimistic  (ANEES 71.7 vs chi2 interval [2.91, 3.09])

An elevated pairwise NEES certifies overconfidence with no reference consulted. Filter A is aligned to filter B directly and the difference is scored under the summed covariances. Common-mode error and an understated reference covariance both push the statistic down, so the verdict is a lower bound on the miscalibration.

---

Your filter doesn't write SQUARE yet?

smfeval needs your filter's per-pose covariance, not just its poses, but you do not have to adopt the SQUARE format to provide it. Two on-ramps are documented in SQUARE_spec.md.

• Wide TUM. Standard TUM pose columns plus the 21 row-major lower-triangle entries of the 6x6 tangent covariance (29 columns total).
• Sidecar file. Plain TUM poses plus --cov cov.txt with timestamp c11 c21 c22 ... c66 rows.

smfeval nees est.tum ref.tum --cov est.cov --est-body-frame imu --ref-body-frame imu

Most filters compute a covariance internally and never publish it. For four popular LiDAR-inertial filters the export already exists. exporters/ carries the audited few-line diff that makes FAST-LIO2, Faster-LIO, Point-LIO, and I2EKF-LO publish their belief, each with its pinned upstream commit, a bag-to-SQUARE converter, and a validation run on a named public sequence. Contributions follow the PR template, with smfeval validate --strict as the mechanical gate.

The full report

smfeval score est.SQUARE ref.tum produces the complete analysis.

=== smfeval scoring report ===

Synchronization
  Mode:                   nearest
  Pairs matched:          309 / 310
  Dropped:                1
  Timestamp gap (ms):     median 0.04, p95 7.57, p99 8.55
  Sync risk (v·Δt / σ):   median 0.0099, p95 1.6363, p99 1.7755
                          [warning] 91 pairs (29.4%) exceed risk 0.3

Alignment
  Gauge (declared):       se3
  Mode applied:           se3   (6 DoF)
  Fitted Δxyz:            (-27.8424, 24.9711, 5.6239) m
  Fit residual (m):       median 0.0094, p95 0.0262
                          6 DoF removed over 32 m of trajectory

Scores
  Translation CRPS:           mean 0.004 m   [95% CI 0.003, 0.006]   (n=309)
                              median 0.003, std 0.003, min 0.001, max 0.014
                              block length (Politis–White): 24.4
  Energy score:               mean 0.009 m   [95% CI 0.006, 0.011]   (n=309)
                              median 0.006, std 0.006, min 0.002, max 0.028
                              block length (Politis–White): 24.6
  Log score (translation):    mean -8.017   [95% CI -10.644, -5.103]   (n=309)
                              median -10.892, std 7.064, min -13.701, max 19.477
                              block length (Politis–White): 24.3
  Interval score:             mean 0.057   [95% CI 0.021, 0.095]   (n=309)
                              median 0.010, std 0.092, min 0.008, max 0.402
                              block length (Politis–White): 24.3

Calibration
  90% Mahalanobis coverage:  55.0%     (nominal 90.0%)
  Coverage test (binomial): p = 0.000  (n=309)  [warning] possible miscalibration
  Translation z-score:    mean 1.63, std 1.02

Diagnoses (attribution → action)
  [warning] sync_risk
      A competing confounder: timestamp-matching error shrinks short-window Σ_rel the same way local over-confidence does.
      · 29.4% of pairs exceed sync risk 0.3
      → Re-score with --sync=interpolate_ref to separate sync from a genuine calibration fault before trusting short-horizon verdicts.

Recommendations
  - 29.4% of pairs have sync risk > 0.3; consider cross-checking with --sync=interpolate_ref to confirm calibration findings.
  - 6 DoF removed over 32 m of trajectory; post-alignment residuals are biased low. Consider --n_to_align to fit on a prefix and score on the remainder.
  - Coverage significantly below nominal (binomial p < 0.05) — the filter is over-confident (claimed Σ too tight, reference falls outside the predicted intervals); widen process noise. Miscalibration is unlikely to be explained by sync error alone.

Point-LIO on Oxford Spires christ-church-03, reproduced from tests/fixtures/regression/real_point_lio. The report is built from:

• synchronization and alignment diagnostics;
• translation proper scoring rules (CRPS, energy score, Gaussian log score with its exact calibration/sharpness split, interval score), each with a stationary-bootstrap confidence interval;
• ellipsoidal coverage with its exact binomial test, and windowed relative-pose calibration (--rpe-window);
• track-frame bias/variance attribution;
• structured failure-mode diagnoses with recommended actions.

Only translation is scored, not orientation: a proper score on SO(3) needs a belief density whose normaliser is intractable for the natural rotation families, so rotation is left to future work (see docs/metrics.rst).

smfeval score --json prints the structured report to stdout, and --json-out writes it to a file. Both follow docs/report.schema.json.

Why several scores? Each proper rule touches a different part of the predictive translation distribution (bulk shape, tails, a chosen coverage level), so no single number suffices. docs/metrics.rst explains every metric and how to read it. SQUARE_spec.md documents the format and conventions.

Commands

Verb	What it does
smfeval nees est ref	three-line calibration verdict (median NEES, scale gap k, coverage)
smfeval pair a b	no-reference pairwise verdict (lower bound on miscalibration)
smfeval score est ref	full scoring report (--json/--json-out for machines)
smfeval validate file	header/row sanity checks (--strict is the exporter gate)

Development

uv sync && uv run pytest

That is enough to run the tests. The repo also ships a Nix dev shell (nix develop, or direnv allow for the .envrc) which additionally pins the ruff and pyright that CI runs, so lint and typecheck results match locally.

The full gate, as CI runs it — from inside the dev shell, which supplies the pinned ruff and pyright:

uv run pytest                             # 3.10 and 3.12 in CI
ruff check smfeval tests scripts
ruff format --check smfeval tests scripts
make typecheck                            # pyright
make docs                                 # sphinx, warnings are errors
uv run python scripts/check_exporters.py  # exporters/ layout

Without the dev shell, run the lint and typecheck steps at the versions .github/workflows/test.yml pins (uvx ruff@<RUFF_VERSION>, uvx pyright@<PYRIGHT_VERSION>) — that file is the only place the versions appear, and a pins CI job asserts they still match what flake.nix installs, so a nix flake update that moves either one fails until the pins follow it.

Docs live under docs/. The test suite includes property-based invariants (hypothesis), seeded Monte Carlo power tests of the verdict machinery itself (tests/test_power.py), and golden reports scored against real Oxford Spires excerpts (tests/fixtures/regression/). After an intentional change to report output, regenerate the goldens with UPDATE_FIXTURES=1 uv run pytest tests/test_regression.py and review the diff.

Citation

If you use smfeval, please cite the accompanying paper. GitHub's Cite this repository button reads CITATION.cff.

Rønning, O., Saqib, U., and Wąsowski, A. You Should Be Properly Scoring Your Odometry. arXiv:2609.25900, 2026. https://arxiv.org/abs/2609.25900

@misc{ronning2026properly,
  title         = {You Should Be Properly Scoring Your Odometry},
  author        = {R{\o}nning, Ola and Saqib, Usama and W{\k{a}}sowski, Andrzej},
  year          = {2026},
  eprint        = {2609.25900},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  doi           = {10.48550/arXiv.2609.25900},
  url           = {https://arxiv.org/abs/2609.25900}
}

Provenance

smfeval grew out of a case study of uncertainty calibration in LiDAR-inertial odometry. The trajectory data used in fixtures and the notebook derives from the Oxford Spires Dataset (CC BY-NC-SA 4.0; see the data license notes in those directories).