Skip to content

MAINT: Faster SQLite fixtures - #2640

Merged
Roman Lutz (romanlutz) merged 3 commits into
microsoft:mainfrom
romanlutz:romanlutz-faster-sqlite-test-fixtures
Sep 16, 2026
Merged

Roman Lutz (romanlutz) merged 3 commits into
microsoft:mainfrom
romanlutz:romanlutz-faster-sqlite-test-fixtures

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

Ordinary unit tests replayed the full Alembic migration history for every sqlite_instance, adding substantial overhead across thousands of tests. Build and validate a pristine SQLite template once per pytest worker, then copy it with SQLite's backup API into a fresh database for each test.

  • Keep sqlite_instance function-scoped, with a new real SQLiteMemory instance, engine, results directory, and the existing threaded-session behavior. The template has its own engine and is query-only; it never becomes the shared singleton.
  • Restore the previous SQLiteMemory and CentralMemory registrations after each test without clearing unrelated singleton registrations. Fixture teardown owns cleanup of its instances, avoiding process-exit callbacks that would retain thousands of objects. This is a separate lifecycle safeguard, not a separately claimed speedup.
  • Leave production constructors, migrations, schema checks, and reset_database() unchanged. Tests that exercise those paths still execute them. No workflow tracing settings, sharding, dependencies, or coverage thresholds change.

Some constructor and identity tests now explicitly request patch_central_database even though their assertions do not query the database. Constructing a target calls CentralMemory.get_memory_instance() in PromptTarget.__init__; those tests previously relied on memory left behind by earlier tests. The added function/class markers remove that test-order dependency without introducing a global autouse fixture.

Local measurements used Windows, Python 3.12.13, SQLite 3.50.4, and unchanged coverage 7.13.1 CTracer settings:

Workload Before After
Fixture setup median, no coverage 212.28 ms 1.13 ms
Fixture setup median, CTracer coverage 574.83 ms 2.38 ms
Same 176 existing memory/target/scorer tests, CTracer coverage 167.95 s 8.87 s

Fixture medians use 20 samples after two warmups and measure the complete setup. One-time template setup was measured separately at 272 ms without coverage and 786 ms with coverage per worker. These are local measurements, not an estimate of end-to-end CI savings.

Tests and Documentation

  • Added 14 regression cases covering row/schema/temp-table/view/result-directory isolation, semantic schema and Alembic revision fidelity, prior singleton/CentralMemory restoration, unrelated registrations, connection cleanup on success and failure, collectability, real resets, and concurrent writers. The fixture infrastructure has 100% line and branch coverage in a focused run.
  • Full make unit-test-cov-xml equivalent using the unchanged Makefile target: 17,107 passed, 124 skipped, 7 subtests passed; 91.96% line coverage, above the unchanged 78% gate. Run on Windows/Python 3.12 with the complete default locked dependencies, including feedgen.
  • Four-worker memory/migration/initialization/target/scorer selection: 882 passed, 1 skipped. All affected consumer modules plus regressions: 677 passed on four workers.
  • The matched 176-test subset lost 22 incidental constructor/singleton/reset/cleanup coverage lines; all 22 are covered in the passing full suite. No migration revision script lost coverage in that comparison. The configured 90% diff gate passed at 93%, including pre-existing base/origin differences. Patch-only diff coverage is not applicable because this patch changes only tests and documentation.
  • Ruff, formatting, the new fixture/regression type checks, documentation validation, and all applicable commit hooks passed. Expanded type checking of the consumer modules reports four pre-existing mock-return diagnostics, reproduced from the unchanged starting commit.
  • Updated the contributor guide with fixture ownership and explicit memory-dependency guidance. No notebooks changed, so JupyText was not run. Linux/all-extra CI and other Python versions were not run locally.

Migrate and validate a private SQLite template once per worker, then back up the schema into fresh per-test memory instances. Scope singleton and CentralMemory registration and keep cleanup owned by each fixture.

Cover schema and row isolation, migration/reset behavior, thread access, and resource cleanup. Declare explicit memory fixtures for tests that relied on state left by earlier tests, and document the lifecycle.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz Roman Lutz (romanlutz) changed the title MAINT: Speed up isolated SQLite unit test fixtures MAINT: Faster SQLite fixtures Sep 12, 2026
Declare the existing CentralMemory fixture for the 18 optional target tests that relied on state leaked by the old SQLite fixture. Document all-extra validation for fixture changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread doc/contributing/5_unit_tests.md
Clarify explicit memory dependencies, isolated SQLite use, and all-extra validation for shared fixture changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Sep 16, 2026
Merged via the queue into microsoft:main with commit 7570f84 Sep 16, 2026
49 checks passed
@romanlutz
Roman Lutz (romanlutz) deleted the romanlutz-faster-sqlite-test-fixtures branch September 16, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants