`sqlmesh test` should support `--use-project-index` for scoped loads

[cmgoffena13]

Summary

sqlmesh lint --use-project-index --model xxx loads only the selected models and their upstream dependencies. sqlmesh test always fully loads the project, then filters which tests run. For a pre-commit hook (or sqlmesh test models/orders.sql), parsing every model is the dominant cost. Test YAML discovery is cheap; model load is not.

The project index is a loader feature. Lint grew the flag first. Unit tests need the same subgraph: the model under test plus transitive upstream for schema resolution. They do not need downstream models.

Current behavior

• The CLI group callback constructs Context with load=True before test runs
• Context.test() only filters self._model_test_metadata
• linter.use_project_index does not apply to test (and should not; that config key is the wrong place)

Proposed CLI

sqlmesh test --local --use-project-index models/orders.sql
sqlmesh test --local --use-project-index tests/test_items.yaml
sqlmesh test --local --use-project-index models/a.sql tests/test_b.yaml

Load order should match lint:

1. Parse args (model paths, test YAML paths, --select-model)
2. Resolve to model FQNs (index maps relative path → FQN; test YAML has model:)
3. load(model_fqns=…, use_project_index=True)
4. Run the union of those tests

No paths / no --select-model still loads everything and refreshes the index (same as sqlmesh lint with no --model ).

Missing or stale index falls back to a full load and rebuilds it. Cross-project dependencies already force that fallback.

Config

Do not reuse linter.use_project_index. CLI flag first. A top-level use_project_index later is fine. Probably best to add in a flag specific for test as well for granular control.

Acceptance

• sqlmesh test --use-project-index models/a.sql does not parse unrelated model files when the index is warm
• Upstream of a is loaded so SELECT * / column resolution still works
• A selected test whose model is missing after a scoped load errors, does not warn-and-skip (create_test currently returns None and the suite can pass)
• Bare sqlmesh test --use-project-index still runs the full suite
• --local and --use-project-index compose; one does not imply the other

Extra

The baseline above still calls load_model_tests(), which globs every tests/test*.yaml, YAML-parses the full body (including fixture rows, with variable substitution), and keeps it all in memory. For a big suite, a staged models/a.sql still opens every test file even when only a's tests will run.

Would be really nice for big projects: put the join in the project index, not the fixtures.

• Store model FQN → test file paths (staged model → open only those YAMLs)
• Store test file path → model FQNs (staged YAML → load those models + upstream)
• Do not cache input/output rows; those load when the test actually runs
• Invalidate when tests/ changes (file set + mtimes), including a model: retarget, and bump the index version

Then sqlmesh test --local --use-project-index models/a.sql skips state, unrelated SQL, and unrelated test YAML. Wiring the flag without this extra still reads the whole suite.

Not required for the first cut. Optional follow-on in this issue if the index work is already in flight.