Skip to content

CONFORMANCE.md

Latest commit

 

History

History
572 lines (495 loc) · 43.2 KB

CONFORMANCE.md

File metadata and controls

572 lines (495 loc) · 43.2 KB
Raw

Conformance coverage

The MetaObjects standard ships 29 shared conformance corpora under fixtures/. Every port runs every corpus that is applicable to it and asserts the same expected behaviour against the same fixtures. This page is the inverse index: fixture → feature doc + per-port pass status, and it is the single place per-corpus counts are maintained (the port READMEs deliberately point here instead of restating them).

Not every corpus is a five-port corpus, and that asymmetry is deliberate rather than a gap — schema migration is TypeScript-owned (ADR-0015), Kotlin runs on the JVM and inherits the Java loader/render/extract engines instead of re-implementing them, and a few corpora gate tooling that only one port ships. The dark cells below say which is which.

If you are coming from a feature doc's ## Verified by section, you are in the right place. If you are wondering whether a particular fixture has a human-readable explanation somewhere, look it up in the "Fixture-to-doc mapping" table below.

Per-corpus totals

Counts are fixture directories (or scenario files, where a corpus is file-shaped); regenerate with ls -d fixtures/<corpus>/*/ | wc -l for directory-shaped corpora, find fixtures/<corpus> -name "*.yaml" | wc -l for file-shaped ones.

Corpus Fixtures TS Java Kotlin C# Python
fixtures/conformance/ (metamodel) 380 ✓ ✓ inherits via metadata-ktx ✓ ✓
fixtures/yaml-conformance/ 16 16 / 16 15 / 16 (1 ledgered: yaml-quoted-leading-zero — Java pipeline strips quotes off "007") inherits via Java 15 / 16 (1 ledgered: error-yaml-coerced-hex-in-string — YamlDotNet doesn't coerce 0xFF) 16 / 16
fixtures/verify-conformance/ 37 ✓ ✓ inherits via Java ✓ ✓
fixtures/verify-strict-conformance/ 1 ✓ — — — ✓
fixtures/render-conformance/ 16 ✓ ✓ inherits via Java ✓ ✓
fixtures/extract-conformance/ 48 ✓ ✓ inherits the shared JVM engine ✓ ✓
fixtures/output-prompt-conformance/ 17 ✓ ✓ ✓ ✓ ✓
fixtures/persistence-conformance/ 54 (48 query + 6 migration) all 54 (the 21 report-* also on SQLite, MySQL and D1) 48 query (migrations TS-only, ADR-0015); the 21 report-* also on MySQL 48 query (via Exposed); the 21 report-* also on SQLite and MySQL 48 query; the 21 report-* also on SQLite and MySQL 48 query (Postgres only: pg8000)
fixtures/api-contract-conformance/ 91 (31 core + 10 tph + 9 m2m + 2 jsonb + 2 write-through + 12 projection + 25 report) ✓ (Fastify reference + generated lane); report/ also on SQLite, MySQL and D1 ✓ (embedded HTTP + JDBC) ✓ (embedded HTTP + Exposed) ✓ (HttpListener + Npgsql); report/ also on SQLite and MySQL ✓ (FastAPI + pg8000)
fixtures/validation-conformance/ 42 cases ✓ (generated Zod + run-time runValidators) ✓ ✓ ✓ ✓ (generated Pydantic + run-time run_validators)
fixtures/registry-conformance/ 1 canonical manifest ✓ (reference emitter) ✓ ✓ ✓ ✓
fixtures/object-model-conformance/ 1 shared metadata fixture (per-port scenarios) ✓ ✓ ✓ ✓ ✓
fixtures/codegen-conformance/ 5 shared inputs; shared-provided-enum also carries the enum-type decision table every port answers ✓ ✓ ✓ ✓ ✓
fixtures/template-codegen-conformance/ 3 ✓ ✓ ✓ ✓ ✓
fixtures/template-output-render-conformance/ 5 ✓ ✓ ✓ ✓ ✓
fixtures/generator-registry-conformance/ 1 canonical manifest ✓ ✓ ✓ ✓ ✓
fixtures/provider-composition-conformance/ 9 (5 error-shape + 4 compose-load) ✓ ✓ — (JVM registry via Java) ✓ ✓
fixtures/source-resolution-conformance/ 25 cases ✓ (reference implementation) ✓ inherits via Java ✓ ✓
fixtures/scope-conformance/ 10 cases ✓ (reference implementation) — — — ✓
fixtures/dependency-conformance/ 23 cases ✓ (reference implementation) — (Phase 2) — (Phase 2) — (Phase 2) ✓ (2 of 23 assert a documented Python-only vocabulary gap instead of the corpus's full contract — see below)
fixtures/agent-context-conformance/ 4 ✓ (the emitter is TS-owned) — — — —
fixtures/metamodel-docs/ 1 ✓ (docs emit is TS-owned) — — — —
fixtures/fmt-conformance/ (#304 — meta fmt) 12 ✓ (reference) ✓ inherits via Java ✓ ✓
fixtures/field-lint-conformance/ (the verify field authoring lint) 15 ✓ (reference) ✓ inherits via Java ✓ ✓
fixtures/requirement-check-conformance/ (the verify requirement gate, ADR-0057) 43 cases ✓ (reference) ✓ inherits via Java (one Maven metaobjects:verify goal) ✓ ✓
fixtures/requirement-test-identity-conformance/ (the identity, skip state and digest of each generated requirement test, and the filter seam, ADR-0057) 26 cases ✓ (reference) ✓ identity function inherits via Java; emitted names asserted by its own generator test ✓ ✓
fixtures/naming-conformance/ 8 cases ✓ ✓ inherits via Java (RouteNaming.pluralize) ✓ ✓
fixtures/codegen-noop/ (FR-044 — reporting vocabulary: what is lowered, what stays inert) 1 model pair (reporting/with vs reporting/without) ✓ (codegen + migrate) ✓ ✓ ✓ ✓
fixtures/cube-model/ (FR-044 Plan 4 — the cube-model generator's golden Cube files, plus the canonical model's golden) 46 cases (36 trees + 10 errors) + 1 canonical golden ✓ (the exporter is TypeScript-only, spec R6; its cube lane also loads the files into a real Cube) — — — —

A ✓ means the port runs the corpus green; an explicit n / m is used where a port carries a ledgered divergence. The two ledgered YAML fixtures are documented library-vs-pipeline divergences (see the _comment block in each port's yaml-conformance-expected-failures.json for the full reconciliation note). They are tracked as known-gaps rather than silently patched — the runner treats listed fixtures as passing, but a future port-level reconciliation pass would close them.

Core corpora and template quality checks

Since ADR-0034 Amendment 3, the corpora above do two different jobs. Only the first is a promise to adopters.

  • Core — the contract. conformance/, yaml-conformance/, registry-conformance/, validation-conformance/, provider-composition-conformance/, source-resolution-conformance/, scope-conformance/, dependency-conformance/, object-model-conformance/, render-conformance/, template-codegen-conformance/ (the Mustache engine), template-output-render-conformance/, output-prompt-conformance/, extract-conformance/, verify-conformance/, verify-strict-conformance/, persistence-conformance/ (runtime reads and writes, and the TS-owned migration scenarios), agent-context-conformance/, metamodel-docs/, fmt-conformance/ (the canonical serializer, surfaced per-file — ADR-0034's "canonical format" is core), field-lint-conformance/ (the advisory field lint every port's verify prints) and requirement-check-conformance/ (the requirement gate every port's verify runs; a drift gate is core, and it is not ejectable in any port). A red cell here is a MetaObjects bug.
  • Template quality checks — not a promise. The generated lane of api-contract-conformance/, generator-registry-conformance/ (stable generator names), codegen-noop/ (vocabulary with no lowering yet emits nothing), cube-model/ (the Cube exporter's output, a TypeScript reference helper), requirement-test-identity-conformance/ (which tests the requirement-tests reference generator yields from a ledger and what each is called; the text it writes is not pinned), the codegen-compile gate and the eject contract (below). They check that the reference generators are correct starting points; an adopter's ejected copy is theirs and is not gated. The hand-rolled reference-server lane of api-contract-conformance/ still pins the wire contract that the runtimes and the client speak, which is core.

Split coverage: where a feature is gated, and where it deliberately is not

The matrix is corpus-shaped, so a feature whose coverage splits — across several corpora, or between its vocabulary and its behaviour — never gets a row of its own, and its boundary ends up discoverable only from the feature doc. The split ones are recorded here instead.

The eject contract (ADR-0034 Amendment 8). A gate, not a corpus: it has no fixtures of its own, and reads generator-registry-conformance/registry.json.

  • Gated in all five ports. Each port's eject-contract test asserts that every generator it ships is in the registry (census), that every registered generator ships eject source unless the shared file exempts it (completeness), and that each generator ejected alone compiles or loads, references nothing in the codegen packages outside the engine surface and emits what the packaged generator emits (depth). TypeScript: cli/test/eject-contract.test.ts. C#: EjectContractTests. Java: EjectContractTest in maven-plugin. Kotlin: EjectContractTest in codegen-kotlin. Python: tests/conformance/test_eject_contract.py.
  • Not yet true everywhere, and the shared file says where. eject.pending lists every generator that still fails an assertion. A port's test fails when an unlisted gap appears and when a listed one closes without its entry being removed.
  • Not gated: an adopter's owned copy. The contract is about what the reference hands over.

Capability requirements (features/requirements.md):

  • Vocabulary — gated in all five ports. The types, their attributes and their closed enums travel in registry-conformance's byte-matched manifest, which every port reproduces exactly, and accept/reject behaviour is pinned by requirement-* fixtures in fixtures/conformance/. A port that drifts on what it will load fails.
  • Checks — gated in all five ports. Every port's verify runs the requirement gate (ADR-0057, which reversed a TypeScript-only split recorded here). requirement-check-conformance pins each diagnostic's code, severity, path and message text and the summary counts against the TypeScript reference. Kotlin has no CLI of its own and runs the gate through the Java Maven goal. verify-conformance still holds no requirement cases: that corpus is template drift, and the gate has its own.
  • Generated tests — identity gated in all five ports, text in none. requirement-test-identity-conformance pins which tests a ledger yields, each one's id, witness key, skip state and digest, and what a project's filter is shown. The file a port writes is its reference helper's output and is not a cross-port contract.
  • Authoring lint — TypeScript only, by decision. The seven WARN_REQUIREMENT_* authoring advisories are printed by the Node meta verify and by no other port (ADR-0057, Consequences). They never fail a build, and no corpus holds them. The structured --format json|toon payload is TypeScript-only too.

Stated as mechanisms rather than as a list of attribute names on purpose — the requirement vocabulary has a breaking change scheduled (FR-038), which moves what the manifest contains without moving these boundaries.

The Cube exporter (features/cube-export.md):

  • Mapping — gated in TypeScript only, by decision. fixtures/cube-model/ holds 46 cases, 36 with the exact tree the generator writes and 10 with the exact error message, plus the canonical model's golden. codegen-ts/test/cube/cube-model-corpus.test.ts and cube-model-canonical.test.ts run them. No other port has an exporter (spec R6), so no other port makes a claim the corpus would have to check.
  • Cube accepts the output — queried on Postgres, compiled in both dialects. The cube lane (scripts/ci-local.sh --only cube, which needs Docker) loads the canonical golden and the 36 cases that hold a tree into a pinned Cube (cubejs/cube:v1.7.43) and requires each to compile, the three MySQL cases included: compiling a model runs no SQL, so the lane's Postgres data source serves them too. For the canonical reports it compares the Cube query with the report view (a @spine report through its Cube view, its empty spine rows included), and for the escaping and measure-default cases it reads the SQL back from Cube's /v1/sql. A second live file (cube-model-mysql.live.ts) runs the MySQL path: the canonical golden generated with dialect: "mysql" is loaded into the same pinned Cube over a real MySQL 8.4 holding the report views buildReportViews lowers (fixtures/cube-model/canonical/seed.mysql.sql), and each of the twenty-three served reports' Cube query is compared with SELECT * from its view, with the pre-aggregation used asserted. That is the smaller of the two honest options: the three MySQL cases (dimension-time-mysql, measure-mysql, measure-boolean-mysql) are still compiled, not executed, and the goldens hold their SQL. The lane runs Cube in development mode, so Cube's production mode with a separate Cube Store is not gated.
  • The lane is a gate, not a second corpus. Like the codegen-compile gate, it reuses fixtures/persistence-conformance/canonical/meta.fitness.json and the committed schema.postgres.sql rather than adding a model beside them; only the seed rows are its own (fixtures/cube-model/canonical/seed.sql).

Does the generated code compile (codegen-compile-conformance, all five ports):

  • Not a corpus, so it has no row. It reuses fixtures/persistence-conformance/canonical/meta.fitness.json rather than adding a kitchen sink beside the one that already exists — 18 entities, two view-backed projections (one restating an entity's enum through extends), M:N junctions including a self-join, a 5-entity TPH hierarchy whose subtypes are an FK and M:N target and declare an M:N of their own, a @required + @default subtype field, jsonb, isArray, currency, decimal, and AllTypes carrying every persistable field subtype. A second such model would drift from the first, and this one is maintained by the other corpora already. Shapes an adopter's build tripped over are folded INTO it, not into a second model.
  • Why it exists. Every corpus above gates BEHAVIOUR. None asks whether the emitted code builds — so four "generated code does not compile" defects shipped in 1.0.4 with the whole matrix green, because gen exits 0 in all four cases and the adopter's build is the first thing that disagrees. The gate found five more across the ports on its first runs.
  • Each port uses its own compiler, so it is the same question rather than the same code: ts.createProgram, Roslyn, javac via ToolProvider, KotlinCompilation, and — Python having no static compiler — importing the generated package (which is what resolves the imports BETWEEN modules) plus ruff F821 for the branches importing never executes.
  • Every port excludes its framework-bound route tier — TS routesFile, C# RoutesGenerator, Java SpringControllerGenerator, Kotlin KotlinSpringControllerGenerator. Those imports are not on an in-memory compile's classpath and stubbing them drowns the signal; that tier is compiled and booted over real HTTP in the api-contract integration lane instead. The boundary is one cross-port rule, not four local concessions.
  • The prompt tier is out of scope here because the fitness corpus declares no template.* nodes. Including those generators would emit nothing and read as coverage that is not there.

Do features work in combination, judged independently (TypeScript only, the integration-tests package, real Postgres):

  • Why it exists. Every corpus above exercises each feature on its own, and every one was green while the first real application found nine defects in an afternoon — each where two features meet (a reference onto a TPH subtype, a required default on one, an M:N declared on one). Two of them were invisible for a structural reason as well: meta verify --db diffs a database against the same buildExpectedSchema that wrote its migration, so a dropped FK looked in sync; and a dropped .references() compiles.
  • The independent oracle (src/independent-oracle.ts) restates, from the metadata alone, which table stores each object's rows, which columns and FKs that table has, and which routes the API serves — and reads the answer from pg_catalog, from Drizzle's own getTableConfig, and from the booted Fastify app, never through migrate-ts or codegen. independent-oracle-pg.test.ts asks it of a hand-written TPH model, including an M:N traversal against real rows, and of the committed canonical/schema.postgres.sql every port's query runner executes.
  • The feature-combination gate (feature-combinations-pg.test.ts) enumerates feature axes — hierarchy shape, where a link is declared, what it points at, link kind, link-table key, a required or defaulted field — and runs a pairwise covering set of models (23 today) through generate, tsc with routes included, migrate, and boot, judged by the oracle. It found three defects on its first run that no other gate could see.
  • Known defects are a ledger, not a skip. A check that fails for a recorded defect passes only if every failure line matches that defect's signature, and a recorded defect that stops reproducing fails its case. The ledger can only shrink.

How to tell a deliberate split from a real parity gap, since the two look identical in the matrix — both show one port covered and four blank. Ask what the uncovered ports claim. For the requirement authoring lint they claim nothing: no other port's verify prints one of those advisories, exactly as the feature doc says. (The requirement checks were a split of this kind until ADR-0057 moved them into every port, and they stopped being one the moment a second port made the claim.) Contrast {{#hasField}} in 0.23.1, where the JVM emitted has<Field>() onto generated payload records and verify accepted the section, while no render engine in any port implemented the other half — two ports shipping halves of one promise, with no fixture that could see it. A split is deliberate when no port makes a claim the corpus would have to check; it is a gap when some port already made one.

A hazard for whoever removes an attribute — and it has now fired once. Contract prose lives inside attr descriptions in expected-registry.json, where five ports byte-match it and nothing else gates it. @trackedBy still carries "NOT resolved by verify, which does not reach the network". @verifiedBy carried "verify checks each name EXISTS and is not skipped; it never runs them" — and that attribute was retired in 0.24.0 (FR-038), which deleted the only gated statement of that guarantee along with it.

That removal was done to the rule: the sentence was rehomed in the same change, to spec/capability-ledger.md (§ "verifiedBy — RETIRED") and to the migration guide, before the description was deleted. Do the same for the next one. Sweep every description being removed for contract prose and rehome it in the same change, not after — the same class as the ADR-0047 renumbering trap, where a string that reads like prose is actually a gated artifact.

Per-port runners + commands:

Port Metamodel + YAML + render + verify Persistence API contract
TypeScript cd server/typescript && bun test (per-package, ~3s) scripts/integration-test.sh ts (needs Docker) cd server/typescript/packages/integration-tests && bun test test/api-contract.test.ts (needs Docker)
Java cd server/java && mvn -pl metadata test (and per-tier -pl render, etc.) scripts/integration-test.sh java (needs Docker) mvn -f server/java/integration-tests/pom.xml test -Dtest=ApiContractConformanceTest (needs Docker)
Kotlin cd server/java && mvn -pl codegen-kotlin test (snapshot suite) mvn -f server/java/integration-tests-kotlin/pom.xml test (needs Docker) mvn -f server/java/integration-tests-kotlin/pom.xml test -Dtest=ApiContractConformanceTest (needs Docker)
C# dotnet test (per project) scripts/integration-test.sh csharp (needs Docker) dotnet test server/csharp/MetaObjects.IntegrationTests/MetaObjects.IntegrationTests.csproj --filter "FullyQualifiedName~ApiContractConformanceTest" (needs Docker)
Python pytest (per package) scripts/integration-test.sh python (needs Docker) cd server/python && uv run --extra integration pytest tests/integration/test_api_contract.py (needs Docker)

The persistence-conformance corpus is intentionally on-demand — none of the unit-test runners (bun test, dotnet test, pytest, mvn test) pull Docker. scripts/integration-test.sh is the entry point and is wired into docs/RELEASING.md §2b as the pre-latest gate.

Fixture-to-doc mapping

fixtures/conformance/ — metamodel loader + canonical serializer (380)

Fixture prefix Feature doc
loader-basic-*, error-parse-* features/loaders.md
extends-*, error-extends-* features/entities.md (inheritance)
identity-*, subtype-entity-*, subtype-value-* features/entities.md (identity / entity vs value)
attr-*, error-attr-*, error-reserved-word-as-attr features/entities.md (attributes)
doc-common-attrs-* features/entities.md (documentation common attrs)
auto-set-on-* features/entities.md (auto-set timestamps)
attr-filter-*, loader-filterable-*, warning-filterable-*, layout-data-grid-*, error-data-grid-* features/entities.md (filter / sort / grid)
overlay-* features/entities.md (overlay / merge)
merge-three-way-no-conflict, error-merge-conflict-attr, merge-conflict-unmarked-attr-redeclaration, warning-duplicate-declaration features/loaders.md (multi-file merge attribution, FR5c)
field-string-*, field-decimal-*, field-object-storage-*, error-field-object-storage-* features/field-types.md
currency-* features/field-types.md (currency)
enum-*, error-enum-*, warning-enum-* features/field-types.md (enum)
source-rdb-*, source-db-table-*, source-db-view-*, source-multi-source-*, error-source-* features/source-kinds.md
relationship-*, error-unknown-relationship-* features/relationships.md
template-*, error-template-* features/templates-and-payloads.md
origin-*, error-origin-* features/templates-and-payloads.md (payload origins)
projection-*, error-projection-*, field-readonly-on-view-projection features/source-kinds.md (projections + the object taxonomy, ADR-0028)
requirement-*, error-unknown-attr-requirement features/requirements.md (vocabulary only — the gate has its own corpus, requirement-check-conformance/; see "Split coverage" above)
reporting-*, error-dimension-*, error-measure-*, error-ratio-*, error-segment-*, error-report-*, error-relative-date-* features/reporting.md (the dimension / measure / segment / object.report vocabulary and its load-time rules; design in the FR-044 spec)
smoke-empty-metadata features/entities.md

fixtures/yaml-conformance/ (16)

All 16 fixtures → features/yaml-authoring.md. The corpus splits into 9 happy-path fixtures (sigil-free attrs, array suffix, anchor/alias, block scalars, mixed bare-and-prefixed, quoted leading zero, an overlay declared before its base in one file, etc.) and 7 error-yaml-* fixtures that pin the YAML 1.1 coercion guards (bool / null / hex in string contexts; numeric in enum contexts; reserved-as-attr).

fixtures/render-conformance/ (16)

All 16 fixtures → features/templates-and-payloads.md (render engine output section). 4 are end-to-end shape examples (prompt / email / spreadsheet / CSV-injection escape); 11 pin Mustache-engine semantics — dotted-path lookup, parent-context fallthrough, falsy/empty-array section behavior, inverted sections, nested partials, standalone-tag whitespace stripping, raw-HTML bypass, trailing-newline preservation, unicode multibyte handling, and the -last list-separator accessor.

fixtures/verify-conformance/ (37)

All 37 fixtures → features/migrations-and-drift.md (template drift section — Renderer.verify). A node of a fixture's payload.json field tree may carry "isArray": true; with fields it is a list of objects, without it a list of scalars. The six verify-positional-accessor-* cases pin where -last is available.

fixtures/persistence-conformance/ (54 — 48 query + 6 migration)

Engines beyond Postgres (FR-044 reports). The twenty-one queries/report-*.yaml scenarios also run on SQLite and MySQL 8.4 wherever a port's runtime has the engine, and on D1's local runtime in TypeScript (no cloud account): TypeScript on all four, C# and Kotlin on SQLite and MySQL, Java on MySQL. Stated limitations, not gaps: Java's OMDB has no SQLite driver, and Python's runtime ships a Postgres driver only (pg8000), so Python runs the corpus on Postgres alone. Adding a driver to run a view-reading test is not worth the surface. Each engine executes an artifact TypeScript produced (canonical/schema.sqlite.sql from meta migrate --dialect sqlite; canonical/schema.mysql.sql from the adopter-written tables plus buildReportViews), held equal to its generator by a test, so no port writes view SQL. The expect blocks stay the Postgres wire ones; engine spelling is normalised on the actual side only (a SQLite ratio is a REAL, a MySQL DECIMAL keeps its scale, a MySQL DATETIME is the UTC wall clock), and report-relative-date carries a per-engine seed (seed-data-engine). op: relate and the write ops stay Postgres-only. See features/reporting.md.

fixtures/api-contract-conformance/ (91)

All 91 scenarios → features/api-contract.md (cross-port REST API URL grammar + JSON wire format). Verifies every backend's emitted CRUD routes answer identically over HTTP — list / get / create / patch+put / delete, plus pagination (limit/offset), sort (sort=field:dir), the withCount=1 envelope, the not_found / invalid_sort error envelopes, and the 201 / 204 status codes.

The corpus also covers the 9 cross-port filter operators (eq, ne, gt, gte, lt, lte, in, like, isNull) plus the implicit-AND combinator and 2 error shapes (invalid_filter_field / invalid_filter_op) under the URL grammar ?filter[<field>][<op>]=<value> (FR-009). On top of the 31 core scenarios the corpus carries six sub-corpora — tph/ (10, single-table inheritance), m2m/ (9 — 3 plain, 5 gating TPH x M:N together, the combination each corpus alone could not reach, and 1 pinning the collection-URL spelling), jsonb/ (2, typed value-object columns), write-through/ (2, table-write + view-read entities), projection/ (12, a read-only view answers reads and refuses writes with 405; a projection with no declared identity has no item route; a projection keyed on a field not named id is addressed by it, whether its identity names @fields or derives them; decimal and float fields filter) and report/ (25, FR-044: a view-backed object.report is listed, filtered, sorted and paged on its derived fields, answers POST with 405 and mounts no /{id}; a @spine report keeps a row with no facts, and a measure with @default filters and sorts as its default). All 5 ports — TS, Java, Kotlin, C#, Python — run it in BOTH lanes: a hand-rolled reference server and the port's own GENERATED API artifact booted over HTTP. write-through/, projection/ and report/ run the generated lane only, on all five ports: what they test is whether a port's generator emits the routes. The report/ sub-corpus is the one that reads real views, so it also runs on SQLite and MySQL (report/schema.sqlite.sql, report/schema.mysql.sql, both TypeScript-produced) in TypeScript and C#, and on D1's local runtime in TypeScript. Java, Kotlin and Python serve that corpus from seeded rows behind their repository seam, so no engine reaches them and the extra engines are not applicable there.

fixtures/source-resolution-conformance/ (25 cases)

All 25 cases → features/metadata-sources.md (how a declared sources set resolves to a file list). Companion to scope-conformance/ below — sources decides which files are read, scope filters what is emitted from them. The corpus is file-shaped: one committed cases.json, read directly by every port's runner, with no per-port fixture and no ledger.

It pins the resolved file SET for a declared sources list — the default directory, replacement-not-merge, the relative-path base (the directory HOLDING .metaobjects/, never the process cwd), recursive directory walking, case-insensitive extension matching, union-with-de-duplication, and every error condition (an unresolvable path, an unsupported resource/package kind, and a malformed config — "expectError": true pins only that resolution RAISES, since which error code it raises with is deliberately NOT a cross-port contract; see the corpus README). All four CLI surfaces run it — TypeScript (the reference implementation), C#, Python, and Java (Kotlin inherits it, since Kotlin has no CLI entry point of its own and runs through the same Maven plugin as Java): server/typescript/packages/sdk/test/source-resolution-conformance.test.ts, server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs, server/python/tests/conformance/test_source_resolution_conformance.py, and server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java.

fixtures/scope-conformance/ (10 cases)

All 10 cases → features/metadata-sources.md (the scope pattern grammar). The corpus is file-shaped: one committed cases.json, read directly by every port's runner, with no per-port fixture and no ledger.

It pins the semantics of a consumer's include/exclude scope over fully-qualified names — * matches within one :: segment and never crosses it; a segment that is exactly ** matches one or more whole segments (so acme::** does not match the bare acme); every other character is literal, regex metacharacters included; an absent or empty include means everything; multiple include patterns are a union and exclude is applied after it; and matching is case-sensitive. These are exactly the rules four independent implementations would otherwise each get slightly wrong — the failure mode that produced the cross-port LIKE/ILIKE divergence fixed in 0.21.6.

TypeScript and Python run it today. The reference implementation is server/typescript/packages/metadata/src/scope.ts (compilePattern / compileScope / matchesScope; @metaobjectsdev/sdk re-exports it unchanged, since it moved there from sdk when FR-023 needed it from codegen-ts without a sdk dependency), and the corpus was authored against it. Python's port (server/python/src/metaobjects/scope.py, matches_scope using re.fullmatch) runs the same corpus. Java, Kotlin and C# have no runner yet; when each gains one, this corpus is what it implements against — it exists now precisely so those three land on one grammar rather than three.

fixtures/dependency-conformance/ (23 cases)

All 23 cases → features/metadata-dependencies.md (declaring a dependency, meta deps sync, the committed snapshot + lock, default exclusion of imported metadata, overlay/extends across the boundary, and every load- and resolution-time failure). File-shaped like scope-conformance/ above: one committed cases.json, no per-port fixture, no ledger.

TypeScript (the reference implementation) and Python both run all 23 cases, identically — every case's expectFiles/expectImported/expectSelected/ expectMigrateGoverned/expectLoadError assertions apply to both runners with no exemption. The two overlay cases that need a view child as incidental content use view.currency (the one concrete view.* subtype registered cross-port), not view.text — a prior revision used view.text, which Python did not register then, and carried a since-discharged allowlist for it. Java, Kotlin and C# have no runner — Phase 1a is TypeScript + Python only; those three ports arrive in Phase 2.

ADR-0055 (deferred overlay application) — SHIPPED in every port. The nine cases added for it (overlay-mixed-file-base-in-later-file, overlay-mixed-file-base-in-earlier-file, overlay-same-file-before-base, overlay-nested-under-plain-parent-base-later, overlay-applies-after-all-plain-declarations, overlay-two-overlays-source-order, error-overlay-no-target, error-overlay-no-target-nested, and the YAML twin yaml-overlay-before-base-same-file) are green on TypeScript, C#, Java, Kotlin (via the JVM loader) and Python, with no entry in any expected-failures ledger — they ran rather than being excused. The fixtures were committed while three ports were still red, and the red was recorded here rather than in a ledger, because ledgering a fixture the design intends to pass is how a port quietly stays behind.

fixtures/naming-conformance/ (8 cases)

All 8 cases → the already-plural pluralize fix (CHANGELOG ## [Unreleased]): an entity whose name is already a plural noun (Stats, Settings, Series) used to get double-pluralized (ProgramPurchaseStats → ProgramPurchaseStatses) in every port's REST collection path, generated hook/query/finder/list names, and DbSet/collection names. File-shaped like scope-conformance/ above: one committed already-plural-pluralize.json, read directly by every port's naming unit-test file, no per-port fixture, no ledger.

Each case carries BOTH axes the fix touches: apiPlural (the API/code-surface pluralizer every generator-facing use now gets) and legacyPlural (the FROZEN default-physical-table-name pluralizer, deliberately NOT fixed, so an adopter's live database never sees a proposed rename). Three already-plural cases (Stats, Settings, Series) prove the two axes now DIVERGE; three look-alike cases (Status, Address, Alias — each ends in "...s" preceded by one of the excluded letters s/u/i/a) prove both axes stay IDENTICAL to the pre-fix behavior; two baseline cases (Category, Box) don't end in "s" at all before pluralizing and are untouched by either axis.

No HTTP server, no database — this corpus is a pure function check. The route-spelling half of the general pluralization rule is additionally gated end-to-end over HTTP by fixtures/api-contract-conformance/m2m/scenarios/route-spelling-multiword-collection.yaml (the PostCategory multi-word/consonant-y scenario, predating this fix); the default-physical-table-name axis has no HTTP-observable surface at all, so it can only be proven at the pluralizer-function level, which is what this corpus does for both axes in one data file.

All five ports run it (Kotlin inherits, since KotlinNaming.collectionSegment delegates to the same RouteNaming.pluralize Java runs): server/typescript/packages/metadata/test/naming.test.ts, server/csharp/MetaObjects.Codegen.Tests/CSharpNamingTests.cs, server/java/codegen-base/src/test/java/com/metaobjects/generator/util/RouteNamingTest.java

  • server/java/metadata/src/test/java/com/metaobjects/source/Fr016SourcePhysicalNameTest.java (API-surface and frozen-legacy halves split across the two modules that already own those two functions), and server/python/tests/codegen/test_route_path_naming.py + server/python/tests/unit/test_fr016_source_name_and_kind_aliases.py (same split).

fixtures/requirement-check-conformance/ (43 cases)

All 43 cases → features/requirements.md ("The gate in every port"). Each case is a directory: input/ metadata, an optional options.json (libraries, requireImplementers) and an expected.json holding the gate's diagnostics (severity, code, requirement path, message text) and the ledger summary, or null for a model with no requirement. The expectations are written from the TypeScript reference and a port that disagrees with one is wrong. The case list, and what each pins, is in the corpus README.

Every port's verify is held to it: server/typescript/packages/cli/test/requirement-check-conformance.test.ts, server/python/tests/conformance/test_requirement_check_conformance.py, server/java/metadata/src/test/java/com/metaobjects/conformance/RequirementCheckConformanceTest.java (Kotlin inherits it, through the shared Maven goal) and server/csharp/MetaObjects.Conformance.Tests/RequirementCheckConformanceTests.cs.

fixtures/requirement-test-identity-conformance/ (26 cases)

All 26 cases → features/requirements.md ("Generated requirement tests and witnesses"). It pins the identity of every test the requirement-tests generator yields from a ledger (package, path, unit, id, witness key, status, skip state and digest), witness-key collisions, and the view a project's filter receives, through eight named filters each port's runner implements in its own language. It does not pin the text of a generated file, the uncovered warning, or a requirement name outside ASCII. The case list is in the corpus README.

All five ports run it: server/typescript/packages/codegen-ts/test/requirement-test-identity-conformance.test.ts, server/java/metadata/src/test/java/com/metaobjects/conformance/RequirementTestIdentityConformanceTest.java, server/python/tests/conformance/test_requirement_test_identity_conformance.py and server/csharp/MetaObjects.Conformance.Tests/RequirementTestIdentityConformanceTests.cs. Kotlin's identity function is the Java one; the names its generator emits are asserted by server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinRequirementTestsGeneratorTest.kt.

fixtures/cube-model/ (46 cases)

All 46 cases and the canonical golden → features/cube-export.md. Each case is the smallest model for one mapping rule (an entity's cube, a join, an alias cube, a dimension type, a measure, a measure's @default, a segment, a relative date, a rollup, a @spine report's Cube view, the escaping rules) or for one ERR_CUBE_* refusal, and holds exactly one of an expected file tree and an expected error message. The case list, with the rule each pins, is the corpus README.

One port runs it: TypeScript, in server/typescript/packages/codegen-ts/test/cube/cube-model-corpus.test.ts. The cube lane (server/typescript/packages/integration-tests/cube-live/cube-model.live.ts) loads the same files into a real Cube.

Orphaned fixtures (tested but not yet documented)

The fixtures in the nine corpora mapped above (metamodel 380 + yaml 16 + verify 37

  • render 16 + persistence 54 + api-contract 91 + source-resolution 25 + scope 10 + dependency 23) each map to a feature doc, and so do the two requirement corpora and the cube-model corpus, whose case lists live in their own READMEs. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider composition, agent context, docs emit) rather than user-facing metamodel behaviour, so they have no feature-doc row.

If you add a new fixture and don't see a clear home for it, either:

  1. Add a section to the closest matching feature doc and reference it from its ## Verified by block, or
  2. Open an issue if the fixture exercises behavior the feature docs haven't yet described.

Orphaned docs (documented but no fixture coverage)

None of the 8 feature docs lack fixture coverage. The closest case is the high-level overview material in each doc (e.g., the "Anatomy of an entity" explainer in entities.md) — these are pedagogical, not behavioral, and don't require pinning.

See also