Skip to content

refactor(migrate-apim): decide from rules.json, not from plan.py - #69

Open
Facundo Santiago (santiagxf) wants to merge 6 commits into
fix/policy-resolutionfrom
fix/rules-as-contract
Open

Facundo Santiago (santiagxf) wants to merge 6 commits into
fix/policy-resolutionfrom
fix/rules-as-contract

Conversation

@santiagxf

@santiagxf Facundo Santiago (santiagxf) commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Stacked on #67 — review that first; this branches from it and merges after it.

capabilities/rules.json is meant to be the executable form of how APIM maps to AI Gateway, so that
"a new gateway release is a data change". Two things stopped it being that.

Twelve decisions about model APIs were written in plan.py. What an unreadable published surface
costs, what a translating caller dialect costs, session affinity, the model catalogue, alias naming,
model and provider registration, the gateway identity, backend reachability — all inline, where
compile can say nothing about them and changing one is a code change. ARCHITECTURE.md described
assets[] as covering models and matching on modelsStatus and surfaceStatus; neither identifier
existed anywhere except the test suite's example of a rule no site can select.

A rule's subject was implied by which array it sat in. match did three unrelated jobs under one
name: family named the subject in assets[] (mcpServers) and was a predicate on a field in
policies[] (authentication); element held element names while the record's field is name;
hasExpression was computed, not read; and anyElement was a fallback dressed as a predicate.

What a rule looks like now

{
  "selects": "sessionAffinity",
  "verdict": "gap",
  "itemKind": "accept-loss",
  "proposal": "accept-loss",
  "options": ["accept-loss", "retain-in-apim", "block"],
  "rationale": "…"
}
  • selects names the question. A subject is a question, not an asset type — one model API raises
    six and they are decided separately. That is the shape the structural rules already had, and the
    alternative does not work: under a single selects: modelApi, alias naming and model registration
    pin no predicates, so no site can tell them apart.
  • where holds field predicates, each one a field of the subject's inventory record.
  • derived holds predicates computed from a record rather than read off it.
  • fallback is a flag, because matching everything is not a statement about the element.

55 rules across 20 subjects and 25 selection sites.

The dividing line

Rules carry the verdict, item kind, capability, proposal and option vocabulary. Code keeps building
target objects and evidence, and keeps prose that interpolates the inventory — how many operations
have no target, which dialect this backend speaks, which identities the source carries. _policies
already split it there; this extends that convention rather than inventing one.

What compile now holds

Check Catches
Every selection site has at least one rule A question the planner asks and the data does not answer — exactly how the twelve got into plan.py
Every where key names a field inventory.schema.json declares A rule matching on a field no record carries, which looks selectable and never fires. rule-reachable cannot see this
Ties count derived, not just where Two rules alike in where and differing only in what is computed read as one, and one silently won

Writing the field check found the schema wrong rather than the rules: the collector has always
written service.networking — including the targetRequirement that decides backend reachability —
and endpoints.surface, which decides what of an API can be published. Neither was declared, and with
no additionalProperties: false the drift was invisible. Both are declared now.

One decision changes, and it needs signing off

backendReachability with needsSubnet: false proposed private-ingress while offering
new-delegated-subnet / public-egress / retain-in-apim / block — the recommended value was not
among the choices.
Both branches were handed the same option list. Verified against 746e364.

Moving the decision into data is what exposed it: compile's proposal-is-an-option check refuses to
write a proposal nobody can select. private-ingress is now offered in that branch, which is a real
change to what an operator sees and the only one in this PR.

The rules have to actually decide

The first pass moved the decisions into rules.json and then kept building the option lists in code
at several sites, so editing those rules changed nothing — the data was decoration. Worse, the check
built to catch exactly that could not see it: a shared _ASSET_DECIDED tuple declared options and
rationale as read at every asset site, and rule-field-unread is satisfied by the declaration. A
false declaration defeated the mechanism meant to prevent false declarations, which is the failure
dispatch.py exists to stop, reappearing inside it.

Each site now declares only what it reads. The property is tested by doctoring every rule's options
to a canary and asserting it reaches the plan.

Nothing else is decided differently

That is the property this change has to hold, and it is held by two tests rather than by reading a
diff that rewrites every rule.

tests/golden/decisions.json records every element the collector can name — with and without an
expression — against the decision the rules give it, plus the whole plan built from the fake ARM
fixture. A rule is identified there by what it decides, never by how it is matched. The snapshot was
taken before the first edit and is unchanged by all three commits.

A snapshot only proves the cases its fixture reaches, and that is not a technicality here. One
decision moved wrongly: session affinity lost block — the option that stops a migration — and gained
one that never existed. The fixture has no pool using affinity, so the plan snapshot stayed green.
So every moved decision is now pinned directly, transcribed from the code that held it, in
tests/test_decisions_moved_intact.py. That pin caught a second defect on its first run: the
catalogue rule naming no discovery source was unreachable, because the rule that names one is written
first and selection is first-match.

Also fixed, both found by writing the checks rather than by reading: models.status: "catalogued"
was missing from inventory.schema.json while resolve.py wrote it and plan.py branched on it, so
migrate-apim validate rejected an inventory the tool had just produced — the same hole
auth.mode: undetermined left. And two checker bugs: _check_uniqueness named ties over the wrong
key set, and the field-path check accepted any path whose tail matched anywhere in the schema.

Testing

713 passed, 1 skipped; compile --strict exits 0; pyflakes clean. The no-change property is
verified at each commit, not only at the tip, and tests/golden/decisions.json has not been touched
since it was taken.

All offline — no Azure resources, nothing run against a live APIM instance or gateway.

Known limit

check_subjects_have_rules verifies each subject has a rule, not that every case a site asks for
has one. Deleting one of the three modelCatalogue rules still compiles, and the planner falls back
to the rule.get(field, default) defaults still present at each site. The pins in
test_decisions_moved_intact.py catch it; compile does not. Closing it means dropping the
code-side defaults or teaching compile which cases each site asks for.

Deferred

Collapsing assets[] / structural[] / policies[] into one rules[], now that selects
discriminates. It would also make _check_uniqueness per-subject rather than per-section, which is
the granularity that catches collisions. Touches _SELECTED_SECTIONS and compile's reporting, so it
is a later step.

🤖 Generated with Claude Code

A rule's subject was implied by which of three arrays it sat in, which
left `match` doing three unrelated jobs under one name. `family` named
the subject in `assets[]` (`mcpServers`) and was a predicate on a field
in `policies[]` (`authentication`) -- the same key, opposite roles,
disambiguated only by position. `element` held element *names* while the
record's field is `name`, a binding that existed only inside the
resolver. `hasExpression` was derived rather than read off any record,
and `anyElement` was not a predicate at all but a fallback wearing one's
clothes.

So the four are separated. `selects` names the subject, `where` holds
field predicates, `derived` holds what is computed, and `fallback` is a
flag. Each selection site declares which it is, and -- for `where` keys
-- the inventory path the key is compared against, so a later commit can
check a rule against the schema the way `check_capabilities` already
checks a declared CLI option against the surface snapshot.

The diagnostic improves on its own: a rule naming a subject nothing
decides about is now reported as unreachable by reading the rule, where
before it had to be inferred from a value no site compares against.

Nothing is decided differently. `tests/golden/decisions.json` records
every element the collector can name -- with and without an expression
-- against the decision the rules give it, plus the whole plan built
from the fake ARM fixture. A rule is identified there by what it
decides, never by how it is matched, because how it is matched is
exactly what this changes. The snapshot was taken before the first edit
and is unchanged by it.

That test is the reason this commit is reviewable: the diff rewrites 39
rules and 14 selection sites, which is not something anyone can check by
reading.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ARCHITECTURE.md` has described `assets[]` as covering models and
modelProviders, matching on `modelsStatus` and `surfaceStatus`, since
before any of that existed. Neither identifier appears in the code, the
rules or the schema; `modelsStatus` appears once, in the suite's example
of a rule no site can select. Twelve decisions about the asset this tool
exists to migrate were written in `plan.py`, where `compile` can say
nothing about them and a gateway release that changed one would be a
code change.

They are rules now: what the published surface costs when it cannot be
read and when it is partly publishable, what a translating or
indeterminate caller dialect costs, session affinity, the two model
catalogue cases, model and provider registration, alias naming, a tool
list gathered with the API's policies bypassed, the gateway identity,
and backend reachability. Ten subjects, sixteen rules.

A subject is a question, not an asset type -- `publishedSurface`, not
`modelApi`. That is the shape the structural sites already had, and the
alternative does not work: under one `selects: modelApi`, alias naming
and model registration pin no keys, so no site can tell them apart and
`_check_uniqueness` is right to reject them.

Rules carry the verdict, kind, capability, proposal and the option
vocabulary. Code keeps building objects and evidence, and keeps the
prose that interpolates the inventory -- how many operations have no
target, which dialect this backend speaks, which identities the source
carries. `_policies` already split it there.

**One decision moved wrongly and the fixture hid it.** Session affinity
lost `block` -- the option that stops a migration -- and gained one that
never existed. The plan snapshot stayed green because the fake ARM
fixture has no pool using affinity, and a snapshot only proves the cases
its fixture reaches. So every moved decision is now pinned directly,
transcribed from the code that held it, in `test_decisions_moved_intact`.
That pin caught a second one on its first run: the catalogue rule that
names no discovery source was unreachable, because the rule that names
one is written first and `first` is first-match.

`_check_uniqueness` judged ties on `where` alone, so two rules alike
there and differing only in what is computed read as one. It counts
`derived` now, which is what makes the two catalogue rules a tie worth
reporting rather than a silent overwrite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two invariants, both read off the registry the planner itself uses.

**Every selection site has at least one rule.** A site with no rule is a
question the planner asks and the data does not answer, which is exactly
how twelve decisions came to live in `plan.py` while `ARCHITECTURE.md`
described them as data. Declaring a site without writing its rule now
fails the build. The other direction was already covered: a rule no site
selects is `rule-reachable`.

**Every `where` key names a field some record actually has.** The
binding between a key and the field it is compared against used to live
inside the resolver, so a rule could match on a field no record carries
and never fire -- the one failure `rule-reachable` cannot see, because
the key is one a site does read. Each site declares the path; this
checks it against `inventory.schema.json`, the way `check_capabilities`
already checks a declared CLI option against the surface snapshot.

Writing that check found the schema wrong rather than the rules. The
collector has always written `service.networking` -- including the
`targetRequirement` the planner reads to decide backend reachability --
and `endpoints.surface`, which decides what of an API can be published.
Neither was declared, and with no `additionalProperties: false` to catch
it, the drift was invisible. Both are declared now, which is the same
defect this whole change is about: a document asserting a contract
nothing held it to.

`ARCHITECTURE.md` described `assets[]` as matching on `modelsStatus` and
`surfaceStatus`. Neither identifier existed anywhere but the suite's
example of a rule nothing can select. It now describes what is there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ten decisions moved into `rules.json`, but seven of the sites kept
writing their own option list beside the rule that now carried one. The
file read as the contract while the code still was it: editing the
session-affinity options, or the networking ones, changed nothing in any
plan. `rule-field-unread` could not see it either, because each of those
sites declared that it read `options` -- and a shared `_ASSET_DECIDED`
tuple excused a field on every rule because some other branch happened
to read it, which is the defect the per-branch `plan._structural`
selectors exist to avoid.

So each site declares what its own branch takes off the rule, and takes
it. Where what a choice costs is this asset's own number -- how many
operations `split` leaves behind, which dialect `move-callers` moves
them to, how many models the catalogue returned -- the rule carries the
choice as a bare string and the planner composes that one sentence,
which is what the bare-string option form has always been for.

`callerDialect` becomes two sites for the same reason: one selector for
both cases would have declared a `proposal` the indeterminate branch
never reads.

The sentences no item could ever carry -- the ones that hold for every
provider, model or tool list while the planner always composes its own
from the asset -- move to `note`, the key addressed to whoever reads the
file rather than to the planner.

One decision had to change, and only one. `backendReachability` proposed
`private-ingress` for a source needing no subnet while offering the
four subnet options in both cases, so its recommended value was one
nobody could settle on; `proposal-is-an-option` rejects writing that
down. The proposal joins the options rather than the four being
narrowed: which of them an operator can live with is theirs to say.

The fixture plan is byte-identical, so `test_decisions_unchanged.py`
still holds. What that fixture cannot reach -- affinity, an
indeterminate dialect, a user-assigned identity, a private-networking
requirement -- is pinned instead, and every site is now asked the other
question too: say something different in the rule, and the plan has to
say it back.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two checks were weaker than they read.

`rule-unique` judges a tie over `where` and `derived` together -- it has
to, or the two catalogue rules and the two identity rules look like one
rule written twice -- but it named the tie over `where` alone. The one
between the catalogue rules printed as `{'discoverySource': None}`,
which is a tie on nothing, and what a site could not tell apart is
exactly what a reader has to see to fix it.

`where-key-field` accepted any path whose tail appeared anywhere in
`inventory.schema.json`, so `assets.models[].endpoints.surface.status`
would have passed on the strength of some other record having a
`surface.status` -- the drift the check exists to find. `_schema_paths`
resolves `$ref`s, so every shape is reachable by its full route from the
root and the whole path can be required; all nine shipped paths already
match exactly.

The check counts subjects, not sites, so it now says so: two sites
deciding `callerDialect` are one question answered in the file.

`ARCHITECTURE.md` and `dispatch.py` still described rules as matching on
`match` keys and `structural[]` as a list of kinds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`resolve.py` writes `models.status: "catalogued"` when a provider's own
catalogue is read in process, and `plan.py` branches on it to raise the
catalogue question. `inventory.schema.json` listed four statuses and not
that one, so `migrate-apim validate` rejected an inventory the tool had
just produced.

The same hole `auth.mode: undetermined` left, and invisible for the same
reason: nothing declares `additionalProperties: false`, no fixture
resolves a catalogue, and nothing asked the schema about one. Writing
the where-key check against this schema is what made it worth asking.

Declared with what a resolve run writes beside it -- the catalogue it
read and where from -- and with the service identity the planner reads
to decide `gatewayIdentity`. The test resolves the fake inventory and
validates the result, which is the case that was never built.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant