Repository navigation
refactor(migrate-apim): decide from rules.json, not from plan.py - #69
Open
Facundo Santiago (santiagxf) wants to merge 6 commits into
Open
Facundo Santiago (santiagxf) wants to merge 6 commits into
Facundo Santiago (santiagxf) wants to merge 6 commits into
Conversation
A rule's subject was implied by which of three arrays it sat in, which left `match` doing three unrelated jobs under one name. `family` named the subject in `assets[]` (`mcpServers`) and was a predicate on a field in `policies[]` (`authentication`) -- the same key, opposite roles, disambiguated only by position. `element` held element *names* while the record's field is `name`, a binding that existed only inside the resolver. `hasExpression` was derived rather than read off any record, and `anyElement` was not a predicate at all but a fallback wearing one's clothes. So the four are separated. `selects` names the subject, `where` holds field predicates, `derived` holds what is computed, and `fallback` is a flag. Each selection site declares which it is, and -- for `where` keys -- the inventory path the key is compared against, so a later commit can check a rule against the schema the way `check_capabilities` already checks a declared CLI option against the surface snapshot. The diagnostic improves on its own: a rule naming a subject nothing decides about is now reported as unreachable by reading the rule, where before it had to be inferred from a value no site compares against. Nothing is decided differently. `tests/golden/decisions.json` records every element the collector can name -- with and without an expression -- against the decision the rules give it, plus the whole plan built from the fake ARM fixture. A rule is identified there by what it decides, never by how it is matched, because how it is matched is exactly what this changes. The snapshot was taken before the first edit and is unchanged by it. That test is the reason this commit is reviewable: the diff rewrites 39 rules and 14 selection sites, which is not something anyone can check by reading. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ARCHITECTURE.md` has described `assets[]` as covering models and modelProviders, matching on `modelsStatus` and `surfaceStatus`, since before any of that existed. Neither identifier appears in the code, the rules or the schema; `modelsStatus` appears once, in the suite's example of a rule no site can select. Twelve decisions about the asset this tool exists to migrate were written in `plan.py`, where `compile` can say nothing about them and a gateway release that changed one would be a code change. They are rules now: what the published surface costs when it cannot be read and when it is partly publishable, what a translating or indeterminate caller dialect costs, session affinity, the two model catalogue cases, model and provider registration, alias naming, a tool list gathered with the API's policies bypassed, the gateway identity, and backend reachability. Ten subjects, sixteen rules. A subject is a question, not an asset type -- `publishedSurface`, not `modelApi`. That is the shape the structural sites already had, and the alternative does not work: under one `selects: modelApi`, alias naming and model registration pin no keys, so no site can tell them apart and `_check_uniqueness` is right to reject them. Rules carry the verdict, kind, capability, proposal and the option vocabulary. Code keeps building objects and evidence, and keeps the prose that interpolates the inventory -- how many operations have no target, which dialect this backend speaks, which identities the source carries. `_policies` already split it there. **One decision moved wrongly and the fixture hid it.** Session affinity lost `block` -- the option that stops a migration -- and gained one that never existed. The plan snapshot stayed green because the fake ARM fixture has no pool using affinity, and a snapshot only proves the cases its fixture reaches. So every moved decision is now pinned directly, transcribed from the code that held it, in `test_decisions_moved_intact`. That pin caught a second one on its first run: the catalogue rule that names no discovery source was unreachable, because the rule that names one is written first and `first` is first-match. `_check_uniqueness` judged ties on `where` alone, so two rules alike there and differing only in what is computed read as one. It counts `derived` now, which is what makes the two catalogue rules a tie worth reporting rather than a silent overwrite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two invariants, both read off the registry the planner itself uses. **Every selection site has at least one rule.** A site with no rule is a question the planner asks and the data does not answer, which is exactly how twelve decisions came to live in `plan.py` while `ARCHITECTURE.md` described them as data. Declaring a site without writing its rule now fails the build. The other direction was already covered: a rule no site selects is `rule-reachable`. **Every `where` key names a field some record actually has.** The binding between a key and the field it is compared against used to live inside the resolver, so a rule could match on a field no record carries and never fire -- the one failure `rule-reachable` cannot see, because the key is one a site does read. Each site declares the path; this checks it against `inventory.schema.json`, the way `check_capabilities` already checks a declared CLI option against the surface snapshot. Writing that check found the schema wrong rather than the rules. The collector has always written `service.networking` -- including the `targetRequirement` the planner reads to decide backend reachability -- and `endpoints.surface`, which decides what of an API can be published. Neither was declared, and with no `additionalProperties: false` to catch it, the drift was invisible. Both are declared now, which is the same defect this whole change is about: a document asserting a contract nothing held it to. `ARCHITECTURE.md` described `assets[]` as matching on `modelsStatus` and `surfaceStatus`. Neither identifier existed anywhere but the suite's example of a rule nothing can select. It now describes what is there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ten decisions moved into `rules.json`, but seven of the sites kept writing their own option list beside the rule that now carried one. The file read as the contract while the code still was it: editing the session-affinity options, or the networking ones, changed nothing in any plan. `rule-field-unread` could not see it either, because each of those sites declared that it read `options` -- and a shared `_ASSET_DECIDED` tuple excused a field on every rule because some other branch happened to read it, which is the defect the per-branch `plan._structural` selectors exist to avoid. So each site declares what its own branch takes off the rule, and takes it. Where what a choice costs is this asset's own number -- how many operations `split` leaves behind, which dialect `move-callers` moves them to, how many models the catalogue returned -- the rule carries the choice as a bare string and the planner composes that one sentence, which is what the bare-string option form has always been for. `callerDialect` becomes two sites for the same reason: one selector for both cases would have declared a `proposal` the indeterminate branch never reads. The sentences no item could ever carry -- the ones that hold for every provider, model or tool list while the planner always composes its own from the asset -- move to `note`, the key addressed to whoever reads the file rather than to the planner. One decision had to change, and only one. `backendReachability` proposed `private-ingress` for a source needing no subnet while offering the four subnet options in both cases, so its recommended value was one nobody could settle on; `proposal-is-an-option` rejects writing that down. The proposal joins the options rather than the four being narrowed: which of them an operator can live with is theirs to say. The fixture plan is byte-identical, so `test_decisions_unchanged.py` still holds. What that fixture cannot reach -- affinity, an indeterminate dialect, a user-assigned identity, a private-networking requirement -- is pinned instead, and every site is now asked the other question too: say something different in the rule, and the plan has to say it back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two checks were weaker than they read.
`rule-unique` judges a tie over `where` and `derived` together -- it has
to, or the two catalogue rules and the two identity rules look like one
rule written twice -- but it named the tie over `where` alone. The one
between the catalogue rules printed as `{'discoverySource': None}`,
which is a tie on nothing, and what a site could not tell apart is
exactly what a reader has to see to fix it.
`where-key-field` accepted any path whose tail appeared anywhere in
`inventory.schema.json`, so `assets.models[].endpoints.surface.status`
would have passed on the strength of some other record having a
`surface.status` -- the drift the check exists to find. `_schema_paths`
resolves `$ref`s, so every shape is reachable by its full route from the
root and the whole path can be required; all nine shipped paths already
match exactly.
The check counts subjects, not sites, so it now says so: two sites
deciding `callerDialect` are one question answered in the file.
`ARCHITECTURE.md` and `dispatch.py` still described rules as matching on
`match` keys and `structural[]` as a list of kinds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`resolve.py` writes `models.status: "catalogued"` when a provider's own catalogue is read in process, and `plan.py` branches on it to raise the catalogue question. `inventory.schema.json` listed four statuses and not that one, so `migrate-apim validate` rejected an inventory the tool had just produced. The same hole `auth.mode: undetermined` left, and invisible for the same reason: nothing declares `additionalProperties: false`, no fixture resolves a catalogue, and nothing asked the schema about one. Writing the where-key check against this schema is what made it worth asking. Declared with what a resolve run writes beside it -- the catalogue it read and where from -- and with the service identity the planner reads to decide `gatewayIdentity`. The test resolves the fake inventory and validates the result, which is the case that was never built. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #67 — review that first; this branches from it and merges after it.
capabilities/rules.jsonis meant to be the executable form of how APIM maps to AI Gateway, so that"a new gateway release is a data change". Two things stopped it being that.
Twelve decisions about model APIs were written in
plan.py. What an unreadable published surfacecosts, what a translating caller dialect costs, session affinity, the model catalogue, alias naming,
model and provider registration, the gateway identity, backend reachability — all inline, where
compilecan say nothing about them and changing one is a code change.ARCHITECTURE.mddescribedassets[]as covering models and matching onmodelsStatusandsurfaceStatus; neither identifierexisted anywhere except the test suite's example of a rule no site can select.
A rule's subject was implied by which array it sat in.
matchdid three unrelated jobs under onename:
familynamed the subject inassets[](mcpServers) and was a predicate on a field inpolicies[](authentication);elementheld element names while the record's field isname;hasExpressionwas computed, not read; andanyElementwas a fallback dressed as a predicate.What a rule looks like now
{ "selects": "sessionAffinity", "verdict": "gap", "itemKind": "accept-loss", "proposal": "accept-loss", "options": ["accept-loss", "retain-in-apim", "block"], "rationale": "…" }selectsnames the question. A subject is a question, not an asset type — one model API raisessix and they are decided separately. That is the shape the structural rules already had, and the
alternative does not work: under a single
selects: modelApi, alias naming and model registrationpin no predicates, so no site can tell them apart.
whereholds field predicates, each one a field of the subject's inventory record.derivedholds predicates computed from a record rather than read off it.fallbackis a flag, because matching everything is not a statement about the element.55 rules across 20 subjects and 25 selection sites.
The dividing line
Rules carry the verdict, item kind, capability, proposal and option vocabulary. Code keeps building
target objects and evidence, and keeps prose that interpolates the inventory — how many operations
have no target, which dialect this backend speaks, which identities the source carries.
_policiesalready split it there; this extends that convention rather than inventing one.
What
compilenow holdsplan.pywherekey names a fieldinventory.schema.jsondeclaresrule-reachablecannot see thisderived, not justwherewhereand differing only in what is computed read as one, and one silently wonWriting the field check found the schema wrong rather than the rules: the collector has always
written
service.networking— including thetargetRequirementthat decides backend reachability —and
endpoints.surface, which decides what of an API can be published. Neither was declared, and withno
additionalProperties: falsethe drift was invisible. Both are declared now.One decision changes, and it needs signing off
backendReachabilitywithneedsSubnet: falseproposedprivate-ingresswhile offeringnew-delegated-subnet / public-egress / retain-in-apim / block— the recommended value was notamong the choices. Both branches were handed the same option list. Verified against
746e364.Moving the decision into data is what exposed it:
compile'sproposal-is-an-optioncheck refuses towrite a proposal nobody can select.
private-ingressis now offered in that branch, which is a realchange to what an operator sees and the only one in this PR.
The rules have to actually decide
The first pass moved the decisions into
rules.jsonand then kept building the option lists in codeat several sites, so editing those rules changed nothing — the data was decoration. Worse, the check
built to catch exactly that could not see it: a shared
_ASSET_DECIDEDtuple declaredoptionsandrationaleas read at every asset site, andrule-field-unreadis satisfied by the declaration. Afalse declaration defeated the mechanism meant to prevent false declarations, which is the failure
dispatch.pyexists to stop, reappearing inside it.Each site now declares only what it reads. The property is tested by doctoring every rule's options
to a canary and asserting it reaches the plan.
Nothing else is decided differently
That is the property this change has to hold, and it is held by two tests rather than by reading a
diff that rewrites every rule.
tests/golden/decisions.jsonrecords every element the collector can name — with and without anexpression — against the decision the rules give it, plus the whole plan built from the fake ARM
fixture. A rule is identified there by what it decides, never by how it is matched. The snapshot was
taken before the first edit and is unchanged by all three commits.
A snapshot only proves the cases its fixture reaches, and that is not a technicality here. One
decision moved wrongly: session affinity lost
block— the option that stops a migration — and gainedone that never existed. The fixture has no pool using affinity, so the plan snapshot stayed green.
So every moved decision is now pinned directly, transcribed from the code that held it, in
tests/test_decisions_moved_intact.py. That pin caught a second defect on its first run: thecatalogue rule naming no discovery source was unreachable, because the rule that names one is written
first and selection is first-match.
Also fixed, both found by writing the checks rather than by reading:
models.status: "catalogued"was missing from
inventory.schema.jsonwhileresolve.pywrote it andplan.pybranched on it, somigrate-apim validaterejected an inventory the tool had just produced — the same holeauth.mode: undeterminedleft. And two checker bugs:_check_uniquenessnamed ties over the wrongkey set, and the field-path check accepted any path whose tail matched anywhere in the schema.
Testing
713 passed, 1 skipped;compile --strictexits 0; pyflakes clean. The no-change property isverified at each commit, not only at the tip, and
tests/golden/decisions.jsonhas not been touchedsince it was taken.
All offline — no Azure resources, nothing run against a live APIM instance or gateway.
Known limit
check_subjects_have_rulesverifies each subject has a rule, not that every case a site asks forhas one. Deleting one of the three
modelCataloguerules still compiles, and the planner falls backto the
rule.get(field, default)defaults still present at each site. The pins intest_decisions_moved_intact.pycatch it;compiledoes not. Closing it means dropping thecode-side defaults or teaching
compilewhich cases each site asks for.Deferred
Collapsing
assets[]/structural[]/policies[]into onerules[], now thatselectsdiscriminates. It would also make
_check_uniquenessper-subject rather than per-section, which isthe granularity that catches collisions. Touches
_SELECTED_SECTIONSand compile's reporting, so itis a later step.
🤖 Generated with Claude Code