Repository navigation
fix(migrate-apim): stop the planner silently losing policies - #66
Open
Facundo Santiago (santiagxf) wants to merge 5 commits into
Open
Facundo Santiago (santiagxf) wants to merge 5 commits into
Facundo Santiago (santiagxf) wants to merge 5 commits into
Conversation
Factoring one `llm-token-limit` into a fragment and including it from several APIs is how APIM shares a policy, and it was the one shape the planner could not place. A fragment is its own document in the inventory, so its elements were classified once, scoped to the fragment, and `_apis_in_scope` had no case for `fragment`: it fell through to the empty set, no model registration matched, and the item shipped as ready carrying an object with no model on it. At apply time that object raises on the missing `modelName` and the policies phase breaks out of its step loop, skipping every policy behind it. Re-planning could not help -- it produces the same object -- and the reason it gave was wrong as well as unhelpful: for a fragment there is something to carry the policy, the planner just had no path from it to the including APIs. The path is the `fragmentRefs` the collector already records from `<include-fragment>`, read backwards. Each including scope is then resolved as itself, so a fragment included globally covers every model and one included by two APIs covers both, and a fragment included by a fragment is followed through. Resolved in `_apis_in_scope` rather than by expanding the fragment into one batch member per including scope, which was the other way to do it. A member is a source policy, and the fragment is one source policy however many APIs include it: expanding it would have reported four occurrences of a limit written once, inflated the variants a gateway-wide baseline records collapsing, and left each member naming an API scope while pointing at the fragment's document, which is where `ip-filter` has to find its address ranges. The baseline path never consulted API scope and still does not. A fragment nothing includes keeps the stranded object: unreferenced, it governs nothing, and that is the genuinely empty scope the message was written for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`plan` sets `configurationComplete: false` with an `incompleteReason` on an object it cannot shape in full, and nothing read either field back. The condition surfaced only when the step ran: the builder refused, reporting whatever cause it could work out, and a failed step stops its phase -- so a problem the planner had already detected and explained cost the rest of a phase to rediscover, in worse words. Preflight now enumerates them. Blocking, because the builder cannot produce the object either and the run stops at it either way, which is the bargain `plannerGap` already makes. Scoped to `steps_for`, so the objects checked are the objects that run: one on work no phase performs cannot fail a phase, and blocking on it would stop a migration over work the operator was going to apply by hand, which `notExecutable` already names. The message quotes `incompleteReason` and `fullConfigurationAt` rather than restating them, since the planner is the only thing that knows why; two planner paths mark an object incomplete from empty settings alone and write no reason, and those report the silence instead of an empty quote. The three fields are a contract between `plan` and `apply` and now a blocking one, so the plan schema declares them. The rest of an object stays open, because its shape is the capability's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`handled` is the strongest claim the planner makes about a policy element: another object carries this, so there is nothing to decide. `_policies` acted on it before `_apply_partial_when` ran, so it was a claim about the element name rather than about the element. `set-backend-service` is where that fails. The rule says "Backend selection becomes the alias backend pool", and the pool is built from the APIM backend pool, which carries no conditional logic at all. So `<set-backend-service backend-id="@(expression)" />`, and the same element inside a `choose`, were classified and then dropped: no item, no gap, and nothing in the plan recording that routing logic had existed. The same project rule that `collect` answered when a backend id built inside a policy expression was invisible -- a failed inference must not read as a confident negative -- was unanswered here one level up. Analyse first, exclude after. `partialWhen` already expresses "this mapping is lossy for this element" in data, so it gains two conditions beside `attributeIsExpression`: `hasExpression`, and `underConditional`, read off the path, which is the only place the enclosing `choose` survives normalization. Any one of them demotes; requiring all would let an element that trips a single condition read as cleanly mapped. So the routing becomes a `partial` / `accept-loss` item proposing `compensate`, carrying `alias.create` so the plan says what did travel, with accept, compensate and retain-in-apim spelled out. Not a `gap`: the backends and their priority and weight do reach the alias, and blocking a migration over a `choose` between two members of one pool is the false alarm that teaches people to ignore the plan. Not a note either -- `needs-decision` is this plan's word for a fact somebody has to dispose of, and anything weaker is not counted, not carried across a regeneration, and not read by `apply`. Nothing changes for a `handled` element that is plain and unconditional, or for one whose rule declares no condition: `authentication-managed-identity` is a sound `handled` -- the provider item genuinely carries it -- and is still excluded, expression or no expression. The trigger is per-rule data, not a blanket policy, so plans do not fill with items for mappings that really did hold. The schema closes `partialWhen` and the `partial` overlay. A condition key the planner does not know would never fire while reading as though it did, which is the defect `additionalProperties: false` is there to catch. 559 passed, 1 skipped. compile --strict exits 0; no selector changed, since `partial` and `partialWhen` were already in the policy sites' `reads`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three fixes land as one chain and nothing tested the joins. A policy factored into a fragment now reaches the models of every including API, and the one case left over -- a fragment nothing includes, which governs nothing -- is caught by preflight in the planner's own words instead of failing a step and skipping every policy behind it. Pinned where the two meet, because each half passes its own tests with the other half absent. The preflight section listed the conditions `--preflight` refuses on and was three short, two of them from earlier work. A list that reads as complete and is not teaches an operator to trust it. Also drops a `label` in `_policies` that nothing has read since the item stopped carrying one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_policies` dropped any element `policy_rule` did not answer for, and it did not answer for ten of them: authentication-basic, authentication-certificate and get-authorization-context, which carry backend credentials; check-header and validate-client-certificate, which gate callers; emit-metric and log-to-eventhub; mcp-server and validate-mcp-request; set-service-url; and every policy APIM ships from now on. None of them reached the plan in any form. An `<authentication-basic>` was read, classified, and discarded in silence. The selection order made it worse than it reads. An element carrying a policy expression fell into the hasExpression pass and was reported as a gap, so what vanished completely were the elements with no expression in them -- the simpler the policy, the more certainly it disappeared. Closed in the three places it can be closed, because no one of them is enough. A terminal rule, tried after the expression pass and matching anything, makes "no rule" unreachable at runtime; what reaches it is a blocking gap, since the alternative is migrating past a policy nobody has read. It is data, not a default in the planner, so what an unknown element costs stays editable. A net makes a poor mapping, so `compile` holds every element in policy.FAMILIES to a rule of its own or a family rule and fails naming any left to the terminal rule. FAMILIES is committed data, so the question is decidable offline, which is what compile is for. The eleven are answered by five family rules and one element rule rather than eleven entries, so the next member of a family is answered before it is written. And the plan now counts what became of every element walked, under exactly one disposition each, with the unrecognised ones named and located. Three dispositions raise no item; while they were uncounted, a plan that had classified an element and discarded it could not be told from one that had never seen it. `walked` is the sum, so the planner can be checked against the inventory instead of trusted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical planner defects can silently omit conditional routing, misapply token limits across workloads, and preserve saved decisions even when governing predicates or effective inclusion scopes change.
13 open issues
Apply compensation and exclude retained workloads · New Block preflight when custom rules leave policies unresolved · New Include effective fragment includers in decision evidence · New Preserve workspace-qualified API scope for fragments · New Respect effective policy inheritance when resolving fragments · New Prevent overlapping fragment limits from weakening model policies · New Recognize retry and wait as policy statement wrappers · New Preserve conditional context when expanding policy fragments · New Include branch predicates in conditional-routing evidence · New Qualify fragment identities by workspace · New Exclude unselected APIs from actionable policy decisions · New Clarify partial IP-filter failures and source corrections · New Retain parent API identity for operation-scoped fragments · New
What changed in this PR
Hardens the migrate-apim workflow for APIM-to-AI Gateway migrations by making policy treatment visible and checking incomplete executable objects before apply.
Changes:
- Resolves fragment policies to including APIs and surfaces conditional routing as migration decisions.
- Adds fallback mappings, policy-vocabulary checks, and per-element coverage accounting.
- Adds incomplete-object preflight checks, with supporting schemas, tests, and documentation.
| File | Description |
|---|---|
tools/migrate-apim/tests/test_plan.py |
Adds routing, coverage, and fragment-scope regression cases. |
tools/migrate-apim/tests/test_compile.py |
Tests vocabulary coverage and terminal-rule checks. |
tools/migrate-apim/tests/test_capability.py |
Updates fallback and known-policy selection tests. |
tools/migrate-apim/tests/test_apply.py |
Tests incomplete-object blocking and fragment integration. |
tools/migrate-apim/schemas/rules.schema.json |
Defines conditional overlays and terminal-rule matching. |
tools/migrate-apim/schemas/plan.schema.json |
Describes coverage and object-completeness fields. |
tools/migrate-apim/README.md |
Documents mappings, fragment handling, coverage, and preflight. |
tools/migrate-apim/migrate_apim/plan.py |
Tracks policy dispositions and resolves fragment includers. |
tools/migrate-apim/migrate_apim/compile.py |
Checks policy-vocabulary coverage and terminal-rule presence. |
tools/migrate-apim/migrate_apim/capability.py |
Shares four-pass policy selection with compilation. |
tools/migrate-apim/migrate_apim/apply.py |
Reports and blocks executable objects marked incomplete. |
tools/migrate-apim/capabilities/rules.json |
Adds missing mappings, fallback coverage, and routing-loss decisions. |
tools/migrate-apim/ARCHITECTURE.md |
Explains four-pass selection and policy coverage. |
Review effort: Max
Comment on lines
+371
to
+372
| "value": "compensate", | ||
| "consequence": "Say what the condition was for in the priority and weight of the alias backend models, as far as it can be said that way." |
Comment on lines
+950
to
955
| if not rule: | ||
| # Unreachable while rules.json carries a terminal rule, and | ||
| # counted rather than deleted because that is a property of | ||
| # the data, which `compile` enforces and this does not. | ||
| self._count_element("unresolved", document, element) | ||
| continue |
Comment on lines
+1510
to
+1513
| for include_level, include_name in _fragment_includers( | ||
| name or "", self.inventory.get("policies") or [] | ||
| ): | ||
| apis = self._apis_in_scope(include_level, include_name) |
Comment on lines
+1510
to
+1513
| for include_level, include_name in _fragment_includers( | ||
| name or "", self.inventory.get("policies") or [] | ||
| ): | ||
| apis = self._apis_in_scope(include_level, include_name) |
Comment on lines
+1513
to
+1515
| apis = self._apis_in_scope(include_level, include_name) | ||
| if apis is None: | ||
| return None |
| holds = ( | ||
| (isinstance(value, str) and value.lstrip().startswith("@")) | ||
| or (condition.get("hasExpression") and bool(element.get("expressionIds"))) | ||
| or (condition.get("underConditional") and _under_conditional(element)) |
Comment on lines
+2106
to
+2109
| by_fragment: dict[str, list[dict[str, Any]]] = {} | ||
| for document in documents: | ||
| for name in document.get("fragmentRefs") or []: | ||
| by_fragment.setdefault(name, []).append(document.get("scope") or {}) |
Comment on lines
+290
to
+294
| "match": { | ||
| "family": "authentication" | ||
| }, | ||
| "verdict": "gap", | ||
| "itemKind": "accept-loss", |
Comment on lines
+272
to
+274
| "Re-plan from a current inventory, or exclude the items named above to leave their " | ||
| "objects out of the run. Applied as it stands, each one fails its step and stops the " | ||
| "rest of its phase.", |
| seen.add(nested) | ||
| pending.append(nested) | ||
| continue | ||
| entry = (scope.get("level"), scope.get("name")) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Closes #58, closes #59, closes #60, closes #65.
The planner could lose a policy four different ways:
In every case the plan still looked fine. This fixes all four.
#58 — resolved in
_apis_in_scope()by walkingfragmentRefsbackwards, rather than expanding a fragment into one batch member per including API, which is the other option the issue suggests.#59 — the verdict check moved after
_apply_partial_when(), andpartialWhenlearnedhasExpressionandunderConditional. The trigger is rule data rather than element names in the planner, soauthentication-managed-identityand a plain<set-backend-service backend-id="pool"/>behave exactly as before.#65 — three places, because none of them is enough alone: a terminal rule in
rules.jsonsoNonecan't happen at runtime, acompilecheck so that rule can't quietly become the mapping, and a coverage tally so the planner can be checked against the inventory instead of trusted. The ten unanswered elements are covered by five family rules and one element rule, so the next member of each family is answered before anyone writes it.#60 — blocking, and it reads from
steps_for()rather than re-deriving which items become steps. Two copies of that predicate drifted once already, and an object no phase will run can't fail a phase — blocking on it would stop a migration over work you were going to do by hand anyway.One test spans them. #60 is the safety net for the other three, and each half passes its own tests with the other absent, so that join was untested. Now: a fragment an API includes raises nothing to block on, and a fragment nothing includes stops at preflight naming the fragment, instead of taking out the phase.
Testing
583 passed, 1 skipped,compile --strictclean.All offline — no Azure resources created, nothing run against a live APIM instance or gateway. These are contract bugs, decidable against the committed catalog, the CLI surface snapshot and the fake ARM fixture. The
exportjob covers the other half by installing the pinned wheel.Not fixed here
#63 — nothing checks that the
partialWhenconditionsrules.schema.jsonaccepts are the ones_apply_partial_when()actually implements. It's already onmain, but #59 adds two conditions to that vocabulary, so this PR widens the gap. Fix it here or follow up?🤖 Generated with Claude Code