diff --git a/tools/migrate-apim/ARCHITECTURE.md b/tools/migrate-apim/ARCHITECTURE.md index 5249413..6ab50e2 100644 --- a/tools/migrate-apim/ARCHITECTURE.md +++ b/tools/migrate-apim/ARCHITECTURE.md @@ -27,7 +27,9 @@ Declares how APIM shapes map to target capabilities. Never versioned separately; - **`structural[]`** — Rules for cross-asset things (workspaces, subscriptions, telemetry, coverage gaps). -- **`policies[]`** — How APIM policies map to AI Gateway policies. Matched by element name, then by family, with fallbacks for expressions and unknowns. +- **`policies[]`** — How APIM policies map to AI Gateway policies. Four passes in order: by element name, then by family, then a fallback for anything carrying a policy expression, then one terminal rule (`match: {anyElement: true}`) that answers for everything else. + + The last pass is what makes an unmapped element impossible rather than unlikely: before it, an element matching no rule was dropped where it was classified, and the ones that vanished were those carrying *no* expression, since an expression at least reached the pass above. It is a net and not a mapping, so `compile` holds every element in `policy.FAMILIES` to a rule of its own or a family rule and fails if one is left to it. The planner counts what reaches it into `policyCoverage.unrecognised` on the plan. - **`publishableRoutes[]`** — The four routes AI Gateway publishes (`/openai/v1/chat/completions`, `/v1/messages`, etc.). Used to decide what part of a model API surface can move. diff --git a/tools/migrate-apim/README.md b/tools/migrate-apim/README.md index 6c006be..cbcdea7 100644 --- a/tools/migrate-apim/README.md +++ b/tools/migrate-apim/README.md @@ -436,9 +436,29 @@ model, or a tool server. APIM has well over a hundred. Most of the migration's r | `azure-openai-content-safety`, `llm-content-safety` | — | **Gap.** Content-safety flags exist in the CLI but are not supported on typed `aigateways`. The policy also pulls in a separate Content Safety backend, surfaced as a `supportingBackends` dependency | | Transformations: `set-body`, `set-header`, `rewrite-uri`, `json-to-xml` | — | **Gap.** No request/response rewriting | | `send-request`, `return-response`, custom `@{…}` expressions | — | **Gap.** `agentHooks` is the nearest mechanism, and only for tool traffic | +| `authentication-basic`, `authentication-certificate`, `get-authorization-context` | — | **Gap.** A provider authenticates with a managed identity or an API key; `model-provider create/update` exposes no basic credential, no client certificate, and no credential-manager equivalent | +| `check-header`, `validate-client-certificate` | — | **Gap.** Caller authentication is Entra auth plus RBAC, or an API key. A gate on a shared header secret or a client certificate is neither | +| `emit-metric`, `log-to-eventhub` | — | **Gap.** `telemetry-exporter create` takes a kind and an endpoint, not a metric to emit or a sink to write to | +| `set-service-url` | — | **Gap.** An alias backend pool holds backends the gateway knows; overriding the URL outright names no backend | +| `mcp-server` | `mcp create` | **Handled.** Collected as an MCP asset from the API that carries it, and registered by the MCP server item | +| `validate-mcp-request` | — | **Gap.** `mcp create` registers a server, its endpoints and its versions; guarding a tool call on the way through is not part of that surface | | `cors`, `trace`, correlation headers | — | **Not applicable** | +| Anything else APIM ships | — | **Gap, blocking.** Reported rather than assumed harmless — see below | | — | `costLimit` | AI Gateway capability with no APIM source; an upgrade, not a gap | +**Nothing is dropped for want of a rule.** Rule selection ends in a terminal rule that matches +anything the four passes above it did not, so an element this catalog has never heard of reaches the +plan as a blocking decision instead of being discarded where it was classified. That hole was real: +eleven elements matched no rule at all, three of them carrying backend credentials, and because an +element containing an expression at least fell into the pass above, the policies that vanished most +completely were the simplest ones. + +The net is not allowed to become the mapping. `compile` holds every element in `policy.FAMILIES` — +the vocabulary the collector classifies against — to a rule of its own or a rule for its family, and +fails naming any that are left to the terminal rule. What legitimately reaches it is counted on the +plan under `policyCoverage`, which accounts for every element walked under exactly one disposition +and sums them, so a reader can check the planner against the inventory rather than trust it. + ### Known structural gaps These are not policy details; they change what a migration *means*, and each one should surface as @@ -759,9 +779,15 @@ checked, so a journal carrying a stale or wrong phase status repairs itself on t There is no transaction across steps, so the cost of finding a problem halfway is a half-built gateway. `--preflight` refuses while any of these stands: the plan names no target; an item still needs a decision; a settled decision has been orphaned; a referenced credential is missing or not -mechanically retrievable; or a `Microsoft.ApiManagement/service` shares the target's name, which -matters because several CLI commands dispatch by name without `--ids`. Manual prerequisites are -raised as notes and acknowledged with `--acknowledge `. +mechanically retrievable; the plan describes work it cannot itself produce; a step names a provider +nothing in the plan registers; an object the planner marked as not fully configurable would become a +step; or a `Microsoft.ApiManagement/service` shares the target's name, which matters because several +CLI commands dispatch by name without `--ids`. Manual prerequisites are raised as notes and +acknowledged with `--acknowledge `. + +That last one is the plan's own judgement, read back. `plan` records why it could not finish shaping +an object, and until `apply` consulted it the condition surfaced as a step failing part-way through +a phase — under a cause the builder guessed at, with every step queued behind it skipped. ### Phases @@ -792,7 +818,8 @@ Phase models -- EXECUTING **`policies` writes two different things**, because the gateway holds policy in two places. A token limit lives on a *model registration*, so an APIM policy scoped to an API, a product or the service -is expanded at planning time into one object per model it covered — one object is one +is expanded at planning time into one object per model it covered — a policy factored into a +*fragment* covers whatever includes it, and is followed there first — one object is one `az aigateway model policy set`. Policies batch by counter key, so a batch can hold two limits with different ceilings; each member keeps its own. On a real instance that was 10000/minute on one API and 1000 on another, and shaping only the first member applied one ceiling to both. diff --git a/tools/migrate-apim/capabilities/rules.json b/tools/migrate-apim/capabilities/rules.json index 8c0bc8d..a6307c0 100644 --- a/tools/migrate-apim/capabilities/rules.json +++ b/tools/migrate-apim/capabilities/rules.json @@ -277,6 +277,61 @@ "rationale": "No schema, header, or status validation on the target.", "proposal": "retain-in-apim" }, + { + "match": { + "family": "backendAuthentication" + }, + "verdict": "gap", + "itemKind": "accept-loss", + "rationale": "A model provider authenticates with a managed identity or an API key. `model-provider create` and `model-provider update` expose no basic credential, no client certificate, and nothing equivalent to an APIM credential manager, so a backend secured any other way has no target. `authentication-managed-identity` is the one member of this family that does map, and it has a rule of its own.", + "proposal": "retain-in-apim" + }, + { + "match": { + "family": "authentication" + }, + "verdict": "gap", + "itemKind": "accept-loss", + "rationale": "Caller authentication on the gateway is Entra auth plus data-plane RBAC, or an API key. A gate on a shared secret in a header, or on a client certificate, is neither. `validate-jwt` and `validate-azure-ad-token` are the members of this family that do map, and they have a rule of their own.", + "proposal": "retain-in-apim" + }, + { + "match": { + "family": "observability" + }, + "verdict": "gap", + "itemKind": "accept-loss", + "rationale": "The gateway reports through a telemetry exporter it owns: `telemetry-exporter create` takes a kind and an endpoint, not a metric to emit or a sink to write to. There is no Event Hub destination and no way to author a custom metric. Where the gateway's own telemetry reports is a separate decision, on the telemetry destination item.", + "proposal": "accept" + }, + { + "match": { + "element": [ + "mcp-server" + ] + }, + "verdict": "handled", + "capabilityId": "mcp.create", + "rationale": "The tool server this policy exposes is collected as an MCP asset from the API that carries it, and registered by the MCP server item rather than raised again here." + }, + { + "match": { + "family": "mcp" + }, + "verdict": "gap", + "itemKind": "accept-loss", + "rationale": "`mcp create` registers a tool server, its endpoints and its supported versions. Guarding a tool call on the way through is not part of that surface. `mcp-server`, which declares the server itself, is carried by the MCP server item and has a rule of its own.", + "proposal": "retain-in-apim" + }, + { + "match": { + "family": "routing" + }, + "verdict": "gap", + "itemKind": "accept-loss", + "rationale": "An alias backend pool holds backends the gateway knows, with a priority and a weight each. Overriding the destination URL outright names no backend, so there is nothing for the pool to hold. `set-backend-service` and `forward-request` choose among backends and have a rule of their own.", + "proposal": "retain-in-apim" + }, { "match": { "element": [ @@ -300,7 +355,30 @@ }, "verdict": "handled", "capabilityId": "alias.create", - "rationale": "Backend selection becomes the alias backend pool." + "rationale": "Backend selection becomes the alias backend pool.", + "partialWhen": { + "hasExpression": true, + "underConditional": true + }, + "partial": { + "itemKind": "accept-loss", + "options": [ + { + "value": "accept", + "consequence": "Every call takes the alias backend pool in its own order, whatever the source would have chosen for it." + }, + { + "value": "compensate", + "consequence": "Say what the condition was for in the priority and weight of the alias backend models, as far as it can be said that way." + }, + { + "value": "retain-in-apim", + "consequence": "Keep this workload on APIM, where the routing logic still runs." + } + ], + "proposal": "compensate", + "rationale": "The source decides where the request goes with logic the target has no place for: a policy expression, a branch of a `choose`, or both. The alias backend pool carries the set of backends and their priority and weight, not the rule that picked among them." + } }, { "match": { @@ -325,9 +403,19 @@ }, "verdict": "gap", "itemKind": "accept-loss", - "note": "Last of the three passes in capability.policy_rule, so it is reached only by an element with no rule of its own and no family rule. The ordering is in the code, not in a priority here.", + "note": "Third of the four passes in capability.policy_rule, so it is reached only by an element with no rule of its own and no family rule. The ordering is in the code, not in a priority here.", "rationale": "A policy expression doing more than reading a well-known context property has no equivalent. The target's policy surface is six declarative types.", "proposal": "retain-in-apim" + }, + { + "match": { + "anyElement": true + }, + "verdict": "gap", + "itemKind": "accept-loss", + "note": "Last of the four passes in capability.policy_rule, and the one that makes an unmapped element impossible rather than merely unlikely. `compile` holds every element in policy.FAMILIES to a rule of its own or a family rule, so nothing reaching here is a policy the tool has ever been taught; it is a policy APIM has that this catalog does not know about. Deliberately blocking: the alternative is a migration that silently drops a policy nobody has read.", + "rationale": "This policy is not in the vocabulary this catalog was written against, so nothing can be said about what it does or whether the target has an equivalent. It is reported rather than assumed harmless.", + "proposal": "retain-in-apim" } ], "structural": [ diff --git a/tools/migrate-apim/migrate_apim/apply.py b/tools/migrate-apim/migrate_apim/apply.py index 30e4db8..02125f7 100644 --- a/tools/migrate-apim/migrate_apim/apply.py +++ b/tools/migrate-apim/migrate_apim/apply.py @@ -245,6 +245,35 @@ def finding(code: str, blocking: bool, message: str, remedy: str = "") -> None: "Re-plan from a current inventory, or settle the items that register those providers.", ) + # Work the plan has already judged unappliable. `plan` detects it, explains + # it and writes the reason on the object, and nothing read it back: the + # condition surfaced as a step failing part-way through a phase, under a + # builder's guess at a cause the plan could name exactly. A failed step stops + # its phase, so one such object also costs every step queued behind it. + # + # Asked of `steps_for` rather than re-derived from the items, so the objects + # checked are the objects that run. One no phase performs cannot fail a + # phase, and blocking on it would stop a whole migration over work the + # operator was going to apply by hand -- which `notExecutable` already names. + incomplete = [ + (step, obj) + for step in steps_for(plan) + for obj in step.objects + # Only an explicit false. The field is absent from most objects, and + # reading absence as incomplete would block every plan there is. + if obj.get("configurationComplete") is False + ] + if incomplete: + finding( + "incompleteObject", True, + f"{len(incomplete)} object(s) the plan marked as not fully configurable: " + + "; ".join(_incomplete_note(step, obj) for step, obj in incomplete[:3]) + + ("..." if len(incomplete) > 3 else ""), + "Re-plan from a current inventory, or exclude the items named above to leave their " + "objects out of the run. Applied as it stands, each one fails its step and stops the " + "rest of its phase.", + ) + unbuildable: dict[str, list[str]] = {} for item in plan.get("items") or []: capability_id = item.get("capabilityId") @@ -277,6 +306,30 @@ def finding(code: str, blocking: bool, message: str, remedy: str = "") -> None: return findings +def _incomplete_note(step: Step, obj: dict[str, Any]) -> str: + """One unappliable object, in the planner's own words. + + The reason is quoted rather than restated, because the plan knows it and a + builder does not: `model policy set` can say only that an object names no + model registration, where the plan can say which scope covered none. + `fullConfigurationAt` comes with it, since the remedy usually starts at the + policy document the element was read from. + """ + label = _object_label(obj) + where = f"{step.item_id} ({label})" if label else step.item_id + reason = str(obj.get("incompleteReason") or "").strip() + if not reason: + # Two planner paths mark an object incomplete from empty settings alone + # and write no reason. Reporting that silence beats quoting an empty + # string, and naming a cause this check cannot see would be a guess. + reason = "the plan marked it incomplete and said no more" + if not obj.get("settings"): + reason += "; it carries no settings" + at = obj.get("fullConfigurationAt") or {} + source = " ".join(str(part) for part in (at.get("policyId"), at.get("path")) if part) + return f"{where}: {reason}" + (f" [{source}]" if source else "") + + def _role_grant_check( plan: dict[str, Any], target: Target, runner: Runner ) -> list[dict[str, Any]]: diff --git a/tools/migrate-apim/migrate_apim/capability.py b/tools/migrate-apim/migrate_apim/capability.py index e008180..71f8d23 100644 --- a/tools/migrate-apim/migrate_apim/capability.py +++ b/tools/migrate-apim/migrate_apim/capability.py @@ -24,11 +24,12 @@ from .azcli import az_argv from .dispatch import selector -# `policy_rule` runs three passes in order: an element-specific rule wins, then -# a family rule, then the expression fallback. Each pass is its own selection -# site, so each declares what it looks at. +# `policy_rule` runs four passes in order: an element-specific rule wins, then a +# family rule, then the expression fallback, then a terminal rule that matches +# anything. Each pass is its own selection site, so each declares what it looks +# at. # -# All three hand the chosen rule back to the same reader in `plan._policies`, so +# All four hand the chosen rule back to the same reader in `plan._policies`, so # they read the same fields; they differ only in how the rule is found. _POLICY_FIELDS = ( "verdict", "itemKind", "capabilityId", "target", "proposal", "rationale", @@ -46,6 +47,50 @@ "policies", keys=("hasExpression",), site="capability.policy_rule expression", reads=_POLICY_FIELDS, ) +# The pass that makes "no rule" unreachable. Everything above it answers some +# part of APIM's policy vocabulary; this answers the rest, including whatever +# APIM ships next. Without it an element nothing matched was dropped where it +# was classified, so a policy the tool did not recognise left no trace at all -- +# and the elements that reached it were the ones carrying no expression, since +# an expression at least fell into the pass above. The simpler the policy, the +# more completely it vanished. +POLICY_TERMINAL = selector( + "policies", keys=("anyElement",), values={"anyElement": (True,)}, + site="capability.policy_rule terminal", reads=_POLICY_FIELDS, +) + + +def resolve_policy_rule( + rules: list[dict[str, Any]], element_name: str, *, family: str = "", + has_expression: bool = False, +) -> dict[str, Any] | None: + """The rule governing one policy element, by the four passes in order. + + A module-level function because `compile` has to ask the same question of a + raw rules document, and asking it a second way is how the two answers drift. + """ + for rule in rules: + if not POLICY_BY_ELEMENT.governs(rule): + continue + names = (rule.get("match") or {}).get("element") + if names and element_name in names: + return rule + for rule in rules: + if not POLICY_BY_FAMILY.governs(rule): + continue + match = rule.get("match") or {} + if match.get("family") and match["family"] == family: + return rule + if has_expression: + for rule in rules: + if not POLICY_BY_EXPRESSION.governs(rule): + continue + if (rule.get("match") or {}).get("hasExpression"): + return rule + for rule in rules: + if POLICY_TERMINAL.governs(rule): + return rule + return None CAPABILITIES_DIR = Path(__file__).resolve().parent.parent / "capabilities" # Discovered rather than written down. The catalog is renamed on every CLI @@ -219,28 +264,25 @@ def policy_rule(self, element_name: str, *, family: str = "", has_expression: bo ``@(context.Subscription.Id)`` still maps to ``tokenLimit`` -- only the partitioning needs choosing, which the rule expresses through ``partialWhen``. The expression rule is the fallback for elements with no - mapping of their own, where arbitrary C# genuinely has no target. + mapping of their own, where arbitrary C# genuinely has no target, and the + terminal rule catches whatever reaches neither. """ - rules = self.rules.get("policies") or [] - for rule in rules: - if not POLICY_BY_ELEMENT.governs(rule): - continue - names = (rule.get("match") or {}).get("element") - if names and element_name in names: - return rule - for rule in rules: - if not POLICY_BY_FAMILY.governs(rule): - continue - match = rule.get("match") or {} - if match.get("family") and match["family"] == family: - return rule - if has_expression: - for rule in rules: - if not POLICY_BY_EXPRESSION.governs(rule): - continue - if (rule.get("match") or {}).get("hasExpression"): - return rule - return None + return resolve_policy_rule( + self.rules.get("policies") or [], element_name, + family=family, has_expression=has_expression, + ) + + def terminal_policy_rule(self) -> dict[str, Any] | None: + """The rule that answers for an element nothing else recognises. + + Named apart from the rest because reaching it is not the same as being + mapped: `compile` holds every element the tool knows of to a rule of its + own, so arriving here means the vocabulary has grown. + """ + return next( + (rule for rule in self.rules.get("policies") or [] if POLICY_TERMINAL.governs(rule)), + None, + ) def service_only_entry(self, entry_id: str) -> dict[str, Any] | None: """A capability the gateway has and the CLI cannot author.""" diff --git a/tools/migrate-apim/migrate_apim/compile.py b/tools/migrate-apim/migrate_apim/compile.py index 84f540c..3019a52 100644 --- a/tools/migrate-apim/migrate_apim/compile.py +++ b/tools/migrate-apim/migrate_apim/compile.py @@ -33,7 +33,13 @@ from typing import Any from . import dispatch -from .capability import CAPABILITIES_DIR, DEFAULT_CATALOG +from .capability import ( + CAPABILITIES_DIR, + DEFAULT_CATALOG, + POLICY_TERMINAL, + resolve_policy_rule, +) +from .policy import FAMILIES, family_of # Importing these registers every selection site in dispatch.REGISTRY. The # import is the point: the contract checked below is the one the planner runs. @@ -427,6 +433,50 @@ def _explain_unreachable(rule: dict[str, Any], section: str, label: str, out: Co ) +def check_policy_vocabulary(rules: dict[str, Any], out: Compilation) -> None: + """Every policy element the collector can name must be answered deliberately. + + `policy.FAMILIES` is the vocabulary this tool classifies against: committed + data, so the question is decidable here, offline, with no APIM to ask. The + terminal rule answers for anything outside it, which is what keeps an + unrecognised policy out of silence -- but it would equally answer for an + element somebody simply never wrote a rule for, and then the safety net + becomes the mapping. Eleven elements lived there, three of them carrying + backend credentials. + + So the net is held to its purpose: a known element reaching it is an error, + and the fix is a rule of its own or a rule for its family. + """ + policies = rules.get("policies") or [] + out.cover( + f"{len(FAMILIES)} policy elements the collector classifies: each answered by a " + f"rule of its own or a family rule, none left to the terminal rule" + ) + + terminal = next((rule for rule in policies if POLICY_TERMINAL.governs(rule)), None) + if terminal is None: + out.error( + "policy-terminal", + "no rule matches `anyElement`, so an element outside the known vocabulary resolves " + "to nothing and is dropped where it is classified. Add the terminal rule.", + ) + return + + unanswered: dict[str, list[str]] = {} + for name in sorted(FAMILIES): + rule = resolve_policy_rule(policies, name, family=family_of(name), has_expression=False) + if rule is terminal: + unanswered.setdefault(family_of(name), []).append(name) + + for family, names in sorted(unanswered.items()): + out.error( + "policy-vocabulary", + f"family {family!r}: {', '.join(names)} reach the terminal rule, which exists for " + f"policies this catalog has never heard of. Give the family a rule, or each element " + f"one of its own.", + ) + + def _check_uniqueness(rules: dict[str, Any], out: Compilation) -> None: """Selection is first-match or last-write; either way a tie is a silent bug. @@ -581,6 +631,7 @@ def compile_contract( check_capabilities(catalog, surface, out) check_pins(catalog, catalog_path, rules, surface, out) check_rules(catalog, rules, out) + check_policy_vocabulary(rules, out) if newest_release: check_release_pin(catalog, newest_release, out) return out diff --git a/tools/migrate-apim/migrate_apim/plan.py b/tools/migrate-apim/migrate_apim/plan.py index f23a85c..bd0074d 100644 --- a/tools/migrate-apim/migrate_apim/plan.py +++ b/tools/migrate-apim/migrate_apim/plan.py @@ -122,6 +122,9 @@ def __init__( self.previous = previous or {} self.items: list[dict[str, Any]] = [] self.prerequisites: list[dict[str, Any]] = [] + # What became of every policy element walked, one bucket each. + self.policy_coverage: dict[str, int] = {} + self.unrecognised_policies: list[dict[str, Any]] = [] # A model API names the backend it routes to; the provider asset built # from that backend is what knows the protocol it speaks. self._provider_by_backend: dict[str, dict[str, Any]] = { @@ -211,6 +214,15 @@ def build(self) -> dict[str, Any]: "plannerGaps": len(planner_gaps), }, "prerequisites": self.prerequisites, + # What became of every policy element the inventory holds. The + # dispositions are exclusive and `walked` is their sum, so a reader + # can check the planner against the inventory instead of trusting + # that an element which raised no item was one that needed none. + "policyCoverage": { + "walked": sum(self.policy_coverage.values()), + "dispositions": dict(sorted(self.policy_coverage.items())), + "unrecognised": self.unrecognised_policies, + }, "plannerGaps": planner_gaps, # Settled decisions whose item no longer exists. Dropping them quietly # would lose work across a tool change and re-ask as if never asked. @@ -914,19 +926,49 @@ def _agent(self, agent: dict[str, Any]) -> None: # ----------------------------------------------------------- policies def _policies(self) -> None: - """One item per distinct policy treatment, batched across assets.""" + """One item per distinct policy treatment, batched across assets. + + Every element walked leaves a count behind, in `policyCoverage`. Three + of the dispositions here end the loop without raising anything, and for + as long as they were uncounted the plan could not be told apart from one + that had never seen the element at all. The tally is what makes "nothing + was dropped" a thing a reader can check rather than take on trust. + """ batches: dict[str, dict[str, Any]] = {} + terminal = self.caps.terminal_policy_rule() for document in self.inventory.get("policies") or []: for element in document.get("elements") or []: if not _is_policy_statement(element): + # Not a decision of its own: the configuration of the + # statement above it, which is classified in its own right. + self._count_element("configuration", document, element) continue has_expression = bool(element.get("expressionIds")) rule = self.caps.policy_rule( element["name"], family=element.get("family", ""), has_expression=has_expression ) - if not rule or rule["verdict"] in {"notApplicable", "handled"}: + if not rule: + # Unreachable while rules.json carries a terminal rule, and + # counted rather than deleted because that is a property of + # the data, which `compile` enforces and this does not. + self._count_element("unresolved", document, element) continue + # Analysed before it is excluded, because `handled` is a claim + # about the element and not every element it covers makes it + # true. A backend chosen inside a `choose` was excluded on the + # strength of an alias pool that holds the backends and not the + # logic that picked among them, and left no trace in the plan. rule = _apply_partial_when(rule, element) + if rule["verdict"] in {"notApplicable", "handled"}: + self._count_element(rule["verdict"], document, element) + continue + # One bucket per element, so the counts sum to what was walked. + # An unrecognised element still becomes an item -- it is counted + # apart because reaching the terminal rule says the vocabulary + # has grown, which is not the same as having been mapped. + self._count_element( + "unrecognised" if rule is terminal else "classified", document, element + ) signature = _policy_signature(element, rule, has_expression) batch = batches.setdefault(signature, {"rule": rule, "element": element, "hasExpression": has_expression, "members": []}) @@ -945,7 +987,6 @@ def _policies(self) -> None: rule = batch["rule"] element = batch["element"] members = batch["members"] - label = element["name"] + (" with expression" if batch["hasExpression"] else "") self._add( item_id=ids.item("policy", signature), source={"assetId": None, "assetKind": "policy", "aspect": element["name"]}, @@ -961,8 +1002,7 @@ def _policies(self) -> None: if rule.get("target") else f"`{element['name']}` is carried by another object." ), - options=[_opt(value, "") for value in rule.get("options") or []] - or _loss_options(rule["verdict"]), + options=_options_of(rule) or _loss_options(rule["verdict"]), evidence={ "element": element["name"], "target": rule.get("target"), @@ -975,6 +1015,25 @@ def _policies(self) -> None: blocking=rule["verdict"] == "gap", ) + def _count_element( + self, disposition: str, document: dict[str, Any], element: dict[str, Any] + ) -> None: + """Account for one policy element under exactly one disposition. + + The unrecognised ones are listed as well as counted, because a number + alone cannot be acted on: knowing that one element was not in the + vocabulary is only useful alongside which element, and where it is + written. + """ + self.policy_coverage[disposition] = self.policy_coverage.get(disposition, 0) + 1 + if disposition in {"unrecognised", "unresolved"}: + self.unrecognised_policies.append({ + "element": element["name"], + "family": element.get("family"), + "policyId": document["id"], + "path": element["path"], + }) + # --------------------------------------------------------- structural def _structural(self) -> None: @@ -1440,6 +1499,22 @@ def _apis_in_scope(self, level: str | None, name: str | None) -> set[str] | None return None if level == "api": return {name or ""} + if level == "fragment": + # A fragment is not a scope anything runs at: it covers whatever + # includes it. Read as a scope of its own it covered no API, so a + # token limit factored into a shared fragment -- the way an operator + # is meant to share one -- reached no model registration and the + # policy stranded. Each including scope is resolved as itself, so a + # fragment included globally still covers everything. + covered: set[str] = set() + for include_level, include_name in _fragment_includers( + name or "", self.inventory.get("policies") or [] + ): + apis = self._apis_in_scope(include_level, include_name) + if apis is None: + return None + covered |= apis + return covered if level == "workspace": return { _api_of((item.get("source") or {}).get("assetId") or "") @@ -1892,9 +1967,22 @@ def _namespace_endpoints(server: dict[str, Any], endpoints: list[dict[str, Any]] endpoint["namespace"] = base if index == 0 else f"{base}-{index + 1}" +# Control flow that wraps a statement without being its configuration. A +# statement inside one of these runs on some requests and not others, and the +# element itself records nothing about that: the path is the only place the +# branch survives normalization. +_CONDITIONAL_WRAPPERS = {"choose", "when", "otherwise"} + # Sections and control flow wrap policy statements; anything else that wraps an # element makes that element its configuration, not a policy of its own. -_STATEMENT_WRAPPERS = {"inbound", "backend", "outbound", "on-error", "fragment", "choose", "when", "otherwise"} +_STATEMENT_WRAPPERS = { + "inbound", "backend", "outbound", "on-error", "fragment", *_CONDITIONAL_WRAPPERS, +} + + +def _ancestors(element: dict[str, Any]) -> list[str]: + """The elements enclosing this one, outermost first, occurrence index removed.""" + return [segment.split("[", 1)[0] for segment in (element.get("path") or "").split("/")[:-1]] def _is_policy_statement(element: dict[str, Any]) -> bool: @@ -1904,22 +1992,38 @@ def _is_policy_statement(element: dict[str, Any]) -> bool: ```` child is how the header is configured. Walking it separately produced phantom items named after configuration elements. """ - ancestors = [segment.split("[", 1)[0] for segment in (element.get("path") or "").split("/")[:-1]] - return all(name in _STATEMENT_WRAPPERS for name in ancestors) + return all(name in _STATEMENT_WRAPPERS for name in _ancestors(element)) + + +def _under_conditional(element: dict[str, Any]) -> bool: + """Whether a branch of the source policy decides if this element runs at all.""" + return any(name in _CONDITIONAL_WRAPPERS for name in _ancestors(element)) def _apply_partial_when(rule: dict[str, Any], element: dict[str, Any]) -> dict[str, Any]: - """Demote a native mapping to partial only when the rule's condition holds. + """Demote a mapping to partial when one of the rule's conditions holds. A token limit maps natively; it becomes a decision only because its caller - partition is an expression APIM evaluates and the target cannot. + partition is an expression APIM evaluates and the target cannot. A backend + selection is carried by the alias pool; it becomes a decision when the + source picked the backend with an expression or inside a ``choose``, + because the pool holds the backends and not the rule that chose among them. + + Any one condition demotes. Requiring all of them would let an element that + trips a single one read as cleanly mapped, which is the reading this exists + to prevent. """ condition = rule.get("partialWhen") if not condition: return rule attribute = condition.get("attributeIsExpression") value = (element.get("attributes") or {}).get(attribute or "") - if not (isinstance(value, str) and value.lstrip().startswith("@")): + holds = ( + (isinstance(value, str) and value.lstrip().startswith("@")) + or (condition.get("hasExpression") and bool(element.get("expressionIds"))) + or (condition.get("underConditional") and _under_conditional(element)) + ) + if not holds: return rule return {**rule, "verdict": "partial", **(rule.get("partial") or {})} @@ -1987,6 +2091,39 @@ def _api_of(asset_id: str) -> str: return (asset_id.split(":", 1)[-1]).rsplit("/", 1)[-1] +def _fragment_includers( + fragment: str, documents: list[dict[str, Any]] +) -> list[tuple[str | None, str | None]]: + """The scopes whose policy includes a fragment, as (level, name) pairs. + + The inventory holds a fragment as a document of its own, and the only link + back to the policies that run it is the `fragmentRefs` the collector records + from ``. A fragment included by a fragment is followed too, + so what comes back is always a scope something runs at. APIM accepts a cycle + of includes even though it cannot execute one, so the walk guards against + revisiting a fragment rather than trusting the source not to contain one. + """ + by_fragment: dict[str, list[dict[str, Any]]] = {} + for document in documents: + for name in document.get("fragmentRefs") or []: + by_fragment.setdefault(name, []).append(document.get("scope") or {}) + + scopes: list[tuple[str | None, str | None]] = [] + pending, seen = [fragment], {fragment} + while pending: + for scope in by_fragment.get(pending.pop(0), []): + if scope.get("level") == "fragment": + nested = scope.get("name") or "" + if nested not in seen: + seen.add(nested) + pending.append(nested) + continue + entry = (scope.get("level"), scope.get("name")) + if entry not in scopes: + scopes.append(entry) + return scopes + + # How strict one policy is against another of the same type, lowest first. Only # used to choose which of several source policies becomes the one gateway-wide # baseline, where the conservative choice is the one no source exceeded. diff --git a/tools/migrate-apim/schemas/plan.schema.json b/tools/migrate-apim/schemas/plan.schema.json index 8a117ca..4c9bc62 100644 --- a/tools/migrate-apim/schemas/plan.schema.json +++ b/tools/migrate-apim/schemas/plan.schema.json @@ -71,6 +71,38 @@ } }, + "policyCoverage": { + "description": "What became of every policy element the inventory holds. Three dispositions raise no item, and while they were uncounted a plan that had classified an element and discarded it was indistinguishable from one that had never seen it. The dispositions are exclusive and `walked` is their sum, so the planner can be checked against the inventory rather than trusted.", + "type": "object", + "required": ["walked", "dispositions", "unrecognised"], + "properties": { + "walked": { + "description": "Every element accounted for, which is the sum of the dispositions.", + "type": "integer", + "minimum": 0 + }, + "dispositions": { + "description": "classified: raised an item. handled / notApplicable: the rule says another object carries it, or that there is nothing to carry. configuration: a child element that configures the statement above it. unrecognised: outside the vocabulary this catalog knows, answered by the terminal rule. unresolved: matched no rule at all, which a rules document carrying a terminal rule cannot produce.", + "type": "object", + "additionalProperties": {"type": "integer", "minimum": 0} + }, + "unrecognised": { + "description": "The elements counted as unrecognised or unresolved, named and located. A count alone cannot be acted on.", + "type": "array", + "items": { + "type": "object", + "required": ["element", "policyId", "path"], + "properties": { + "element": {"type": "string"}, + "family": {"type": ["string", "null"]}, + "policyId": {"type": "string"}, + "path": {"type": "string"} + } + } + } + } + }, + "plannerGaps": { "description": "Work an item commits to that the plan cannot produce: a required field no source supplied, or a settled decision that yields no target object. These are defects in this tool rather than decisions for the operator, and they block apply, so work that will not happen is visible before the first write instead of being discovered when a phase produces nothing.", "type": "array", @@ -187,7 +219,31 @@ "type": "object", "properties": { "capability": {"type": "string"}, - "objects": {"type": "array"} + "objects": { + "type": "array", + "items": { + "description": "One call's inputs. The fields are the capability's own and are left open here, except for the three below, which any object may carry and which apply reads whatever the capability is.", + "type": "object", + "properties": { + "configurationComplete": { + "description": "False when the planner could not shape the whole object. Preflight blocks on one that would become a step: the builder cannot produce it either, and a step that fails part-way through a phase leaves a half-built gateway.", + "type": "boolean" + }, + "incompleteReason": { + "description": "Why the object is incomplete, in the planner's words, because only the planner knows. Quoted by preflight, which can otherwise say no more than that the object is incomplete.", + "type": "string" + }, + "fullConfigurationAt": { + "description": "The source document and element path the object was shaped from, so whatever did not survive the shaping can be read where it is whole.", + "type": "object", + "properties": { + "policyId": {"type": ["string", "null"]}, + "path": {"type": ["string", "null"]} + } + } + } + } + } } }, "blocking": {"type": "boolean"}, diff --git a/tools/migrate-apim/schemas/rules.schema.json b/tools/migrate-apim/schemas/rules.schema.json index 57cf274..69d2583 100644 --- a/tools/migrate-apim/schemas/rules.schema.json +++ b/tools/migrate-apim/schemas/rules.schema.json @@ -59,6 +59,40 @@ "note": {"type": "string"} } }, + "partialWhen": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "description": "What makes this mapping lossy for one element. Any one condition that holds demotes the rule to partial and overlays `partial` on it; requiring all of them would let an element that trips a single condition read as cleanly mapped. Each condition is named by `plan._apply_partial_when`, and a key it does not know would never fire while reading as though it did, so the set is closed here. The booleans take `true` only, for the same reason: there is no condition that holds by an element lacking an expression.", + "properties": { + "attributeIsExpression": { + "type": "string", + "description": "The attribute whose value is a policy expression rather than a literal." + }, + "hasExpression": { + "const": true, + "description": "The element evaluates a policy expression anywhere in it." + }, + "underConditional": { + "const": true, + "description": "A `choose` branch in the source decides whether the element runs at all." + } + } + }, + "partialOverlay": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "description": "The rule as it reads once `partialWhen` holds, overlaid on the rule itself. The verdict is already partial by then, so it is not restated here.", + "properties": { + "itemKind": {"$ref": "#/$defs/itemKind"}, + "capabilityId": {"type": "string"}, + "options": {"$ref": "#/$defs/options"}, + "proposal": {"type": "string"}, + "rationale": {"type": "string"}, + "target": {"type": "string"} + } + }, "partialNeedsOptions": { "if": {"properties": {"verdict": {"const": "partial"}}, "required": ["verdict"]}, "then": {"required": ["options"]} @@ -122,6 +156,7 @@ "required": ["match", "verdict"], "additionalProperties": false, "allOf": [{"$ref": "#/$defs/partialNeedsOptions"}], + "dependentRequired": {"partial": ["partialWhen"]}, "properties": { "match": { "type": "object", @@ -130,7 +165,11 @@ "properties": { "element": {"type": "array", "items": {"type": "string"}, "minItems": 1}, "family": {"type": "string"}, - "hasExpression": {"type": "boolean"} + "hasExpression": {"type": "boolean"}, + "anyElement": { + "const": true, + "description": "The terminal rule, tried last and matching anything the passes above did not. There is exactly one; a second is a tie `compile` rejects. It exists so that an element no rule recognises still reaches the plan, rather than being dropped where it was classified." + } } }, "verdict": {"$ref": "#/$defs/verdict"}, @@ -141,8 +180,8 @@ "rationale": {"type": "string"}, "note": {"type": "string"}, "target": {"type": "string"}, - "partial": {"type": "object"}, - "partialWhen": {"type": "object"}, + "partial": {"$ref": "#/$defs/partialOverlay"}, + "partialWhen": {"$ref": "#/$defs/partialWhen"}, "batchOn": {"type": "array", "items": {"type": "string"}} } } diff --git a/tools/migrate-apim/tests/test_apply.py b/tools/migrate-apim/tests/test_apply.py index 96fc9f1..28da003 100644 --- a/tools/migrate-apim/tests/test_apply.py +++ b/tools/migrate-apim/tests/test_apply.py @@ -552,6 +552,160 @@ def test_a_gap_blocks_before_anything_is_written(self, settled, inventory): assert gap["blocking"] is True +class TestIncompleteObjectsStopPreflight: + """`plan` marks an object it cannot fully configure and records why. + + Nothing read that back, so a condition the planner had already detected and + explained surfaced as a step failing mid-phase, under whatever cause the + builder guessed at. + """ + + REASON = ("No model registration in this plan falls under api 'aoai-api', so there is " + "nothing to carry the policy.") + + def _stranded(self, settled, *, reason=REASON): + """A policy object the plan says it cannot build, as `plan` shapes one.""" + broken = copy.deepcopy(settled) + item = next(i for i in broken["items"] if i.get("capabilityId") == "model.policy") + obj = item["target"]["objects"][0] + obj.pop("modelName", None) + obj.pop("providerName", None) + obj["configurationComplete"] = False + obj.pop("incompleteReason", None) + if reason: + obj["incompleteReason"] = reason + return broken, item + + def test_an_object_the_plan_cannot_build_blocks_before_the_first_write( + self, settled, inventory + ): + broken, _ = self._stranded(settled) + findings = preflight(broken, inventory, runner=_Runner()) + incomplete = next(f for f in findings if f["code"] == "incompleteObject") + assert incomplete["blocking"] is True + + def test_the_finding_carries_the_planners_reason_and_where_to_read_the_source( + self, settled, inventory + ): + broken, item = self._stranded(settled) + policy_id = item["target"]["objects"][0]["fullConfigurationAt"]["policyId"] + [incomplete] = [f for f in preflight(broken, inventory) if f["code"] == "incompleteObject"] + assert self.REASON in incomplete["message"] + assert item["id"] in incomplete["message"] + assert policy_id in incomplete["message"] + + def test_an_object_marked_incomplete_without_a_reason_says_so_rather_than_quoting_nothing( + self, settled, inventory + ): + broken, item = self._stranded(settled, reason=None) + [incomplete] = [f for f in preflight(broken, inventory) if f["code"] == "incompleteObject"] + assert item["id"] in incomplete["message"] + assert "said no more" in incomplete["message"] + + def test_objects_that_carry_no_completeness_flag_are_not_called_incomplete( + self, settled, inventory + ): + # Most objects never carry the field. Reading its absence as a false + # would raise the finding on every plan, which is the same as never. + codes = {f["code"] for f in preflight(settled, inventory)} + assert "incompleteObject" not in codes + + def test_only_the_first_few_are_listed_and_the_rest_are_counted(self, settled, inventory): + broken = copy.deepcopy(settled) + for item in broken["items"]: + for obj in (item.get("target") or {}).get("objects") or []: + obj["configurationComplete"] = False + [incomplete] = [f for f in preflight(broken, inventory) if f["code"] == "incompleteObject"] + assert incomplete["message"].endswith("...") + + def test_an_incomplete_object_no_phase_will_run_does_not_stop_the_migration( + self, settled, inventory, no_builder_for + ): + # The item becomes no step, so nothing it holds can fail mid-phase, and + # blocking here would stop a whole migration over work the operator was + # going to apply by hand anyway. notExecutable already names it. + broken, _ = self._stranded(settled) + no_builder_for("model.policy") + codes = {f["code"] for f in preflight(broken, inventory)} + assert "incompleteObject" not in codes + assert "notExecutable" in codes + + def test_the_fields_apply_blocks_on_are_declared_in_the_plan_schema(self, settled): + # A cross-module contract nothing declared is how these three came to be + # written by the planner and read by nobody for as long as they were. + from migrate_apim.cli import validate_document + + broken, item = self._stranded(settled) + assert validate_document(broken) == [] + item["target"]["objects"][0]["configurationComplete"] = "no" + assert validate_document(broken) != [] + + def test_left_to_run_such_an_object_fails_its_step_under_a_borrowed_reason( + self, settled, inventory + ): + # What preflight now gets in front of. The step fails -- and a failed + # step stops its phase -- reporting whatever the builder could work out, + # while the explanation sits unread on the object a few lines away. + broken, _ = self._stranded(settled) + executor = Executor(broken, inventory, new_journal(broken), runner=_Runner(), + execute=True, skip_rehearsal=True) + step = next(s for s in steps_for(broken) if s.capability_id == "model.policy") + executor.run_step(step) + assert step.status == "failed" + assert self.REASON not in step.detail + + +class TestAFragmentReachesPreflightRatherThanAPhase: + """The planner's fragment handling and preflight's check, as one chain. + + A policy factored into a fragment used to reach no model registration, and + the object that stranded on it stopped the policies phase. The planner now + follows the fragment to the APIs that include it, and what is left over -- + a fragment nothing includes, which governs nothing -- stops at preflight + carrying the reason the planner wrote. Each half is tested where it lives; + this is the join, which is the behaviour the two were for. + """ + + def _planned(self, inventory, *, included_by): + document = copy.deepcopy(inventory) + document["policies"].append({ + "id": "policy:fragment/shared-limits", "stats": {}, "fragmentRefs": [], + "scope": {"level": "fragment", "name": "shared-limits"}, + "elements": [{ + "section": "fragment", "path": "fragment/llm-token-limit[1]", + "name": "llm-token-limit", "family": "tokenLimit", "depth": 1, + "attributes": {"tokens-per-minute": "7000", "counter-key": "shared"}, + "text": None, "expressionIds": [], "namedValueRefs": [], "childNames": [], + }], + }) + if included_by: + including = next(d for d in document["policies"] if d["id"] == included_by) + including["fragmentRefs"] = [*(including.get("fragmentRefs") or []), "shared-limits"] + plan = build_plan(document, CapabilityMap.load(), target=TARGET) + for item in plan["items"]: + if item["status"] == "needs-decision": + item["status"] = "adapted" + item["decision"] = item["proposal"] or "accept" + return plan, document + + def test_a_fragment_an_api_includes_raises_nothing_to_block_on(self, inventory): + plan, document = self._planned(inventory, included_by="policy:api/aoai-api") + codes = {f["code"] for f in preflight(plan, document) if f["blocking"]} + assert "incompleteObject" not in codes + + def test_a_fragment_nothing_includes_stops_before_the_first_write(self, inventory): + # Unreferenced, it governs nothing, so the object is honestly stranded. + # What changed is where that is discovered: preflight, in the planner's + # words, rather than a step failing with every policy behind it skipped. + plan, document = self._planned(inventory, included_by=None) + [incomplete] = [ + f for f in preflight(plan, document) if f["code"] == "incompleteObject" + ] + assert incomplete["blocking"] is True + assert "fragment 'shared-limits'" in incomplete["message"] + assert "policy:fragment/shared-limits" in incomplete["message"] + + class TestAliasVerification: """An alias read back reports a different shape from the one it was sent.""" diff --git a/tools/migrate-apim/tests/test_capability.py b/tools/migrate-apim/tests/test_capability.py index 1f89dfa..88fdab6 100644 --- a/tools/migrate-apim/tests/test_capability.py +++ b/tools/migrate-apim/tests/test_capability.py @@ -129,8 +129,26 @@ def test_expression_is_the_fallback_for_unmapped_elements(self, caps): assert rule["verdict"] == "gap" assert (rule.get("match") or {}).get("hasExpression") is True - def test_an_unmapped_element_without_an_expression_matches_nothing(self, caps): - assert caps.policy_rule("some-future-policy", family="other") is None + def test_an_unmapped_element_without_an_expression_reaches_the_terminal_rule(self, caps): + # It used to match nothing, and `plan._policies` dropped what matched + # nothing where it stood. An element carrying an expression at least + # reached the pass above, so the policies that vanished most completely + # were the simplest ones. + rule = caps.policy_rule("some-future-policy", family="other") + assert rule is caps.terminal_policy_rule() + assert rule["verdict"] == "gap" + + def test_only_an_element_outside_the_vocabulary_reaches_the_terminal_rule(self, caps): + # The terminal rule is the net, not the mapping. `compile` holds every + # element in policy.FAMILIES to a rule of its own or a family rule. + from migrate_apim.policy import FAMILIES, family_of + + terminal = caps.terminal_policy_rule() + reached = sorted( + name for name in FAMILIES + if caps.policy_rule(name, family=family_of(name)) is terminal + ) + assert reached == [] def test_plumbing_is_explicitly_not_applicable(self, caps): for name in ("base", "include-fragment", "choose", "set-variable", "cors"): diff --git a/tools/migrate-apim/tests/test_compile.py b/tools/migrate-apim/tests/test_compile.py index 423057d..790b4f1 100644 --- a/tools/migrate-apim/tests/test_compile.py +++ b/tools/migrate-apim/tests/test_compile.py @@ -454,6 +454,51 @@ def test_a_selector_rejects_a_value_it_never_compares_against(self): assert not site.governs({"match": {"family": "models"}}) +class TestTheKnownPolicyVocabularyIsAnswered: + """The terminal rule is a net, and a net makes a poor mapping. + + Eleven elements matched no rule at all and were dropped where they were + classified, three of them carrying backend credentials. A terminal rule + stops the dropping, but it would just as happily answer for an element + somebody never got round to -- so what it may answer for is held down here. + """ + + def _without(self, rules, **match): + trimmed = copy.deepcopy(rules) + trimmed["policies"] = [ + rule for rule in trimmed["policies"] + if not all((rule.get("match") or {}).get(k) == v for k, v in match.items()) + ] + return trimmed + + def test_an_element_left_to_the_terminal_rule_is_an_error(self, compile_with, shipped): + _, rules, _ = shipped + result = compile_with(rules=self._without(rules, family="backendAuthentication")) + assert "policy-vocabulary" in codes(result) + [diagnostic] = [d for d in result.errors if d.check == "policy-vocabulary"] + # Named, because "something is unmapped" is not a fix. + assert "authentication-basic" in diagnostic.message + assert "backendAuthentication" in diagnostic.message + + def test_the_shipped_rules_leave_nothing_to_it(self, compile_with): + assert "policy-vocabulary" not in codes(compile_with()) + + def test_losing_the_terminal_rule_is_an_error_of_its_own(self, compile_with, shipped): + # Without it an unknown element resolves to nothing, and resolving to + # nothing is what the planner used to drop in silence. + _, rules, _ = shipped + result = compile_with(rules=self._without(rules, anyElement=True)) + assert "policy-terminal" in codes(result) + + def test_a_second_terminal_rule_is_a_tie_and_rejected(self, compile_with, shipped): + # Selection is first-match, so two of them means one is never reached. + _, rules, _ = shipped + doubled = copy.deepcopy(rules) + terminal = next(r for r in doubled["policies"] if (r.get("match") or {}).get("anyElement")) + doubled["policies"].append(copy.deepcopy(terminal)) + assert "rule-unique" in codes(compile_with(rules=doubled)) + + class TestACleanRunSaysWhatItChecked: """A compile that only speaks up on failure leaves the reader of a green run unable to tell thorough from vacuous -- and this contract shipped a @@ -462,7 +507,7 @@ class TestACleanRunSaysWhatItChecked: def test_every_check_reports_what_it_covered(self, compile_with, shipped): documented = shipped[0]["cliEvidence"]["documentedVersion"] covered = compile_with(newest_release=f"cli-v{documented}").covered - assert len(covered) == 5, covered + assert len(covered) == 6, covered def test_the_counts_are_the_real_ones(self, compile_with, shipped): catalog, rules, surface = shipped diff --git a/tools/migrate-apim/tests/test_plan.py b/tools/migrate-apim/tests/test_plan.py index 8533d2e..e25769e 100644 --- a/tools/migrate-apim/tests/test_plan.py +++ b/tools/migrate-apim/tests/test_plan.py @@ -236,6 +236,104 @@ def test_plumbing_raises_nothing(self, plan): assert not elements & {"base", "include-fragment", "choose", "when", "set-variable"} +def _routed(inventory, *, path, attributes, expressions=(), name="set-backend-service", + api="aoai-api"): + """A copy of the inventory with one routing element added to an API policy.""" + changed = copy.deepcopy(inventory) + document = next(d for d in changed["policies"] + if (d.get("scope") or {}).get("level") == "api" + and (d.get("scope") or {}).get("name") == api) + document["elements"].append({ + "section": "inbound", "path": path, "name": name, + "family": "routing", "depth": path.count("/") + 1, "attributes": attributes, + "text": None, "expressionIds": list(expressions), "namedValueRefs": [], "childNames": [], + }) + return changed + + +def _routing_items(plan): + return [i for i in plan["items"] + if i["source"]["assetKind"] == "policy" + and i["evidence"]["element"] in {"set-backend-service", "forward-request"}] + + +class TestRoutingLogicLeavesATrace: + """`handled` claims another object carries the element, and for a backend + picked by an expression or inside a `choose` no object does: the alias pool + is built from the APIM backend pool, which holds none of that logic. The + element was excluded before anything looked at it, so a routing decision + the target cannot make produced no item, no gap and no mention.""" + + def test_a_plain_backend_selection_is_still_carried_by_the_alias( + self, inventory, capabilities): + routed = _routed(inventory, path="inbound/set-backend-service[1]", + attributes={"backend-id": "aoai-pool"}) + assert _routing_items(build_plan(routed, capabilities)) == [] + + def test_a_backend_chosen_by_an_expression_becomes_a_decision( + self, inventory, capabilities): + routed = _routed(inventory, path="inbound/set-backend-service[1]", + attributes={"backend-id": "@(context.Variables[\"region\"])"}, + expressions=["expr:route"]) + [item] = _routing_items(build_plan(routed, capabilities)) + assert item["verdict"] == "partial" + assert item["kind"] == "accept-loss" + assert item["status"] == "needs-decision" + assert item["proposal"] == "compensate" + assert {o["value"] for o in item["options"]} == { + "accept", "compensate", "retain-in-apim" + } + + def test_a_backend_chosen_inside_a_choose_becomes_a_decision( + self, inventory, capabilities): + # No expression on the element itself: the branch that selects it is + # the whole of what will not travel. + routed = _routed(inventory, path="inbound/choose[1]/when[1]/set-backend-service[1]", + attributes={"backend-id": "aoai-east"}) + [item] = _routing_items(build_plan(routed, capabilities)) + assert item["verdict"] == "partial" + assert item["evidence"]["hasExpression"] is False + assert item["evidence"]["members"][0]["path"].startswith("inbound/choose[1]/when[1]/") + + def test_the_decision_says_what_became_of_the_routing(self, inventory, capabilities): + routed = _routed(inventory, path="inbound/choose[1]/otherwise[1]/set-backend-service[1]", + attributes={"backend-id": "aoai-west"}) + plan = build_plan(routed, capabilities) + [item] = _routing_items(plan) + # The alias is what carries the part that does travel, so the item + # points at it rather than reading as a loss with no destination. + assert item["capabilityId"] == "alias.create" + assert "alias backend pool" in item["rationale"] + assert all(option["consequence"] for option in item["options"]) + assert validate_document(plan) == [] + + def test_it_asks_once_however_many_occurrences(self, inventory, capabilities): + routed = _routed(inventory, path="inbound/choose[1]/when[1]/set-backend-service[1]", + attributes={"backend-id": "aoai-east"}) + routed = _routed(routed, api="support-mcp", + path="inbound/choose[1]/when[1]/set-backend-service[1]", + attributes={"backend-id": "support-backend"}) + [item] = _routing_items(build_plan(routed, capabilities)) + assert {m["policyId"] for m in item["evidence"]["members"]} == { + "policy:api/aoai-api", "policy:api/support-mcp" + } + + def test_a_sound_handled_element_is_left_alone(self, inventory, capabilities): + # authentication-managed-identity really is carried by the provider + # item, expression or no expression. Raising it here would put an item + # in every plan for something nothing was lost to. + routed = _routed( + inventory, name="authentication-managed-identity", + path="inbound/choose[1]/when[1]/authentication-managed-identity[1]", + attributes={"resource": "@(context.Variables[\"scope\"])"}, + expressions=["expr:scope"], + ) + plan = build_plan(routed, capabilities) + assert not [i for i in plan["items"] + if i["source"]["assetKind"] == "policy" + and i["evidence"]["element"] == "authentication-managed-identity"] + + class TestStructural: def test_a_non_default_workspace_must_be_collapsed(self, plan): [item] = items_of(plan, "collapse") @@ -1085,6 +1183,181 @@ def test_a_scope_with_no_models_says_so_instead_of_vanishing(self, inventory, ca assert "nothing to carry" in stranded[0]["incompleteReason"] +class TestEveryPolicyElementIsAccountedFor: + """A plan that classified an element and discarded it used to be + indistinguishable from one that had never seen it. + + Three dispositions end the walk without raising an item, and a fourth -- + matching no rule at all -- ended it without raising anything anywhere. The + tally is what turns "nothing was dropped" from a claim into a sum. + """ + + def _unknown(self, inventory, name="quantum-shield", family="other"): + document = copy.deepcopy(inventory) + target = next(d for d in document["policies"] if d["id"] == "policy:api/aoai-api") + target["elements"].append({ + "section": "inbound", "path": f"inbound/{name}[1]", "name": name, + "family": family, "depth": 1, "attributes": {"mode": "strict"}, + "text": None, "expressionIds": [], "namedValueRefs": [], "childNames": [], + }) + return document + + def test_the_dispositions_sum_to_the_elements_the_inventory_holds( + self, inventory, capabilities + ): + coverage = build_plan(copy.deepcopy(inventory), capabilities)["policyCoverage"] + walked = sum(len(d.get("elements") or []) for d in inventory["policies"]) + assert coverage["walked"] == walked + # Exclusive buckets, or the sum says nothing. + assert sum(coverage["dispositions"].values()) == walked + + def test_an_element_outside_the_vocabulary_is_named_and_located( + self, inventory, capabilities + ): + document = self._unknown(inventory) + coverage = build_plan(document, capabilities)["policyCoverage"] + assert coverage["dispositions"]["unrecognised"] == 1 + [found] = coverage["unrecognised"] + assert found["element"] == "quantum-shield" + assert found["policyId"] == "policy:api/aoai-api" + assert found["path"] == "inbound/quantum-shield[1]" + + def test_it_reaches_the_plan_as_a_decision_rather_than_a_silence( + self, inventory, capabilities + ): + document = self._unknown(inventory) + plan = build_plan(document, capabilities) + item = next( + i for i in plan["items"] if (i.get("source") or {}).get("aspect") == "quantum-shield" + ) + assert item["verdict"] == "gap" + assert item["blocking"] is True + assert item["status"] == "needs-decision" + + def test_a_recognised_element_is_not_counted_as_unrecognised(self, inventory, capabilities): + # The net must not quietly become the mapping: `set-header` has a family + # rule, so it is classified even though it raises an accept-loss item. + document = self._unknown(inventory, name="set-header", family="transform") + coverage = build_plan(document, capabilities)["policyCoverage"] + assert coverage["unrecognised"] == [] + + +def policy_document(inventory, policy_id, scope, *, includes=None): + """The inventory's document for one policy scope, created if the fixture has none.""" + document = next((d for d in inventory["policies"] if d["id"] == policy_id), None) + if document is None: + document = {"id": policy_id, "scope": scope, "elements": [], "fragmentRefs": [], "stats": {}} + inventory["policies"].append(document) + if includes: + document["fragmentRefs"] = [*(document.get("fragmentRefs") or []), includes] + return document + + +def token_limit_in_fragment(inventory, name, *, counter_key="shared-counter"): + """A fragment carrying one `llm-token-limit`, the way an operator shares one.""" + document = policy_document( + inventory, f"policy:fragment/{name}", {"level": "fragment", "name": name} + ) + document["elements"].append({ + "section": "fragment", "path": "fragment/llm-token-limit[1]", + "name": "llm-token-limit", "family": "tokenLimit", "depth": 1, + "attributes": {"tokens-per-minute": "7000", "counter-key": counter_key}, + "text": None, "expressionIds": [], "namedValueRefs": [], "childNames": [], + }) + return document + + +def fragment_item(plan, name="shared-limits"): + return next( + item for item in plan["items"] + if item.get("capabilityId") == "model.policy" + and any(member["policyId"] == f"policy:fragment/{name}" + for member in item["evidence"]["members"]) + ) + + +class TestAPolicyFactoredIntoAFragment: + """A fragment is how APIM shares one policy across several APIs. + + The inventory holds a fragment as a document scoped to itself, and resolved + as a scope of its own it covers no API. So a token limit an operator had + factored out reached no model registration: the item read as ready, carried + an object with no model on it, and the first such object stopped the + policies phase for every policy behind it. + """ + + def test_its_limit_lands_on_the_models_of_every_including_api(self, inventory, capabilities): + document = copy.deepcopy(inventory) + token_limit_in_fragment(document, "shared-limits") + policy_document(document, "policy:api/aoai-api", {"level": "api", "name": "aoai-api"}, + includes="shared-limits") + policy_document(document, "policy:api/finance/model-api", + {"level": "api", "name": "model-api"}, includes="shared-limits") + item = fragment_item(build_plan(document, capabilities)) + assert item["status"] == "ready", "a literal counter key maps natively" + objects = item["target"]["objects"] + # Both including APIs, and every registration each of them makes: the + # pooled `gpt-5` is registered once per account and limited on both. + assert {obj["modelName"] for obj in objects} == { + "gpt-5", "text-embedding-3-large", "claude-opus-5"} + assert len(objects) == 4, objects + assert all(obj["configurationComplete"] for obj in objects), objects + + def test_a_fragment_included_through_another_fragment_still_reaches_the_api( + self, inventory, capabilities + ): + document = copy.deepcopy(inventory) + token_limit_in_fragment(document, "shared-limits") + policy_document(document, "policy:fragment/outer", {"level": "fragment", "name": "outer"}, + includes="shared-limits") + policy_document(document, "policy:api/aoai-api", {"level": "api", "name": "aoai-api"}, + includes="outer") + objects = fragment_item(build_plan(document, capabilities))["target"]["objects"] + assert {obj["modelName"] for obj in objects} == {"gpt-5", "text-embedding-3-large"} + + def test_a_cycle_of_includes_terminates(self, inventory, capabilities): + """APIM accepts a fragment including one that includes it back, and stores it.""" + document = copy.deepcopy(inventory) + token_limit_in_fragment(document, "shared-limits") + policy_document(document, "policy:fragment/shared-limits", + {"level": "fragment", "name": "shared-limits"}, includes="outer") + policy_document(document, "policy:fragment/outer", {"level": "fragment", "name": "outer"}, + includes="shared-limits") + policy_document(document, "policy:api/aoai-api", {"level": "api", "name": "aoai-api"}, + includes="outer") + objects = fragment_item(build_plan(document, capabilities))["target"]["objects"] + assert {obj["modelName"] for obj in objects} == {"gpt-5", "text-embedding-3-large"} + + def test_a_fragment_the_global_policy_includes_covers_every_model( + self, inventory, capabilities + ): + """Narrowing the union would still produce objects, so the count is the check.""" + document = copy.deepcopy(inventory) + token_limit_in_fragment(document, "shared-limits") + policy_document(document, "policy:global", {"level": "global", "name": None}, + includes="shared-limits") + built = build_plan(document, capabilities) + registered = { + (obj["name"], obj["providerName"]) for item in built["items"] + if item.get("capabilityId") == "model.create" + for obj in (item.get("target") or {}).get("objects") or [] + } + covered = {(obj["modelName"], obj["providerName"]) + for obj in fragment_item(built)["target"]["objects"]} + assert covered == registered + + def test_a_fragment_nothing_includes_still_says_what_became_of_its_policy( + self, inventory, capabilities + ): + """Unreferenced, it governs nothing, and that is the genuinely empty scope.""" + document = copy.deepcopy(inventory) + token_limit_in_fragment(document, "shared-limits") + objects = fragment_item(build_plan(document, capabilities))["target"]["objects"] + assert [obj.get("modelName") for obj in objects] == [None] + assert objects[0]["configurationComplete"] is False + assert "nothing to carry" in objects[0]["incompleteReason"] + + class TestTheGatewayWideBaseline: """One baseline, however many source policies.