instancetype: UpdateInstanceTypes drops known SKUs on a partial resourceSKUs response, causing spurious NodeClaim drift

[Harshithraj24]

Description

DefaultProvider.UpdateInstanceTypes (pkg/providers/instancetype/instancetypes.go) refreshes the in-memory instance-type catalog on a timer (InstanceTypesRefreshInterval = 12h) by doing a live Azure fetch and then unconditionally replacing the catalog with the result. The only safeguard is a check for a totally empty result:

skus := cache.List(ctx, skewer.ResourceTypeFilter("virtualMachines"))
for i := range skus {
    ...
    if !skus[i].HasLocationRestriction(p.region) && p.isSupported(&skus[i], vmsize) {
        instanceTypes[skus[i].GetName()] = &skus[i]
    }
}
if len(instanceTypes) == 0 {          // ~L463: only guard
    return fmt.Errorf("no instance types found")
}
...
p.instanceTypesInfo = instanceTypes    // ~L474: full replace, no regression guard

The Azure resourceSKUs API (and the per-SKU HasLocationRestriction / capability metadata) is eventually consistent — a single refresh can return an HTTP 200 with a valid but incomplete SKU set, or transiently flag a family/zone as restricted. When that happens, the affected SKUs silently disappear from instanceTypesInfo. List() and Get() then stop offering them, so every NodeClaim on those SKUs is marked Drifted with InstanceTypeNotFound, even though the SKU is compiled-in, generally available, and unrestricted. Because the refresh is on a 12h timer, the catalog stays wrong — and those NodeClaims keep drifting/churning — until the next complete refresh, up to 12h later. It then self-heals with no restart.

This is distinct from the UnavailableOfferings path: that correctly marks an offering unavailable for a TTL. Here the instance type is dropped from the catalog entirely, which is what triggers drift.

Reproduction

1. Let the provider populate its catalog from a complete fetch (contains, say, Standard_E32pds_v6).
2. On the next UpdateInstanceTypes, have the resource client return a response that omits that SKU (simulating an eventually-consistent/partial resourceSKUs response, or a transient HasLocationRestriction on that family).
3. Get("Standard_E32pds_v6") now returns not-found and List() no longer offers it → existing NodeClaims of that type are flagged Drifted: InstanceTypeNotFound.
4. The catalog stays in this state until a subsequent complete refresh (up to InstanceTypesRefreshInterval later).

Impact

• Spurious Drifted/InstanceTypeNotFound on healthy NodeClaims → unnecessary node replacement / churn.
• Long recovery window (up to 12h) driven by the refresh interval, not by how long Azure's response was actually incomplete.
• The AWS provider does not have this class of issue because its DescribeInstanceTypes pagination returns an error on a failed page (return fmt.Errorf(...)), so a failure preserves the previous catalog. Azure's failure mode here is a successful-but-incomplete response, which produces no error to key off — so the empty-only guard doesn't catch it.

Proposed fix

Make the catalog update regression-resistant instead of trusting any non-empty fetch:

1. Gross-truncation guard — reject a fetch that returns far fewer SKUs than the previous catalog (e.g. < prev/2); keep last-known-good and return an error so the controller retries on backoff rather than pinning a truncated catalog for the full refresh interval.
2. Per-SKU absence hysteresis — do not drop a previously-known SKU on a single refresh. Track consecutive absence per SKU and only remove it after it has been absent for N consecutive refreshes (e.g. 2). A transient miss is retained (no drift); a genuinely retired SKU still ages out.

This keeps additions immediate (fast-positive) while making removals require confirmation (slow-negative), which is the appropriate reconciliation model for an eventually-consistent source. It has no steady-state behavior change when fetches are complete.

Happy to open a PR with this + unit and integration tests (the integration test drives UpdateInstanceTypes against a fake resource client that omits a SKU on one call and asserts it is retained, then removed only after two consecutive omissions). Let me know if the maintainers prefer a different shape (e.g. threshold as an option, or routing transient unavailability through UnavailableOfferings instead of dropping from the catalog).

[Harshithraj24]

Update on the proposed fix — I've prototyped and validated a defense-in-depth version that I'm happy to upstream:

1. In-reconcile retry (fetch-level). resourceSKUs is eventually consistent, so a fetch can return HTTP 200 with a subset (no error to key off). When a fetch comes back smaller than the last good catalog, UpdateInstanceTypes re-fetches a small bounded number of times (2, 300ms backoff) within the same reconcile and keeps the most complete result before committing — riding out a transient partial in the current cycle rather than waiting for the 12h refresh. Happy path is unchanged: a fetch at or above the previous size commits with zero retries and zero added latency.

2. Gross-truncation guard. Reject a fetch that returns far fewer SKUs than the previous catalog (keep last-known-good, return an error so the controller retries on backoff). Calibrated against the previous catalog size, not the static requested SKU set — a real region's filtered live count is much smaller than the requested universe, so sizing off the latter would reject every healthy fetch.

3. Per-SKU absence hysteresis. Don't drop a previously-known SKU on a single refresh; only remove after N consecutive absent refreshes.

4. In-use pin. Never remove a SKU that is still depended on by running capacity, regardless of absence count — so even a sustained multi-refresh regression can't drift a live node. (In our fork this reads a hibernation-pool availability checker; upstream this could key off NodeClaims referencing the instance type. Removals then only apply to unused SKUs, where the offerings/UnavailableOfferings layer backstops any launch attempt.)

Tests: unit for the merge/hysteresis + pin, and an integration test driving the real UpdateInstanceTypes against a fake resource client that (a) omits a SKU on one call and asserts the retry recovers it, and (b) omits it persistently and asserts it's retained-then-removed / pinned.

If the maintainers are open to it I'll open a PR. Happy to adjust the shape — e.g. thresholds/retry counts as options, or routing transient unavailability through UnavailableOfferings instead of dropping from the catalog. Let me know your preference on (4) in particular (pool-checker vs NodeClaim-reference vs offerings-based).

[tallaxes]

The Azure resourceSKUs API (and the per-SKU HasLocationRestriction / capability metadata) is eventually consistent — a single refresh can return an HTTP 200 with a valid but incomplete SKU set, or transiently flag a family/zone as restricted.

Harshith (@Harshithraj24) if you have any evidence for this claim, could you please share it? Thanks!

[heoelri]

We observed two managed AKS Node Auto Provisioning incidents that appear related to this issue's transient catalogue/removal scenario. We cannot prove that the underlying resourceSKUs response was partial because the managed controller does not expose the raw response, but the resulting drift/self-heal behavior is similar.

Environment details (identifiers omitted):

• AKS Kubernetes: 1.36.3
• Managed Karpenter/NAP: 1.14.0-aks (from the controller's discovered karpenter version log)
• Previous managed version: 1.13.1-aks; the four-cluster dev fleet moved to 1.14.0-aks on 2026-08-04/05
• Azure provider catalogue logs consistently reported skuCount=871 and instanceTypeCount=486
• NodePool requirements: amd64, linux, on-demand, SKU family D|E|F, CPU Gt 4
• No customer NodeOverlay resources are defined

Incident 1 — 2026-08-31 UTC

12:50:56  discovered SKUs: 871; discovered instance types: 486
12:51:01  updated instance types
13:07:07  two healthy NodeClaims marked RequirementsDrifted
13:07:14  NodePool begins reporting NoCompatibleInstanceTypes
13:37:54  managed Karpenter controller restarts
13:37:59  catalogue rediscovered: 871 SKUs / 486 instance types
13:38:04  updated instance types; NoCompatibleInstanceTypes continues
13:51:19  last NoCompatibleInstanceTypes occurrence
13:52:14  drift condition removed as "not drifted"

The condition lasted about 45 minutes and continued for 14 minutes after the restart. No configuration changed and no node was deleted.

Incident 2 — 2026-09-01 UTC

12:18:53  healthy NodeClaims marked RequirementsDrifted; first NoCompatibleInstanceTypes
12:22:03  one workload node tainted for disruption
12:22:05  provisioner logs "nodepool requirements filtered out all instance types"
12:22:56  VM deletion begins
12:23:12  node and NodeClaim deleted
12:22:05–12:38:24  three workload pods repeatedly cannot be scheduled
12:38:26  drift condition removed as "not drifted"; no restart/config change
12:38:34  new NodeClaim computed
12:38:35  Standard_D8als_v6 selected
12:38:40  replacement launched
12:39:29  replacement initialized

The same NodeClaim that survived the incident had a karpenter.sh/nodepool-hash matching the NodePool and labels satisfying every requirement (arch=amd64, os=linux, capacity-type=on-demand, sku-family=D, sku-cpu=8). It was nevertheless marked RequirementsDrifted and later changed back to not drifted without any object update.

We also ruled out the normal external causes:

• quota snapshots before, during, and after each window had identical payload hashes;
• no QuotaExceeded, InsufficientCapacity, AllocationFailed, ZonalAllocationFailed, or SkuNotAvailable occurred;
• no Azure VM creation was attempted while the pool reported zero compatible types;
• az vm list-skus showed 265 unrestricted x64 D/E/F SKUs with more than four vCPUs in the region;
• Standard_D8als_v6 was selected immediately after recovery with unchanged configuration.

The concerning part is the interaction with disruption: in incident 2, the provisioner was already reporting zero compatible instance types at 12:22:05, but the disruption path still deleted the existing node at 12:23:12.

Could the maintainers advise whether a partial resourceSKUs refresh can leave the total logged catalogue counts unchanged while dropping or restricting one or more previously used SKUs? It would also be useful to know whether the managed 1.14.0-aks build can temporarily expose an empty resolved instance-type set through the NodeOverlay InstanceTypeStore without returning UnevaluatedNodePoolError.

We can provide sanitized controller/event exports and object YAML through an appropriate private AKS support channel if useful. No subscription or resource identifiers are included here.

[kave-h]

Another managed-NAP data point matching this issue's shape. We can't see the raw resourceSKUs response either, but we ran an active allocation probe that rules out Azure capacity as the cause.

Environment: AKS 1.35.4, managed NAP (nodeProvisioningMode: Auto), germanywestcentral, on-demand only. NodePool lists D{4,8,16}ads_v5, D{4,8,16}ads_v6, D{4,8,16}ds_v5 (+ E equivalents). NAP controller version not visible to us.

Symptom: since 2026-09-02, D8ads_v5 — and later D16ads_v5 — are never selected, while D4ads_v5 continues to be selected normally. Launches succeed on the next-priced SKU on the first attempt with no Warning event.

Node-hours per rung (kube-state-metrics):

day      D4ads_v5  D8ads_v5  D16ads_v5
08-30          96        96         24
09-01         120       145         17
09-02         150         0         24
09-03         182        11         43
09-04          41         0          0

2026-09-04 UTC — three refusals bracketed by successful allocations of the same SKU:

09:40:00  capacity reservations 6/6 Succeeded (D4ads_v5 + D8ads_v5 x zones 1,2,3)
12:48:32  NodeClaim created -> 12:48:40 Launched  Standard_D8ads_v6  zone 1
12:51:37  capacity reservation Standard_D8ads_v5 zone 1: Succeeded
13:13:59  NodeClaim -> Standard_D8ds_v5
14:00:50  NodeClaim -> Standard_D4ads_v5   (control: this rung works throughout)
14:14:57  NodeClaim -> Standard_D16ds_v5

Every launch went DisruptionLaunching -> Launched=True in 8-12s, first attempt, no InsufficientCapacityError. Prices at the time of each choice: D8ads_v5 $0.500 vs D8ds_v5 $0.544 / D8ads_v6 $0.552; D16ads_v5 $1.000 vs D16ds_v5 $1.088 — the cheaper SKU was the one not selected.

Ruled out:

• SKU present in the NodePool's node.kubernetes.io/instance-type requirement throughout.
• Quota: Standard DADSv5 Family vCPUs 32/350.
• Policy: Allowed vm SKUs is DoNotEnforce at our management group.
• az vm list-skus -l germanywestcentral reports no restrictions on these SKUs (checked 09-03 10:00Z and 09-04 08:39Z).
• Pricing: v5 is below v6 in both germanywestcentral ($0.500 / $0.552) and eastus ($0.412 / $0.456), so no pricing source ranks v6 first.

Differences from the incidents reported above: ours is per-SKU rather than pool-wide (NoCompatibleInstanceTypes never occurred — Intel and v6 stayed selectable), and it has persisted ~2.5 days for D8ads_v5, well beyond the 12h InstanceTypesRefreshInterval. D4ads_v5 did recover on its own at 07:50Z after being unselected the previous evening, which matches the self-heal-on-complete-refresh behaviour described here.

Happy to provide sanitized event exports through a private channel.