Static NodePools churn NodeClaims when all compatible offerings are unavailable

[matthchr]

Problem

A static NodePool (spec.replicas set) can enter an unbounded NodeClaim create/fail/delete loop when all offerings compatible with its requirements are unavailable.

This happens without pending pods:

1. Core static provisioning sees fewer NodeClaims than the desired replica count.
2. It creates replacement NodeClaims directly from the NodePool template.
3. Azure CloudProvider.Create() filters instance types using cached quota and UnavailableOfferings.
4. With no compatible available offering, launch returns InsufficientCapacityError.
5. Lifecycle deletes the failed NodeClaim.
6. The deletion immediately retriggers static provisioning.

The loop causes Kubernetes API churn, events, and controller log spam. Azure normally rejects these claims from local caches before making a Machine API request, so the primary waste is control-plane churn rather than repeated VM creation calls.

Code-path gap

Dynamic provisioning calls CloudProvider.GetInstanceTypes() and runs scheduling/available-offering filtering before CreateNodeClaims().

Static provisioning in core skips that path:

static.provisioning.Reconcile
  -> NewNodeClaimTemplate
  -> CreateNodeClaims
  -> lifecycle
  -> CloudProvider.Create

NodeClaimTemplate.ToNodeClaim() explicitly leaves instance selection to cloudprovider.Create() for static claims. The provider therefore discovers unavailability only after the Kubernetes NodeClaim exists.

AWS behaves the same way. It mirrors its three-minute insufficient-capacity cache into Offering.Available and rejects subsequent launches locally with CompatibleAvailableFilter, reducing EC2 calls but not preventing static NodeClaim churn.

Possible fixes

The primary fix likely belongs in core Karpenter:

1. Preferred: Before static top-up, call CloudProvider.GetInstanceTypes() and apply the NodePool requirements plus compatible available-offering filtering. Create no NodeClaim when all compatible offerings are unavailable.
2. Extract a provider-neutral preflight helper shared by dynamic and static provisioning so their availability semantics cannot diverge.
3. Add per-NodePool bounded/exponential backoff after insufficient-capacity failures as defense in depth.
4. Coalesce or rate-limit deletion-triggered static reconciles to cap worst-case churn.

Azure-provider-specific backoff could be an interim mitigation, but provider launch filtering alone cannot prevent core from creating the NodeClaim object.

Expected behavior

When all offerings compatible with a static NodePool are temporarily unavailable, Karpenter should create no additional NodeClaims, expose a rate-limited condition/event, and retry after a bounded interval or when offering availability changes.