Propagate cancellation through Community Image Gallery pagination

[theunrepentantgeek]

🤖 TheUnrepentantRobot says:

Version

Karpenter Version: current main (352646f599290c3a6d2f9710fb535a7b288f88ed); the affected CIG path is also present in v1.14.0

Kubernetes Version: N/A (Azure API cancellation issue)

Expected Behavior

Cancellation of an image-resolution or NodeClass reconciliation context should cancel all Azure Community Image Gallery page requests promptly. No Azure SDK pagination call should replace the caller's context with context.Background().

Repeated canceled reconciles should release their worker goroutines and temporary image/version objects.

Actual Behavior

The CIG path discards the caller context at multiple levels:

• provider.List passes ctx to listCIG at pkg/providers/imagefamily/nodeimage.go:113-117;
• listCIG declares the context as _ context.Context at line 153;
• getCIGImageID and latestNodeImageVersionCommunity accept no context;
• latestNodeImageVersionCommunity calls pager.NextPage(context.Background()) at line 194.

If an Azure Community Gallery page request stalls, canceling the NodeClass reconcile, shutting down the manager, or timing out a higher-level operation does not interrupt it. The reconcile worker and its object graph remain retained until the Azure call eventually returns. Enough blocked calls can consume the controller's concurrent reconcile workers and increase memory use.

This affects the Community Image Gallery path (USE_SIG=false), which is particularly relevant to self-hosted deployments. The SIG path already passes the caller context to nodeImageVersions.List.

Steps to Reproduce the Problem

1. Extend the Community Gallery image version fake so NextPage blocks until the context passed to it is canceled.
2. Call NodeImageProvider.List with USE_SIG=false and a cancelable context.
3. Wait until NextPage has started, then cancel the caller context.
4. Observe that the current implementation remains blocked because the pager received context.Background().
5. Repeat with multiple concurrent NodeClass reconciles and confirm that workers/goroutines accumulate.
6. Force garbage collection and verify that the retained goroutines keep the associated image and NodeClass state live.

Suggested remediation:

• thread ctx through listCIG, getCIGImageID, and latestNodeImageVersionCommunity;
• pass that context to every pager.NextPage call;
• return cancellation errors with operation and image-definition context;
• add a regression test that fails unless cancellation releases the blocked pager promptly.

Acceptance criteria:

• canceling NodeImageProvider.List cancels an in-flight CIG page request;
• manager shutdown cannot leave CIG pagination goroutines blocked on a background context;
• tests cover cancellation on the first and a later page;
• SIG behavior remains unchanged.

Resource Specs and Logs

Example configuration surface:

USE_SIG=false

The issue is independent of a particular NodePool or pod. It is exercised when resolving a Community Image Gallery image for an AKSNodeClass.

Relevant code:

pkg/providers/imagefamily/nodeimage.go:76-121
pkg/providers/imagefamily/nodeimage.go:153-166
pkg/providers/imagefamily/nodeimage.go:182-204

No production log uniquely identifies the blocked request because cancellation is currently disconnected from the pager.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment