Bound asynchronous instance promise polling and retained goroutines

[theunrepentantgeek]

🤖 TheUnrepentantRobot says:

Version

Karpenter Version: current main (352646f599290c3a6d2f9710fb535a7b288f88ed); the affected paths are also present in v1.14.0

Kubernetes Version: N/A (code-level lifecycle issue)

Expected Behavior

Every asynchronous VM or AKS Machine create operation should have a finite lifetime. If Azure never completes a long-running operation, the provider should cancel or time out the poll, release the goroutine and retained request state, report the failure, and run the normal cleanup path.

Repeated stalled creates should therefore converge to a bounded goroutine count and live heap after their provisioning deadline.

Actual Behavior

CloudProvider.handleInstancePromise launches one goroutine for every NodePool-managed create and waits for instancePromise.Wait() without adding a provider-controlled deadline:

• pkg/cloudprovider/cloudprovider.go:264-315
• pkg/providers/instance/vminstance.go:903-941
• pkg/providers/instance/aksmachineinstance.go:552-583

The VM promise and non-batched AKS Machine promise call the Azure SDK's PollUntilDone with only a polling frequency. defaultPollerOptions() in pkg/providers/instance/poller.go:25-31 does not set an operation deadline. The captured reconcile context is normally not canceled merely because Create returns, so an Azure operation that remains in progress indefinitely can retain:

• the promise goroutine and wait-group entry;
• the Azure SDK poller and HTTP pipeline state;
• the NodeClaim and launch template captured by the closure;
• the selected instance type and its scheduling requirements;
• AKS Machine templates and related SDK objects.

The header-batched AKS Machine path is different: MachineCache.PollUntilDone has a 15-minute timeout. VM-based modes and non-batched aksmachineapi do not have an equivalent explicit bound.

This is both a memory-retention and shutdown concern. WaitForInstancePromises() cannot complete while one of these polls is permanently stalled.

Steps to Reproduce the Problem

1. Add a fake VM poller whose PollUntilDone blocks until its context is canceled and never reports completion.
2. Invoke the NodePool-managed create path repeatedly with distinct NodeClaims.
3. Allow each Create call to return after launching its promise waiter.
4. Delete or cancel the originating NodeClaims without canceling the process context.
5. Confirm that waiter goroutines and instancePromiseWg entries remain live indefinitely.
6. Repeat the same test for the non-batched AKS Machine API promise.
7. Force multiple garbage collections and compare runtime.NumGoroutine, heap object count, and live heap before and after the creates.

A production-oriented soak can use an Azure transport/fake that continuously returns an in-progress LRO response. The test should demonstrate that memory and goroutines converge after a configured provisioning timeout.

Suggested remediation:

• create a dedicated context with a finite create/poll timeout for every promise;
• use that context for both SDK polling and follow-up error retrieval;
• ensure timeout/cancellation enters the existing promise error and cleanup path;
• make the bound consistent across aksscriptless, bootstrappingclient, aksmachineapi, and aksmachineapiheaderbatch where practical;
• expose a metric for active instance promise waiters and timeout outcomes.

Acceptance criteria:

• stalled VM and AKS Machine polls terminate after a documented bound;
• no promise goroutine remains after timeout and cleanup;
• shutdown waiting cannot block indefinitely on a stalled Azure LRO;
• tests cover VM-based and both AKS Machine create dispatch strategies.

Resource Specs and Logs

No Kubernetes resource specification is required to trigger the underlying lifecycle issue. Any NodePool-managed NodeClaim that reaches asynchronous instance creation can exercise it.

Relevant code:

pkg/cloudprovider/cloudprovider.go:264-315
pkg/providers/instance/vminstance.go:903-941
pkg/providers/instance/aksmachineinstance.go:489-523
pkg/providers/instance/aksmachineinstance.go:552-583
pkg/providers/instance/poller.go:25-31

No production log uniquely identifies retained pollers today. Adding active-promise and timeout metrics would make this observable.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment