Skip to content

fix(nebius): prevent orphaned resources after failed provisioning and termination - #162

Draft
mmoulikk wants to merge 2 commits into
mainfrom
BREV-10538/nebius-orphan-network-fix
Draft

mmoulikk wants to merge 2 commits into
mainfrom
BREV-10538/nebius-orphan-network-fix

Conversation

@mmoulikk

@mmoulikk mmoulikk commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Issue

Nebius creates a dedicated network, subnet and disk for every VM.

When provisioning failed after Nebius had already created the VM, cleanup could miss the VM ID. The VM remained attached to the subnet, so subnet deletion failed and the network was left behind.

Normal termination could also leave resources behind when the VM deletion wait consumed most of the caller's timeout.

These orphaned resources continue consuming Nebius quotas, especially VPC network and subnet pool quotas, and can eventually block new instance provisioning.

Root cause

  • CreateInstance saved the VM ID only after operation.Wait() succeeded.
  • Network, subnet and disk IDs could be discarded when their async operations failed.
  • Cleanup could continue without first confirming that the VM was gone.
  • Subnet, network and disk cleanup could inherit an expired or nearly expired caller context.
  • One slow cleanup operation could consume the time needed by the remaining resources.

Changes

  • Preserve VM, network, subnet and disk IDs directly from their operation objects, including partial-failure paths.
  • Run failed-create cleanup on a detached, bounded 3-minute context.
  • Confirm the VM is gone before deleting dependent resources.
  • If normal termination times out, retry the exact VM ID using the detached cleanup context.
  • Delete resources in dependency order: VM, subnet, network, then disk.
  • Give VM, subnet, network and disk cleanup separate time limits.
  • Retry only temporary Nebius delete errors: FailedPrecondition and Unavailable.
  • Continue attempting network cleanup when subnet deletion reports an error.
  • Keep the original provisioning error visible and log cleanup errors separately.

This is bounded best-effort cleanup. It does not reclaim resources orphaned before this change.

Verification

dev1 validation

Deployed cloud 0e9dc95 through dev-plane c3c23940f (v0-brev-10538-2).

Tested 25 H200, L40S and H100 environments:

  • 21 failed creates entered cleanup.
  • 4 successful VMs completed normal UI deletion.
  • No subnet or network cleanup errors.
  • Read-only Nebius CLI checks confirmed all 84 captured resources were deleted.

Four disk waits timed out locally, but all four disks were later confirmed deleted by Nebius.

Datadog evidence:

@mmoulikk mmoulikk self-assigned this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant