[paolosalvatori]

Motivation

Every run of the training job in the GPU node pools tutorial waited 5 to 6 minutes before its pod started, though the note under How to run it said that happened only the first time.
The GPU node downloaded pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime (about 3 GB) again on every run, because the cluster autoscaler deleted the idle node, and the image with it, once the previous job was done.
06-simulate-eviction.sh waited twice. For a demo, that is too long.

AKS keeps a node's images through a scale-down when the pool's Scale-down Mode is Deallocate.
The cluster autoscaler then deallocates the idle node instead of deleting it, and the node's OS disk keeps the images its pods pulled.
The next time a pod needs the pool, the autoscaler starts the node again, before it adds any new node.
Spot pools have supported it since it became generally available, and it requires a managed OS disk.

Changes

• scripts/00-variables.sh: adds GPU_NODE_SCALE_DOWN_MODE="Deallocate" and GPU_NODE_OS_DISK_TYPE="Managed", with a comment on what the mode does and why an eviction still deletes the node.
• scripts/01-add-gpu-node-pool.sh:
  • Adds the pool with --scale-down-mode and --node-osdisk-type.
  • An existing pool's OS disk type, which can't change, joins the VM size and priority it must match.
  • An existing pool gets the scale-down mode with az aks nodepool update --scale-down-mode.
  • The printed pool shows osDiskType and scaleDownMode.
• scripts/05-run-training-job.sh and scripts/06-simulate-eviction.sh: their comments and messages say that:
  • a run may start the node an earlier run left deallocated;
  • an eviction deletes its node, because the pool's eviction policy (Delete) applies to it rather than its scale-down mode, so the node that replaces it pulls the image again.
• README.md:
  • The note now says that the first run in a cluster waits for the download and the next runs don't, except on the node that replaces the evicted one.
  • A new A node that keeps its images paragraph explains the mode, its cost (the OS disk, and a deallocated Spot node still counts against the Spot vCPU quota) and why the OS disk is managed.
  • The scripts table, the eviction paragraph and the resources follow.

The first commit corrected the note for the emulator as it is today; the second rewrites it for the new behaviour.

Tests

• Tested on LocalStack for Azure run from the source of localstack/localstack-pro pull request 10317, on a new cluster created by scripts/01-user-assigned-managed-identity.sh:
  • Setup: 01 to 04 passed. The pool reports osDiskType: Managed, scaleDownMode: Deallocate, scaleSetPriority: Spot and scaleSetEvictionPolicy: Delete.
  • First 05-run-training-job.sh: 423 s, of which 5m20.526s were the new node's image pull. 35 s after the job ended, the autoscaler deallocated the node: its container exited, the Node went NotReady with the Cloud Controller Manager's shutdown taint, the VMSS instance reported PowerState/deallocated, and the pool reported count: 0.
  • 06-simulate-eviction.sh: passed in 657 s. Its pod started on the restarted node 31 s after its creation, with Container image "pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime" already present on machine. The eviction deleted that node, and the replacement pod's new node pulled the image in 5m53.577s.
  • Second 05-run-training-job.sh: 154 s, on the restarted node, with the image already present.
• On localstack/localstack-azure:latest (build 2026.11.0.dev770), before this change, a later run's pod spent 5m12.656s pulling the image. Until pull request 10317 ships, the emulator accepts --scale-down-mode Deallocate but deletes the idle node as before.
• bash -n passes on the changed scripts. pre-commit run: gitleaks passed, zizmor skipped. The note renders as a Note alert through GitHub's markdown API.
• Not run on Azure: adding a Spot Standard_NV18ads_A10_v5 pool with --scale-down-mode Deallocate --node-osdisk-type Managed. Microsoft Learn documents both flags, and Scale-down Mode's support for Spot pools.

Related

• localstack/localstack-pro pull request 10317: the emulator's implementation of Scale-down Mode Deallocate. This pull request should merge once that one ships in localstack/localstack-azure:latest.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FvNuJKW5UZQ1Rce8qnG2GR

[Copilot]

🟡 Changes recommended

The tutorial overstates image-download requirements for reruns and should qualify those claims to account for cached-image reuse when no fresh GPU node is needed.

1 open issue

• Limit the download requirement to fresh GPU nodes · New

What changed in this PR

This documentation-only PR updates the GPU node pools tutorial to explain recurring PyTorch image download delays as GPU nodes are removed and recreated.

Changes:

• Replaces the first-run warning with recurring download expectations for training and eviction scripts.
• Explains node-local image caching and the different idle scale-down delays in LocalStack and AKS.

File	Description
tutorials/​gpu-node-pools/​README.md	Expands the startup-delay note with image lifecycle and script-specific download expectations.

_{Review effort: Balanced}