[paolosalvatori]

Motivation

The training job's pod in the GPU node pools tutorial can't start until its GPU node has downloaded the job's image, pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime, about 3 GB.
The pod stays Pending meanwhile, so 05-run-training-job.sh keeps printing (pods: Pending) for minutes and seems stuck.
In a run of the tutorial on LocalStack for Azure, the pod was Pending for 6m40s: 45 s for the cluster autoscaler to add the GPU node, then 5m54.879s for the kubelet to pull the image.

Changes

tutorials/gpu-node-pools/README.md gets a GitHub note under How to run it, after the list of commands, which says:

• the first time you run the tutorial, the job's pod can take a few minutes to start running, because the new GPU node downloads the image, which can take between 5 and 10 minutes depending on the network bandwidth, and the pod stays Pending until then;
• every node the cluster autoscaler adds to the GPU pool starts without the image, so the wait comes back whenever a run finds the pool scaled back to zero, and in 06-simulate-eviction.sh for the node that replaces the evicted one;
• kubectl get events --namespace gpu-training --watch shows when the download starts (Pulling image) and when it ends (Successfully pulled image).

Tests

• Measured on LocalStack for Azure (localstack/localstack-azure:latest), from the cluster events and the emulator log of one run of 05-run-training-job.sh: the pod was created at 14:28:01Z and bound to aks-gpupool-9590704-vmss000000 at 14:28:46Z, the kubelet logged Successfully pulled image "pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime" in 5m54.879s (5m54.879s including waiting). Image size: 3337943678 bytes. and started the container at 14:34:42Z, and the autoscaler removed the idle node at 14:36:35Z.
• Checked why the wait comes back: each emulated node is a k3d container whose image store (/var/lib/rancher/k3s) is a volume of its own, and the image archives the emulator preloads into new nodes are its add-ons and the k3s system images only, with no PyTorch image.
• Rendered the note with GitHub's markdown API: it renders as a Note alert.
• pre-commit run --files tutorials/gpu-node-pools/README.md: gitleaks passed, zizmor skipped (no workflow file).
• Not run: a second run of 05-run-training-job.sh to time the repeated download, and a run on Azure.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FvNuJLPJxRzyEhHVHgyFCH

[Copilot]

🟢 Approval recommended

No active review issues are reported for the documentation update clarifying image-download delays and how to monitor progress.

No open issues 🎉

What changed in this PR

Clarifies expected image-download delays in the GPU node pools tutorial so users can distinguish a normal startup wait from a stalled training job.

Changes:

• Adds a note explaining the 5–10 minute PyTorch image download and why it can recur after scale-to-zero or node eviction.
• Documents how to watch Kubernetes events for image-download progress.

File	Description
tutorials/​gpu-node-pools/​README.md	Explains image-pull delays, recurring waits, and the command for monitoring progress.

_{Review effort: Balanced}