From 53a18282701da62ccceb04891e8f093a9ca99193 Mon Sep 17 00:00:00 2001 From: Paolo Salvatori Date: Fri, 6 Nov 2026 16:00:54 +0100 Subject: [PATCH] Note that the GPU training job's pod can wait minutes for its PyTorch image Each new GPU node downloads pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime, about 3 GB, before the job's pod starts, and the pod stays Pending meanwhile, so 05-run-training-job.sh seems stuck. On LocalStack for Azure the pod was Pending for 6m40s: 45 s for the cluster autoscaler to add the GPU node, then 5m54.879s for the kubelet to pull the image. A note under "How to run it" now says so, says the wait comes back on every new GPU node, and shows how to see the download start and end. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01FvNuJLPJxRzyEhHVHgyFCH --- tutorials/gpu-node-pools/README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tutorials/gpu-node-pools/README.md b/tutorials/gpu-node-pools/README.md index 3ccce61..dd22d73 100644 --- a/tutorials/gpu-node-pools/README.md +++ b/tutorials/gpu-node-pools/README.md @@ -80,6 +80,12 @@ cd gpu-node-pools/scripts # ./07-cleanup.sh # when finished ``` +> [!NOTE] +> The first time you run the tutorial, the job's pod can take a few minutes to start running: the new GPU node has to download the pod's image, `pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime` (about 3 GB), which can take between 5 and 10 minutes depending on your network bandwidth. +> Until the download completes, the pod stays `Pending`, and `05-run-training-job.sh` keeps printing `(pods: Pending)`. +> Every node the cluster autoscaler adds to the GPU pool starts without the image, so the wait comes back whenever a run finds the pool scaled back to zero, and in `06-simulate-eviction.sh` for the node that replaces the evicted one. +> To check that the pod is downloading its image rather than stuck, run `kubectl get events --namespace gpu-training --watch` in another terminal: it shows `Pulling image` when the download starts and `Successfully pulled image` when it ends. + ### Kubernetes manifests Every manifest in [scripts/](scripts/) is valid, applyable YAML whose environment-specific fields are empty strings; the scripts fill them in with `yq`, which is why the same files work against the emulator and against Azure without edits.