Repository navigation
CI: pytest-gpu-omp-amd segfaults in the examples step when two runs overlap on ese-mi50 #3049
Description
Symptom
pytest-gpu-omp-amd (workflow pytest-gpu.yaml, self-hosted runner ese-mi50-A, label AMDGPU) fails with Process completed with exit code 139 in the Test examples step, after both pytest suites in the same job have passed (257 passed, then 46 passed). The crashing example differs from run to run:
| Run | Branch | Crashing example |
|---|---|---|
| 38176097603 (11 Oct 21:22) | JDBetteridge/update_actions |
tti_example.py::test_tti_stability[centered-(51, 51)] |
| 38180644984 (11 Oct 22:29) | JDBetteridge/ci_refactor |
viscoacoustic_example.py::test_viscoacoustic_stability[1-sls-(16, 16, 16)] |
| 38425837724 (13 Oct 15:53) | fix-cse-indexedpointer (#3047) |
tti_example.py::test_tti_stability[staggered-(16, 16, 16)] |
| 38426576406 (13 Oct 15:58) | JDBetteridge/ci_refactor |
tti_example.py::test_tti_stability[centered-(16, 16, 16)] |
Each is Fatal Python error: Segmentation fault inside Operator.apply (mpiexec noticed that process rank 0 ... on node ese-mi50 exited on signal 11).
Pattern
Every failure above is one of a pair of runs that overlapped on the runner (the job takes ~65 min; the pairs started 67 min and 5 min apart). Every run that had the runner to itself passed: the last 12 main pushes (17 Sep – 12 Oct), the #3043 run (10 Oct) and a dependabot run (9 Oct). The workflow's concurrency group only de-duplicates runs of the same branch, so runs from different PRs can execute on ese-mi50 at the same time.
A codegen defect would crash the same kernel each time; a different, randomly placed example dying only when two jobs share the one MI50 points at GPU/driver resource contention (two processes on one device). For #3047 specifically, the generated code of the crashing operator is byte-identical with and without that PR's change.
Suggested fix
Serialise jobs on the AMD runner, e.g. a repo-wide concurrency: { group: amdgpu-runner } on the pytest-gpu-omp-amd job (without cancel-in-progress), or configure the runner with a single job slot. Alternatively, find the actual device-level failure (the segfault reason is not logged; running the examples step with HSA/HIP error logging enabled when two jobs overlap would show whether it is an allocation failure).
Not reproducible from a single run, so a reproducer is "start two CI-gpu runs within an hour of each other".
🤖 Generated with Claude Code
Activity
ggorman commentedon Oct 13, 2026
Confirming data point: the
pytest-gpu-omp-amdjob of #3048 (run 38436497341, started 17:10 on 13 Oct, with no other CI-gpu run onese-mi50for its duration) passed, including the examples step. Same day, same runner, same code family as the two 15:53/15:58 failures above; the only difference is that it ran alone.JDBetteridge commentedon Oct 14, 2026
This appears to be caused by specifying
--network=host --ipc=hostwhen launching the docker container. MPI calls are not coordinated and cause the segfault. This is a result of my cargo-culting the previous assumed good docker CI configuration.Since
mi50is the bottleneck for OSS CI I am determined to avoid serialising this stepAttempted fix here
ggorman commentedon Oct 15, 2026
Resolved by #3045 (dropping
--network=host --ipc=hostfrom the container launch).pytest-gpu-omp-amdhas passed on main in both runs since it merged (c6a977c, 2707ac9). Reopen if the overlap segfault recurs.