CI: raise self-hosted job timeout to 1380m to absorb Phoenix embers queue wait

[sbryngelson]

Phoenix GPU jobs run under QOS embers (priority weight 0). With ~475 inferno jobs pending at ~231k priority, ours (priority ~20) only start on leftover GPUs, and queue waits of ~7h are now routine. timeout-minutes counts that wait, so jobs get killed shortly after they finally start. Example: carbon-surface-v1 bench gpu-acc (SLURM 13821471) sat PENDING 03:22Z→10:13Z, ran ~1h, then the 480m GitHub timeout cancelled it. The runner's scancel cleanup is why these show up as CANCELLED by <uid> in sacct.

This raises the job-level timeout for the self-hosted Test Suite, Case Opt, and Benchmark jobs from 480m to 1380m (23h). SLURM walltime limits are unchanged (3–4h); only the queue wait we tolerate grows. It stays under 24h because GitHub fails jobs that wait >24h for a runner and GITHUB_TOKEN expires at 24h.

Tradeoffs, to revisit if they bite:

• Phoenix runners (10, all busy) are held longer per job, so more backlog shows as GitHub "queued" rather than SLURM PENDING. Our SLURM age priority is 0, so submitting earlier wasn't buying anything anyway.
• Master-push runs are never cancelled, so a burst of merges can occupy runners for up to a day.
• Genuine failures behind a long queue take longer to report.

Experimental; revert if runner starvation gets worse than the timeouts it removes.

Acknowledgement

• I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-03T15:03:54.575388Z	a3b421a	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

Copilot review overview

🟢 Approval recommended

The limited timeout changes match the stated purpose, preserve SLURM walltimes, and have no identified blocking issues.

Review effort: Balanced
Findings: None

What changed in this PR

Extends self-hosted CI timeouts to tolerate long Phoenix SLURM queue waits.

Changes:

• Raises Test Suite, Case Opt, and Benchmark job timeouts from 480 to 1380 minutes.
• Updates related script comments; SLURM walltimes and script behavior remain unchanged.

File	Description
.github/​workflows/​test.yml	Extends Test Suite and Case Opt timeouts.
.github/​workflows/​bench.yml	Extends Benchmark timeout.
.github/​scripts/​submit-slurm-job.sh	Updates timeout reference in a comment.
.github/​scripts/​run_parallel_benchmarks.sh	Updates timeout reference in a comment.
.github/​scripts/​monitor_slurm_job.sh	Updates queue-wait timeout commentary.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.80%. Comparing base (c7feed2) to head (a3b421a).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1942   +/-   ##
=======================================
  Coverage   62.80%   62.80%           
=======================================
  Files          86       86           
  Lines       22385    22385           
  Branches     3304     3304           
=======================================
  Hits        14060    14060           
  Misses       6073     6073           
  Partials     2252     2252           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.