Repository navigation
ci(gz): bound and retry the Gazebo apt install (24% of runs stalled to the 45-min timeout) - #603
Merged
Merged
Conversation
… to the 45-min timeout and ejected queued PRs Measured over the last 21 gazebo.yml runs: "Install Gazebo Harmonic" takes 1-3 min normally (max 11), but 5 of 21 (24%) stalled on the hosted runner's azure.archive.ubuntu.com mirror until the job's 45-min timeout cancelled them. They were still receiving one package every ~2 s, so no per-request apt timeout could fire. Under the merge queue each stall EJECTS the PR being built: #595 (17:48Z) and #571 (22:27Z) on 2026-10-22. apt_retry: each attempt capped at 10 min with `timeout`; downloaded .debs stay in apt's cache so a retry resumes; a retry moves off the azure mirror to archive.ubuntu.com. A real failure still fails (the third attempt's status is the step's). Exercised with stubbed sudo/apt-get: hang-then-ok rc=0 after 2 attempts + 1 mirror switch; always-fail rc=1 after 3; ok rc=0 after 1. Three capped attempts plus the usual ~6 min build stay inside the 45-min job. soak.yml carries two copies of the same install; not changed here (not a required gate, so it ejects nothing). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured
Over the last 21
gazebo.ymlruns, theInstall Gazebo Harmonicstep usually takes 1–3 min (11 min at most). 5 of 21 (24%) stalled until the job's 45-minute timeout cancelled them.Every stall was on the hosted runner's
azure.archive.ubuntu.commirror, and packages were still arriving about every 2 s. Because the connection never goes silent, no per-request apt timeout can catch it.Under the merge queue, each stall ejects the PR being built. That happened twice on 2026-10-22: #595 at 17:48Z and #571 at 22:27Z. In both cases Kani, CI and the verification gate had passed in the queue.
Change
apt_retrywraps every apt call in this step:timeout..debs stay in apt's cache, so a retry resumes instead of starting over.archive.ubuntu.com.Three capped attempts plus the usual ~6 min build fit inside the 45-min job.
Exercised (stubbed
sudo/apt-get, the function extracted from this workflow)actionlintis clean. The diff has no Kani-scope hits.Not changed:
soak.ymlhas two copies of the same install. It is not a required gate, so a stall there ejects nothing; it is a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG