Skip to content

fix(idle-watchdog): self-test raced the clock under load (job 122420732235) - #6483

Closed
noahgift wants to merge 1 commit into
mainfrom
fix/idle-watchdog-selftest-flake
Closed

noahgift wants to merge 1 commit into
mainfrom
fix/idle-watchdog-selftest-flake

Conversation

@noahgift

Copy link
Copy Markdown
Member

idle_watchdog.sh --self-test failed twice in CI job 122420732235 (workspace-test shard 2/4, run on #6438) while the intel runner was heavily loaded (load ~97 on 32 cores):

  • TERM exits 143: got '124'
  • within grace: got 0

What raced

  1. The script. The TERM trap only signalled the child's process group. If TERM arrived outside wait -n (for example during logs_size), the loop kept going, so a step that ignores TERM ran on to the idle limit and the watchdog exited 124 instead of 143. That also pushed the grace row past its 8 s bound. The trap now sets termed, and the loop checks it on the next pass.
  2. The self-test. It slept a fixed 0.7 s before sending TERM. It now waits for real state instead: a marker file written by the step, and the watchdog's tick-sleep child. The "within grace" and "timeout fired at its own deadline" rows stop their clock at the exit, before the settle sleeps and pgrep calls.

No rows were removed or loosened, and the 8 s bounds are unchanged.

Evidence

  • Under load (64 yes loaders, load 40–115 on 32 cores): the first 20-run set went 19/20 and exposed a third clock race in the timeout-deadline row, which this PR also fixes. After that fix, 20/20 at load 70–78. A cosmetic seq → {1..1200} change came after that set and was checked with a single passing run only.
  • Mutation: changing exit 143 to exit 142 turns the "exits 143" row red. Removing kill -KILL turns the "KILLed after grace" rows red. Both were reverted.
  • Lint: cargo fmt --all -- --check is clean. bashrs lint exits 0 with 0 errors.

This file runs inside a CI shard, so treat the change as gate content: it needs a quorum and a non-author sign-off before arming. Auto-merge is not armed.

🤖 Generated with Claude Code

…32235)

A TERM that reached the watchdog outside `wait -n` (trap ran during
logs_size or the loop head) only signalled the child group; a step that
ignores TERM kept the watchdog looping until the idle limit, so the
self-test saw exit 124 instead of 143 and a >8s "grace" time. The trap
now records `termed`, so the loop honours it on its next pass.

The self-test also slept a fixed 0.7s before sending TERM and timed the
deadline rows across settle sleeps and pgrep. It now waits on state (the
step's marker file and the watchdog's tick sleep) and stops the clock at
the exit. No row is removed or loosened.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=6483 head=7d00e558b01075ace9d590ee2bbb3fb65e8e2e34 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

@noahgift

Copy link
Copy Markdown
Member Author

EMG-004 §2.2 (U5): superseded by Epic PR #6484 (epic/emg4-apr-1 @ a2fb135).

@noahgift noahgift closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant