Skip to content

ci: MG x86-main guard-cargo p50 2635s and guard-tree p50 1895s; 7 of 18 runs hit the 3600s section cap #5739

Open

Description

@noahgift

Measured 2026-10-19 over the last 18 completed merge_group x86-main runs (job logs, lines 'section X: ... in Ns'), plus run 39214997865 (both 3600s).

section p50 min max at 3600s cap
guard-cargo ~2635s 1825s 3600s 7 of 18
guard-tree ~1895s 957s 3600s 2 of 18 (5 more above 2700s)

For comparison sov.test 638s, sov.lint 657s in the same run. The guards are on the critical path of every merge-group pass (the Sections step is the longest).

Split CPU vs wait: not_measured. The job logs carry no load or PSI lines, and the runner host is not this box. Indirect evidence for wait/contention: the same hermetic script check_crux_inference_judge.sh was 508s+ in passing run 39196197124 and killed at 2400s in 39214997865 with identical code; 12 sections run concurrently on the one runner. #5709 alone passed (39214996476: guard-cargo 2602s, guard-tree 1800s), so the failure is not that PR.

Proposed: (1) have fat_driver log /proc/pressure/{cpu,io} and loadavg at each 300s 'still running' line so the split is measurable; (2) per-section concurrency cap or move guard-cargo off the shared runner. Refs flow rows; filed by aprender-w5032.

Activity

  1. added this to the 0.73.0 milestone on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions