feat(actions): size arc-job-report runners from the job's 7-day history

[JesperTerkelsen]

Summary

Follow-up to #381. The runner suggestion is now based on the job's last 7 days of runs, and memory and CPU are treated differently. That's because runners have a memory limit but only a CPU request: too little memory gets the job OOM-killed, while too little CPU only makes it slower.

Why one run isn't enough: cache hits change a job's peak memory a lot. Over 6 hours of monorepo-typescript runs:

Job	Runs	Peak memory p50	Max	Runs whose own peak suggests a size the worst run would OOM on
Typecheck	417	1.5 GB	13.1 GB	84%
Build (staging-blind)	417	0.9 GB	4.8 GB	78%
E2E Tests (hub)	346	7.2 GB	10.7 GB	11%

The rule (suggest() in job_resources.py):

• Memory is a hard floor. The size must hold 1.3 × the job's 7-day peak. If it doesn't, the suggestion is ⬆️ needs memory. An OOM kill in this run always moves up a size.
• CPU is a labelled trade-off:
  • ⬆️ CPU-bound, faster when the job's median CPU is at least 85% of the runner's vCPU.
  • A smaller size must still cover the job's median CPU. It is fits when it also covers the peak, and trial: may be slower when it doesn't.
• Smaller sizes need 5 runs in the window. This replaces the 2-minute cut-off, since the history covers short jobs. Bigger sizes don't wait.

The suggestion table now shows the reason, the 7-day peak memory, median and peak CPU, and the run count.

Where the history comes from: the arc:job_memory_working_set_bytes:max1m and arc:job_cpu_usage_cores:rate1m recording rules in https://github.com/monta-app/terraform/pull/261. The report runs four instant queries per run, all grouped by job_name, that read only the selected jobs' recorded series. It never joins raw cAdvisor series at read time; doing that over 7 days OOM-killed the mimir-internal query-frontends today.

Local path: job_resources.py now calls the same Grafana datasource-proxy URL as CI, through gcx api. gcx metrics query silently widened the step to fit about 100 points, so long jobs were coarser locally than in CI.

Testing

• python3 -m unittest discover -s .github/actions/arc-job-report -p 'test_*.py': 12 tests. They cover every branch of suggest() (memory floor, OOM, CPU-bound, fits/trial, the 5-run gate, runners outside the catalog), the escaping in the history query, and the new table header and row.

• job_resources.py against https://github.com/monta-app/monorepo-typescript/actions/runs/38407948093 with no recorded history yet: every job is keep and nothing errors.

• The same run's arc_job_report.py --dry-run, with history stubbed from the 6-hour numbers above:

  • Build: 8cpu → 4cpu, fits
  • Lint: 8cpu → 4cpu, trial
  • Knip: 4cpu → 2cpu, fits
  • Format: 4cpu → 2cpu, trial
  • Test shards: 8cpu → 16cpu, CPU-bound, faster
  • Typecheck: stays on 8cpu, because its 13.1 GB peak needs 17 GB

  That run caught a name clash with the comment's existing HISTORY regex, which is fixed, and the render test now checks the header.

Gaps

• Needs https://github.com/monta-app/terraform/pull/261 applied. Until then the history queries return nothing and the report suggests only memory and CPU-bound upsizes from the current run. It doesn't fail. There's no backfill, so downsizes start about a week after the rules are applied.
• Jobs are matched by repository + job_name. Two workflows in one repo with the same job name share a history, which can only make the memory floor higher.
• The Test gate job and the Test (shard N) jobs are separate names, so they have separate histories. The table row still shows the shard job as "Test", as before.

🤖 Generated with Claude Code