From 20e3d3556ff3dd0d2c919279d93908d3e2e5ddda Mon Sep 17 00:00:00 2001 From: Jesper Terkelsen Date: Tue, 13 Oct 2026 16:56:44 +0200 Subject: [PATCH] feat(actions): size arc-job-report runners from the job's 7-day history One run is not safe to size on: cache hits change a job's peak memory a lot (monorepo Typecheck peaks at 1.5 GB in a typical run, 13.1 GB in the worst), so a cached run suggested a runner the next uncached run would be OOM-killed on. The suggestion now reads the job's last 7 days from the arc:job_memory_working_set_bytes:max1m and arc:job_cpu_usage_cores:rate1m recording rules (monta-app/terraform) and treats the two resources differently, since runners have a memory limit but only a CPU request: - memory is a hard floor: 1.3 x the 7-day peak, and an OOM kill always moves up a size; - CPU is a labelled trade-off: "CPU-bound, faster" when the median is at least 85% of the runner's vCPU, and a smaller size must cover the median, marked "trial: may be slower" when the peak would not fit. Smaller sizes need 5 runs in the window, which replaces the 2-minute cut-off. The local path now calls the datasource proxy through gcx api, like CI, because gcx metrics query widens the step to about 100 points. Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/actions/arc-job-report/README.md | 47 +++++--- .../actions/arc-job-report/arc_job_report.py | 26 ++--- .../actions/arc-job-report/job_resources.py | 105 ++++++++++++++---- .../arc-job-report/test_arc_job_report.py | 73 ++++++++++-- CLAUDE.md | 6 +- 5 files changed, 191 insertions(+), 66 deletions(-) diff --git a/.github/actions/arc-job-report/README.md b/.github/actions/arc-job-report/README.md index c3d7d6d..3441d52 100644 --- a/.github/actions/arc-job-report/README.md +++ b/.github/actions/arc-job-report/README.md @@ -14,9 +14,10 @@ Expanded, it has: [ARC Job Resources dashboard](https://montaapp.grafana.net/d/arc-job-resources). Sharded jobs (` - shard N`) are merged into one row with the time range and a link per shard number; the slowest shard is bold. -- **Suggested runner changes:** only jobs where another catalog size would fit, - marked ⬇️ (smaller) or ⬆️ (bigger), with peak memory and CPU; the job name - links to its dashboard. Left out entirely when every runner fits. +- **Suggested runner changes:** only jobs where another catalog size is + suggested, marked ⬇️ (smaller) or ⬆️ (bigger) with the reason, the job's peak + memory, median and peak CPU and number of runs over the last 7 days; the job + name links to its dashboard. Left out entirely when every runner is right. - **Run history:** one row per run (commit, ✅/❌, wall time, change vs the previous run, slowest job), so speed changes can be traced to commits. The history is stored as JSON in a hidden HTML comment and re-rendered on every @@ -69,25 +70,35 @@ an ARC runner (`arc-*` / `self-hosted-*`) are skipped. Metrics come from cAdvisor and kube-state-metrics in `mimir-internal`, scraped every **15 s**. -- **Peak memory is reliable.** It is the highest working-set sample over the job. -- **CPU is not, for short jobs.** It is the highest 1 min average, so short bursts - are averaged away. In service-ocpi, the CPU-based rule suggested 4 vCPU for the - integration shards, but a trial run made every shard 17–97 s slower. Trial a - smaller runner before moving to it. -- **Jobs under 2 minutes are left out of the suggestions.** They get too few - samples. There is no point waiting for them: a runner pod's samples stop once - its job ends and the pod is deleted. - -The suggested size is the smallest catalog runner with memory ≥ 1.3 × peak and -vCPU ≥ 1.2 × the CPU p95. +Suggestions use the job's runs over the **last 7 days** (same repository and job +name), not just this run: cache hits make one run's peak misleading. Typecheck in +monorepo-typescript peaks at 1.5 GB in a typical run and 13.1 GB in the worst. The +history comes from two recording rules in monta-app/terraform +(`global/grafana/teams/sre/alerts_arc.tf`): `arc:job_memory_working_set_bytes:max1m` +and `arc:job_cpu_usage_cores:rate1m`. They join each pod's cAdvisor series to its +GitHub job once a minute, so the report never joins raw series at read time. + +Memory and CPU are treated differently because the runners have a memory limit +and only a CPU request: + +- **Memory is a hard floor.** A size must hold 1.3 × the job's 7-day peak, or the + job is OOM-killed. A job that needs more gets ⬆️ *needs memory*, and so does + any job that was OOM-killed in this run. +- **CPU only costs time.** A job whose median CPU is at least 85% of its runner's + vCPU gets ⬆️ *CPU-bound, faster*. A smaller size must still cover the job's + median CPU; it is *fits* if it also covers the peak, and *trial: may be slower* + if not. In service-ocpi, a 4 vCPU runner made every integration shard 17–97 s + slower, so trial a smaller runner before moving to it. +- **Smaller sizes need 5 runs** in the window, so rare jobs and new recording + rules don't produce a downsize from one run. Bigger sizes don't wait. ## Files - `arc_job_report.py` — builds and posts the comment. -- `job_resources.py` — the per-job analysis. It is `infra/arc/scripts/job-resources.py` - from monta-app/kube-manifests, plus a `GRAFANA_TOKEN` path that queries Prometheus - through Grafana's datasource proxy instead of `gcx`. Keep the two in sync. -- `test_arc_job_report.py` — unit tests for the comment rendering: +- `job_resources.py` — the per-job analysis and the size suggestion. It queries + Prometheus through Grafana's datasource proxy, with `GRAFANA_TOKEN` in CI or + `gcx api` locally. +- `test_arc_job_report.py` — unit tests for the comment rendering and the suggestion: `python3 -m unittest discover -s .github/actions/arc-job-report -p 'test_*.py'` ## Running locally diff --git a/.github/actions/arc-job-report/arc_job_report.py b/.github/actions/arc-job-report/arc_job_report.py index 8f812bf..8e8ad94 100644 --- a/.github/actions/arc-job-report/arc_job_report.py +++ b/.github/actions/arc-job-report/arc_job_report.py @@ -16,12 +16,13 @@ import sys from datetime import datetime -from job_resources import ARC_RUNNER, analyse, jobs_for, sh +from job_resources import MEMORY_HEADROOM, MIN_RUNS, WINDOW, analyse_run, sh MARKER = "" HISTORY = re.compile(r"", re.S) SHARD = re.compile(r"^(.*) - shard (\d+)$") RESULT = {"success": "✅", "failure": "❌"} +WHY = {"memory": "needs memory", "faster": "CPU-bound, faster", "fits": "fits", "trial": "trial: may be slower"} # ponytail: keeps the last 50 runs so the comment stays far below GitHub's 65,536-character limit MAX_HISTORY = 50 @@ -79,17 +80,15 @@ def job_rows(results): def change(r): - if r["duration_s"] < 120: - return None - runner, suggested = size(r["runner"]), size(r.get("suggested_runner") or r["runner"]) - return None if suggested == runner else (runner, suggested) + return r.get("suggested_runner") and (size(r["runner"]), size(r["suggested_runner"])) def suggestion_row(r): runner, suggested = change(r) arrow = "⬇️" if rank(suggested) < rank(runner) else "⬆️" - return (f"| [{name(r)}]({r['dashboard']}) | {arrow} {runner} → {suggested} | {r['cpu_cores']['max']:.1f} | " - f"{r['memory_gib']['peak']:.1f} GiB |") + h = r["history"] + return (f"| [{name(r)}]({r['dashboard']}) | {arrow} {runner} → {suggested} | {WHY[r['reason']]} | " + f"{h['peak_mem_gib']:.1f} GiB | {h['cpu_p50']:.1f} / {h['cpu_peak']:.1f} | {h['runs']} |") def history_row(h, previous): @@ -127,11 +126,13 @@ def render(run, results, previous_body): "|---|---|", *job_rows(results), "", - *(["💡 **Suggested runner changes.** Memory is the exact peak; CPU is the max of 1 min averages, so " - "trial before downsizing on it. Jobs under 2 min have too few samples and are left out.", + *([f"💡 **Suggested runner changes.** Memory is a hard limit: a size must hold {MEMORY_HEADROOM} × the " + f"job's peak over the last {WINDOW}, or the job is OOM-killed. CPU only costs time: a smaller size " + "must cover the job's median CPU, and is a *trial* when its peak would not fit. " + f"Smaller sizes are only suggested after {MIN_RUNS} runs in the window.", "", - "| Job | Change | CPU | Memory |", - "|---|---|---|---|", + f"| Job | Change | Why | Peak memory ({WINDOW}) | CPU median / peak | Runs ({WINDOW}) |", + "|---|---|---|---|---|---|", *map(suggestion_row, changed), ""] if changed else []), f"", @@ -156,8 +157,7 @@ def main(): if not prs: sys.exit(f"no pull request found for run {run_id}") pr = prs[0]["number"] - results = [analyse(j) for j in jobs_for(args[0]) - if j.get("runner_name") and ARC_RUNNER.match(j["runner_name"]) and j.get("completed_at")] + results = analyse_run(args[0]) comments = json.loads(sh("gh", "api", "--paginate", "--slurp", f"repos/{repo}/issues/{pr}/comments")) existing = next((c for page in reversed(comments) for c in reversed(page) if MARKER in c["body"] and c["user"]["type"] == "Bot"), None) diff --git a/.github/actions/arc-job-report/job_resources.py b/.github/actions/arc-job-report/job_resources.py index 8423170..8c7d7bf 100644 --- a/.github/actions/arc-job-report/job_resources.py +++ b/.github/actions/arc-job-report/job_resources.py @@ -7,14 +7,14 @@ Needs `gh` (authenticated) and either GRAFANA_TOKEN (a Grafana service-account token with read access to the Prometheus datasource) or `gcx` (a context with that datasource). A job's runner name (GitHub API `runner_name`) is its ARC pod name; metrics are scraped every 15 s. - -Copied from monta-app/kube-manifests infra/arc/scripts/job-resources.py @ dd43b0172; the only -change is the GRAFANA_TOKEN query path in prom(). +A job's history across runs comes from the arc:job_* recording rules (monta-app/terraform, +global/grafana/teams/sre/alerts_arc.tf). """ import json import math import os import re +import statistics import subprocess import sys import urllib.parse @@ -33,23 +33,51 @@ ("arc-x64-4cpu-12gb", "amd64", 4, 12), ("arc-x64-8cpu-24gb", "amd64", 8, 24), ] +JOB_MEMORY = "arc:job_memory_working_set_bytes:max1m" +JOB_CPU = "arc:job_cpu_usage_cores:rate1m" +WINDOW = "7d" +MIN_RUNS = 5 +MEMORY_HEADROOM = 1.3 +CPU_BOUND = 0.85 def sh(*args): return subprocess.run(args, capture_output=True, text=True, check=True).stdout -def prom(expr, start, end, step=15): +def get(api, **params): + path = f"/api/datasources/proxy/uid/{DATASOURCE}/api/v1/{api}?{urllib.parse.urlencode(params)}" token = os.environ.get("GRAFANA_TOKEN") if token: - query = urllib.parse.urlencode({"query": expr, "start": start, "end": end, "step": step}) - req = urllib.request.Request(f"{GRAFANA}/api/datasources/proxy/uid/{DATASOURCE}/api/v1/query_range?{query}", - headers={"Authorization": f"Bearer {token}"}) + req = urllib.request.Request(GRAFANA + path, headers={"Authorization": f"Bearer {token}"}) with urllib.request.urlopen(req, timeout=30) as resp: return json.load(resp)["data"]["result"] - out = sh("gcx", "metrics", "query", "-d", DATASOURCE, expr, - "--from", str(start), "--to", str(end), "--step", f"{step}s", "-o", "json") - return json.loads(out)["data"]["result"] + return json.loads(sh("gcx", "api", path))["data"]["result"] + + +def prom(expr, start, end, step=15): + return get("query_range", query=expr, start=start, end=end, step=step) + + +def label(value): + return value.replace("\\", "\\\\").replace('"', '\\"') + + +def history(repo, names): + if not names: + return {} + regex = "|".join(re.sub(r"([.*+?^${}()|\[\]\\])", r"\\\1", n) for n in sorted(set(names))) + sel = f'repository="{label(repo)}", job_name=~"{label(regex)}"' + + def by_job(expr): + return {r["metric"]["job_name"]: float(r["value"][1]) for r in get("query", query=expr)} + + mem = by_job(f"max by (job_name) (max_over_time({JOB_MEMORY}{{{sel}}}[{WINDOW}]))") + cpu_p50 = by_job(f"quantile by (job_name) (0.5, quantile_over_time(0.5, {JOB_CPU}{{{sel}}}[{WINDOW}]))") + cpu_peak = by_job(f"max by (job_name) (max_over_time({JOB_CPU}{{{sel}}}[{WINDOW}]))") + runs = by_job(f"count by (job_name) (count by (job_name, workflow_run_id) (count_over_time({JOB_MEMORY}{{{sel}}}[{WINDOW}])))") + return {n: {"runs": int(runs.get(n, 0)), "peak_mem_gib": mem[n] / 2**30, + "cpu_p50": cpu_p50.get(n), "cpu_peak": cpu_peak.get(n)} for n in mem} def values(result): @@ -75,15 +103,27 @@ def jobs_for(url): return [json.loads(line) for line in out.splitlines() if line] -def suggest(arch, peak_mem_gib, p95_cpu): - need_mem, need_cpu = peak_mem_gib * 1.3, max(p95_cpu * 1.2, 1) - for name, a, cpu, mem in CATALOG: - if a == arch and mem >= need_mem and cpu >= need_cpu: - return name - return next((n for n, a, _, _ in reversed(CATALOG) if a == arch), None) - - -def analyse(job): +def suggest(runner, arch, peak_mem_gib, cpu_p50, cpu_peak, runs, oom): + sizes = [c for c in CATALOG if c[1] == arch] + now = next((c for c in sizes if runner.startswith(c[0] + "-")), None) + if not now: + return None + i = sizes.index(now) + fits = [c for c in sizes if c[3] >= peak_mem_gib * MEMORY_HEADROOM] + need = sizes.index(fits[0]) if fits else len(sizes) - 1 + if oom: + need = max(need, i + 1) + if need > i: + return sizes[min(need, len(sizes) - 1)][0], "memory" + if cpu_p50 >= CPU_BOUND * now[2] and i + 1 < len(sizes): + return sizes[i + 1][0], "faster" + smaller = next((c for c in fits if c[2] >= cpu_p50), now) + if runs < MIN_RUNS or sizes.index(smaller) >= i: + return None + return smaller[0], "fits" if smaller[2] >= cpu_peak else "trial" + + +def analyse(job, hist): pod, start, end = job["runner_name"], ts(job["started_at"]), ts(job["completed_at"]) sel = f'cluster="internal", namespace="arc", pod="{pod}"' lo, hi = start - 60, end + 60 @@ -110,13 +150,22 @@ def analyse(job): cpu_sorted = sorted(cpu) p95 = cpu_sorted[min(len(cpu_sorted) - 1, math.ceil(0.95 * len(cpu_sorted)) - 1)] peak_mem = max(mem) / 2**30 + h = hist.get(job["name"], {}) + runs = h.get("runs", 0) + week = {"runs": runs, "peak_mem_gib": round(max(peak_mem, h.get("peak_mem_gib", 0)), 2), + "cpu_p50": round(h["cpu_p50"] if runs >= MIN_RUNS and h.get("cpu_p50") is not None + else statistics.median(cpu), 2), + "cpu_peak": round(max(max(cpu), h.get("cpu_peak") or 0), 2)} res.update({ "cpu_cores": {"avg": round(sum(cpu) / len(cpu), 2), "p95": round(p95, 2), "max": round(max(cpu), 2), "request": max(req) if req else None}, "memory_gib": {"peak": round(peak_mem, 2), "limit": round(max(lim) / 2**30, 1) if lim else None}, "oom_kills": int(max(oom)) if oom else 0, - "suggested_runner": suggest(res["node"]["arch"], peak_mem, p95), + "history": week, }) + res["suggested_runner"], res["reason"] = suggest( + pod, res["node"]["arch"], week["peak_mem_gib"], week["cpu_p50"], week["cpu_peak"], runs, + res["oom_kills"]) or (None, None) notes = [] if duration < 120: notes.append("short job: few samples, low confidence") @@ -129,12 +178,18 @@ def analyse(job): return res +def analyse_run(url): + jobs = [j for j in jobs_for(url) + if j.get("runner_name") and ARC_RUNNER.match(j["runner_name"]) and j.get("completed_at")] + hist = history(re.search(r"github\.com/([^/]+/[^/]+)/", url).group(1), [j["name"] for j in jobs]) + return [analyse(j, hist) for j in jobs] + + def main(): args = [a for a in sys.argv[1:] if not a.startswith("--")] if not args: sys.exit(__doc__) - results = [analyse(j) for j in jobs_for(args[0]) - if j.get("runner_name") and ARC_RUNNER.match(j["runner_name"]) and j.get("completed_at")] + results = analyse_run(args[0]) if "--json" in sys.argv: print(json.dumps(results, indent=2)) return @@ -145,10 +200,12 @@ def main(): print(f"\n{r['job']} ({r['duration_s']}s, {r['conclusion']})\n runner {r['runner']}\n" f" node {n['instance_type']} {n['capacity']} ({n['vcpu']} vCPU)") if "cpu_cores" in r: - c, m = r["cpu_cores"], r["memory_gib"] + c, m, h = r["cpu_cores"], r["memory_gib"], r["history"] print(f" cpu avg {c['avg']} p95 {c['p95']} max {c['max']} cores (request {c['request']})\n" f" memory peak {m['peak']} GiB of {m['limit']} GiB limit OOM kills: {r['oom_kills']}\n" - f" suggest {r['suggested_runner']}") + f" {WINDOW} {h['runs']} runs, peak memory {h['peak_mem_gib']} GiB, " + f"CPU median {h['cpu_p50']} peak {h['cpu_peak']} cores\n" + f" suggest {r['suggested_runner'] or 'keep'} {r['reason'] or ''}") if r.get("note"): print(f" note {r['note']}") print(f" graphs {r['dashboard']}") diff --git a/.github/actions/arc-job-report/test_arc_job_report.py b/.github/actions/arc-job-report/test_arc_job_report.py index 0ab7db2..571a30e 100644 --- a/.github/actions/arc-job-report/test_arc_job_report.py +++ b/.github/actions/arc-job-report/test_arc_job_report.py @@ -5,6 +5,7 @@ sys.path.insert(0, str(Path(__file__).parent)) from arc_job_report import render # noqa: E402 +from job_resources import history, suggest # noqa: E402 def run(number, attempt=1, ended="2026-09-28T11:07:46Z", conclusion="success"): @@ -13,10 +14,14 @@ def run(number, attempt=1, ended="2026-09-28T11:07:46Z", conclusion="success"): "run_started_at": "2026-09-28T11:00:00Z", "updated_at": ended} -def job(name, duration, suggested="arc-arm64-4cpu-12gb"): - return {"job": name, "duration_s": duration, "runner": "arc-arm64-8cpu-24gb-bwt29-runner-2t8gj", +POD = "arc-arm64-8cpu-24gb-bwt29-runner-2t8gj" + + +def job(name, duration, suggested=None, reason=None): + return {"job": name, "duration_s": duration, "runner": POD, "dashboard": f"https://grafana/{name.replace(' ', '')}", "cpu_cores": {"avg": 1.0, "max": 2.0}, - "memory_gib": {"peak": 3.0, "limit": 24.0}, "suggested_runner": suggested} + "memory_gib": {"peak": 3.0, "limit": 24.0}, "suggested_runner": suggested, "reason": reason, + "history": {"runs": 40, "peak_mem_gib": 6.5, "cpu_p50": 1.2, "cpu_peak": 5.1}} def history_rows(body): @@ -48,23 +53,73 @@ def test_keeps_a_job_named_like_the_shards_out_of_the_shard_row(self): self.assertIn("| Unit Tests · shards [0](https://grafana/UnitTests-shard0) " "**[1](https://grafana/UnitTests-shard1)** | 1m 40s – 1m 50s |", body) - def test_marks_re_runs_and_lists_only_jobs_over_two_minutes_with_a_change(self): - fits = job("Unit Tests (Fast Feedback)", 150, suggested="arc-arm64-8cpu-24gb") + def test_marks_re_runs_and_lists_only_jobs_with_a_change_and_why(self): body = render(run(12, attempt=2, conclusion="failure"), - [job("Compile", 150), job("Plan test shards", 15), fits], None) + [job("Compile", 150, "arc-arm64-4cpu-12gb", "trial"), job("Plan test shards", 15)], None) self.assertIn("| ❌ | 7m 46s (re-run) | – |", history_rows(body)[0]) - self.assertIn("| [Unit Tests](https://grafana/UnitTests(FastFeedback)) | 2m 30s |", body) - self.assertIn("| [Compile](https://grafana/Compile) | ⬇️ 8cpu-24gb → 4cpu-12gb | 2.0 | 3.0 GiB |", body) + self.assertIn("1.3 × the job's peak over the last 7d", body) + self.assertIn("| Job | Change | Why | Peak memory (7d) | CPU median / peak | Runs (7d) |", body) + self.assertIn("| [Compile](https://grafana/Compile) | ⬇️ 8cpu-24gb → 4cpu-12gb | trial: may be slower | " + "6.5 GiB | 1.2 / 5.1 | 40 |", body) self.assertNotIn("[Plan test shards](https://grafana/Plantestshards) | ⬇️", body) self.assertIn("· 💡 1 runner change", body) def test_omits_the_suggestion_table_when_every_runner_fits(self): - body = render(run(12), [job("Compile", 150, suggested="arc-arm64-8cpu-24gb")], None) + body = render(run(12), [job("Compile", 150)], None) self.assertNotIn("Suggested runner changes", body) self.assertNotIn("💡", body) +def sized(peak_mem_gib=3.0, cpu_p50=1.0, cpu_peak=2.0, runs=40, oom=0, runner=POD): + return suggest(runner, "arm64", peak_mem_gib, cpu_p50, cpu_peak, runs, oom) + + +class SuggestTest(unittest.TestCase): + def test_memory_is_a_hard_floor_with_headroom(self): + self.assertEqual(sized(peak_mem_gib=19.0), ("arc-arm64-16cpu-48gb", "memory")) + self.assertIsNone(sized(peak_mem_gib=18.0, cpu_p50=5.0)) + + def test_an_oom_kill_always_moves_up_a_size(self): + self.assertEqual(sized(peak_mem_gib=10.0, oom=1), ("arc-arm64-16cpu-48gb", "memory")) + + def test_cpu_bound_most_of_the_job_suggests_a_bigger_size_as_faster(self): + self.assertEqual(sized(peak_mem_gib=5.0, cpu_p50=7.0, cpu_peak=8.0), ("arc-arm64-16cpu-48gb", "faster")) + + def test_downsizes_to_the_smallest_size_holding_memory_and_median_cpu(self): + self.assertEqual(sized(peak_mem_gib=2.0, cpu_p50=0.8, cpu_peak=1.5), ("arc-arm64-2cpu-4gb", "fits")) + self.assertEqual(sized(peak_mem_gib=2.0, cpu_p50=1.5, cpu_peak=5.0), ("arc-arm64-2cpu-4gb", "trial")) + self.assertEqual(sized(peak_mem_gib=2.0, cpu_p50=3.0, cpu_peak=5.0), ("arc-arm64-4cpu-12gb", "trial")) + + def test_never_downsizes_on_fewer_runs_than_the_minimum(self): + self.assertIsNone(sized(peak_mem_gib=2.0, runs=4)) + self.assertEqual(sized(peak_mem_gib=30.0, runs=0), ("arc-arm64-16cpu-48gb", "memory")) + + def test_leaves_runners_outside_the_catalog_alone(self): + self.assertIsNone(sized(peak_mem_gib=40.0, runner="self-hosted-arm64-2xl-f968t-runner-878jv")) + + +class HistoryTest(unittest.TestCase): + def test_selects_the_jobs_by_escaped_name_and_reads_every_series_per_job(self): + import job_resources + queries = [] + + def fake_get(api, query): + queries.append(query) + value = {"max by (job_name) (max_over_time(arc:job_memory": 2 * 2**30} + v = next((v for k, v in value.items() if query.startswith(k)), 3) + return [{"metric": {"job_name": "Build (staging)"}, "value": [0, str(v)]}] + + job_resources.get, real = fake_get, job_resources.get + try: + h = history("monta-app/repo", ["Build (staging)", "Build (staging)", 'Lint "x"']) + finally: + job_resources.get = real + + self.assertEqual(h, {"Build (staging)": {"runs": 3, "peak_mem_gib": 2.0, "cpu_p50": 3.0, "cpu_peak": 3.0}}) + self.assertIn('job_name=~"Build \\\\(staging\\\\)|Lint \\"x\\""', queries[0]) + + if __name__ == "__main__": unittest.main() diff --git a/CLAUDE.md b/CLAUDE.md index e314606..cdabf3d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,11 +2,13 @@ ## Last Documentation Update - **Date**: 2026-10-13 -- **Latest SHA**: 7ac4f51 (check for newer commits) -- **Changes**: `argocd-resource-name` scopes the deploy's ArgoCD wait to one Deployment +- **Latest SHA**: a7aa579 (check for newer commits) +- **Changes**: `arc-job-report` sizes runners from a job's 7-day history: memory is a hard floor, CPU a labelled speed trade-off ## Recent Changes (2026-10-13) 1. **`argocd-resource-name` on the v2 deploy workflows**: `deploy-generic-v2.yml` and `deploy-kotlin-v2.yml` accept `argocd-resource-name` and pass it through `component-deploy-v2.yml` to `argocd-wait-sync`'s existing `resource-name` (kind stays `Deployment`). The wait then requires the expected revision, `Synced`, and that Deployment healthy, and ignores the app's aggregate health. Without it, any Degraded resource in the app fails the deploy on the first poll: service-monta-factory run 38388140776 failed with its Deployment `Healthy` because a CronJob's last Job had failed. Empty (the default) keeps tracking the whole app. +1. **`arc-job-report` reads `mimir-internal` at 15 s**: ARC's cAdvisor and kube-state-metrics series stopped arriving in `grafanacloud-prom` around 2026-10-01, so every job came back `no metrics` and no runner was ever suggested. `job_resources.py` queries `mimir-internal` (uid `ffzoywkuhm5tsd`) at a 15 s step with `rate[1m]`; the old 60 s / `rate[2m]` reported 2-minute averages and hid CPU bursts (monorepo Lint p90 peak 1.8 → 3.3 cores). +1. **`arc-job-report` sizes runners from the job's last 7 days**: one run is not safe to size on, because cache hits change a job's peak memory a lot (monorepo Typecheck: 1.5 GB typical, 13.1 GB worst). The history comes from the `arc:job_memory_working_set_bytes:max1m` and `arc:job_cpu_usage_cores:rate1m` recording rules in monta-app/terraform, which join each runner pod to its GitHub job (`repository`, `job_name`, `workflow_run_id`) once a minute. Doing that join at read time over 7 days OOM-killed the mimir-internal query-frontends, so the report must only read the recorded series. Runners have a memory limit but only a CPU request, so memory is a hard floor (1.3 × the 7-day peak; an OOM kill always moves up a size) and CPU is a labelled trade-off: ⬆️ *CPU-bound, faster* when the median is ≥ 85% of the runner's vCPU; a smaller size must cover the median and is *fits* or *trial: may be slower* depending on the peak. Smaller sizes need 5 runs in the window; there is no backfill, so they start a week after the rules are applied. The old 2-minute cut-off is gone, since the history covers short jobs. The local path uses `gcx api` against the same datasource-proxy URL as CI, because `gcx metrics query` silently widens the step to about 100 points. ## Recent Changes (2026-10-07) 1. **Terraform plans are stored beside the state, not as Actions artifacts**: `terraform-stack.yml` writes the saved plan to `s3:///tfplans//.tfplan`, read from the initialised backend in `.terraform/terraform.tfstate`, and the apply downloads and then deletes it. A saved plan is a zip of the full prior state (`tfstate`, `tfstate-prev`) plus every variable value, `sensitive` or not, and Actions artifacts are downloadable by anyone with read access to the repository — the whole org. Verified locally: a `sensitive` data-source value was absent from the text plan and present in `terraform show -json tfplan`. The shared `tfplans/` prefix is what lets one bucket lifecycle rule expire unapplied plans and the noncurrent versions left by the post-apply delete (S3 lifecycle filters only match a literal key prefix); the cost is that roles scoped to their own state prefix need `tfplans//*` added. Stacks must use the `s3` backend; anything else fails with `Unsupported backend`. `plan-retention-days` and the `Slug the stack path` step are gone, and callers no longer need `actions: write`. The runner needs the AWS CLI. Callers pinned to an older SHA keep uploading artifacts until they bump, and a PR planned under the old pin finds no plan after the bump, so it needs a fresh plan or a `recover`.