fix(actions): read arc-job-report metrics from mimir-internal at 15 s

.github/actions/arc-job-report/README.md

@@ -62,21 +62,20 @@ an ARC runner (`arc-*` / `self-hosted-*`) are skipped.
 | Input | Required | Description |
 |---|---|---|
 | `github-token` | yes | Reads the run's jobs and writes the comment. Needs `actions: read` and `pull-requests: write`. |
-| `grafana-token` | yes | Grafana service-account token that can read the `grafanacloud-prom` datasource. |
+| `grafana-token` | yes | Grafana service-account token that can read the `mimir-internal` Prometheus datasource. |
 | `run-url` | no | Run to report on. Defaults to the run that triggered the `workflow_run` event. |
 
 ## How to read the numbers
 
-Metrics come from cAdvisor and kube-state-metrics, scraped every **60 s**.
+Metrics come from cAdvisor and kube-state-metrics in `mimir-internal`, scraped every **15 s**.
 
 - **Peak memory is reliable.** It is the highest working-set sample over the job.
-- **CPU is not, for short jobs.** It is the highest 60 s average, so short bursts
+- **CPU is not, for short jobs.** It is the highest 1 min average, so short bursts
   are averaged away. In service-ocpi, the CPU-based rule suggested 4 vCPU for the
   integration shards, but a trial run made every shard 17–97 s slower. Trial a
   smaller runner before moving to it.
 - **Jobs under 2 minutes are left out of the suggestions.** They get too few
-  samples. Jobs under a minute have none at all (CPU needs two samples), so they
-  show `–`. There is no point waiting for them: a runner pod's samples stop once
+  samples. There is no point waiting for them: a runner pod's samples stop once
   its job ends and the pod is deleted.
 
 The suggested size is the smallest catalog runner with memory ≥ 1.3 × peak and

.github/actions/arc-job-report/action.yml

@@ -10,7 +10,7 @@ inputs:
     required: true
   grafana-token:
     description: >-
-      Grafana service-account token with read access to the `grafanacloud-prom` datasource,
+      Grafana service-account token with read access to the `mimir-internal` Prometheus datasource,
       for example `secrets.GRAFANA_CLOUD_TOKEN`.
     required: true
   run-url:

.github/actions/arc-job-report/arc_job_report.py

@@ -127,7 +127,7 @@ def render(run, results, previous_body):
         "|---|---|",
         *job_rows(results),
         "",
-        *(["💡 **Suggested runner changes.** Memory is the exact peak; CPU is the max of 60 s averages, so "
+        *(["💡 **Suggested runner changes.** Memory is the exact peak; CPU is the max of 1 min averages, so "
            "trial before downsizing on it. Jobs under 2 min have too few samples and are left out.",
            "",
            "| Job | Change | CPU | Memory |",

.github/actions/arc-job-report/job_resources.py

@@ -6,8 +6,7 @@
 
 Needs `gh` (authenticated) and either GRAFANA_TOKEN (a Grafana service-account token with read
 access to the Prometheus datasource) or `gcx` (a context with that datasource).
-A job's runner name (GitHub API `runner_name`) is its ARC pod name; metrics are scraped every 60 s,
-so jobs shorter than ~2 minutes get only a sample or two.
+A job's runner name (GitHub API `runner_name`) is its ARC pod name; metrics are scraped every 15 s.
 
 Copied from monta-app/kube-manifests infra/arc/scripts/job-resources.py @ dd43b0172; the only
 change is the GRAFANA_TOKEN query path in prom().
@@ -22,7 +21,7 @@
 import urllib.request
 from datetime import datetime
 
-DATASOURCE = "grafanacloud-prom"
+DATASOURCE = "ffzoywkuhm5tsd"  # mimir-internal
 GRAFANA = "https://montaapp.grafana.net"
 ARC_RUNNER = re.compile(r"^(self-hosted|arc)-")
 # (name, arch, vCPU, memory GiB) -- keep in sync with infra/arc/README.md
@@ -40,7 +39,7 @@ def sh(*args):
     return subprocess.run(args, capture_output=True, text=True, check=True).stdout
 
 
-def prom(expr, start, end, step=60):
+def prom(expr, start, end, step=15):
     token = os.environ.get("GRAFANA_TOKEN")
     if token:
         query = urllib.parse.urlencode({"query": expr, "start": start, "end": end, "step": step})
@@ -88,7 +87,7 @@ def analyse(job):
     pod, start, end = job["runner_name"], ts(job["started_at"]), ts(job["completed_at"])
     sel = f'cluster="internal", namespace="arc", pod="{pod}"'
     lo, hi = start - 60, end + 60
-    cpu = values(prom(f'sum(rate(container_cpu_usage_seconds_total{{{sel}, container=~"runner|dind"}}[2m]))', lo, hi))
+    cpu = values(prom(f'sum(rate(container_cpu_usage_seconds_total{{{sel}, container=~"runner|dind"}}[1m]))', lo, hi))
     mem = values(prom(f'sum(container_memory_working_set_bytes{{{sel}, container=~"runner|dind"}})', lo, hi))
     req = values(prom(f'max(kube_pod_container_resource_requests{{{sel}, container="runner", resource="cpu"}})', lo, hi))
     lim = values(prom(f'max(kube_pod_container_resource_limits{{{sel}, container="runner", resource="memory"}})', lo, hi))
@@ -120,7 +119,7 @@ def analyse(job):
     })
     notes = []
     if duration < 120:
-        notes.append("short job: only a few 60 s samples, low confidence")
+        notes.append("short job: few samples, low confidence")
     if res["node"]["vcpu"] and p95 >= 0.85 * res["node"]["vcpu"]:
         notes.append("CPU-bound on this node: a larger size may make it faster")
     if res["oom_kills"]: