Repository navigation
fix(actions): read arc-job-report metrics from mimir-internal at 15 s - #381
Merged
Merged
Conversation
ARC cAdvisor and kube-state-metrics stopped landing in grafanacloud-prom around 2026-10-01, so every job in the report came back as "no metrics" and no runner suggestions were made. Query mimir-internal instead. mimir-internal scrapes every 15 s, so query at a 15 s step with a 1 min rate window instead of 60 s / 2 min, which caught only 2-minute averages and hid CPU bursts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JesperTerkelsen
marked this pull request as ready for review
October 13, 2026 14:25
JesperTerkelsen
requested review from
maoanran
and removed request for
a team
October 13, 2026 14:25
sudheer-monta
approved these changes
Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The ARC job report has had no resource data since about 2026-10-01. ARC's cAdvisor and kube-state-metrics series stopped arriving in
grafanacloud-prom; they now live inmimir-internal. Every job came back asno metrics, so the comments showed durations only and never suggested a runner change. An example is https://github.com/monta-app/monorepo-typescript/pull/11680#issuecomment-6147620737.This PR:
mimir-internal(uidffzoywkuhm5tsd) instead ofgrafanacloud-prom.rate[1m], matching the actual 15 s scrape interval (about 20 samples per 5 min on runner pods). It previously used 60 s /rate[2m]. At the old resolution, peak CPU was effectively a 2-minute average and hid bursts. Across about 280 runs of each monorepo PR job, p90 peak CPU goes from 1.8 → 3.3 cores for Lint, 2.9 → 5.7 for Typecheck and 2.7 → 5.2 for the E2E jobs. A 30 s window is higher still: 5.0, 7.5 and 6.1.action.ymland the comment text from "60 s" to 15 s.Testing
job_resources.pyagainst https://github.com/monta-app/monorepo-typescript/actions/runs/38407948093:no metricsfor every job.python3 -m unittest discover -s .github/actions/arc-job-report -p 'test_*.py'passes.Gaps
GRAFANA_CLOUD_TOKENservice account can querymimir-internal, because reading datasource permissions needs a scope I don't have. If it can't, the request raises and the report job fails visibly; the first monorepo run after merge will tell.60s, which makes$__rate_interval4 min on every dashboard using it, including ARC Job Resources. That is a separate change to a shared datasource and isn't part of this PR.🤖 Generated with Claude Code