[luthermonson]

What this is

Design doc + a reviewable, flag-off foundation for a self-healing runner version subsystem. It addresses the runner-floor stall: GitHub's undocumented minimum runner version moved past what the fleet runs, every JIT runner registered → was refused → exited 0, and the fleet sat idle for ~a day because (a) nothing distinguished "ran a job and exited" from "never got a job and exited" and (b) the only fix was a rebuild.

Nothing is wired. cmd/ephemerd and pkg/scheduler do not call any of this; runtime.Config.RunnerOverride is nil everywhere. [runner.self_heal] enabled, image_override and auto_upgrade all default to false. mage/download/download.go (RunnerVersion, pinnedSHA256, Alpine pins) and .github/workflows/ are untouched. Draft on purpose — please review the design before anything gets wired.

Correction from live fleet debugging (2026-11-10)

The first draft assumed the runner the fleet runs is the embedded tarball. For Linux — the dominant path — it is not. runner-ci-linux is FROM ghcr.io/actions/actions-runner:latest; isOfficialRunnerImage (pkg/runtime/runtime.go) treats it as official, so Create runs the IMAGE's /home/runner/run.sh with needsRunnerMount=false. The embedded runner is only ever mounted for custom images and on Windows/macOS. The image was at 2.336.0 (below floor) and every Linux job died regardless of the embedded pin.

Also confirmed on metal: patching the image's overlayfs lower layer in place is unreliable (page cache serves the stale runner to new container mounts). So the Linux mechanism is a bind/overlay mount of a verified fresh runner tree over /home/runner, using the per-job overlay machinery prepareJobRunnerTree already provides for custom images — not a layer edit. The doc now has a dedicated section (The Linux official-image path) and this PR implements the seam for it.

Design — docs/arch/self-healing-runner.md

Mechanisms, all preserving SHA pinning (there is a dedicated section walking every path):

1. Boot-time verify — re-hash the embedded archive and any cached one, probe the official Linux image's /home/runner (read-only view snapshot + Runner.Listener.deps.json, which carries "Runner.Listener/2.337.0" — verified against the real archive), compare all against a minisign-signed runner-manifest.json (schema: version, floor, channel stable|canary, rollback, published_at, per-platform {url, sha256}), refuse work below floor (cordon, not thrash).
2. Zero-work detector + circuit breaker — registered → self-exit within 120 s → no in_progress/busy signal naming that runner. exit_code is explicitly not an input. K=3 consecutive per platform opens the breaker (provisioning suspended, pool degraded, alert fired once), 15 m cooldown → one half-open probe. Optional _diag grep corroborates the deprecation message.
   3a. Image override (Linux) — when the image's runner is below floor (or the breaker is open with a newer tree on the node), engage an overlay of the best verified runner tree over /home/runner on every subsequent job; the tree is chowned to the image's runner uid/gid (probed) so _work/ can be created; loud fixed-string alert runner image override engaged: IMAGE RUNNER IS STALE + ephemerd_runner_image_override_active=1 until a rebuilt image is probed ≥ floor. A healthy image is never masked; a failed probe never overrides on a guess.
   3b. Break-glass verified auto-upgrade (host daemon only) — when 3a needs a tree the node doesn't have (today's shape: image 2.336.0 AND embedded 2.337.0 both below floor): resolve from the signed manifest AND GitHub GET …/actions/runners/downloads (sha256_checksum); both must agree on version+hash; fetch https-only with a free-space guard and size cap; stream-hash; verify-before-commit into a content-addressed store bounded to current+previous; atomic pointer swap; the next job uses it (via 3a on the image path, via the selector on mounted paths); alert; auto-PR with a deadline (designed, not implemented).

Host coverage for 3a: Linux hosts directly; macOS via the virtio-fs-shared data dir; Windows-host WSL VMs are a documented gap (the VM cannot see the host's tree; needs a host→VM verified blob push over Dispatch — deferred).

GitHub primitives were checked against live docs/API (runners/downloads schema; assets[].digest populated on v2.338.2; <!-- BEGIN SHA os-arch --> release-notes markers) with confidence noted per item. The doc also records the known prereq that mayfly never renders metrics bind_addr, so nothing is scraped fleet-wide yet.

Implemented

Package / file	What
pkg/runnerversion/manifest.go	Manifest, strict Validate (schema=1, X.Y.Z, floor ≤ version, https URLs whose basename must equal the upstream asset name, lower-hex sha)
pkg/runnerversion/minisign.go + keys/runner-manifest.pub	Pure-Go minisign verify (Ed25519 + BLAKE2b-512 prehash, global signature, key-id check). The embedded key is a placeholder; EmbeddedPublicKey() returns ErrNoPublicKey and every verify fails closed until an operator generates one offline (minisign -G). Sign() exists so tests round-trip; cross-checking against the real minisign CLI is a follow-up.
pkg/runnerversion/loader.go	HTTPManifestLoader (https-only, bounded, verify-then-parse)
pkg/runnerversion/store.go	CAS store: Put hashes the stream and commits only on match; Extract; atomic Swap/Rollback; Verify re-hashes at boot and drops corrupt blobs; Prune bounds to current+previous; temp cleanup; replay guard
pkg/runnerversion/fetch.go	https-only, diskspace.Check floor, Content-Length + limited-reader cap, single-flight per SHA
pkg/runnerversion/github.go	GitHubSource over go-github v72 ListRunnerApplicationDownloads / org variant
pkg/runnerversion/upgrader.go	two-source agreement → fetch → extract → swap; ErrDisabled before any I/O when AutoUpgrade=false; channel, replay, downgrade-without-rollback refusals
pkg/runnerversion/selector.go	embedded ∪ verified-cache for the mounted paths; strictly-higher cache wins; BootVerify now also runs the image probe and returns BootReport.Image (the override decision)
pkg/runnerversion/imageprobe.go	ProbeRunnerTree / ParseListenerDepsVersion: version + owner uid/gid of any runner tree
pkg/runnerversion/override.go	ImageOverride (engaged state the runtime consults; Engage fires the alert + gauge), DecideImageOverride (the decision table in the doc), PrepareOverrideTree (chown to the image's runner uid/gid)
pkg/runtime/runneroverride.go	RunnerOverrideSource interface + planRunnerMount, the pure decision that replaced the inline needsRunnerMount expression in Create. With a nil source the plan is the historical rule verbatim; with an engaged override and an official image on Linux it mounts the override tree at /home/runner through the existing prepareJobRunnerTree overlay. dind bindMappings follow the plan's mount.
pkg/runtime/runnerprobe_linux.go (+ _other.go)	Runtime.ProbeImageRunner: read-only view snapshot of the image rootfs + mount.WithTempMount (same technique as patchSnapshotRunner) + ProbeRunnerTree(/home/runner); view removed afterwards
pkg/runnerversion/detector.go	zero-work verdicts, per-platform breaker (closed/open/half-open), OnDegraded fired once per trip
pkg/runnerversion/diag.go	_diag/Runner_*.log tail scan for the refusal line
pkg/metrics/runnerversion.go	ephemerd_runner_version_info{version,source} (source now includes image-override), ephemerd_runner_pin_age_seconds, ephemerd_runner_zero_work_total{platform}, ephemerd_runner_breaker_open{platform}, ephemerd_runner_image_version_info{image,version}, ephemerd_runner_image_override_active, ephemerd_runner_manifest_errors_total{reason}
pkg/config/selfheal.go	[runner.self_heal] table incl. image_override; defaults; Validate(); one field added to RunnerConfig
pkg/runner/version.go	new file only: exported ParseVersion, ParseVersionString, CompareVersions, Manager.EmbeddedArchive, ExtractArchive. runner.go is byte-identical to main.

The one edit to existing behaviour-bearing code

pkg/runtime/runtime.go Create: needsRunnerMount := (customImage || windows) && … became plan := planRunnerMount(…) and the three uses of r.cfg.RunnerDir/r.cfg.RunnerMount in the mount block and dind bindMappings read from plan. TestPlanRunnerMount pins the historical rule (custom/official/Windows/unconfigured) and the new case. Config.RunnerOverride is nil in main.go, so the compiled daemon behaves exactly as before.

Tests

Unit tests for every file above (go test ./pkg/runnerversion/... ./pkg/metrics/... ./pkg/config/... + TestPlanRunnerMount in pkg/runtime + the three new pkg/runner tests), -race clean on pkg/runnerversion, go vet clean on windows and CGO_ENABLED=0 GOOS=linux, golangci-lint (repo config) clean on the touched packages, whole-module go build clean. Covered: every manifest validation branch; minisign round-trip in both modes, tampered data / tampered trusted comment / wrong key; store verify-before-commit leaves nothing behind on mismatch or oversize, swap/rollback/prune bounds, corrupt-blob drop, re-extract from a good blob, temp cleanup; fetcher https-only, low-disk refusal without touching the network, single-flight (5 callers → 1 request); upgrader disabled touches nothing, sources-disagree refuses without fetching, agreed-hash-but-bad-bytes refuses, replay, channel, downgrade/rollback; selector floor rules and the image path (both-below-floor → NeedFetch; cache lifts → Override; probe failure → no override); DecideImageOverride table incl. the exact 2026-11-10 shape; ImageOverride engage/clear + gauges; deps.json parse/probe; detector verdicts, threshold-once alert, half-open probe reopen/close, metrics.

Not unit-testable here: Runtime.ProbeImageRunner (needs containerd — belongs in test/e2e), the actual overlay-over-/home/runner job run (e2e on a Linux node: start a job on runner-ci-linux with the override engaged and assert Runner.Listener version from _diag), the real embedded archive path (TestEmbeddedArchiveMatchesFindTarball runs its full assertion only after mage download, like the existing TestFindTarball_PicksHighestVersion), and a real GitHub runners/downloads call (stubbed via the struct go-github returns).

Integration points for Phase 1 (not done here)

1. cmd/ephemerd/main.go passes rm.Dir() as RunnerDir at ~L357 (worker) and ~L677 (host). Those become sel.Dir() with sel := &runnerversion.Selector{Embedded: rm, Store: store}; it returns exactly rm.Dir() when there is no better verified tree.
2. runtime.Config.RunnerOverride = imageOverride (*runnerversion.ImageOverride) in the same runtime.New call — nil today.
3. sel.ProbeImage = func() (TreeInfo, error) { return rt.ProbeImageRunner(ctx, defaultLinuxImage) } and the boot/poll handler that turns BootReport.Image into Engage / Clear / fetch-then-engage.

Selector wraps *runner.Manager and does not change findTarball's own choice among embedded archives. If Phase 1 changes how runner.Version or the embed dir is populated, Selector.Select()/BootVerify are the only consumers of runner.Version / EmbeddedArchive() to reconcile.

Remaining wiring (next increment): Detector.Registered/JobStarted/Exited from trackRunning / handleInProgress / the two wait-goroutines (killedByUs = jobCtx.Err() != nil || orphan sweep || drain), Detector.Allow(platform) in admitDispatch, the manifest poll loop + alert routing, the auto-PR, and the Windows-host VM blob push.

Open questions for review

• Key custody: who holds the minisign secret key, and is ephpm/ephemerd releases (runner-manifest/<n> tag) the right place to publish the manifest?
• Should image_override be allowed to mount the embedded runner over the image without auto_upgrade (no download involved, just a newer tree we already pinned)? The code allows it when image_override=true; the doc describes it that way.
• Detector defaults (120 s window, K=3, 15 m cooldown) are proposed from the outage shape; happy to tune.
• Whether the auto-PR should reuse ci/runner-bump-workflow's editor rather than a second one.

https://claude.ai/code/session_01CYdPQrMwFwZJrsnAswyH9d