[luthermonson]

Turns the self-healing runner version subsystem from #220 (library code, flag-off, no callers) into a first-class, default-on product behavior for v0.3.0. Eight commits, in dependency order: config → key → manifest pipeline → controller/runtime → scheduler → daemon wiring → e2e/CI → docs.

What is now wired and on by default

A node with no [runner.self_heal] table at all gets, on both the host scheduler daemon and the in-VM --containerd-only worker:

• Boot-verify (runnerversion.Controller, started from serve right after runtime + providers): loads the signed manifest (keeps the last verified one across fetch failures, ignores other channels), re-hashes the embedded runner, verifies the cache, probes the official Linux image's /home/runner, and re-runs hourly and on every breaker trip.
• Zero-work detector + circuit breaker (scheduler.Config.RunnerHealth): Registered from trackRunning (only for providers that report runner names), JobStarted from the in_progress event and from any Busy busy-probe answer (so a webhook outage does not trip it on runners that are working), Exited from the three wait-goroutines with killedByUs = jobCtx.Err() != nil, Allow(platform) as the last gate in admitDispatch. A refused job goes on the retry ladder, not the 30-minute reconcile.
• Mount-over-stale-official-image: runtime.Config.RunnerOverride is populated with one value implementing both seams — ImageOverride for official images and the new MountedRunnerSource (verified-cache runner on custom-image/Windows paths on the next job, no restart). Engage resets the linux breaker.
• Break-glass two-source fetch: on the host through the first GitHub provider's authenticated client (runners/downloads cross-check); in the VM worker when the staged config carries a PAT. Any missing dependency disables the fetch (logged once) and the override falls back to the embedded tree.
• Kill-switch: [runner.self_heal] enabled = false leaves RunnerOverride and RunnerHealth nil — byte-for-byte the pre-0.3.0 daemon. enabled/auto_upgrade/image_override are *bool (nil → true; sub-switches false whenever the kill-switch is), the same convention as [vm.linux] enabled. Validate() is now called from config.Load and fails fast; manifest_url has a built-in default so an empty table validates.

Signed-manifest pipeline

• pkg/runnerversion/keys/runner-manifest.pub is a real key now (id CCFDB74C12C7F433, pinned by TestEmbeddedPublicKeyIsReal). Generated with the new mage manifest:keygen (pure-Go minisign keypair; unencrypted secret-key layout compatible with minisign -G -W).
• mage manifest:build composes dist/runner-manifest.json from mage/download's pinned table (RunnerVersion, new RunnerFloor = 2.338.0, pinnedSHA256 via RunnerArtifacts()) — hashes live in exactly one place. manifest:sign signs with RUNNER_MANIFEST_MINISIGN_KEY; manifest:verify checks the pair against the compiled-in key (the CI gate that the secret matches the trust root).
• .github/workflows/runner-manifest.yml (release: published + workflow_dispatch with floor/channel/rollback inputs) publishes both files to a rolling prerelease with a fixed tag (runner-manifest, canary: runner-manifest-canary), make_latest: false so it can never take the releases/latest slot ephemerd upgrade follows. That URL is the config default.

Fixes found along the way

• The sticky breaker caveat from the feat(runner): self-healing runner version subsystem (foundation, flag-off) #220 review: a verified manifest is authoritative and clears an override engaged on the breaker alone; no manifest never clears (so an outage cannot flap it and sacrifice three runners per poll). Breaker open + manifest-says-healthy is not an override; the log says to republish with a higher floor.
• ProbeImageRunner never applied the containerd namespace to its context (latent feat(runner): self-healing runner version subsystem (foundation, flag-off) #220 bug); it now does, holds a lease over the view so GC cannot remove it mid-probe, and SweepRunnerProbeViews removes leaked ephemerd-runner-probe-* views at boot.
• Overlay ownership: overlayfs takes the merged root's owner from the upperdir root (which prepareJobRunnerTree always creates, as root). Under the override the official image's uid-1001 runner could not have written .runner/_work. matchTreeOwner now chowns the per-job upper root to the lowerdir's owner (no-op for the root-owned custom-image path). The e2e asserts the write as uid 1001.

Required manual step (one time)

Add the repository secret RUNNER_MANIFEST_MINISIGN_KEY on ephpm/ephemerd: the contents of the runner-manifest.key file that pairs with the compiled-in public key. It was generated off-repo and is not in this PR or the tree. (If it is ever lost: mage manifest:keygen, commit the new .pub, update the pinned id in minisign_test.go, replace the secret.)

Before nodes can boot-verify

The manifest must be published once: run the Runner manifest workflow (workflow_dispatch, defaults) or publish any ephemerd release (which triggers it). Until then every node logs "manifest unavailable; no verified manifest yet (floor unknown)" hourly at warn, and self-heals on the breaker path only (override from the embedded tree; no fetch, since nothing verified says what to fetch).

Not fully proven locally

• The containerd e2e (test/e2e/runner_override_test.go: job on ghcr.io/actions/actions-runner:2.336.0 reports the image's runner without the override and the host's newer runner with it, writes as uid 1001 into the per-job upperdir, never into the shared tree; Clear restores; probe leaks no view) is gated linux && e2e && privileged like the rest of the suite. It compiles under those tags (vetted cross-OS) and ci.yml runs it verbosely so its PASS or SKIP is on the record — on the current unprivileged CI runner that will be a SKIP, as for the other privileged e2e. Run mage e2eall on a privileged box for the on-metal proof.
• The publish workflow has not run (needs the secret). Its three mage targets are covered end-to-end by mage/manifest tests with a throwaway key.
• make_latest/overwrite_files behavior of softprops/action-gh-release@v2 is from its documented inputs, not exercised here.

Deferred (documented in the arch doc)

Auto-PR for the pin; ephemerd_runner_pin_age_seconds; override/breaker state on /healthz and the Status RPC; host→VM blob push for Windows/macOS hosts (the VM fetches for itself when it has a token); an explicit cordon on below-floor-with-fetch-unavailable (the breaker stops the thrash instead).

Verification

go build ./...; go vet (windows + CGO_ENABLED=0 GOOS=linux, incl. the e2e,privileged tags); golangci-lint 2.11.4 on every touched package: 0 issues; go test on config, runnerversion (incl. 10 new controller tests), runtime, scheduler (x2), metrics, github, providers/github, cmd/ephemerd, mage/download, mage/manifest: all green.

Follow-up (approved additions)

Break-glass fetch of GitHub's latest (2393284)

The reactive loop could not heal the case it exists for: GitHub ships a new runner and raises its floor before anyone re-signs the manifest — Upgrader.Run refused with ErrSourcesDisagree (GitHub ahead of the signed pin) and the fleet sat protected-but-stuck. Upgrader.breakGlassEligible/breakGlass is now the ONE path where GitHub's REST API is the sole authority for a runner binary, gated hard:

• the zero-work breaker for this runner's platform class is OPEN (BreakGlassOpen, wired by the controller; nil on the VM worker) — sources disagreeing with the breaker closed still refuse and alert; steady state is not loosened;
• roll-forward only: GitHub strictly newer than the signed manifest and than the runner the node already has (cache or embedded). GitHub equal-with-other-hash or behind the pin is refused as spoof/regression;
• one shot per breaker cooldown (ErrBreakGlassCooldown);
• auto_upgrade, max_archive_mb, min_free_gb, SHA pin (GitHub's sha256_checksum over the whole stream before extraction) unchanged.

Audit trail: sidecar Sources = ["github-runners-downloads"] only (Sidecar.IsBreakGlass); manifest not recorded as applied; ephemerd_runner_version_info{source="github-breakglass"} (selector reads the sidecar on every Select, so it survives restarts); ephemerd_runner_breakglass_active gauge + _total counter; fixed WARN self-heal fetched un-manifest-signed runner from GitHub to clear an open breaker; re-sign the manifest (and rebuild the image) to make this permanent, repeated each reconcile. On the image path the break-glass tree is mounted over the image (DecideImageOverride: breaker open AND node holds a tree newer than the manifest's version) and kept until a manifest covers it. Revert to signed: a re-signed manifest naming a version ≥ the break-glass one makes alreadyCurrent record the manifest as a second source on the same blob (no download), label → verified-cache, gauge → 0; a newer manifest is a normal two-source upgrade. Decision-matrix tests: pkg/runnerversion/breakglass_test.go (a–e), TestControllerBreakGlassThenManifestCatchUp, TestDecideImageOverride break-glass cases.

Poll demoted to a 6h backstop (a4979fc)

DefaultSelfHealPollInterval 1h → 6h (poll_interval still configurable). The breaker's OnDegraded → Kick is the primary trigger; the poll only propagates a known-good pin to not-yet-failing nodes and catches a dropped kick.

---

https://claude.ai/code/session_01CYdPQrMwFwZJrsnAswyH9d