Repository navigation
fix(daemon): never kill a live daemon for being slow, and survive it when one is replaced - #142
Merged
Merged
Conversation
A live-but-slow daemon was SIGTERMed two seconds after it missed two 1.5s /__ping__ probes (2026-10-19, this machine). Three concurrent CLI calls timed out, `_classify` folded probe kind `timeout` into FOREIGN, `ensure()` spawned a `serve`, and that serve's port reclaim treated one negative 1.0s ping as proof the incumbent was stale. The daemon was never wedged: it was still accepting ws clients and it shut down gracefully on the signal. - verdict state UNRESPONSIVE: `timeout`/`garbage` are no longer FOREIGN. `ensure()` takes no action for it (same early return as UNDECIDED), so silence is never grounds to spawn or to signal. FOREIGN keeps its meaning: something answered HTTP, just not our pong. - escalation before concluding: `_ipc.probes_until_answered` re-asks inside a bounded 6s window, so a merely busy daemon answers and is diagnosed up/stale. Only silence for the whole window is concluded. Cost is paid only by the pathological shape: refused->DOWN 0.00s, healthy 1-50ms, silent holder 7.5s single / 9.0s confirm. - the reclaim gates on that same escalated evidence, and records its decision as a `LIFECYCLE reclaim` line (pid, endpoint, ports, per-probe kind@latency, reason, initiator) through `_ipc.log_lifecycle`'s direct file append. It used to leave no trace at all: it ran 39 lines before `_wire_logging()`, and ensure() spawns serve with stdout/stderr=devnull. `_wire_logging()` now runs first and is idempotent. - the explicit paths (`stop`, and `restart`'s incumbent stop) may now act on proven identity when the ping fails: `_stale.unresponsive_daemon` requires lsof + a browserwright cmdline + our own runtime dir, and death is then measured against the process table. Before this, a wedged daemon could not be stopped by any verb while the implicit path killed on the weakest possible evidence. Tests: 31 assertions fail against the pre-change source and pass after. mise run test: 1012 passed / 105 skipped (daemon), 338 (skill), 5 (pi). mise run lint: clean.
Three agent calls died on 2026-10-19 with "no browserwright extension is connected ... open chrome://extensions" while the extension was connected and healthy: the daemon was being replaced under the call and had pulled its own relay away first. Every word of that error blamed the one layer that was fine. - the daemon now names its own reason: `DaemonShuttingDown` (a marker carried through the JSON-RPC error envelope) is raised from `ensure_session_drivable` both on arrival and inside the readiness wait, and the flag is set first thing in `_graceful_shutdown`. The extension text is untouched and still used when the extension really is what is missing. - the client survives the replacement: `CDPTransportClosed` separates "the socket died with no reply" from "the daemon answered with an error", and `_call_until_daemon_returns` retries only the two seams that run strictly before a request can reach the executor — the `ensureExecutor` verb and the `/exec` connect. Each attempt re-resolves the endpoint (the cached ws url is invalidated), so the retry dials the daemon that answers now. - the send/recv pair still has NO retry, and that is the rule the change is built around: once a request frame may be on the wire, replaying it would run the caller's code a second time. - ~9s of retries (13 x 0.75s, the cold-start precedent), capped by whatever the caller's `--timeout` has left (ADR-0014). A first-attempt "nothing is running" is still refused immediately; retrying it would only make every no-daemon call ~9s slower to say the same thing. - exhausted retries name the daemon, state that the browser and extension are not involved, append a one-shot `/__ping__` observation (ADR-0013 rule 3), and end with `fix="retry the command"`. Tests: 2 new daemon tests quote the exact incident message and fail at HEAD; 4 of the 8 new client tests fail at HEAD. Existing tests changed only where a signature or a budget moved; `test_adr0013_diagnose_before_blame.py` now also scans `daemon/errors.py`, which widens the banned-word net. mise run test: 1012 passed / 107 skipped (daemon), 346 (skill), 5 (pi). mise run lint: clean.
A readable, repeatable demonstration that the 2026-10-19 failure no longer
reproduces, and the cheapest way for a reviewer to see the new rule work:
same four endpoint shapes, one short probe each (the only evidence the old code
had) against what `ensure()` concludes.
slow (3s) but alive probe=timeout ensure()=up spawn=no stop=no
accepts, never answers probe=timeout ensure()=unresponsive spawn=no stop=no
healthy probe=ours ensure()=up spawn=no stop=no
nothing listening probe=refused ensure()=down spawn=yes stop=no
Hermetic by construction: it sets `BS_HOME`/`TMPDIR` aside before importing the
package and replaces `daemon_endpoint`, `_spawn_detached` and `run_verb` with
fakes, so it cannot touch the machine-global daemon. That is the lesson from
this incident's own investigation: an earlier version of this replay ran a dev
checkout (version 0.0.0) against the default endpoint, which the lifecycle
correctly read as a stale daemon and replaced — killing a live one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
On 2026-10-19 an agent-driven browser call on this machine died with:
Every word of that was wrong. The extension was connected and healthy the whole
time. A machine-global daemon that was alive but momentarily slow (heavy
machine load; it had been idle for ~53 minutes) was SIGTERMed two seconds after
three concurrent CLI calls missed two 1.5s
/__ping__probes. It was neverwedged: it was still accepting ws clients, and it shut down gracefully on the
signal.
Root cause — three defects, all confirmed by tracing every signal path
daemon_lifecycle._classifyfolded probe kindstimeout/garbageintoFOREIGN, andensure()spawns a replacementserveforFOREIGN. ADR-0013rule 2 requires an unprovable holder to be
needs-human, never an automatickill or replacement target — and its "two consecutive agreeing probes" guard
is blind here, because two timeouts agree with each other.
_reclaim_stale_daemon_ports— run by every freshserve— had no secondprobe, no agreement rule and no window. Its identity gates were strict
(
lsof+browserwrightcmdline + same runtime dir); its liveness evidencewas one ping. That is the path that killed the daemon.
to find: it ran 39 lines before
_wire_logging(), so its warnings reachedonly
logging.lastResort, andensure()spawnsservewithstdout=DEVNULL, stderr=DEVNULL. An ensure-spawned serve could kill a daemonand leave no trace anywhere. (The first hypothesis — "zero reclaim lines in
nine days of logs proves the path never ran" — was wrong for exactly this
reason.)
And the asymmetry underneath all three: the explicit verbs refused to signal
anything they could not ping, so a genuinely wedged daemon could not be stopped
by any verb, while the implicit path killed on the weakest possible evidence.
What changed
426ac5b— silence is not proof of a strangerUNRESPONSIVEfortimeout/garbage;FOREIGNkeeps itsmeaning ("something answered HTTP, just not our pong").
ensure()takes noaction for it — no spawn, no stop.
_ipc.probes_until_answeredre-asks inside abounded 6s window, so a merely busy daemon answers and is diagnosed
up/stale. Only continuous silence for the whole window is concluded.a
LIFECYCLE reclaimline (pid, endpoint, ports, per-probekind@latency,reason, initiator) through
_ipc.log_lifecycle's direct file append._wire_logging()now runs first and is idempotent.stopandrestart's incumbent stop may act onproven identity when the ping fails (
_stale.unresponsive_daemon: lsof +browserwright cmdline + our own runtime dir), with death measured against the
process table. The "never signal a process we cannot prove is browserwright"
rule is unchanged.
71ae538— survive a replaced daemon, and stop blaming the extensionDaemonShuttingDown(a marker carried throughthe JSON-RPC error envelope), raised on arrival and inside the readiness
wait, set first thing in
_graceful_shutdown. The extension text is untouchedand still used when the extension really is what is missing.
CDPTransportClosedseparates "the socketdied with no reply" from "the daemon answered with an error", and
_call_until_daemon_returnsretries only the two seams that run strictlybefore a request can reach the executor — the
ensureExecutorverb and the/execconnect — re-resolving the endpoint on every attempt.around: once a request frame may be on the wire, replaying it would run the
caller's code twice.
caller's
--timeouthas left (ADR-0014). A first-attempt "nothing is running"is still refused immediately.
involved, append a one-shot
/__ping__observation (ADR-0013 rule 3), and endwith
fix="retry the command".992a238…— "no LaunchAgent" is a supported ownership moderestartno longer requires a plist: without one it stops the incumbent,spawns a detached
serve, and proves the launchd path's own post-condition(new pid ≠ old pid,
/__status__reports the expected version), naming fivedistinct ways to fail.
install/uninstalland the launchd path areunchanged. (Before this,
upgrade-global— which callsrestart— died atthe restart step on a plist-less machine.)
daemon/config.pyfalls back to$BS_HOME/config.toml(resolved through the product's own home, so
tests/conftest.py'sBS_HOMEredirect keeps the suite off a developer's real config).
daemon_urldeliberately still requires a named file, so an auto-discovered config can
never re-point every client.
LIFECYCLE signal sig=SIGTERM …),without duplicating the CLI's
LIFECYCLE stopwhen a verb asked for it.daemon ownershipand the
signallifecycle event.Verification
mise run test: daemon 1034 passed / 105 skipped, skill 346 passed,evals 13/13, pi 5/5 (~49s).
mise run lint: clean.Red before green, verified three ways: reverting
src/fails every newassertion (31 in the first change alone); targeted mutations (drop the version
proof, drop the exited-child observation, drop the stop-line suppression, drop
the config fallback, drop the
run_servewiring) each fail exactly the testthat pins them.
The incident itself, replayed —
scripts/replay-daemon-silence-incident.py(hermetic; it neutralises
daemon_endpoint/_spawn_detached/run_verbandredirects
BS_HOME/TMPDIR):Row 1 is the fix: the same endpoint still reads
timeoutto the one shortprobe the old code acted on, and
ensure()no longer kills or replaces it.Row 4 is the guarantee that recovery still works.
What this does NOT do
reclaimed. The fix raises the evidence bar from "one negative 1s ping" to
"7.5s of continuous silence", it does not make a frozen daemon immortal. Client
survival (
71ae538) is what makes the remaining case non-fatal.confirm=True) instead of being killed. No other shape pays: a healthy daemonis 1–50ms, a dead one 0ms.
restart's proof window is stilltimeout=5s(unchanged from the launchdpath), so a very slow cold start can report a failed restart while the new
daemon is still coming up.
facade_hostdurability and plist-lessrestartonlyreach a machine's global install once this is released.
Follow-ups (not in this PR)
write
facade_hostinto$BS_HOME/config.toml,restart(now plist-less),verify both the loopback and the tailnet endpoint, then
uninstall.