Recovery Reconciler: reconcile accepted tasks to verified terminal outcomes

[dnth]

Summary

Track the broader Recovery Reconciler: a FirstMate-owned supervision loop that continuously reconciles every accepted task toward a verified terminal outcome.

No accepted task may remain non-terminal indefinitely without observable progress and a clearly identified next owner.

This is broader than #155. Issue #155 should remain the first narrow, deterministic recovery adapter for persistent delivered-no-turn incidents. This issue owns normalized task evidence, task-wide reconciliation, semantic ambiguity handling, shadow/advisory rollout, and eventual bounded autonomy.

Tracking/specification only: creating this issue does not authorize implementation, destructive recovery, merge, or deployment. A separate captain instruction is required.

Problem

FirstMate can truthfully report durable delivery while a worker stops starting turns, finishes without reporting, waits indefinitely, or enters a contradictory state. A relaunch mechanism alone is insufficient because FirstMate must first determine whether work is complete, progressing, legitimately waiting, safely recoverable, or requires a decision.

A process restart is not recovery. Recovery requires task-relevant downstream progress and preservation of task identity and evidence.

Ownership boundary

FirstMate remains the sole lifecycle authority for:

• task identity and terminal state;
• worktrees, commits, uncommitted changes, instructions and spawn generations;
• validation/branch custody;
• receipts, completion gates, PR state and merge authority;
• recovery execution, retry budgets, audit and escalation.

The reconciler must extend existing FirstMate owners rather than become a competing orchestrator. Hermes supplies captain intent and receives durable results. Jev may classify bounded semantic ambiguity, but it cannot grant completion or authorize mutation.

Proposed state machine

1. Observe — bind canonical home, task, generation, endpoint, worktree, pending instruction, applicable completion contract and evidence provenance.
2. Suspect — a supported progress or delivery threshold has expired; observation failures remain explicit.
3. Reconcile — completion wins over replacement; productive work and bounded external waits return to observation; unresolved decisions route to their existing owner.
4. Candidate — an explicitly supported deterministic recovery predicate holds and task-wide budget remains.
5. Reserve — acquire FirstMate authority, task/generation exclusion and authoritative validation/branch custody reservation; recheck current evidence at the effect boundary.
6. Execute — journal intent before lifecycle effects and use the existing recovery actuator.
7. Observe replacement — verify the target instruction was handled and the expected downstream effect advanced.
8. Recovered, retry eligible, or escalated with work preserved.

Represent completion as stages rather than one boolean: implementation evidenced, delivery ready, awaiting authorized landing, and landed.

Classification contract

Deterministic policy owns:

• COMPLETE: all applicable stage-specific gates and authority checks pass.
• DONE_BUT_NOT_REPORTED: a ready/landed stage is proven but its delivery receipt is absent.
• STILL_WORKING: positive current activity or attributed validation/CI progress.
• WAITING_EXTERNAL: an identified dependency with a bounded recheck policy.
• WEDGED_RETRYABLE: a supported persistent wedge plus custody, generation, preservation and budget gates.
• BLOCKED_NEEDS_DECISION: a current keyed decision or authoritative blocked gate.
• FAILED_NONRETRYABLE: an explicit terminal cause or exhausted automatic policy.
• UNOBSERVABLE: required evidence is missing, stale, conflicting or failed.

These dimensions may overlap and therefore need precedence or separate structured fields, not one unconstrained model label.

Jev boundary

Jev is optional and advisory, introduced only for measured semantic ambiguity after deterministic evidence is exhausted.

• Feed it bounded, redacted, source-labeled evidence and closed choices including insufficient evidence.
• Persist prompt/model versions and evidence snapshot identity; discard stale responses.
• Never accept model-supplied commands, paths, ownership facts or policy exceptions.
• Confidence is not a custody certificate, completion gate or restart authorization.
• Timeout, model failure and low confidence leave deterministic supervision intact.

Do not call Jev on deterministic completion or #155's supported no-turn fast path.

Progress and retry policy

Use at most two automatic relaunches per task lifetime, spanning generations and incident identities. Reserve attempts durably before lifecycle effects. An uncertain crash boundary consumes the reserved attempt; a positively proven pre-effect refusal may release it.

Strong incident-recovery evidence:

• the target durable instruction is handled by the successor and its expected downstream effect is visible; or
• the relevant completion gate passes.

Activity that may cancel recovery but does not prove task success includes a correlated turn, attributed validation advancement, a newly evidenced criterion, or exact PR/check advancement for the current head.

Do not reset retry authority for a new PID, generation, turn count, enqueue, duplicate receipt, log timestamp, dirty-flag toggle, arbitrary HEAD change, repeated validation loop, or Jev confidence.

Rollout

1. Workers wedge with delivered-no-turn despite truthful receipts; need custody-aware auto-recovery #155 deterministic adapter — narrow supported cases, atomic custody, durable journal and post-action verification.
2. Shadow mode — record recommendation, evidence freshness, actual FirstMate action and later outcome; measure false restarts, missed recovery, duplicates, time to progress and unobservable rate.
3. Advisory mode — send bounded recommendations through existing durable supervision; FirstMate revalidates every precondition.
4. Bounded autonomy — enable only demonstrated deterministic cases with an opt-out and task-wide cap.
5. Add Jev only for a measured ambiguity class; never promote Jev-assisted restart merely from confidence.

Acceptance criteria

• One FirstMate-owned reconciliation state machine with explicit owner boundaries.
• Structured, provenance-bearing snapshots distinguish missing evidence from negative evidence.
• Completion uses the task's applicable receipt, validation, PR and merge-authority contract.
• Recovery candidates are generation-bound, idempotent and protected against duplicate/concurrent detectors.
• Every mutation requires atomic custody/exclusion and a fresh effect-boundary recheck.
• Worktree, commits, staged/untracked content, instructions and valid task-scoped evidence survive recovery.
• Retry budget is durable across generations and cannot be replenished by superficial activity.
• A live replacement without target-instruction/downstream progress is not marked recovered.
• Observation failures produce UNOBSERVABLE, not automatic restart.
• Jev cannot grant completion, custody, merge authority or mutation permission.
• Shadow metrics and adverse-race tests support any move to bounded autonomy.
• Exhaustion preserves work and escalates with incident, custody, attempt and consequence evidence.

Likely integration points

• bin/fm-watch.sh and bin/fm-task-inbox-lib.sh for suspicion/candidate notification.
• Existing durable wake reconciliation for FirstMate-owned decision and audit.
• bin/fm-crew-state.sh, bin/fm-receipt-check.sh and PR/validation owners for structured evidence.
• bin/fm-control.sh, bin/fm-spawn.sh and shared custody/lease owners for the transactional effect boundary.
• Existing stuck-crewmate policy and behavioral test suites; avoid duplicating their policy in a new checker.

Relationship to other work

• Depends on and generalizes Workers wedge with delivered-no-turn despite truthful receipts; need custody-aware auto-recovery #155; Workers wedge with delivered-no-turn despite truthful receipts; need custody-aware auto-recovery #155 remains independently testable without model behavior.
• PR fix(omp): prevent false-success fallback receipts #153 made no-turn receipts truthful, enabling deterministic detection.
• feat: return inbox results to the originating Hermes session #152 / PR feat: return durable inbox results to Hermes sessions #154 cover trusted-local Hermes inbox result correlation and return delivery, not recovery authorization.

Review provenance

Architecture review completed 2026-09-20 against a4fa70111b2f20ebc1fd20e97f4dda9078a94dad by an Astra medium scout. Verdict: revise the safety contract before implementation. The review was read-only; it did not reproduce a new wedge or run validation.