diff --git a/CHANGELOG.md b/CHANGELOG.md index 69921ab7e..421131a67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,31 @@ # Changelog +## v1.0.20 (2026-10-09) +> **Release focus:** This release focuses on reliability and resilience across agent sessions, fixing issues that could cause stalled work, shared slots, and interrupted sessions while adding automatic retries for connection errors and unavailable AI providers. It also improves day-to-day usability with multi-item deletion, a dedicated work item tab, an interview "Thinking…" indicator, and clearer guidance from the review queue and doctor. +### Features +- The dispatcher now refuses to start downtime work when slot info is unreliable, preventing two agents from sharing one slot. (WL-0MUXVPXAZ005RESW) +- The interview command now shows a "Thinking…" indicator during AI requests so it no longer looks frozen. (WL-0MUX2W8IN005RW66) +- Keeps work flowing when the sprint view is empty, while sharing leadership fairly. (WL-0MUU4XFU90008E24) +- Delete multiple work items in one command with a single sync. (WL-0MUI83ZTX005RN93) +- Stalled work now resumes automatically instead of being left unfinished. (WL-0MUMA5OMH0024PN1) +- The client now waits as long as the server asks before retrying, so overnight sessions resume cleanly instead of spamming errors. (WL-0MUR5EGH7003VOHM) +- Doctor now warns you when upgrades are pending instead of falsely reporting no issues. (WL-0MUTVB271007YKJZ) +- Work on a work item now opens in its own tab, keeping all related panes together. (WL-0MUKZGEQ2007FECS) +- Pressing r i now focuses the interview pane so you can answer questions right away. (WL-0MURJ0LFH002O95I) +### Bug Fixes +- Temporary connection errors now retry automatically instead of failing your task. (WL-0MUQO8AEE003G1Z2) +- The HERDR plugin no longer freezes or drops your commands during background updates. (WL-0MUY0RDBQ001H74P) +- Idle mode switching now waits for active work to finish, so your sessions won't be interrupted. (WL-0MUL0KO7Q003O7YJ) +- Prevents stray git settings from breaking agent sessions and corrupting your game files. (WL-0MV0TZEWZ003ZXEB) +- Fixed a test so the r i shortcut correctly focuses the interview pane. (WL-0MUZZW8KJ006HAPF) +- You can now detach items with --parent null instead of creating a broken parent link. (WL-0MUJM2LV1000IHKR) +- Prevents rare crashes when reloading sessions during headless runs. (WL-0MUL1BG09005VRBO) +- Long tasks now retry automatically when no AI provider is available instead of stalling. (WL-0MU56ZSSB0054D8W) +- Audited items no longer lose their ready status after sorting or syncing. (WL-0MU2O8AQ8000ASDO) +- The review queue banner now tells you whether to keep auditing or ship. (WL-0MUJRJCZV000NG5X) +- Fixed slot status cards not showing session info and console errors on the home page. (WL-0MUKY4MAX0008I0L) +- Stage filters now match the real pipeline, so you'll never see empty or broken lists. (WL-0MU624MF7004WYMI) + ## v1.0.19 (2026-10-06) > **Release focus:** This release focuses on reliability and workflow polish across task handling, sessions, and audits, fixing issues like endless retries, lost progress from connection errors, and sync conflicts that dropped completed work. It also improves visibility and speed with priority icons, clearer flag explanations, quick-create options, and more accurate reporting from the latest worklog data. ### Features diff --git a/CLI.md b/CLI.md index 648e53293..e3599b07a 100644 --- a/CLI.md +++ b/CLI.md @@ -107,7 +107,7 @@ Options: - `--description-file ` — Read description from a file (optional). - `-s, --status ` — Status value from config defaults (optional; default: `open`). - `-p, --priority ` — `low|medium|high|critical` (optional; default: `medium`). -- `-P, --parent ` — Parent work item ID (optional). +- `-P, --parent ` — Parent work item ID (optional). Pass `--parent null` (case-insensitive; aliases: `none`, `nil`, `-`, or `""`) to detach the item (set `parentId` to `null`). Without this flag the item is top-level by default. - `--tags ` — Comma-separated tags (optional). - `-a, --assignee ` — Assignee name (optional). - `--stage ` — Stage value from config defaults (optional). @@ -167,6 +167,8 @@ Update fields on one or more existing work items. Accepts multiple IDs. Options > **Reparenting demotion:** when `--parent ` attaches an item to a `completed`/`in_review` parent, the parent is demoted to `open`/`plan_complete` so a finished parent never silently gains uncompleted work. Automation-authored telemetry children (tagged `test-failure`, titled `[test-failure]…`, or created by a known bot identity) are exempt and do not rewind the parent (WL-0MTWU4XUD0001ALR). A demotion records an audit-trail comment on the parent naming the attached child, the actor, and the transition (WL-0MTWU4Y82001B3UH). +> **Detach:** pass `--parent null` (case-insensitive; also accepted: `none`, `nil`, `-`, or `""`) to detach an item, setting its `parentId` to `null`. Use `wl doctor dangling-parents` to detect and optionally fix orphaned parent references (e.g. the legacy `WL-NULL` artifact). + Automatic re-sort: - `wl update` will automatically invoke a re-sort when one or more updated fields are among: `status`, `priority`, `risk`, `effort`, or `stage`. By default this re-sort runs asynchronously so the CLI is not blocked. This helps `wl next` and other selection-based commands reflect recent priority or status changes without requiring a manual `wl re-sort`. @@ -346,19 +348,27 @@ wl audit-unwaive WL-ABC123 wl audit-unwaive WL-ABC123 --json ``` -### `delete` [options] +### `delete` [options] + +Delete one or more work items (marks them as deleted): sets each work item's status to `deleted` in the local database. If you prefer to set the status explicitly, use `wl update -s deleted` instead. -Delete a work item (marks as deleted): this sets the work item status to `deleted` in the local database. If you prefer to set the status explicitly, use `wl update -s deleted` instead. +Every id is processed independently. A failure for one id (for example, not found) does not stop the remaining ids; each id is reported in a per-id result. The command exits non-zero if any id failed and zero when all ids succeed. Descendants of a recursively-deleted parent are reported per id, and the automatic git sync runs exactly once after all deletions in the invocation complete (suppressed by `--no-sync`). Options: -- `--prefix ` — Operate on a specific prefix (optional). +- `--prefix ` — Operate on a specific prefix (optional; applies uniformly to every id in the batch). +- `--no-recursive` — Delete only the specified items, leaving children orphaned (applies to every id). +- `--no-sync` — Skip the automatic post-delete sync for the whole batch. + +JSON mode returns a per-id `results` array plus an overall `success`/`deleted`/`failed` summary. When exactly one id is supplied, the legacy single-id fields (`deletedId`, `deletedWorkItem`, `recursive`) are preserved for backward compatibility. Duplicate ids, and ids already removed as a descendant earlier in the same batch, are reported as `skipped` (`already deleted`) rather than failing the command. Examples: ```sh -wl delete WL-ABC123 # permanently removes the item and its comments -wl --json delete WL-ABC123 # machine-readable confirmation (204 on success) +wl delete WL-ABC123 WL-DEF456 WL-GHI789 # delete several items in one command +wl delete WL-ABC123 --no-recursive # delete only the parent, leaving children +wl delete WL-ABC123 WL-DEF456 --no-sync # delete a batch without syncing +wl --json delete WL-ABC123 # single-id machine-readable confirmation ``` ### `comment` (subcommands) @@ -1031,6 +1041,8 @@ wl doctor upgrade --confirm # Apply pending schema migrations (creates bac Validate work items against config-driven status/stage rules. Reports invalid values or incompatible combinations. +`wl doctor` also reports pending schema migrations and outdated git hooks (read-only) and instructs you to run `wl doctor upgrade`; when validation is clean but upgrades are pending, it prints the pending-upgrade notice instead of a bare `Doctor: no issues found.`. The notice is advisory and does not change the exit code. + For detailed migration policy, backup behavior, and CI guidance, see [DOCTOR_AND_MIGRATIONS.md](DOCTOR_AND_MIGRATIONS.md). Options: @@ -1086,6 +1098,13 @@ Notes: JSON output is a raw array of findings. Each finding includes: `checkId`, `type`, `severity`, `itemId`, `message`, `proposedFix`, `safe`, `context`. +When pending upgrades exist, a synthetic finding with `checkId: "upgrade.pending"` +and `type: "pending-upgrade"` is appended additively to the same array (top-level +shape unchanged). Its `context` carries machine-readable pending-upgrade data: +`pendingMigrations`, `pendingMigrationCount`, `outdatedHooks`, and +`outdatedHookCount`. Consumers grouping findings by work item should skip this +entry (`itemId: null`). + ### `re-sort` [options] Recompute `sort_index` values for active work items (excluding completed/deleted) using the current database values. @@ -1297,8 +1316,32 @@ canonical `## Appendix: Clarifying questions` heading is inserted; when one exists the confirmed questions are merged into it, keeping re-runs idempotent. If no provider is configured, or the request fails, times out or returns -malformed JSON, the command degrades silently to the deterministic-only -behaviour described below — no error is surfaced. +malformed JSON, the command degrades to the deterministic-only behaviour +described below, after printing a brief +`LLM unavailable — using structured evidence.` notice so the operator knows +the questions did not come from the model. + +#### In-flight progress feedback + +Both LLM paths (question extraction and the producer-review explanation) can +block for up to 15 s. To make that visible, the command prints a static +`Thinking…` status message immediately before each request. + +- When stdout is a TTY, an animated spinner is rendered on the same line and + is cleared deterministically before any subsequent output or prompt, so it + never interleaves with the readline prompt. +- When stdout is not a TTY (piped or captured), only the static message is + written — no spinner frames and no control characters — keeping captured + output clean. +- In `--json` mode no status or spinner is emitted at all; stdout stays + byte-for-byte valid JSON. +- When the command falls back to structured evidence (the LLM is unavailable, + times out or errors), it prints `LLM unavailable — using structured + evidence.` once and clears the spinner. An explicit `--no-llm` opt-out does + not print this notice. + +No LLM request is issued — and therefore no feedback is shown — when the +deterministic parser already found clarifying questions. #### When there are no questions to answer @@ -1322,7 +1365,9 @@ large. The summary and raw-output excerpt are individually bounded so a single large audit cannot dominate the prompt. When the LLM is disabled (`--no-llm`) or unreachable the command falls back -silently to the **structured evidence**, in priority order: +to the **structured evidence** (printing the `LLM unavailable — using +structured evidence.` notice for the unreachable case, but not for an +explicit `--no-llm` opt-out), in priority order: 1. the latest audit result (verdict and summary); 2. a durable audit-gap waiver (`auditWaiver`); @@ -1353,7 +1398,8 @@ Options: `needsProducerReview` (boolean), `producerReviewExplanation` (string or `null`; multi-line explanations are preserved), `noSection`, `noQuestions`, `total`, `outstanding` and `allAnswered`. Performs no - prompts, no mutation and (with `--no-llm`) no LLM call. + prompts, no mutation, no progress/status output and (with `--no-llm`) no + LLM call. - `--no-llm` — Disable all LLM use (the producer-review explanation and the LLM-assisted question extraction) and use the structured fallback instead (default: explanation on, extraction off unless enabled). diff --git a/DOCTOR_AND_MIGRATIONS.md b/DOCTOR_AND_MIGRATIONS.md index 5c6a47595..572e71e5e 100644 --- a/DOCTOR_AND_MIGRATIONS.md +++ b/DOCTOR_AND_MIGRATIONS.md @@ -11,7 +11,8 @@ This document describes the `wl doctor` command and the migration policy for Wor - **Status/stage compatibility** — validates every work item's status and stage against the rules defined in `.worklog/config.yaml` (see `docs/validation/status-stage-inventory.md` for the full rule set). - **Dependency edges** — checks that all dependency edges reference existing work items. - **Podcast script `Key Files:`** — flags a `podcast` item at a script-bearing stage (`plan_complete` / `in_review` / `done`) whose `Key Files:` contains no `.podcast.md` path that resolves to an existing file under the worklog root, and any `podcast` item whose listed `.podcast.md` path does not resolve. Detection is artifact-based (filesystem), not stage/status-based. `wl doctor --fix` backfills the path when a matching script is found (see `docs/FILE_PATH_CONVENTION.md`). -- **Pending migrations** — the `upgrade` subcommand detects and applies schema migrations. +- **Pending migrations** — the top-level `wl doctor` reports pending schema migrations and outdated git hooks (read-only) and instructs you to run `wl doctor upgrade`; the `upgrade` subcommand detects and applies them. +- **Outdated git hooks** — the top-level `wl doctor` reports worklog-managed hooks whose committed content differs from the installed hook, alongside pending migrations. - **Stale deleted items** — the `prune` subcommand removes soft-deleted items older than a configurable threshold. - **Audit gaps** — the `audit-gaps` subcommand reports `completed`/`in_review` items with no audit record (read-only). @@ -32,6 +33,53 @@ wl doctor --fix When issues are found, doctor prints each work item ID with its findings and suggested fixes. Findings that require manual intervention are grouped by type at the end. +### Pending-upgrade notice + +`wl doctor` also surfaces pending `wl doctor upgrade` work — schema migrations +and outdated git hooks — before you would otherwise notice it. The check is +**read-only**: it detects what `wl doctor upgrade` would apply but never changes +the database or hooks itself. Exit status stays `0`; the notice is advisory, +mirroring `wl doctor upgrade --dry-run`. + +```bash +$ wl doctor +Doctor: pending upgrades detected. +Pending migrations (2): + - 20260315-add-audit: Legacy: Add audit TEXT column to workitems (now replaced by audit_results table) (safe=true) + - 20260401-add-example: Add example column (safe=true) +Run `wl doctor upgrade` to preview pending upgrades, or `wl doctor upgrade --confirm` to apply them. +``` + +When there are no validation findings and no pending upgrades, doctor still +prints `Doctor: no issues found.` When validation findings exist, the +pending-upgrade notice is appended after the existing findings and manual-fix +sections without altering them. + +In `--json` mode the output remains a single, parseable JSON array of findings +(unchanged top-level shape). Pending-upgrade information is added additively as +a synthetic finding: + +```json +{ + "checkId": "upgrade.pending", + "type": "pending-upgrade", + "severity": "info", + "itemId": null, + "message": "Pending upgrades: 2 pending migrations. Run `wl doctor upgrade`.", + "proposedFix": null, + "safe": false, + "context": { + "pendingMigrations": [{ "id": "20260315-add-audit", "description": "...", "safe": true }], + "pendingMigrationCount": 1, + "outdatedHooks": ["pre-push"], + "outdatedHookCount": 1 + } +} +``` + +Consumers that iterate findings should skip `itemId: null` / `type: +"pending-upgrade"` entries when grouping by work item. + ### Schema migrations (`wl doctor upgrade`) Preview pending migrations before applying: diff --git a/EXAMPLES.md b/EXAMPLES.md index d59ee06e2..1701bdf0a 100644 --- a/EXAMPLES.md +++ b/EXAMPLES.md @@ -71,6 +71,9 @@ worklog update WI-0J8L1JQ3H8ZQ2K6D -s completed -d "Implementation finished and # Change parent (move in hierarchy) worklog update WI-0J8L1JQ3H8ZQ2K6F -P WI-0J8L1JQ3H8ZQ2K6E +# Detach an item (set parentId to null) +worklog update WI-0J8L1JQ3H8ZQ2K6F -P null + # Add tags worklog update WI-0J8L1JQ3H8ZQ2K6D --tags "urgent,reviewed" diff --git a/LOCAL_LLM.md b/LOCAL_LLM.md index 07d4a1e6d..b176232dc 100644 --- a/LOCAL_LLM.md +++ b/LOCAL_LLM.md @@ -229,11 +229,14 @@ Defaults: `baseUrl` `http://192.168.0.199:8000/v1`, `model` `compact`, `timeoutMs` `15000`. Environment variables `LLM_BASE_URL`, `LLM_MODEL`, `LLM_API_KEY` and `LLM_TIMEOUT_MS` are used as fallbacks; config values take precedence over environment variables (matching the `embedding.*` precedence). -Every call is bounded by the timeout and degrades silently to a non-LLM -fallback on error; the fallback reports the flag reason from the structured +While a request is in flight the `wl interview` command prints a static +`Thinking…` status (plus a TTY-only spinner, suppressed for piped output and +`--json`); every call is bounded by the timeout and degrades to a non-LLM +fallback on error, printing a brief `LLM unavailable — using structured +evidence.` notice. The fallback reports the flag reason from the structured evidence first (the persisted audit verdict and summary, then an audit waiver, then the review comments). See [CLI.md](CLI.md#interview-options-id) for the -`wl interview --no-llm` opt-out. +`wl interview --no-llm` opt-out and the full progress-feedback behaviour. --- diff --git a/README.md b/README.md index 133ab7ff9..7a4def253 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ You can get a lot of value from using Worklog as a memory for your agents. But y | [docs/SKILL_AUTHORING.md](docs/SKILL_AUTHORING.md) | Skill authoring guide with script best practices | | [docs/dev/wl-process-healthcheck.md](docs/dev/wl-process-healthcheck.md) | wl process healthcheck watchdog usage (cron/systemd, thresholds) | | [docs/dev/audit-debug-log-retention.md](docs/dev/audit-debug-log-retention.md) | Audit debug log retention policy + runbook: sweep locations/thresholds, dry-run/`--apply`/`--check`, cron install/removal, safety guarantees | -| [docs/dev/downtime-dispatcher.md](docs/dev/downtime-dispatcher.md) | Leader-election + shared-coordination downtime dispatcher architecture (herdr); critical-first dispatch tier + freeze split-by-skill; dedicated Dispatcher workspace with per-project tabs (one tab per work-item prefix) | +| [docs/dev/downtime-dispatcher.md](docs/dev/downtime-dispatcher.md) | Leader-election + shared-coordination downtime dispatcher architecture (herdr); critical-first dispatch tier + freeze split-by-skill; automated per-project tabs (one tab per work-item prefix) and interactive item-ID-tab placement (invoking workspace, fail-open) | | [docs/pane-triage-skill.md](docs/pane-triage-skill.md) | Approval-gated pane-triage skill and the cross-language pane-close bridge that exposes the shared classifier to Python skills | ## Tutorials diff --git a/docs/COLOUR-MAPPING.md b/docs/COLOUR-MAPPING.md index 5fe0291ab..9e43a317f 100644 --- a/docs/COLOUR-MAPPING.md +++ b/docs/COLOUR-MAPPING.md @@ -34,14 +34,13 @@ item (per clarification Q3b on WL-0MSJ2JFMO007PGQ6). |-----------|------------|----------------|-------------| | Unknown/missing priority | Medium (yellow) | `220` | Falls back to medium/yellow | -### Stage Colours (unchanged — used by the stage filter dialog and separators) +### Stage Colours (used by the stage filter dialog and separators) | Stage | CLI Colour | Colour Name | |-------|-----------|-------------| | `idea` | Gray | `gray` | | `intake_complete` | Blue | `blue` | | `plan_complete` | Cyan | `cyan` | -| `in_progress` | Yellow | `yellow` | | `in_review` | Green | `green` | | `done` | White | `white` | diff --git a/docs/dev/downtime-dispatcher.md b/docs/dev/downtime-dispatcher.md index 2e6b41445..f071574f1 100644 --- a/docs/dev/downtime-dispatcher.md +++ b/docs/dev/downtime-dispatcher.md @@ -54,13 +54,16 @@ re-read live on every dispatch cycle, so a change applies without a restart. A caller that passes no limit falls back to the sprint-view default (20), so an unwindowed head is never produced. -**Non-critical work outside the view is never dispatched.** When the head -yields no candidate the dispatcher reports the terminal reason derived from the -in-view flags (`no-candidate`, `review-queue-hold`, `code-freeze`, -`fresh-audit-skip`, `audit-in-flight`, `audit-host-saturated`, `in-flight-hold`, -`wl-error`) and does **not** fall back to a second ranking or an out-of-view -non-critical scan. When every visible item is filtered by a safety gate (or the -view is empty) the outcome is a NOP. +**Non-critical work outside the view is never dispatched by the normal +paths.** When the head yields no candidate the dispatcher reports the terminal +reason derived from the in-view flags (`no-candidate`, `review-queue-hold`, +`code-freeze`, `fresh-audit-skip`, `audit-in-flight`, `audit-host-saturated`, +`in-flight-hold`, `wl-error`) and does **not** fall back to a second ranking or +an out-of-view non-critical scan. When every visible item is filtered by a +safety gate (or the view is empty) the outcome is a NOP — **except** on the +coordination leader path, which has one bounded, own-root escape valve that +then hands off leadership (see *Own-backlog fallback & leadership hand-off* +below). **Critical escape hatch (defensive).** A bounded out-of-window read (`fetchExtendedHerdrItems`, `DOWNTIME_DISPATCH_EXTEND_MAX = 30`) re-reads the @@ -69,7 +72,9 @@ ranking — and returns **only** `critical` items. Because every `critical` item is already mandatory in the view, this escape hatch is normally a no-op; it exists purely as a safety net so a release blocker can never be windowed out. Non-critical items beyond the sprint view are filtered out, so the operator can -never be surprised by hidden-backlog dispatch. The extension runs on both +never be surprised by hidden-backlog dispatch. (The critical-only filter is +lifted **only** for the coordination-mode own-backlog fallback below; the +defensive escape hatch itself stays critical-only.) The extension runs on both dispatch paths; the **TUI worklist is unchanged** (it keeps rendering exactly `browseItemCount` items, plus the mandatory set). @@ -88,6 +93,72 @@ caller keeps its entry / reports `wl-error`); a failed or empty extended critical read degrades to the original terminal reason, so the escape hatch can never convert a defined outcome into a new failure. +### Own-backlog fallback & leadership hand-off (WL-0MUU4XFU90008E24) + +The sprint-view-only window above is the **normal** dispatch contract: the +dispatcher head equals the rendered view and a view with nothing dispatchable +is a NOP. In **coordination/leader mode** the leader additionally has a +bounded, self-scoped escape valve so a fleet does not idle when one root's view +is momentarily empty or fully blocked — with a fairness guard so no instance +can monopolise leadership by repeatedly reaching into its own hidden backlog. + +**Trigger (AC1).** The fallback is evaluated in `dispatchFromCoordination` +**only after** the coordination offer scan yields no dispatchable candidate — +i.e. the same terminal-reason set the view can produce (`no-candidate`, +`review-queue-hold`, `code-freeze`, `fresh-audit-skip`, `audit-in-flight`, +`audit-host-saturated`, `in-flight-hold`, or a genuinely empty view). It never +runs when a normal in-view/coordination offer dispatched. + +**Own root only (AC2).** The fallback reads the leader's **own** worklog root +(`opts.cwd`) only — never another instance's or another worklog root's hidden +backlog. A `{ok:false}` lookup, a thrown error, or an empty extended tail is +fail-open: the original terminal reason is reported unchanged and nothing +panics (AC9). + +**Bounded window, canonical ranking, gates are filters (AC3/AC7/AC9).** The +fallback is a **window extension**, never a second ranking: it re-reads the +SAME canonical ranking path (`getHerdrListHead` → `fetchNextItems` → +`selectWorkItems` → `regroupWorkItems`) with a larger, bounded window +(`current.length + DOWNTIME_DISPATCH_EXTEND_MAX`, `= 30` additional items) and +returns only the newly visible tail. The critical-only filter of the defensive +escape hatch above is lifted for the fallback (any ranked item in the extended +tail is eligible), but every existing safety gate still applies as a +**sequential filter** — classify/producer-review/freshness/caps, +dispatched-marker exclusion, attempt budget, non-terminal cooldown, +code-freeze, audit slot minimum, in-flight (critical) guard, and the +review-queue hold. A gated own-backlog item is skipped exactly like an in-view +one; a lost CAS race tries the next ranked candidate. + +**Leadership hand-off (AC4/AC5/AC6, assumption A3).** A successful dispatch +sourced from the own backlog (and **only** then) is flagged machine-readably on +the outcome (`ownBacklogDispatch: true`). After the claim, dispatch marker and +pane spawn have completed, the worker loop calls the existing +`LeaderElectionManager.releaseLeadership()` **exactly once** and sets +`leaderState = false`, so another instance can win the next election. The +step-down never rolls back the pending dispatch and never orphans the spawned +pane. It is **not** called after a normal in-view dispatch, and **not** called +when the fallback finds nothing — in that case leadership is retained and the +pre-existing terminal reason is reported unchanged. Because there is no +reliable peer-eligibility predicate (coordination entries are removed on +dispatch), the step-down is unconditional after a successful fallback dispatch; +with no peer the same instance simply re-wins the normal election on a later +tick, bounded by the existing election backoff. + +**Scope: coordination/leader mode only.** The fallback lives on the +coordination leader path (`dispatchFromCoordination`) and the step-down is +guarded by a non-null `leaderManager`. **Legacy mode** (no `coordinationDir`; +`leaderManager === null`, `leaderState` hard-coded `true`) is unchanged — there +is no manager and no peer to hand off to, so the direct/legacy path keeps the +strict sprint-view-only behaviour (non-critical out-of-view work is never +dispatched). The TUI worklist is unchanged. + +**No contradiction with WL-0MUNS8X97007C9H9.** The sprint-view-only contract +still governs the dispatcher head and the direct/legacy path; the fallback is a +bounded, own-root, leadership-handing-off exception on the coordination leader +path only. It is *not* a reinstatement of the unconditional non-critical +extension that WL-0MUNS8X97007C9H9 superseded: the window stays bounded, the +ranking is the same, and the fairness guard removes the monopolisation risk. + **Coordination leader (F3, WL-0MTK1ILM2009QYB2):** the shared coordination file holds ONE entry per instance — an **offer** of that instance's own Herdr list head (computed at the owner's check-in by `computeMostImportantItem`, which walks the same Herdr sequence with the @@ -194,7 +265,12 @@ Lifecycle (`packages/herdr/src/coordination.ts`, WL-0MTMPIQBE001J41P non-expirin override / round-robin ordering are retired by WL-0MTK1ILM2009QYB2). Each entry is eligibility re-checked at dispatch time; stale entries are dropped (removed) without a pane or marker (see Lifecycle §2). When a slot - opens the first passing offer dispatches in the entry's `worklogRoot`. + opens the first passing offer dispatches in the entry's `worklogRoot`. If + **no** offer passes, the leader performs one bounded own-backlog fallback + against its **own** root (`opts.cwd`) and, on a successful fallback + dispatch, voluntarily steps down via `releaseLeadership()` so another + instance can take over (WL-0MUU4XFU90008E24; see *Own-backlog fallback & + leadership hand-off* above). 4. The dispatched entry is **removed** from the coordination file. The existing dispatched-marker exclusion and CAS claim mechanisms are preserved unchanged. @@ -392,6 +468,160 @@ tick (live) and wired through `DowntimeWorkerConfig.config().maxAttempts` into `dispatchDowntimeWork`, `computeMostImportantItem`, `runCoordinationCheckIn` and `dispatchFromCoordination`. +### Stalled-work scan and resume (WL-0MUYMBSA90092QV2; parent WL-0MUMA5OMH0024PN1) + +Before this feature the dispatcher always took the next item from the Herdr +head and spawned a **new** pane when local-LLM capacity was idle. An existing +pane holding a non-terminal item whose agent had stopped working (its pi agent +went `idle`, or exited as `done`, without the item reaching a terminal stage) +was ignored, so half-finished work accumulated in workspaces while fresh +capacity was spent on new work. The dispatcher now performs a **machine-wide +stalled-work scan and in-place resume** on every idle dispatch cycle, **before** +selecting a new work item. + +The decision half (`classifyStalledPane`, `scanStalledPanes`) and the resume +half (`resumeStalledPane`, `resumeStalledWorkIfAble`) live in +`packages/herdr/src/stalled-work.ts`; production I/O seams are wired in +`createDowntimeDeps` (`packages/herdr/src/index.ts`). + +#### Ordering relative to new-item dispatch (parent AC3) + +The scan is consulted **once per idle cycle, immediately before new-item +selection**, on every dispatch entry point: + +- `dispatchDowntimeWork` (direct leader dispatch) — after the idle / free-slot + gate and before the Herdr-head filters (`dispatchFromHerdrList`); +- `computeMostImportantItem` (the instance check-in offer) — before reading the + Herdr head, so a resumed pane makes the instance offer **nothing** that cycle + rather than pausing; +- `dispatchFromCoordination` (leader dispatch of a remote offer) — before the + offer's review-queue / in-flight / cooldown filters. + +When the scan yields a resumable candidate and a resume succeeds, the cycle +**short-circuits** with the neutral reason `stalled-resume` +(`{dispatched:false, reason:'stalled-resume'}`; `computeMostImportantItem` +returns `{ok:true, stalledResume:true}`) — **no new item is dispatched in that +cycle**. When no candidate is resumable the existing dispatch path proceeds +unchanged. The LLM idle / free-slot check remains the sole concurrency +limiter; the scan introduces no client-side pane cap. + +#### Stall conditions (parent AC1) + +A pane is **stalled** only when **all** hold: + +1. it hosts a live pi agent — the parsed `herdr pane list` record has an + `agent` value; +2. its label suffix parses to a non-terminal item — status not + `completed`/`deleted`, stage not `in_review`/`done` (reuses the canonical + `isActiveBlocker` predicate); +3. its `agent_status` is neither actively working (active set + `work`/`working`/`busy`/`running`) nor `blocked`; and +4. its not-`working` state has persisted at least the stall threshold + (`downtimeStallThresholdMs`). + +#### Exclusion rules and machine-readable reasons (parent AC4) + +Every skip is **neutral** — never a CLI-error strike, never a `no-candidate` +cooldown, never a duplicate. `classifyStalledPane` returns a single +machine-readable reason: + +| Reason | Meaning | +|---|---| +| `no-agent` | The pane hosts no live pi agent. | +| `no-item-id` | The label has no parseable work-item id. | +| `kind-unknown` | The label carries no recognisable dispatch kind. | +| `item-terminal` | The item is completed/deleted or `in_review`/`done`. | +| `needs-producer-review` | The item is flagged `needsProducerReview === true` (hard exclusion). | +| `agent-working` | The agent is actively working. | +| `agent-blocked` | The agent status is `blocked` (never hijacked). | +| `kind-stale` | The item has already advanced past the pane's dispatched kind. | +| `cooldown-active` | The non-terminal pane-close cooldown holds the `(item, kind)`. | +| `attempt-cap-exhausted` | The per-item/per-kind dispatch-attempt cap is reached. | +| `last-activity-unknown` | No parseable activity timestamp to measure a stall. | +| `stall-threshold-not-elapsed` | Not-`working` for less than the threshold. | + +The dispatched kind is derived from the pane label convention +` triggered …` (the kind token immediately follows +`triggered`), recognising `implement`, `risk-effort`, `intake`, `audit` and +`plan`. Other labels — including a free-form `Manually triggered prompt …` +pane — have no kind and are skipped. + +**Kind vs stage progression (`kind-stale`).** Stage ranks are `idea` 0, +`intake_complete` 1, `plan_complete`/`in_progress` 2, `in_review` 3, `done` 4; +kind ranks are `intake` 0, `plan` 1, `risk-effort`/`implement` 2, `audit` 3. A +pane whose item rank exceeds its kind rank (an `intake` pane on an +`intake_complete` item, an `implement` pane on an `in_review` item) is stale +and skipped. An unknown stage cannot prove advancement, so it fails open (not +stale). + +**Stall anchor.** The not-working duration is measured from the **later** of +the item's `updatedAt` and its most recent non-`spawn-failed`, +non-`enrichment` dispatch marker for that kind, so a pane freshly dispatched +for an old item is not mistaken for a stall. No parseable anchor → skip +(`last-activity-unknown`). + +#### Resume in place (parent AC2) + +Each candidate is resumed **in place** — a new pane is never spawned for the +item; the resume always targets the candidate's `paneId`: + +- a **live agent** receives the literal continuation prompt `continue` + (`herdr agent prompt continue`, `STALLED_RESUME_PROMPT`); +- a pane whose agent has **exited** (`agent_status` `done`/`exited`) has pi + relaunched in the **same** pane continuing its session + (`herdr agent start pi --kind pi --pane -- --session `, or + `-- --continue` when the pane records no session path), after which the + literal `continue` prompt is submitted. + +A successful resume records a dispatch attempt for `(item, kind)` in the item +root's rolling dispatch log, so the existing per-item/per-kind attempt cap +applies to resumes too; the cap is enforced up front and escalates via +`needsProducerReview` when exhausted. Concurrent resumes of the same +`(paneId, itemId)` are prevented by an in-process in-flight guard (a second +attempt is the neutral `concurrent-resume` skip). + +#### Neutral outcome vocabulary and fail-safety (parent AC2/AC6) + +Every not-resumed outcome is neutral — never a strike, never a crash, never a +duplicate. The resume adds these reasons on top of the classifier reasons +above: + +| Reason | Meaning | +|---|---| +| `concurrent-resume` | A resume of the same `(paneId, itemId)` is already in flight. | +| `agent-blocked` | `herdr agent prompt` refused because the agent is `blocked`. | +| `agent-prompt-stalled` | The submission was accepted but the agent did not reach `working`/`blocked` within the CLI's documented 5 s window (`agent_prompt_stalled`). | +| `pane-vanished` | The pane/agent disappeared (e.g. the pane-closure reaper closed it) between scan and resume. | +| `cli-error` | Any other herdr/CLI failure. | + +Fail-safe degradation: an unreadable/unparseable `herdr pane list` yields no +candidates; a per-pane failure (unresolvable root, failed `wl show`, thrown log +read) skips that pane without aborting the scan; unwired `scanStalledPanes` / +`resumeStalledPane` deps and a disabled scan both fall through to normal +dispatch; a thrown resume is neutral and the scan tries the next candidate. A +code-frozen/ambiguous cycle skips `audit` and `implement` panes (freeze +split-by-skill), so a resume can never bypass the freeze. + +#### Cross-root resume contract (parent AC5) + +Each candidate carries the **item's own worklog root**, resolved from the +pane's `cwd` first and then from the pi session-log path in `agent_session` +(`defaultStalledPaneRootResolver`). The resume reads the item and its rolling +log, records the attempt and flags producer review against **that** root — +never the leader's ambient root. When neither `cwd` nor `agent_session` +resolves to a valid worklog root the pane is **skipped** (fail closed) rather +than resumed against the wrong database. + +#### Settings + +Both settings are re-read from the plugin settings on **every idle tick** +(never cached), so a change applies live: + +| Setting | Default | Clamp | +|---|---|---| +| `downtimeStallScanEnabled` | `true` (`DEFAULT_DOWNTIME_STALL_SCAN_ENABLED`) | boolean; `false` disables the scan entirely and falls through to normal dispatch | +| `downtimeStallThresholdMs` | 5 min / `300000` ms (`DEFAULT_DOWNTIME_STALL_THRESHOLD_MS`) | `[60 s, 60 min]` / `[60000, 3600000]` ms (`clampDowntimeStallThresholdMs`); invalid/missing falls back to the default | + ### Pane placement: project workspace first, `Dispatcher` fallback (WL-0MUR5FUWD00024XN) Automated downtime panes spawn **inside the owning project's herdr @@ -580,9 +810,84 @@ Lifecycle (`packages/herdr/src/dispatcher-anchor.ts`): Duplicate `Dispatcher` workspaces are harmless — the persisted anchor file is the authority, not the label count; close surplus idle ones one at a time -without disturbing active `Downtime triggered …` panes. Manual -`open-worklist` / `open-pi-agent` flows are unaffected (this placement applies -only to automated downtime dispatch). +without disturbing active `Downtime triggered …` panes. `open-worklist` / +`open-pi-agent` flows are unaffected by the **automated** placement above +(they carry no work-item ID). The following sub-section documents the +**interactive** counterpart, which does place worklist-dispatched, +ID-carrying panes into an item-ID tab. + +### Interactive dispatch pane placement (WL-0MUYI3JAO002BTNL) + +The sections above describe the **automated downtime** dispatcher. An +operator-driven **interactive** dispatch (a worklist chord resolving to an +ID-carrying command) also routes its pane into a tab labelled with the exact +work-item ID, but it is a deliberately different path: it uses the invoking +pane's **current** workspace and is **fail-open**. It is the manual +counterpart of the automated placement (parent work item +WL-0MUKZGEQ2007FECS). + +| | Automated downtime (above) | Interactive dispatch (this sub-section) | +|---|---|---| +| Trigger | idle dispatcher, `dispatchClaimedTier` | operator chord / worklist command (`onCommand`) | +| Workspace | **project** workspace resolved via `resolveProjectWorkspace` (logical-root match) | **invoking pane's current** workspace (`HERDR_WORKSPACE_ID`) | +| Channels | agent panes (`send-to-pi.sh`) | agent panes (`send-to-pi.sh`) **and** shell/command-output panes (`run-in-pane.sh`) | +| No work-item id | scheduled prompts → `Dispatcher` anchor | current-pane split (legacy behaviour, unchanged) | +| Failure | **fail-closed** — `anchor-unavailable`, no pane | **fail-open** — falls back to the current-pane split, never dropped | + +Lifecycle (`packages/herdr/src/index.ts`; shared anchor helper in +`packages/herdr/src/dispatcher-anchor.ts`): + +1. **Workspace resolution** — `resolveInteractiveWorkspaceId(env)` reads + `HERDR_WORKSPACE_ID` (present in the plugin pane environment); when it is + absent/blank it derives the workspace from the invoking pane id + `HERDR_PANE_ID` (`:`), else returns `null` (→ fail-open). + There is **no** `/proc`-based project resolution on this path (Q2a), and + the pane is never placed in another project's workspace. +2. **Item-ID tab resolution** — `resolveInteractiveItemAnchor(cwd, + workspaceId, itemId, deps?, onCreate?)` wraps the shared + `getItemTabAnchor`, so the tab labelled exactly `` is created on + first use and reused thereafter under the coordination lock with the same + double-check and CLI-shape tolerance as the automated path (the tab label + is the key; no persistence file, no duplicate tab). `onCreate` fires only + when this call actually provisioned the tab — the create-vs-reuse signal + consumed by step 4. **Any** resolution/creation error returns `null` + (fail-open). +3. **Anchor forwarding** — the item-tab anchor is resolved once per dispatch + and forwarded to both channels by the arg builders: + `buildSendToPiArgs(..., anchor)` and `buildRunInPaneArgs(..., anchor)` + append `--anchor ` when it is a non-empty string. + `send-to-pi.sh` already supported `--anchor`; `run-in-pane.sh` gained it — + with `--anchor` it runs `herdr pane split --pane --direction right + --no-focus --cwd ` instead of the legacy `--current`, tolerating the + `pane_id` / `paneId` / `id` split-output variants (raw output logged on + parse failure, mirroring `send-to-pi.sh`). `--no-focus` is preserved by + default, so the item tab never steals focus — **except** for shortcuts + that opt in with `focus: true` (currently `r i`, the producer interview, + WL-0MURJ0LFH002O95I): their dispatch passes `--focus`, and + `run-in-pane.sh` focuses the new pane's item tab (`herdr pane get ` + → `herdr tab focus `) **before** zooming the pane. Tab focus is + required because the pane lives in a tab other than the caller's and + `pane zoom` alone cannot switch tabs; it is best-effort/fail-open (an + unresolvable tab id falls back to the zoom-only behaviour). +4. **Root-pane cleanup** — when this dispatch **created** the tab, the + placeholder root pane (herdr's initial empty pane, used only as the first + split anchor) is closed after the split is **confirmed**: the spawning + script writes its new pane id via `--pane-id-file`, and only then does + `closeInteractiveTabRootPane(anchorPaneId, { cwd }, deps)` run. Cleanup is + **fail-safe** — the anchor is closed only when liveness POSITIVELY + confirms it is not a live tracked pi/shell pane + (`defaultInteractiveLiveTrackedPanesResolver` → `herdr pane list`; live = + a non-terminal pi agent or a `Shell: …` label). Unknown liveness (probe + absent, failed, or unparseable) leaves the anchor open, so a reused tab's + live pane is never closed. Cleanup is also **fail-open** — any error is + swallowed and the dispatch outcome never changes. A later dispatch for the + same item reuses the surviving pane as the split anchor. + +`getItemTabAnchor` is therefore the single shared create-or-reuse mechanism +for both placements; only the workspace choice, failure semantics and +root-pane liveness source differ. The automated path checks the running +**downtime** pane ids for its cleanup liveness; the interactive path checks +the live **tracked pi/shell** panes (step 4). ### No-candidate cooldown & the empty offer file @@ -594,7 +899,10 @@ tracker so a fresh full idle period is required after the pause. A `no-candidate` outcome means the *sprint view* — the live `browseItemCount` window (plus mandatory items) — held nothing dispatchable: the dispatcher does not scan hidden backlog (WL-0MUNS8X97007C9H9; see *Sprint-view-only dispatch -window* above). In +window* above) **except** via the coordination leader's bounded own-backlog +fallback, which dispatches before this reason is returned and then hands off +leadership (WL-0MUU4XFU90008E24; see *Own-backlog fallback & leadership +hand-off* above). In coordination mode (WL-0MTEZ4XZJ006Y9U7) the shared runtime file (`.worklog/downtime-coordination.json`) is an **offer list, not the backlog**: the leader removes each entry after dispatching (see step 4 @@ -666,7 +974,7 @@ gate is the *outermost* audit-concurrency guard, never a replacement): |---|---|---|---| | **Host-wide marker** (this mechanism) | machine-wide, all projects/instances | an audit dispatched by ANY project on this host | skip audit tier; reason `audit-host-saturated` | | **Per-worklog single-flight** (WL-0MT3PHW4I002SNOV) | one project's dispatch log | a non-stale `kind=audit` marker mapping to an `in_progress` item in THIS worklog | skip audit tier; reason `audit-in-flight` | -| **Proxy-slot gating** (`available_slots` / `contention_queue_depth` / per-slot ownership) | the Local Proxy | live slot availability, queue depth, live leases | ineligible/skip audit dispatch (per-tier minimum 2 free slots) or `proxy-contention` | +| **Proxy-slot gating** (per-slot availability / `contention_queue_depth` / per-slot ownership) | the Local Proxy | live slot availability, queue depth, live leases | ineligible/skip audit dispatch (per-tier minimum 2 free slots) or `proxy-contention`; unusable per-slot detail → `slots-unavailable` | | **`AUDIT_PHASE2_PARALLELISM=1`** | the audit skill's child fan-out | Phase 2 deep-analysis children | parent + one child = exactly 2 local slots | **Observability.** The skip is logged to stderr as @@ -995,7 +1303,9 @@ Each machine-wide entry records `worklogRoot` (preferred) + `directory` alias. The single leader dispatches offers in **file order** (each offer is its root's Herdr list head) **across worklogRoots** and spawns each pane in the entry's `worklogRoot`. The slot budget is machine-wide: ONE leader poll → -ONE `freeSlots` snapshot (per-slot or `available_slots`), forwarded to the sole +ONE `freeSlots` snapshot **derived from the per-slot array** — the only +trusted availability source (fail-closed contract, WL-0MUXVPXAZ005RESW) — +forwarded to the sole dispatch call — no per-worklog duplication (F5 WL-0MTII48OV008P2QU; WL-0MT50LKAK001EF5Q single cap source). v1 scope is single-machine; a multi-machine (real flock/NFS) extension is future work. @@ -1028,9 +1338,10 @@ an agent on a tool call (wl, bash, tests) left the slot "free", so multiple The dispatcher does **not** wait out a lease: it dispatches into genuinely free **unowned** slots via the per-slot gate (WL-0MU8807BI008C9ME), and a lease held by a dispatched pane (which already owns its own slot) does not - block the OTHER free unowned slots. Only the **count-based single-slot** - path treats a held lease as blocking (a truly owned sole slot stays - protected). The maximum is pinned in code as + block the OTHER free unowned slots. The lease is a **decision-log signal + only** (`ownerPresent`): the ownership gate itself is derived from the + trusted per-slot array (a slot with a live `owner_session_id` is never + free). The maximum is pinned in code as `LOCAL_DISPATCH_LEASE_MAX_SECONDS`; if `llm-manager` changes the default, revisit that constant and this section so the assumption cannot drift silently again. @@ -1038,23 +1349,21 @@ an agent on a tool call (wl, bash, tests) left the slot "free", so multiple `owner_session_id`; `countFreeUnownedSlots` excludes owned slots from the free count and the per-slot idle tracker resets an owned slot's timer, so a slot with a live lease is never considered available for a new pane. - A single idle-but-owned slot (count-based path) fails closed via the - derived `local_lease_active`. - - **Stale/empty per-slot data (WL-0MUFP30T2003OX1F).** The proxy serves - `slots: []` together with `slots_stale: true` when its fresh `/slots` - query fails: the slot COUNTS come from the last-known cache but the - per-slot detail is unavailable. An empty array is **not** "zero free - slots", so the worker treats stale OR empty `slots` as "no per-slot - identity" and falls back to the count-based path. In that path, with a - multi-slot config (`0 < N < total`) a held lease no longer blocks - dispatch into the proxy-reported spare capacity: the count-based gate - uses `available_slots`, reserving one slot per lease only when the owner - may be idle (`local_active_query !== true` — an active query's slot is - already processing and excluded from the count). A single-slot - (`total_slots = 1`) or `N <= 0` / `N >= total` setup keeps the strict - fail-closed gate. The proxy-side improvement (serve cached per-slot - detail when stale) is tracked separately in `llm-manager`. + + **Fail closed on unusable per-slot detail (WL-0MUXVPXAZ005RESW).** The + proxy serves `slots: []` together with `slots_stale: true` when its fresh + `/slots` query fails: the slot COUNTS come from the last-known cache but + the per-slot detail is unavailable. An empty array is **not** "zero free + slots". Under the fail-closed contract, absent, empty, or stale per-slot + detail is **unusable**: the worker refuses to dispatch and records the + distinct reason `slots-unavailable` (header token `no-slots`), so a + fail-closed stop is distinguishable from `slot-owned` / `proxy-contention` + in the decision log and herdr header. The inaccurate count-based fallback + (and its `countBasedFreeSlots` / `countBasedSpareCapacity` / `leaseReserve` + / `ownerLeaseCount` helpers) has been **removed** — `available_slots` is + never used as a dispatch budget. The proxy-side improvement (serve cached + per-slot detail when stale) is tracked separately in `llm-manager` + (LP-0MUFSVXID0039ZAQ). 3. **Contention feedback (AC6)** — the proxy's LIVE `contention_queue_depth` is parsed; while > 0 the dispatcher backs off with outcome reason `proxy-contention` until the queue drains. The sibling @@ -1184,6 +1493,8 @@ status refresh unchanged at 30s.** | Success-marker staleness window | **24 h** (`downtimeMarkerStaleWindowMs`; clamped to 1 h – 7 d; releases a stranded success marker at an unchanged stage, WL-0MU6UL0RJ008IHGT) | `DEFAULT_DOWNTIME_MARKER_STALE_WINDOW_MS`, `clampDowntimeMarkerStaleWindowMs` (`downtime-worker.ts`) | | Non-terminal pane-close cooldown | **30 min** (`downtimeNonTerminalCooldownMs`; clamped to 1 min – 24 h; holds same-kind re-dispatch after a non-terminal pane close, WL-0MUKYERLZ006ELL5) | `DEFAULT_DOWNTIME_NON_TERMINAL_COOLDOWN_MS`, `clampDowntimeNonTerminalCooldownMs` (`downtime-worker.ts`) | | Per-item/per-kind attempt cap | **3** (`downtimeMaxAttempts`; clamped to 1 – 10; flags `needsProducerReview` and stops re-dispatch of that kind once the cap is reached at the current stage, WL-0MUKYEXMK0033MFK) | `DEFAULT_DOWNTIME_MAX_ATTEMPTS`, `clampDowntimeMaxAttempts` (`downtime-worker.ts`), `countAttempts` (`downtime-log.ts`) | +| Stalled-work scan | enabled (`downtimeStallScanEnabled`; re-read live; `false` disables the pre-dispatch scan and falls through to normal dispatch, WL-0MUYMBK6H005PY16) | `DEFAULT_DOWNTIME_STALL_SCAN_ENABLED` (`downtime-worker.ts`) | +| Stalled-work threshold | **5 min** / `300000` ms (`downtimeStallThresholdMs`; minimum continuous not-`working` duration before an in-place resume; clamped to 60 s – 60 min; invalid/missing falls back to the default, WL-0MUYMBK6H005PY16) | `DEFAULT_DOWNTIME_STALL_THRESHOLD_MS`, `clampDowntimeStallThresholdMs` (`downtime-worker.ts`) | | Pane-closure reaper cadence | **60 s** (`PANE_CLOSE_REAPER_INTERVAL_MS`; gated by `paneCloseEnabled`, WL-0MUJL1NAH0042GOS) | `pane-close-scheduler.ts` | | Pane-closure idle threshold | **0 min** — idle close disabled (`paneCloseIdleThresholdMinutes`; clamped to 0 min – 24 h) | `pane-close-scheduler.ts` | | Pane-closure grace period | **5 min** (`paneCloseGracePeriodMinutes`; clamped to 1 min – 24 h; no pane is eligible for close within this window of first dispatch, WL-0MUMM5IUF003EVT8) | `pane-close-scheduler.ts` | @@ -1202,7 +1513,10 @@ on load (`downtimeNonTerminalCooldownMs`) is likewise configurable and clamped on load (`clampDowntimeNonTerminalCooldownMs`). The per-item/per-kind attempt cap (`downtimeMaxAttempts`) is likewise configurable and clamped on load -(`clampDowntimeMaxAttempts`). +(`clampDowntimeMaxAttempts`). The stalled-work scan enable flag +(`downtimeStallScanEnabled`) and threshold (`downtimeStallThresholdMs`) are +likewise re-read live every tick and clamped on load +(`clampDowntimeStallThresholdMs`). ## Files & runtime artifacts @@ -1212,7 +1526,8 @@ on load | `packages/herdr/src/machine-coordination.ts` | Machine coordination dir resolver (`~/.herdr/downtime` / `HERDR_COORDINATION_DIR`) | | `packages/herdr/src/leader-election.ts` | Lock acquisition, lease management, re-election (machine dir) | | `packages/herdr/src/coordination.ts` | Coordination file read/write (entries, prune, upsert) — machine dir `downtime-coordination.json` | -| `packages/herdr/src/downtime-worker.ts` | Worker tick: election, check-in, idle gate, dispatch (anchor-before-claim) | +| `packages/herdr/src/downtime-worker.ts` | Worker tick: election, check-in, idle gate, dispatch (anchor-before-claim); stalled-work scan wiring (`resumeStalledWorkIfAble`, `STALLED_RESUME_REASON`) | +| `packages/herdr/src/stalled-work.ts` | Stalled-pane classifier + machine-wide scan + in-place resume orchestrator (`classifyStalledPane`, `scanStalledPanes`, `resumeStalledPane`) | | `packages/herdr/src/downtime-log.ts` | Coordination/dispatch rolling logs (per worklog root, retained) | | `packages/herdr/src/pane-close.ts` | Shared pane-closure classifier (`classifySession`, `extractFinalAssistantText`) consumed by the reaper and `pane-triage` (WL-0MUJL1NAH0042GOS) | | `packages/herdr/src/pane-close-reaper.ts` | Closure reaper orchestration + CLI (`runReaper`, `runReaperCli`) | @@ -1315,8 +1630,8 @@ duplicate-dispatch RCA is answerable from the log alone: | Field | Type | Description | |---|---|---| -| `selectionPath` | string | Which loop selected the candidate: `critical-first`, `normal-scan`, `coordination-offer`, `scheduled-prompt`, or `legacy-tier` | -| `selectionReason` | string | Machine-readable reason: e.g. `no-live-pane`, `marker-stale-escalation`, `in-flight-pane` (skip), `non-critical`, `leader-offer`, `scheduled-due`, `critical-tier` | +| `selectionPath` | string | Which loop selected the candidate: `critical-first`, `normal-scan`, `coordination-offer`, `own-backlog` (coordination own-backlog fallback, WL-0MUU4XFU90008E24), `scheduled-prompt`, or `legacy-tier` | +| `selectionReason` | string | Machine-readable reason: e.g. `no-live-pane`, `marker-stale-escalation`, `in-flight-pane` (skip), `non-critical`, `leader-offer`, `own-backlog-fallback` (WL-0MUU4XFU90008E24), `scheduled-due`, `critical-tier` | | `paneId` | string \| null | (Enrichment entry only) the resolved dispatch pane/session id, or `null` when it could not be resolved — never a guess | | `enrichment` | `true` | (Enrichment entry only) discriminator marking a post-spawn enrichment rather than a fresh dispatch | @@ -1462,5 +1777,19 @@ trace. review-queue depth gate rewired onto the live Herdr path, sprint-complete auto-disable removed, marker manual-only) - Package README: `packages/herdr/README.md` → *Downtime worker (local-LLM - idle dispatch)* + idle dispatch)* and *Interactive pane placement (work-item-ID tabs)* +- Work item: **WL-0MUYI3JAO002BTNL** / **WL-0MUYI3K8H007MPKF** *Interactive + item-ID-tab placement for worklist dispatch (both channels) and its + fail-safe placeholder root-pane cleanup* (documented in *Interactive + dispatch pane placement* above) +- Work item: **WL-0MUMA5OMH0024PN1** *When dispatching a new workload first + scan all workspaces and panes for stalled work* (+ its children + WL-0MUYMBK6H005PY16 settings / WL-0MUYMBMCG000Z4WP pane-record enrichment / + WL-0MUYMBO9X000WDF6 classifier / WL-0MUYMBPZ5004LBFX resume orchestrator / + WL-0MUYMBSA90092QV2 dispatch wiring / WL-0MUYMBUAK007H10J verification; + documented in *Stalled-work scan and resume* above) +- Work item: **WL-0MUU4XFU90008E24** *Own-backlog dispatch fallback must + relinquish leadership* (bounded own-root fallback + voluntary + `releaseLeadership()` hand-off; documented in *Own-backlog fallback & + leadership hand-off* above) - Docs work item: **WL-0MT76H3Z900908TV** (this page) \ No newline at end of file diff --git a/docs/icons-design.md b/docs/icons-design.md index 7b348cdda..a23383f9c 100644 --- a/docs/icons-design.md +++ b/docs/icons-design.md @@ -57,7 +57,7 @@ across the CLI (chalk) and TUI rendering paths. It covers: | intake_complete | `📥` | `[INTAKE]` | "Stage: Intake Complete" | | plan_complete | `📋` | `[PLAN]` | "Stage: Plan Complete" | | in_review | `🔍` | `[REVIEW]` | "Stage: In Review" | -| done | `🏁` | `[DONE]` | "Stage: Done" | +| done | `✔️` | `[DONE]` | "Stage: Done" | ## 3. Audit Result Icons diff --git a/package.json b/package.json index 6418e884e..ea92a61a9 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "worklog", - "version": "1.0.19", + "version": "1.0.20", "description": "A simple experimental issue tracker for AI agents", "main": "dist/index.js", "type": "module", diff --git a/packages/herdr/README.md b/packages/herdr/README.md index 73583ed16..224261510 100644 --- a/packages/herdr/README.md +++ b/packages/herdr/README.md @@ -9,12 +9,12 @@ A Herdr plugin that provides a keyboard-navigable work item selection list for b - **View details** — Press Enter on any item to see its full details (description, acceptance criteria, metadata, tags, priority, GitHub issue number, and audit status information such as audit result, review status, and last audit timestamp) - **Audit indicators** — The list view shows audit icons next to `in_review` items (✅ audited, ❌ failed, ❓ unaudited). The metadata section (list-mode panel and detail view) mirrors the list's icons with text labels — the selected item's Stage row uses the same audit-aware `in_review` icon (✅/❌/❓ fresh, ⏳ stale-passed, 🔍 otherwise), and the Audit/Reviewed rows pair their icons with text (e.g. `✅ ready to close`, `❌ needs review`). The detail view additionally shows the last audit timestamp. **Child coverage (WL-0MUBVH8QG0020H9L):** a `completed`/`in_review` child whose direct parent has a fresh audit is *covered* — its list row and metadata Stage row show the parent's audit-result symbol in a dimmed/grey style (visually distinct from a bright own-audit icon) and the metadata panel adds a `Covered by ` row. Coverage is **derived at read time** (`isCoveredByParent` over the parent's audit freshness inputs via the shared `isAuditFresh` predicate) — nothing is persisted and there is no schema migration. A child with its own audit always shows its own verdict; an uncovered child (no own audit, parent demoted/stale) keeps the plain `🔍` stage icon and is never dispatched (see [audit-tier selection](#audit-tier-dispatch)). In text-only (`noIcons`) mode the covered indicator renders as `[COVERED]`. - **Chord shortcuts** — Multi-key chord sequences provide quick actions like updating priorities, stage/status, title, closing/deleting items, running workflows, and toggling review status (configurable via `shortcuts.json`) -- **Command output** — When a chord resolves to a non-`/wl` command (e.g., `!!wl update --priority high`), the resolved command is executed **visibly in a new herdr pane** (see `scripts/run-in-pane.sh`) so the user sees the command line and its output; the wrapper keeps the pane's process alive so the pane stays open for inspection — dismiss it with Enter or close it with `prefix+x`. Panes spawned from the selection list open **without stealing focus**: the list keeps the keyboard focus while the command-output pane opens in the background (see [Design decisions](#design-decisions)). +- **Command output** — When a chord resolves to a non-`/wl` command (e.g., `!!wl update --priority high`), the resolved command is executed **visibly in a new herdr pane** (see `scripts/run-in-pane.sh`) so the user sees the command line and its output; the wrapper keeps the pane's process alive so the pane stays open for inspection — dismiss it with Enter or close it with `prefix+x`. Panes spawned from the selection list open **without stealing focus**: the list keeps the keyboard focus while the command-output pane opens in the background (see [Design decisions](#design-decisions)). When the command carries a work-item ID, the pane is placed in that item's work-item-ID tab in the current workspace (see [Interactive pane placement](#interactive-pane-placement-work-item-id-tabs)). - **Command input form** — When a chord command contains unknown `` placeholders (e.g. `!!wl update --status --stage `), the plugin shows a modal input form so you can fill in the values before the command runs. Known identifiers like `` are still auto-substituted with the selected item's ID. The form is a full-pane page (no border/centering) that wraps text at the pane width and grows downward as content is entered. See [Command input form](#command-input-form). - **Keyboard navigation** — Arrow keys or j/k to navigate (wraps at list boundaries), Page Up/Down, g/G for first/last, Enter to select, Escape to go back. - **Mouse and touch** — Click/tap a row to select; double-click to open detail; mouse wheel or touch-scroll to navigate (list mode) or scroll detail; tap stage options in the filter prompt. Requires SGR mouse reporting (enabled on raw-mode entry, disabled on exit). Terminals without it fall back to keyboard-only. **Alt+m toggles mouse tracking** on/off (WL-0MT0AP2LR000JFWN): while tracking is off, the terminal's native drag-select-to-copy works; press Alt+m again to resume mouse interaction. The current state is shown in the footer (`alt+m mouse on/off`). **Hover tooltip** (WL-0MT9XRZDK006GMUH): hovering the mouse over a row that has an associated agent-pane shows a metadata tooltip in the footer area (ID, Title, Command, Priority, Type, Risk, Effort, Start Time); `Esc` dismisses it until the mouse leaves and re-enters the list. - **Fold indicators** — When the worklist has more items than fit the visible list area, the list shows dim `▼ more` / `▲ more` markers so you always know when items are hidden below the fold or above the current scroll position (WL-0MSG8YXYJ008PWJJ). See [Selection List Behaviour](#selection-list-behaviour). -- **Pi agent pane dispatch** — Agent commands (`/skill:*`, `/intake`, `/plan`) are automatically dispatched to a new pi agent pane opened to the right, where pi receives the command as its initial prompt. Free-form prompts use the `/prompt:` prefix: the routing prefix is stripped so pi receives only the prompt text. The agent pane opens **without stealing focus** from the selection list (see [Design decisions](#design-decisions)). +- **Pi agent pane dispatch** — Agent commands (`/skill:*`, `/intake`, `/plan`) are automatically dispatched to a new pi agent pane opened to the right, where pi receives the command as its initial prompt. Free-form prompts use the `/prompt:` prefix: the routing prefix is stripped so pi receives only the prompt text. The agent pane opens **without stealing focus** from the selection list (see [Design decisions](#design-decisions)). When the command carries a work-item ID, the pane is placed in that item's work-item-ID tab in the current workspace (reused if it already exists) — see [Interactive pane placement](#interactive-pane-placement-work-item-id-tabs). - **Downtime worker (local-LLM idle dispatch)** — During operator idle time the plugin dispatches pi agent panes to run audits/refactors of completed items against the local llama-server (see [Downtime worker](#downtime-worker-local-llm-idle-dispatch)) - **Hydrator (self-healing `in_progress` claims)** — Every 30 seconds the plugin fetches all `in_progress` work items, matches each against live agent panes in the current workspace (the pane title carries the work-item ID), and releases any claim with no matching pane so it re-enters the dispatchable pool: `in_review` items complete, dependency-blocked items are marked `blocked`, everything else returns to `open` at its claimed stage. The check also runs immediately when the worklog tab regains focus, and is fully visibility-gated (a hidden tab spawns zero `wl`/`herdr` processes). See [Hydrator](#hydrator-self-healing-in_progress-claims). - **Mode-switch worker (activity-gated proxy mode switching)** — Automatically switches the llama-proxy between fast (cloud) and cheap (local) modes: agent-route commands fire an immediate fast switch (fail-open), while a full operator-idle window plus a proxy-idle check triggers the cheap switch (fail-closed). The proxy URL reuses `downtimeProxyUrl` and the plugin's switches are manual overrides that the proxy's own time schedule reclaims. See [Mode-switch worker](#mode-switch-worker-activity-gated-proxy-mode-switching-wl-0msn3fwv5008kqe9). @@ -25,7 +25,7 @@ A Herdr plugin that provides a keyboard-navigable work item selection list for b - **Quit** — Press `q` to exit - **Metadata panel** — The selected item's metadata (ID, title, status, stage, priority, type, risk, effort, tags, audit info, and more, plus a **description preview** — first up-to-3 lines of the item's description) is shown in a panel below the list. The list takes as much vertical space as its content needs, up to the full pane height; the metadata panel fills whatever space remains — expanding when the list is short, and sitting at the bottom (minimum 3 rows) when the list is long. The panel scrolls independently with `m`/`M` (down/up) so long metadata never affects list navigation. See [Metadata panel](#metadata-panel). - **Command log** — Every plugin-dispatched command that targets a work item (via `` substitution or an explicit item ID) is recorded to a local JSON log. For `in_progress` items the panel shows the **last command** at the bottom, so you can see exactly what was last dispatched against the item. See [Command log](#command-log). -- **Stage grouping** — Work items are grouped by their Worklog stage (standard lifecycle stages only: `idea`, `intake_complete`, `plan_complete`, `in_progress`, `in_review`, `done` — no custom stage values). Podcast episode items group exactly as their frontmatter stages map 1:1 (PRD §7.2). See [Stage grouping](#stage-grouping). +- **Stage grouping** — Work items are grouped by their Worklog stage (standard lifecycle stages only: `idea`, `intake_complete`, `plan_complete`, `in_review`, `done` — no custom stage values). Actively-worked items are identified by `status = in-progress`, not by a stage. Podcast episode items group exactly as their frontmatter stages map 1:1 (PRD §7.2). See [Stage grouping](#stage-grouping). - **Generic md viewer** — When a work item's description carries a `Key Files:` path to a markdown document (e.g. a podcast episode `.podcast.md`), the detail view renders the file with a generic markdown viewer (frontmatter skipped, full GFM rendering: headings, lists, tables, blockquotes, code, links) as a preview. The description section is rendered with the same markdown renderer. A persistent **Related Docs** table of contents at the top of the detail view lists every `.md` Key File (`↑↓/j:k` to navigate, `Enter` to open in the viewer), and the metadata panel shows a display-only `Related Docs` row. See [Markdown viewer](#markdown-viewer). - **Inline note links** — Inline `[NOTE : ...]` markers (PRD §7.1) render as clickable links to the note work items: the marker is displayed as `↗`, and the note text is never shown in the viewer. Any markdown document opened in the viewer can also be annotated in place (`n,e` add/edit, `n,d` delete); podcast scripts sync notes to the worklog as child work items (PRD §7.3). See [Inline note links](#inline-note-links). - **Code Freeze awareness** — While a ship-it release is in progress the project is in *Code Freeze*: the worklist shows a prominent banner and blocks all implement commands (`/skill:implement*`) with a notice dialog until the release finishes. See [Code Freeze](#code-freeze). @@ -112,7 +112,7 @@ The plugin pane will then be available via the Herdr plugin system. - Press `f`, `d` — Show the downtime dispatches grouped into 4-hour time blocks (see [Recent dispatches view](#recent-dispatches-view)) - Stage, priority, and dispatches filters are **mutually exclusive**: applying one replaces whichever other axis was active (single filter slot, replace semantics). Sprint (`f s s`, `f p s`, or `/wl` with no arguments) clears **all** of them. - Type `/wl dispatches` directly to activate the dispatches view (equivalent to `f d`). - - `/wl ` accepts shorthand aliases (`idea`, `intake`, `plan`, `progress`, `review`) and canonical stage names (`intake_complete`, `plan_complete`, `in_progress`, `in_review`) + - `/wl ` accepts the five CLI stage names (`idea`, `intake_complete`, `plan_complete`, `in_review`, `done`) plus the shorthand aliases `idea`, `intake`, `plan`, and `review`. The removed `progress` alias and the legacy `completed` value no longer apply a filter — they fail soft (no crash, no filter change) exactly like `/wl ` - `/wl --priority ` accepts the canonical priority names `critical`, `high`, `medium`, `low`; unknown values fall back gracefully (no crash, no filter change) - `/wl` with no stage/priority argument returns to the default unfiltered browse list - Filtered views show every root item matching the filter's rule (see [Selection List Behaviour](#selection-list-behaviour)) @@ -121,7 +121,7 @@ The plugin pane will then be available via the Herdr plugin system. 4. Workflow shortcuts (single-key): - Press `c` — Create a new work item - Press `d` — **Toggle downtime dispatch for the current pane** (per-pane only — sibling panes and other worklog roots are unaffected; the header shows `[Downtime Off]` while off and the live status when on. The disable persists across plugin restarts via a `.herdr-downtime-disabled` marker in the worklog root; press `d` again to return to following the global setting. See [Downtime worker](#downtime-worker-local-llm-idle-dispatch)) - - Press `i` — Run the implement workflow on the selected item (intake_complete, plan_complete, in_progress) + - Press `i` — Run the implement workflow on the selected item (intake_complete, plan_complete) - Press `n` — Run the intake workflow on the selected item (idea stage) - Press `p` — Run the plan workflow on the selected item (intake_complete stage) - Press `s` — Insert a search command @@ -129,7 +129,7 @@ The plugin pane will then be available via the Herdr plugin system. 5. Review chords (press `r` then a key): - Press `r`, `p` — Toggle 'Needs Producer Review' flag and add a comment to the selected item - - Press `r`, `i` — Walk through outstanding interview questions on the selected item (`wl interview`) + - Press `r`, `i` — Walk through outstanding interview questions on the selected item (`wl interview`); the new pane is **focused** so you can answer immediately (see **Focused shortcuts via `focus: true`** below) 6. Priority update chords (press `u` then `p` then a priority key): - Press `u`, `p`, `l` — Set priority to low @@ -338,9 +338,15 @@ dispatches; the other herdr instances coordinate instead of polling: (closed/`in_progress`/`done`, audit-now-fresh, `needsProducerReview === true`, above-caps, or otherwise not currently dispatchable) is **removed eagerly without pane or dispatched marker**, and dispatch continues to - the next offer. The dispatched entry is **removed**; its owner re-offers - its next Herdr head at the next check-in. There is **no TTL-based - pruning**. **Every per-entry `wl` invocation — the dispatch-time + the next offer. If no offer is dispatchable, the leader may fall back to + its **own backlog** — a bounded window extension of the same ranking + beyond the sprint view, own root only — and, on a successful own-backlog + dispatch, voluntarily steps down via `releaseLeadership()` so another + instance can take over (WL-0MUU4XFU90008E24; see the *Own-backlog + fallback & leadership hand-off* paragraph below). The dispatched entry is + **removed**; its owner re-offers its next Herdr head at the next + check-in. There is **no TTL-based pruning**. **Every per-entry `wl` + invocation — the dispatch-time `fetchItem`/audit-enrichment lookups, the CAS claim (`wl update`) and the `wl comment add` trail — resolves against the OFFER's own `worklogRoot` via stateless per-call `--worklog-dir` (`buildWlArgsForRoot`, @@ -378,6 +384,19 @@ dispatches; the other herdr instances coordinate instead of polling: cycle). Scheduled prompts (no work-item id) keep the `Dispatcher` anchor. See [docs/dev/downtime-dispatcher.md](../../docs/dev/downtime-dispatcher.md#pane-placement-project-workspace-first-dispatcher-fallback-wl-0mur5fuwd00024xn). +- **Stalled-work scan and resume (WL-0MUYMBSA90092QV2)** — before selecting a + new item on an idle cycle, the dispatcher scans the machine-wide + `herdr pane list` and resumes a stalled pane **in place** (reusing its + session) rather than opening a new one. A pane is stalled when it hosts a + live pi agent, its label parses to a non-terminal work item, its + `agent_status` is neither `blocked` nor actively working, and its + not-working state has persisted at least `downtimeStallThresholdMs` + (default 5 min). A successful resume short-circuits the cycle with the + neutral reason `stalled-resume` (no new-item dispatch); every skip/failure + is neutral (never a strike, never a cooldown, never a duplicate) and + degrades to the unchanged dispatch path. A foreign-root pane is resumed + against the root resolved from the pane's `cwd` / `agent_session`. See + [docs/dev/downtime-dispatcher.md](../../docs/dev/downtime-dispatcher.md#stalled-work-scan-and-resume-wl-0muymbsa90092qv2-parent-wl-0muma5omh0024pn1). Coordination operations (check-ins, elections/takeovers, eligibility drops) are recorded in `.worklog/downtime-coordination.log` — a separate rolling log from the dispatch log, so the dispatch-marker readers never see @@ -409,15 +428,33 @@ WL-0MT3FM8VA005XBHE critical). WL-0MU6UL3GQ0015AA5 AC1/AC4):** the dispatcher head is the sprint view. Both selection paths pass the live per-root `browseItemCount` (clamped 1–50) to `getHerdrListHead`, so the head equals the rendered worklist (mandatory items always included, remaining slots -filled from "other" items). Non-critical work outside the view is never dispatched: when -the head yields no candidate the dispatcher reports the in-view terminal reason -(`no-candidate`, `review-queue-hold`, `code-freeze`, …) and does not scan hidden backlog. +filled from "other" items). Non-critical work outside the view is never dispatched by +the normal paths: when the head yields no candidate the dispatcher reports the in-view +terminal reason (`no-candidate`, `review-queue-hold`, `code-freeze`, …) and does not scan +hidden backlog — except via the coordination own-backlog fallback below, which then hands +off leadership. A bounded out-of-window escape hatch (`DOWNTIME_DISPATCH_EXTEND_MAX`, 30 additional items) re-reads the **same ranking path** but returns **only** `critical` items — because critical items are already mandatory in the view this is normally a no-op, and it can never surface hidden non-critical work. The TUI worklist still renders exactly `browseItemCount` items. +**Own-backlog fallback & leadership hand-off (WL-0MUU4XFU90008E24; +coordination/leader mode only):** when the coordination offer list yields no +dispatchable candidate, the leader may read its **own** backlog beyond the +sprint view — a bounded window extension of the SAME canonical ranking +(`DOWNTIME_DISPATCH_EXTEND_MAX`, 30 additional items), never a second ranking, +and **own-root only** (never another instance's hidden backlog). Every safety +gate still applies as a sequential filter, and the lookup is fail-open (a +`{ok:false}`/thrown/empty tail leaves the original terminal reason unchanged). +On a successful own-backlog dispatch (claim → marker → spawn already complete) +the leader calls `releaseLeadership()` **exactly once** and stops acting as +leader, so another instance can win the next election; it never steps down +after a normal in-view dispatch, and a fallback that finds nothing retains +leadership and the original terminal reason. **Legacy mode** (no +`leaderManager`) is unchanged. See +[docs/dev/downtime-dispatcher.md](../../docs/dev/downtime-dispatcher.md#own-backlog-fallback--leadership-hand-off-wl-0muu4xfu90008e24). + A "valid" audit is defined by the review-icon freshness rule: the audit is current — i.e. the review icon is **neither** the hourglass `⏳` (stale passed) **nor** the magnifying glass `🔍` (no audit / stale failed). Concretely, @@ -474,13 +511,28 @@ without per-slot data it fails closed to all-slots-free for `0 < N < total` kind — a neutral skip (reason `attempt-budget-exhausted`, never a strike, never a no-candidate) reset by a stage advancement (default: `3`, clamped to 1 – 10) +- `downtimeStallScanEnabled` — Enable the pre-dispatch stalled-work scan + (default: `true`). When on, the dispatcher prefers resuming an existing + stalled pane over opening a new one (see the *Stalled-work scan and + resume* bullet above); `false` disables the scan and falls through to the + normal dispatch path +- `downtimeStallThresholdMs` — Minimum continuous not-`working` duration + before a pane is considered stalled and resumed in place (default: + `300000` = 5 minutes, clamped to 60 s – 60 min; invalid/missing falls back + to the default) The worker polls `GET {proxyUrl}/llama/local/status` on the poll interval. -Idle means: llama-server running, no active **local** query (when the proxy -serves `local_active_query` — preferred over the global `active_query`, so -remote-only streams with free local slots do not block dispatch; absent on -pre-fix proxies, the global `active_query` is used as the fallback), no model -switch, no active local lease, and the required free-slot condition met. +Idle means: llama-server running, no active **local** query (the proxy's +`local_active_query` is the sole busy signal, so remote-only streams with free +local slots do not block dispatch; absent on pre-fix proxies it is treated as +busy), no model switch, no active local lease, and the required free-slot +condition met. **Per-slot detail is mandatory (fail-closed contract, +WL-0MUXVPXAZ005RESW):** the free-slot budget always comes from the per-slot +`slots` array, never the `available_slots` count. When per-slot detail is +absent, empty (`slots: []`), or stale (`slots_stale: true`) the dispatcher +refuses to dispatch and records the distinct reason `slots-unavailable` +(header token `no-slots`), so a fail-closed stop is distinguishable from +`slot-owner` / `contention`. **In per-slot identity mode with `0 < N < total` the query/lease checks are relaxed** (spare-capacity dispatch, parent WL-0MT32F90V008UAD2): a query or lease tied to a busy slot is the operator's own session and must not block @@ -528,10 +580,11 @@ period; an ambiguous/unparseable poll also resets all timers (fail-closed). This assumes `slot_id` values are stable across polls (they identify the physical slots on the llama-server). Malformed per-slot data (non-array, missing/empty `slot_id`, non-boolean `is_processing`, or duplicate ids) is -treated as busy (fail-closed). Without per-slot data — or when N is `0` or -≥ `total` — the worker falls back to the count-based all-slots-free logic -with the FULL global checks (server, local query, model switch, lease): N -of `total` slots free never dispatches without per-slot identity. +treated as busy (fail-closed). **Without usable per-slot detail the worker +fails closed** (no dispatch, reason `slots-unavailable` / header token +`no-slots`) rather than falling back to the `available_slots` count. When N +is `0` or ≥ `total`, the full global checks (server, local query, model +switch, lease) apply and the free count still comes from the per-slot array. **Per-tier free-slot minimums** (parent WL-0MT32F90V008UAD2 AC3) — at selection time, against the latest polled status, each dispatch tier @@ -757,6 +810,59 @@ tick), so two instances with identical configuration do not probe in lockstep — other machines get a fair chance to win the dispatch race. The jitter factor is clamped to `[0.5×, 1.5×]` of the configured interval. +### Interactive pane placement (work-item-ID tabs) + +Interactive dispatch from the worklist — an agent command (`/skill:*`, +`/intake`, `/plan`, `/prompt:`) or a shell/command-output command (`!!`/`!` +or a plain command) that carries a work-item ID — opens its pane in a **tab +labelled with the exact work-item ID**, in the **invoking pane's current +workspace** (WL-0MUYI3JAO002BTNL, parent WL-0MUKZGEQ2007FECS). This groups +all of one item's panes together and mirrors the automated downtime +dispatcher's item-ID tabs (see [Downtime worker](#downtime-worker-local-llm-idle-dispatch)), +with three deliberate differences: + +- **Current workspace, not project workspace** — the item tab lives in the + workspace of the pane that invoked the shortcut. The plugin reads + `HERDR_WORKSPACE_ID` from its environment and falls back to the workspace + component of `HERDR_PANE_ID`; there is no `/proc`-based project + resolution on this path. +- **Fail-open** — if the item tab cannot be resolved or created (no work-item + ID, no workspace id, provisioning error), the dispatch falls back to + today's current-pane split. The operator's dispatch is never dropped or + blocked. (Tab provisioning itself stays fail-closed internally: it never + places the pane in a wrong tab.) The automated downtime path, by contrast, + is fail-closed (`anchor-unavailable`). +- **Both channels** — the placement applies to agent panes (`send-to-pi.sh`) + **and** to shell/command-output panes (`run-in-pane.sh`), which gained a + `--anchor ` flag mirroring `send-to-pi.sh` (with `--anchor` it runs + `herdr pane split --pane ` instead of `--current`). + +Semantics: + +- **Create on first use, reuse thereafter** — the tab label is the key + (`herdr tab list`); the shared `getItemTabAnchor` helper creates the tab + under the coordination lock with a double-check, and a second dispatch for + the same workspace + item adds a pane to the existing tab rather than + creating a duplicate. +- **No-ID panes unchanged** — commands with no work-item association (the + unbound **Open Pi Agent** action, the `P n` blank-prompt session, generic + commands) still split the current pane in the current tab. +- **Root-pane cleanup** — when a dispatch creates a fresh item tab, the + tab's placeholder root pane (used only as the first split anchor) is closed + once the new pane has spawned. Cleanup is **fail-safe** (the anchor is + closed only when liveness positively confirms it is not a live tracked + pi/shell pane — a reused tab's live pane survives; unknown liveness leaves + it open) and **fail-open** (an error never blocks or changes the dispatch). + A later dispatch for the same item reuses the surviving pane as the split + anchor. +- **No focus stealing** — item tabs are created with `--no-focus` and the + existing no-focus dispatch default is preserved, so the selection list + keeps the keyboard focus. + +See [docs/dev/downtime-dispatcher.md](../../docs/dev/downtime-dispatcher.md#interactive-dispatch-pane-placement-wl-0muyi3jao002btnl) +for the implementation-level detail and the automated-vs-interactive +comparison. + ### Pane-closure reaper (WL-0MUJL1NAH0042GOS) Agent panes launched in Herdr run `pi` in interactive mode. A skill that @@ -873,9 +979,11 @@ proxy idle state: the dispatcher's query/lease — no longer block the switch: a downtime pane is exactly the work the operator wants running while the proxy runs cheap. Accepted tradeoff: a fast-mode downtime request in flight during the mode - restart is killed and retried by its client. Without per-slot data the - all-slots-free fail-closed fallback is unchanged (server up, no query, - no model switch, no lease, ALL slots free). Any ambiguity — an + restart is killed and retried by its client. Under the shared fail-closed + contract (WL-0MUXVPXAZ005RESW) `evaluateIdle` returns false when per-slot + detail is unusable, so a proxy that cannot serve `slots[]` delays the + cheap switch rather than switching on an unverifiable count. Any + ambiguity — an unparseable status payload (including malformed/ambiguous per-slot identity), a non-2xx admin response, or a fetch failure — is treated as busy (fail-closed) so the proxy is never switched cheap @@ -1392,6 +1500,26 @@ above. The "top N of M" header reflects the **actual displayed count** (N), which may exceed `browseItemCount` when the mandatory set is large. +### Review-queue-depth footer banner + +When the root-only `completed`/`in_review` count reaches `browseItemCount` +(default 20, re-read live from settings), the selection-list footer shows a +review-queue-depth banner (WL-0MTHSHN5V008R5L0). Its copy reflects the audit +state of the queue (WL-0MUDYTQ55002NUSJ): + +- **Outstanding audit** — at least one root-level `in_review` item has no + stored audit or a non-current (stale) audit, per the shared `isAuditFresh` + predicate — the banner reads + `Review queue deep ( of completed/in_review) — focus on audits`. +- **Every item freshly audited** — the banner reads + `Ready to Ship (shortcut 'S')`, prompting the `S` Ship It shortcut. + +The banner is gated by `showHelpText` and is **display-only**: it never +disables downtime dispatch and never writes or clears the +`.herdr-downtime-disabled` marker (queue depth must never halt coordination — +see the sprint-complete auto-disable removal in +[docs/dev/downtime-dispatcher.md](../../docs/dev/downtime-dispatcher.md)). + ### Fold indicators (below-the-fold / above-the-fold markers) When the display-rows list (headings + items) has more rows than fit in @@ -1458,9 +1586,10 @@ first rows), so a long list never scrolls off the top. the row is omitted entirely when there are no `.md` Key Files. The row is **display-only** — opening a document happens from the detail view's Related Docs table of contents (see [Markdown viewer](#markdown-viewer)). -- For items whose stage is `in_progress`, the panel additionally shows - **`Last command:`** — the most recent command the plugin dispatched - against that item (`none yet` until the first dispatch). +- For actively-worked items (`status = in-progress`), the panel + additionally shows **`Last command:`** — the most recent command the + plugin dispatched against that item (`none yet` until the first + dispatch). - When the item has a description, the panel shows a **`Description`** preview: the first up-to-3 non-empty lines of the description (markdown source as-is, each line truncated to the pane width), so you can see what @@ -1478,7 +1607,7 @@ first rows), so a long list never scrolls off the top. Work items are grouped by their Worklog **stage** using the standard lifecycle stages only — `idea`, `intake_complete`, `plan_complete`, -`in_progress`, `in_review`, `done`. No custom stage values are required for +`in_review`, `done`. No custom stage values are required for grouping, so podcast episode items group exactly as their frontmatter `pipeline_stage` maps 1:1 onto the Worklog stages (PRD §7.2). Groups render in the canonical order (Critical → Group N → Idea → Other → In Review) as @@ -1488,10 +1617,12 @@ while a heading is selected collapses/expands that group — collapsed groups hide their items from both the render and navigation (the heading itself stays). Counts reflect the top-level items in the group (post stage-filter) regardless of collapse state. Stage changes re-group items on the next -refresh. Non-critical `in_progress` items join the file-path-partitioned -`Group N` lists alongside `plan_complete`/`intake_complete` items and sort -ahead of them (actively-worked items first); "Other" remains only as a -safety net for unknown/custom stages. +refresh. Non-critical actively-worked items (`status = in-progress`) join +the file-path-partitioned `Group N` lists alongside +`plan_complete`/`intake_complete` items and sort ahead of them +(actively-worked items first); "Other" remains only as a safety net for +unknown/custom stages (including the removed `in_progress`/`completed` +stage values), so legacy rows still render deterministically. **In Review** items are sorted within the group using a deterministic **6-bucket predicate** (WL-0MSLPM5ZB003TADT): @@ -1746,10 +1877,10 @@ packages/herdr/ - **Command routing via callback** — When a chord resolves to a non-`/wl` command, it is passed to an `onCommand` callback (set by the entry point) which routes it by prefix: - `!!`/`!` prefixed commands (shell-executed shortcuts such as audit approve/reject, priority updates, close/delete) are run **visibly in a new herdr pane** via `scripts/run-in-pane.sh` — the wrapper keeps the pane's process alive so the pane stays open (exit status reported; dismiss with Enter or close with `prefix+x`) so the user can inspect the command output. - Everything else is written to stdout with a `CMD:` prefix for the calling framework (Herdr) to execute. -- **Selection-list dispatch keeps focus** — Every pane spawned from the worklist selection list (pi agent panes via `send-to-pi.sh`, and command-output panes via `run-in-pane.sh`) opens **without moving focus** by default (WL-0MSHIA53D009DJOT): the dispatch passes `--no-focus` to both launchers, so the final zoom/focus step is skipped and the selection list keeps the keyboard focus. The user can read dispatch feedback via toasts and inspect the opened pane with herdr pane navigation (`prefix+o`, `prefix+x` to close). The `P n` shortcut opts in to **focus the new pane immediately** (see **Focused shortcuts via `focus: true`** below). Out of scope and unchanged: the downtime worker (already `--no-focus`), the `open-pi-agent` unbound plugin action, and `open.sh`/`toggle.sh`. -- **Pi agent dispatch** — Agent commands (`/skill:*`, `/intake`, `/plan`) are intercepted by the entry point and routed to a new pi agent pane. The `send-to-pi.sh` script splits the current pane to the right, creates a new pane, runs `pi` with the command as the initial prompt, and renames the pane to "Pi Agent". The dispatch passes `--no-focus` (selection-list dispatch keeps focus, see above) unless the shortcut opts in to **focus the new pane** (`focus: true` — only `P n` today, see **Focused shortcuts via `focus: true`** above), in which case `--focus` is passed and the new pane is zoomed. Agent commands are routed before any prefix handling, so they are unaffected by `!!`/`!` processing. +- **Selection-list dispatch keeps focus** — Every pane spawned from the worklist selection list (pi agent panes via `send-to-pi.sh`, and command-output panes via `run-in-pane.sh`) opens **without moving focus** by default (WL-0MSHIA53D009DJOT): the dispatch passes `--no-focus` to both launchers, so the final zoom/focus step is skipped and the selection list keeps the keyboard focus. The user can read dispatch feedback via toasts and inspect the opened pane with herdr pane navigation (`prefix+o`, `prefix+x` to close). The `P n` and `r i` shortcuts opt in to **focus the new pane immediately** (see **Focused shortcuts via `focus: true`** below). Out of scope and unchanged: the downtime worker (already `--no-focus`), the `open-pi-agent` unbound plugin action, and `open.sh`/`toggle.sh`. +- **Pi agent dispatch** — Agent commands (`/skill:*`, `/intake`, `/plan`) are intercepted by the entry point and routed to a new pi agent pane. The `send-to-pi.sh` script splits the current pane to the right (or, when the command carries a work-item ID, the item-ID tab's anchor pane — see [Interactive pane placement](#interactive-pane-placement-work-item-id-tabs)), creates a new pane, runs `pi` with the command as the initial prompt, and renames the pane to "Pi Agent". The dispatch passes `--no-focus` (selection-list dispatch keeps focus, see above) unless the shortcut opts in to **focus the new pane** (`focus: true`, see **Focused shortcuts via `focus: true`** above), in which case `--focus` is passed and the new pane is zoomed. Agent commands are routed before any prefix handling, so they are unaffected by `!!`/`!` processing. - **No-pane dispatch via `open_pane: false`** — A shortcut entry may carry an optional `open_pane: false` flag (WL-0MSJLD1I70045ZUL) to run its command **in the background without opening a pane**: shell (`!!`/`!`) commands execute via detached `bash -c` and agent commands run headless (`pi -p --mode json`, honoring the entry's `model`), with stdout/stderr captured to a per-run log file under `/herdr-background-logs/` (the path is written to stderr so it can be located for inspection). No pane is created, so the work-item ↔ pane association is skipped for agent commands. The bundled quiet state-change shortcuts use it: `a-y` audit approve, `a-r` audit reject, `u-p-*` priority updates, and `x-c`/`x-d` close/delete — they complete silently and the worklist refresh shows the updated state. If a background command exits non-zero, a **failure toast** is shown with the command, its exit code (or termination signal) and a short excerpt of its output (WL-0MUEBQLRD00288VV); the full output remains in the per-run log file. Shortcuts without the flag (all other bundled entries) open a pane exactly as today. -- **Focused shortcuts via `focus: true`** — A shortcut entry may carry an optional `focus: true` flag (WL-0MT70LC6B009TL3Q) to **focus the newly opened pane** immediately after it spawns. When absent or `false` (the default) the selection list keeps focus (the behaviour above). The flag is orthogonal to `open_pane: false` (no pane → focus is moot) and invalid values are logged and treated as absent (no-focus). The only bundled entry with the flag is `P n` (new Pi session — blank `/prompt:`) so pressing `P n` puts the cursor straight in the new Pi pane for immediate typing without an extra `prefix+o` step. Any future shortcut can opt in with the same field without a schema change. +- **Focused shortcuts via `focus: true`** — A shortcut entry may carry an optional `focus: true` flag (WL-0MT70LC6B009TL3Q) to **focus the newly opened pane** immediately after it spawns. When absent or `false` (the default) the selection list keeps focus (the behaviour above). The flag is orthogonal to `open_pane: false` (no pane → focus is moot) and invalid values are logged and treated as absent (no-focus). Focus is applied to the pane's **tab** as well as the pane itself (`run-in-pane.sh` resolves the new pane's tab via `herdr pane get` and runs `herdr tab focus` before the pane zoom, WL-0MURJ0LFH002O95I): a focused ID-carrying dispatch is anchored to the work-item-ID tab (see [Interactive pane placement](#interactive-pane-placement-work-item-id-tabs)), and `pane zoom` alone cannot switch tabs — without the tab focus the new pane would open in a background tab and the selection list would keep focus. The bundled entries with the flag are `P n` (new Pi session — blank `/prompt:`) and `r i` (producer interview — `wl interview`): `P n` puts the cursor straight in the new Pi pane for immediate typing, and `r i` puts the cursor straight in the interview pane so the producer can answer outstanding questions without an extra `prefix+o` step. Any future shortcut can opt in with the same field without a schema change. - **Model lease release on pane close** — Pi agent panes launched by `send-to-pi.sh` or `open-pi-agent.sh` run pi via `shared/run-pi-agent.sh`, which gives the session a deterministic id (`pi --session-id herdr---`) and registers EXIT/TERM/HUP/INT traps. When the pi session ends — normal exit or pane close (`prefix+x`) — the wrapper runs `shared/release-lease-on-exit.mjs`, which posts to the Local Proxy's `POST {baseUrl}/leases/release` using the **same shared implementation** as the Pi extension (`@worklog/shared/lease-release`), so the proxy's dispatch lease is reclaimed promptly instead of lingering until timeout. The release is strictly best-effort: failures (unreachable proxy, missing `~/.pi/agent/models.json`, unconfigured provider) are silently discarded, a 5s request timeout bounds the pane-close path, and the wrapper always propagates pi's exit status (WL-0MSGI7UIH008USVB). - **Worklog tab naming** — `herdr plugin pane open` creates tabs with generated numeric labels. The `open-podcast-editor-tab` action wraps the same pane-open command and renames the created tab to "Worklog" via `herdr tab rename` (socket API, not session-state editing), so the worklog pane is instantly recognisable in the tab row. Each press still opens a new tab; only the label changes. - **Model selection per shortcut** — Each LLM-bound shortcut entry in `shortcuts.json` may carry an optional `model` field (a pi model pattern such as `plan`, `code`, or `author`). When the command is dispatched to the agent channel, `--model ` is forwarded to the spawned `pi` CLI (e.g. `pi --model code '/skill:implement '`), so every workflow runs on an appropriately specialised model without manual model switching. Agent-bound entries without a `model` field default to `plan`; shell (`!!`) and `/wl` filter entries never carry a model and never receive a `--model` flag. The default mapping in `src/shortcuts.json`: `/plan`, `/intake`, `/skill:audit`, `/prompt:` → `plan`; `/skill:implement` → `code`. diff --git a/packages/herdr/scripts/run-in-pane.sh b/packages/herdr/scripts/run-in-pane.sh index d7923239c..a5eb28a3e 100755 --- a/packages/herdr/scripts/run-in-pane.sh +++ b/packages/herdr/scripts/run-in-pane.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash # run-in-pane.sh — Execute a command visibly in a new Herdr pane # -# Splits the current pane to the right, runs the given command through a +# Splits the current pane to the right (or a given anchor pane when +# --anchor is supplied), runs the given command through a # shell in the new pane so its command line and output are visible, renames # the pane, and manages the pane lifecycle: # - any exit status → the pane stays open so the user can read the output; @@ -10,7 +11,7 @@ # with Enter or herdr `close_pane` (default `prefix+x`) # # Usage: -# run-in-pane.sh [--cwd ] [--no-focus] [--pane-name ] +# run-in-pane.sh [--cwd ] [--focus|--no-focus] [--anchor ] [--pane-name ] [--pane-id-file ] # # The command is executed via `bash -c`, so compound commands (`&&`), # single-quoted arguments (e.g. `--summary 'Approved by manual review'`), @@ -24,8 +25,29 @@ # pane opens in the background (WL-0MSHIA53D009DJOT). # Without the flag the new pane is focused as before # (backward compatible). +# --focus Focus the new pane (the default). Accepted explicitly +# so callers that always pass one of the two forms +# (`buildRunInPaneArgs`) never leak the flag into the +# command. When the new pane lands in a DIFFERENT tab — +# an ID-carrying interactive dispatch anchored to the +# work-item-ID tab (WL-0MUYI3JAO002BTNL) — the pane's tab +# is focused first, because `pane zoom` alone cannot +# switch tabs (WL-0MURJ0LFH002O95I). # --pane-name Pane title (overrides RUN_IN_PANE_NAME and the # default "Command Output"; WL-0MSJ4E8UA005KG9Y) +# --anchor Split from the given pane id (e.g. an item-ID tab's +# root pane) instead of the current pane, so an +# ID-carrying interactive dispatch lands in the +# work-item-ID tab (WL-0MUYI3JAO002BTNL). Absent = +# legacy current-pane split. The flag may also be given +# as `--anchor=`. +# --pane-id-file +# Write the new pane ID as JSON ({"pane_id": ""}) to +# immediately after the split succeeds. Mirror of +# send-to-pi.sh's flag: the interactive dispatch uses it +# as a split-confirmation channel to gate the fail-safe +# placeholder root-pane cleanup (WL-0MUYI3K8H007MPKF). +# May also be given as `--pane-id-file=`. # # Everything after the options — including commands whose first token # starts with `--` — is treated as the command to run. @@ -45,6 +67,27 @@ set -uo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" herdr_bin="${HERDR_BIN_PATH:-herdr}" +# Extract the pane id from a `herdr pane split` JSON envelope. Tolerates the +# `pane_id` / `paneId` / `id` key variants and nested `result` envelopes +# (CLI-shape tolerance). Prints the first id found, or nothing when the +# output cannot be parsed (the caller then logs the raw output and fails). +parse_pane_id() { + printf '%s' "$1" | sed -n \ + -e 's/.*"pane_id": *"\([^"]*\)".*/\1/p' \ + -e 's/.*"paneId": *"\([^"]*\)".*/\1/p' \ + -e 's/.*"id": *"\([^"]*\)".*/\1/p' | head -n1 +} + +# Extract the tab id from a `herdr pane get` JSON envelope (`result.pane.tab_id`) +# so the focus path can switch to the tab the new pane lives in +# (WL-0MURJ0LFH002O95I). Tolerates the `tab_id` / `tabId` key variants; prints +# nothing when absent (the caller then skips tab focus, fail-open). +parse_tab_id() { + printf '%s' "$1" | sed -n \ + -e 's/.*"tab_id": *"\([^"]*\)".*/\1/p' \ + -e 's/.*"tabId": *"\([^"]*\)".*/\1/p' | head -n1 +} + # ── In-pane wrapper mode ───────────────────────────────────────────────── # Invoked by the new pane itself (via `herdr pane run ... exec bash # --exec `). Runs the command, reports the @@ -94,17 +137,27 @@ fi # ── Main mode: split, run, rename ──────────────────────────────────────── pane_name="${RUN_IN_PANE_NAME:-Command Output}" # The command is everything after the leading options. Only --cwd, -# --no-focus and --pane-name are parsed as options at the head of argv; -# everything else (including commands whose first token begins with --) is -# the command. +# --focus, --no-focus, --pane-name and --anchor are parsed as options at the +# head of argv; everything else (including commands whose first token begins +# with --) is the command. target_cwd="" no_focus=false +anchor="" +pane_id_file="" while [ $# -gt 0 ]; do case "${1:-}" in --cwd) target_cwd="$2" shift 2 ;; + --focus) + # Explicit focus: the default. Consuming it here is essential — an + # unrecognised --focus would fall through to the command and the pane + # would run `--focus --cwd …` instead of the real command + # (WL-0MURJ0LFH002O95I). + no_focus=false + shift + ;; --no-focus) no_focus=true shift @@ -113,6 +166,22 @@ while [ $# -gt 0 ]; do pane_name="$2" shift 2 ;; + --anchor) + anchor="$2" + shift 2 + ;; + --anchor=*) + anchor="${1#*=}" + shift + ;; + --pane-id-file) + pane_id_file="$2" + shift 2 + ;; + --pane-id-file=*) + pane_id_file="${1#*=}" + shift + ;; *) break ;; @@ -130,21 +199,30 @@ if ! command -v "$herdr_bin" &>/dev/null; then exit 1 fi -# Split the current pane to the right -# Resolve the target CWD for the new pane: --cwd arg > HERDR_RESOLVED_CWD -# > $PWD. The new pane must start in the correct project root; herdr's -# "follow" policy would otherwise inherit the source pane's CWD (e.g. the -# plugin directory). +# Split the target pane to the right. Resolve the target CWD for the new +# pane: --cwd arg > HERDR_RESOLVED_CWD > $PWD. The new pane must start in +# the correct project root; herdr's "follow" policy would otherwise inherit +# the source pane's CWD (e.g. the plugin directory). +# +# --anchor splits THAT pane instead of the current one (an item-ID +# tab's root pane for an ID-carrying interactive dispatch); without it the +# legacy `--current` split is kept (backward compatible). target_cwd="${target_cwd:-${HERDR_RESOLVED_CWD:-$PWD}}" -split_out="$("$herdr_bin" pane split --current --direction right --no-focus --cwd "$target_cwd" 2>/dev/null || true)" +if [ -n "$anchor" ]; then + split_out="$("$herdr_bin" pane split --pane "$anchor" --direction right --no-focus --cwd "$target_cwd" 2>/dev/null || true)" +else + split_out="$("$herdr_bin" pane split --current --direction right --no-focus --cwd "$target_cwd" 2>/dev/null || true)" +fi if [ -z "$split_out" ]; then - echo "Error: Failed to split pane. Ensure you are inside a herdr session." >&2 + echo "Error: Failed to split pane. Ensure you are inside a herdr session (or pass --anchor with a valid pane id)." >&2 exit 1 fi -# Parse the pane_id from JSON output -np="$(printf '%s' "$split_out" | sed -n 's/.*"pane_id":"\([^"]*\)".*/\1/p' | head -n1)" +# Parse the pane_id from JSON output. Tolerates the `pane_id` / `paneId` / +# `id` key variants and nested `result` envelopes (CLI-shape tolerance, +# AC6); on failure the raw output is logged so a shape change is diagnosable. +np="$(parse_pane_id "$split_out")" if [ -z "$np" ]; then echo "Error: Could not determine new pane ID from split output" >&2 @@ -152,6 +230,14 @@ if [ -z "$np" ]; then exit 1 fi +# When --pane-id-file is given, write the pane ID immediately after the split +# succeeds (mirroring send-to-pi.sh) so the caller can confirm the split +# without parsing herdr output itself. Best effort: a failed write must not +# abort the command execution in the new pane. +if [ -n "$pane_id_file" ]; then + printf '{"pane_id":"%s"}\n' "$np" > "$pane_id_file" 2>/dev/null || true +fi + # Run the command through a shell in the new pane. Each argument is # bash-escaped so the pane's shell re-tokenizes it back to a single argv # element (compound commands with && and quoted --summary values survive). @@ -168,7 +254,20 @@ quoted_pane="$(printf '%q' "$np")" # (WL-0MSHIA53D009DJOT): selection-list dispatches pass --no-focus so the # command-output pane opens without stealing focus; the user can focus it # manually with herdr pane navigation. +# +# An ID-carrying interactive dispatch anchors the split to the work-item-ID +# tab (WL-0MUYI3JAO002BTNL), so the new pane can live in a DIFFERENT tab than +# the caller's selection list. `pane zoom` only affects the pane within its +# tab — it cannot switch tabs — so the pane's tab is focused first +# (WL-0MURJ0LFH002O95I). Resolving the tab id is best-effort and fail-open: an +# unavailable id falls back to the legacy zoom-only behaviour (the pane still +# receives keyboard focus when it shares the caller's tab). if [ "$no_focus" = false ]; then + pane_get_out="$("$herdr_bin" pane get "$np" 2>/dev/null || true)" + tab_id="$(parse_tab_id "$pane_get_out")" + if [ -n "$tab_id" ]; then + "$herdr_bin" tab focus "$tab_id" >/dev/null 2>&1 || true + fi "$herdr_bin" pane zoom "$np" --on >/dev/null 2>&1 || true "$herdr_bin" pane zoom "$np" --off >/dev/null 2>&1 || true fi diff --git a/packages/herdr/scripts/tests/test_run_in_pane.sh b/packages/herdr/scripts/tests/test_run_in_pane.sh index 40d99ef25..ceadd648d 100755 --- a/packages/herdr/scripts/tests/test_run_in_pane.sh +++ b/packages/herdr/scripts/tests/test_run_in_pane.sh @@ -8,6 +8,11 @@ # - --no-focus skips the final pane zoom (selection list keeps focus) # - omitting --no-focus preserves the current focus/zoom behavior # - --no-focus does not affect pane creation / command execution +# Covers the --focus flag and tab focus (WL-0MURJ0LFH002O95I): +# - --focus is consumed by the parser, never leaked into the command +# - a focused run focuses the new pane's tab (item-ID anchor placement) +# - --no-focus skips tab focus too +# - tab focus fails open when the tab id cannot be resolved # # Run from the repo root (or anywhere): # bash packages/herdr/scripts/tests/test_run_in_pane.sh @@ -108,6 +113,9 @@ case "\$1" in run) echo "mock: ran" ;; + get) + echo '{"result":{"pane":{"pane_id":"test-pane-main","tab_id":"w1:t9"}}}' + ;; rename|zoom) ;; *) @@ -116,6 +124,16 @@ case "\$1" in ;; esac ;; + tab) + case "\$2" in + focus) + ;; + *) + echo "mock: unknown tab subcommand \$2" >&2 + exit 1 + ;; + esac + ;; *) echo "mock: unknown command \$1" >&2 exit 1 @@ -142,6 +160,86 @@ else fail "without --no-focus the zoom must still run (log: $(cat "$HERDR_LOG" 2>/dev/null))" fi +# --focus is consumed as an option, never passed to the executed command +# (WL-0MURJ0LFH002O95I): before the fix the parser did not know --focus, so it +# fell through and the pane ran `--focus --cwd … ` — the command never +# executed. +rm -f "$HERDR_LOG" +HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --focus --cwd /tmp "echo FOCUS_OK" < /dev/null >/dev/null 2>&1 +if grep -q "pane run" "$HERDR_LOG" 2>/dev/null && grep -q "FOCUS_OK" "$HERDR_LOG" 2>/dev/null && ! grep -q -- "--focus" "$HERDR_LOG" 2>/dev/null; then + pass "--focus is consumed and never leaked into the executed command" +else + fail "--focus must not leak into the command (log: $(cat "$HERDR_LOG" 2>/dev/null))" +fi + +# A focused run focuses the TAB the new pane lives in: an ID-carrying +# interactive dispatch anchors the split to the work-item-ID tab, and +# `pane zoom` alone cannot switch tabs (WL-0MURJ0LFH002O95I). +rm -f "$HERDR_LOG" +HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --focus --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 +if grep -q "tab focus w1:t9" "$HERDR_LOG" 2>/dev/null; then + pass "--focus focuses the new pane's tab (tab focus w1:t9)" +else + fail "--focus must focus the new pane's tab (log: $(cat "$HERDR_LOG" 2>/dev/null))" +fi + +# --no-focus skips tab focus as well, so the selection list keeps focus. +rm -f "$HERDR_LOG" +HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --no-focus --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 +if grep -q "tab focus" "$HERDR_LOG" 2>/dev/null; then + fail "--no-focus must not focus the pane's tab (log: $(cat "$HERDR_LOG" 2>/dev/null))" +else + pass "--no-focus skips tab focus (selection list keeps focus)" +fi + +# The anchored (item-ID tab) focused dispatch focuses that tab. +rm -f "$HERDR_LOG" +HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --anchor w9:p7 --focus --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 +if grep -q "pane split --pane w9:p7" "$HERDR_LOG" 2>/dev/null && grep -q "tab focus w1:t9" "$HERDR_LOG" 2>/dev/null; then + pass "anchored focused run focuses the item tab (--anchor + tab focus)" +else + fail "anchored focused run must focus the item tab (log: $(cat "$HERDR_LOG" 2>/dev/null))" +fi + +# Tab focus is best-effort: when the tab id cannot be resolved (pane get +# fails), the run still succeeds and the pane zoom is attempted. +MOCK_HERDR_NOTAB="$SANDBOX/mock-herdr-notab" +cat > "$MOCK_HERDR_NOTAB" <> "$HERDR_LOG" +case "\$1" in + pane) + case "\$2" in + split) + echo '{"pane_id":"test-pane-main","success":true}' + ;; + run) + echo "mock: ran" + ;; + get) + exit 1 + ;; + rename|zoom) + ;; + *) + exit 1 + ;; + esac + ;; + *) + exit 1 + ;; +esac +MOCK +chmod +x "$MOCK_HERDR_NOTAB" +rm -f "$HERDR_LOG" +HERDR_BIN_PATH="$MOCK_HERDR_NOTAB" bash "$RUN_IN_PANE" --focus --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 +if grep -q "zoom" "$HERDR_LOG" 2>/dev/null && ! grep -q "tab focus" "$HERDR_LOG" 2>/dev/null; then + pass "tab focus fails open when the tab id is unavailable (zoom still runs)" +else + fail "tab focus must fail open when the tab id is unavailable (log: $(cat "$HERDR_LOG" 2>/dev/null))" +fi + # --cwd is still honoured alongside --no-focus (the command still runs) rm -f "$HERDR_LOG" HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --no-focus --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 @@ -151,6 +249,59 @@ else fail "--no-focus must not skip pane run (log: $(cat "$HERDR_LOG" 2>/dev/null))" fi +# --anchor splits the given pane instead of --current (AC2/AC6): +# an ID-carrying interactive dispatch anchors to the item tab's root pane. +# TDD red phase: run-in-pane.sh does not parse --anchor yet, so these cases +# fail until the sibling implementation child (WL-0MUYI3JAO002BTNL) lands. +echo "" +echo "=== Test: --anchor splits the given pane instead of --current ===" + +rm -f "$HERDR_LOG" +HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --anchor w9:p7 --no-focus --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 +if grep -q "pane split --pane w9:p7" "$HERDR_LOG" 2>/dev/null; then + pass "--anchor splits the given pane (--pane w9:p7)" +else + fail "--anchor should split --pane w9:p7 (log: $(cat "$HERDR_LOG" 2>/dev/null))" +fi +if grep -q "pane split --current" "$HERDR_LOG" 2>/dev/null; then + fail "--anchor must not split --current" +else + pass "--anchor does not use --current" +fi + +# Omitting --anchor keeps the current-pane split (backward compatible). +rm -f "$HERDR_LOG" +HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --no-focus --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 +if grep -q "pane split --current" "$HERDR_LOG" 2>/dev/null; then + pass "omitting --anchor keeps the current-pane split" +else + fail "without --anchor the split must use --current (log: $(cat "$HERDR_LOG" 2>/dev/null))" +fi + +# --pane-id-file writes the split pane id immediately after the split +# (WL-0MUYI3K8H007MPKF): the interactive dispatch uses this as the +# split-confirmation channel that gates the fail-safe placeholder cleanup. +echo "" +echo "=== Test: --pane-id-file writes the new pane id after the split ===" + +ID_FILE="$SANDBOX/pane-id.json" +rm -f "$HERDR_LOG" "$ID_FILE" +HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --pane-id-file "$ID_FILE" --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 +if [ -f "$ID_FILE" ] && grep -q '"pane_id":"test-pane-main"' "$ID_FILE"; then + pass "--pane-id-file writes the split pane id JSON" +else + fail "--pane-id-file should write the split pane id (file: $(cat "$ID_FILE" 2>/dev/null))" +fi + +# Omitting --pane-id-file writes nothing (opt-in channel, backward compatible). +rm -f "$ID_FILE" +HERDR_BIN_PATH="$MOCK_HERDR" bash "$RUN_IN_PANE" --cwd /tmp "echo hi" < /dev/null >/dev/null 2>&1 +if [ ! -f "$ID_FILE" ]; then + pass "omitting --pane-id-file writes no file" +else + fail "without --pane-id-file no file should be written" +fi + rm -rf "$SANDBOX" echo "" diff --git a/packages/herdr/shared/run-pi-agent.sh b/packages/herdr/shared/run-pi-agent.sh index e9fd08804..66ae155f8 100644 --- a/packages/herdr/shared/run-pi-agent.sh +++ b/packages/herdr/shared/run-pi-agent.sh @@ -24,6 +24,13 @@ set -uo pipefail +# Defence-in-depth (SA-0MUIZSXEY008NGR8 / WL-0MV0TZEWZ003ZXEB): never carry a +# repository-override git variable into the agent session. Git honours GIT_DIR +# over cwd, so a value leaked in from the launcher environment could redirect a +# `git` command (or a test subprocess) into the live checkout. +unset GIT_DIR GIT_WORK_TREE GIT_CONFIG GIT_CONFIG_GLOBAL GIT_CONFIG_SYSTEM \ + GIT_CONFIG_NOSYSTEM GIT_OBJECT_DIRECTORY GIT_ALTERNATE_OBJECT_DIRECTORIES + lease_session_id="${1:-}" shift || true diff --git a/packages/herdr/shared/send-to-pi.sh b/packages/herdr/shared/send-to-pi.sh index 27f6ac959..8d22e5b81 100755 --- a/packages/herdr/shared/send-to-pi.sh +++ b/packages/herdr/shared/send-to-pi.sh @@ -49,6 +49,13 @@ set -uo pipefail +# Defence-in-depth (SA-0MUIZSXEY008NGR8 / WL-0MV0TZEWZ003ZXEB): do not forward a +# repository-override git variable to the spawned pane. Git honours GIT_DIR over +# cwd, so a leaked value could make an agent/test `git` command operate on the +# live checkout instead of the pane's own project. +unset GIT_DIR GIT_WORK_TREE GIT_CONFIG GIT_CONFIG_GLOBAL GIT_CONFIG_SYSTEM \ + GIT_CONFIG_NOSYSTEM GIT_OBJECT_DIRECTORY GIT_ALTERNATE_OBJECT_DIRECTORIES + # ── Help ──────────────────────────────────────────────────────────────── show_help() { sed -n '/^# Usage/,/^$/p' "$0" | sed 's/^# //; s/^#$//' diff --git a/packages/herdr/src/coordination-cross-root.test.ts b/packages/herdr/src/coordination-cross-root.test.ts index 0980ccdde..3a86f5fec 100644 --- a/packages/herdr/src/coordination-cross-root.test.ts +++ b/packages/herdr/src/coordination-cross-root.test.ts @@ -9,12 +9,20 @@ * worklogRoot. */ -import { describe, it, expect, vi } from 'vitest'; +import { describe, it, expect, vi, beforeEach } from 'vitest'; import { mkdtempSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { dispatchFromCoordination, type DowntimeWorkerDeps, type DowntimeItemInfo } from './downtime-worker.js'; +import { dispatchFromCoordination, type DowntimeWorkerDeps, type DowntimeItemInfo, type HerdrPaneRecord } from './downtime-worker.js'; import type { CoordinationEntry } from './coordination.js'; +import { + scanStalledPanes, + resumeStalledPane, + _resetStalledResumeInFlight, + type StalledScanDeps, + type ResumeStalledPaneDeps, +} from './stalled-work.js'; +import type { DowntimeLogEntry } from './downtime-log.js'; let shared: string; function withShared(fn: () => Promise | void) { @@ -109,6 +117,156 @@ describe('F4 cross-root offer-list dispatch', () => { expect(String(spawn[0])).toContain('WL-IMPL'); })); + // ── Stalled-work cross-root resume (WL-0MUYMBUAK007H10J) ──────────── + // End-to-end through the LEADER path: the real `scanStalledPanes` + + // `resumeStalledPane` orchestrators are wired as the dispatcher deps, so a + // stalled pane for an item in a FOREIGN worklog root is resumed against + // that root (never the leader's) and an unresolvable root fails closed and + // falls through to the normal offer dispatch. + + const STALE = () => new Date(Date.now() - 10 * 60 * 1000).toISOString(); + + function stalledPane(overrides: Partial = {}): HerdrPaneRecord { + return { + paneId: 'pane-foreign', + label: 'Downtime triggered implement Foreign item - WL-FOREIGN', + agent: 'pi', + agentStatus: 'idle', + cwd: '/foreign/repo', + ...overrides, + }; + } + + function foreignItem(id = 'WL-FOREIGN'): DowntimeItemInfo { + return { id, title: 'Foreign item', status: 'open', stage: 'plan_complete', updatedAt: STALE() } as DowntimeItemInfo; + } + + function wireRealScanResume( + scanDeps: StalledScanDeps, + resumeDeps: ResumeStalledPaneDeps, + ): Pick { + return { + scanStalledPanes: (cwd, opts) => scanStalledPanes(scanDeps, cwd, opts), + resumeStalledPane: (cand, opts) => + resumeStalledPane( + { + pane: cand.pane, + item: cand.item, + cwd: cand.cwd, + kind: cand.kind, + nonTerminalCooldownMs: opts.nonTerminalCooldownMs, + thresholdMs: opts.thresholdMs, + }, + resumeDeps, + ), + }; + } + + beforeEach(() => { + _resetStalledResumeInFlight(); + }); + + it('resumes a foreign-root stalled pane against ITS OWN root, not the leader\u2019s', withShared(async () => { + const leaderRoot = '/leader/repo'; + const foreignRoot = '/foreign/repo'; + const prompted: Array<[string, string]> = []; + const fetched: Array<[string, string]> = []; + const recorded: DowntimeLogEntry[] = []; + + const scanDeps: StalledScanDeps = { + listPanes: async () => [stalledPane({ cwd: foreignRoot })], + fetchItem: async (id, cwd) => { + fetched.push([id, cwd]); + return { ok: true, info: foreignItem(id) }; + }, + readEntries: async () => [], + resolveRoot: (pane) => (pane.cwd === foreignRoot ? foreignRoot : null), + }; + const resumeDeps: ResumeStalledPaneDeps = { + promptAgent: async (paneId, text) => { + prompted.push([paneId, text]); + return { ok: true }; + }, + startAgent: async () => ({ ok: true }), + readEntries: async () => [], + recordAttempt: async (cwd, entry) => { + recorded.push({ ...entry, cwd } as DowntimeLogEntry); + }, + markNeedsProducerReview: async () => true, + maxAttempts: 3, + now: () => Date.now(), + }; + + const d = deps({ + fetchItem: vi.fn().mockResolvedValue({ ok: true, info: info({ id: 'WL-OFFER', status: 'open', stage: 'idea' }) }), + ...wireRealScanResume(scanDeps, resumeDeps), + }); + + const out = await dispatchFromCoordination( + d, + [entry('inst-leader', 'WL-OFFER', leaderRoot)], + { model: 'plan', cwd: leaderRoot, coordinationDir: shared, freeSlots: 2 }, + ); + + // A successful resume short-circuits new-item dispatch for the cycle. + expect(out).toEqual({ dispatched: false, reason: 'stalled-resume' }); + // The item was fetched against the PANE'S root, never the leader's. + expect(fetched).toEqual([['WL-FOREIGN', foreignRoot]]); + expect(prompted).toEqual([['pane-foreign', 'continue']]); + // The dispatch attempt was recorded on the FOREIGN root. + expect(recorded).toHaveLength(1); + expect(recorded[0]).toMatchObject({ itemId: 'WL-FOREIGN', kind: 'implement', cwd: foreignRoot }); + // No pane was spawned in the leader's root and the offer was never fetched. + expect(d.spawnAgentPane).not.toHaveBeenCalled(); + const offerFetch = (d.fetchItem as ReturnType).mock.calls; + expect(offerFetch).toHaveLength(0); + })); + + it('fails closed when the pane root is unresolvable and dispatches the offer normally (no wrong-root resume)', withShared(async () => { + const leaderRoot = '/leader/repo'; + const prompted: string[] = []; + const fetchedRoots: string[] = []; + + const scanDeps: StalledScanDeps = { + listPanes: async () => [stalledPane({ cwd: '/definitely/not/a/worklog' })], + fetchItem: async (id, cwd) => { + fetchedRoots.push(cwd); + return { ok: true, info: foreignItem(id) }; + }, + readEntries: async () => [], + // The cwd/session cannot be resolved to an unambiguous worklog root. + resolveRoot: () => null, + }; + const resumeDeps: ResumeStalledPaneDeps = { + promptAgent: async (paneId) => { + prompted.push(paneId); + return { ok: true }; + }, + startAgent: async () => ({ ok: true }), + readEntries: async () => [], + recordAttempt: async () => undefined, + markNeedsProducerReview: async () => true, + maxAttempts: 3, + }; + + const d = deps({ + fetchItem: vi.fn().mockResolvedValue({ ok: true, info: info({ id: 'WL-OFFER', status: 'open', stage: 'idea' }) }), + ...wireRealScanResume(scanDeps, resumeDeps), + }); + + const out = await dispatchFromCoordination( + d, + [entry('inst-leader', 'WL-OFFER', leaderRoot)], + { model: 'plan', cwd: leaderRoot, coordinationDir: shared, freeSlots: 2 }, + ); + + // Fail closed: no resume, no wrong-root fetch, and normal offer dispatch. + expect(prompted).toEqual([]); + expect(fetchedRoots).toEqual([]); + expect(out.dispatched).toBe(true); + expect((d.spawnAgentPane as ReturnType).mock.calls[0][1].cwd).toBe(leaderRoot); + })); + it('worklogRoot preferred over directory (compat) and fetchItem receives worklogRoot', withShared(async () => { const root = '/repo/b'; const e: CoordinationEntry = { instanceId: 'inst', workItemId: 'WL-1', directory: '/legacy', worklogRoot: root, assignedAt: new Date().toISOString(), lastUpdated: new Date().toISOString() }; diff --git a/packages/herdr/src/coordination-dispatch.test.ts b/packages/herdr/src/coordination-dispatch.test.ts index f60fab386..0fbf8ad8a 100644 --- a/packages/herdr/src/coordination-dispatch.test.ts +++ b/packages/herdr/src/coordination-dispatch.test.ts @@ -30,6 +30,7 @@ import { computeMostImportantItem, coordinationTierRank, DOWNTIME_AUDIT_RECENCY_WINDOW_MS, + DOWNTIME_DISPATCH_EXTEND_MAX, type DowntimeWorker, type DowntimeWorkerDeps, type DowntimeItemInfo, @@ -1034,12 +1035,19 @@ describe('runCoordinationCheckIn', () => { function makeStatusPayload(overrides: Record = {}): Record { return { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 4, total_slots: 4, + // Per-slot detail is required under the fail-closed contract + // (WL-0MUXVPXAZ005RESW). + slots: [ + { slot_id: 'slot-1', is_processing: false }, + { slot_id: 'slot-2', is_processing: false }, + { slot_id: 'slot-3', is_processing: false }, + { slot_id: 'slot-4', is_processing: false }, + ], ...overrides, }; } @@ -1905,3 +1913,347 @@ describe('in-flight guard on the coordination offer path (WL-0MUBVKS5I007LSWB / expect(result.kind).toBe('implement'); }); }); + +// ── Own-backlog fallback — leader selection contract (WL-0MUZPBQQ3001DAM6) ── +// Parent WL-0MUU4XFU90008E24: when the sprint-view head yields no dispatchable +// candidate, the coordination-mode leader MAY fall back to its OWN worklog +// backlog (items beyond the sprint-view window). This child pins the SELECTION +// contract only (parent AC1, AC2, AC3, AC9): +// +// * it dispatches an eligible non-critical item that sits beyond the live +// sprint-view window in the LEADER'S OWN root; +// * it never reads another instance's/root's hidden backlog (own `cwd` only); +// * it preserves the canonical ranking (the same `getHerdrListHead` order — +// a bounded window extension, never a second ranking); +// * the fallback window is bounded by `DOWNTIME_DISPATCH_EXTEND_MAX`; +// * it fails open (a `{ok:false}`/throwing/empty lookup leaves the original +// terminal reason unchanged). +// +// RED WITNESS. The fallback does not exist yet, so every body asserts the +// post-implementation behaviour and is committed with `it.fails` — the suite +// stays GREEN while the assertions are RED (the repo's documented TDD pattern; +// see `docs/dev/downtime-dispatcher-post-fix-verification.md`). The +// implementation child WL-0MUUZPBSYW004I5D3 flips each `it.fails` to `it` once +// the fallback lands (the RED→GREEN transition is the dependency evidence). +describe('own-backlog fallback — leader selection contract (WL-0MUZPBQQ3001DAM6)', () => { + // The live sprint-view window used by every case (== DEFAULT_BROWSE_ITEM_COUNT). + const SPRINT = 20; + + /** A full in-view head whose every item is filtered by the producer-review gate. */ + const filteredInView = (): DowntimeHerdrItem[] => + Array.from({ length: SPRINT }, (_, i) => + headItem({ + id: `WL-VIEW-${String(i).padStart(2, '0')}`, + title: `Filtered in-view ${i}`, + status: 'open', + stage: 'plan_complete', + priority: 'high', + risk: 'Low', + effort: 'S', + needsProducerReview: true, + sortIndex: i, + }), + ); + + /** An eligible, dispatchable non-critical implement item. */ + const eligibleBacklog = (id: string, sortIndex = 1000): DowntimeHerdrItem => + headItem({ + id, + title: `Backlog ${id}`, + status: 'open', + stage: 'plan_complete', + priority: 'high', + risk: 'Low', + effort: 'S', + sortIndex, + }); + + /** + * Leader-own-root `getHerdrListHead` mock modelling the canonical ranking: + * a window at/below the sprint view returns only the filtered head; a larger + * window returns the filtered head followed by the ranked out-of-view tail. + */ + const rankedOwnRoot = ( + tail: readonly DowntimeHerdrItem[], + inView: DowntimeHerdrItem[] = filteredInView(), + ) => + vi.fn(async (_cwd: string, limit = SPRINT) => ({ + ok: true as const, + items: limit <= SPRINT ? inView : [...inView, ...tail], + })); + + const dispatch = (deps: DowntimeWorkerDeps, entries: CoordinationEntry[] = []) => + dispatchFromCoordination(deps, entries, { + model: 'plan', + cwd: '/leader-root', + coordinationDir: testDir, + browseItemCount: SPRINT, + }); + + it( + 'AC1: dispatches an eligible non-critical item beyond the sprint-view window from the leader own root', + async () => { + const getHerdrListHead = rankedOwnRoot([eligibleBacklog('WL-BACKLOG-1')]); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + expect(outcome.dispatched).toBe(true); + expect(outcome.kind).toBe('implement'); + expect(outcome.candidate?.id).toBe('WL-BACKLOG-1'); + // Dispatched against the leader's OWN root (claim + spawn cwd). + expect(deps.claimItem).toHaveBeenCalledWith('WL-BACKLOG-1', expect.anything(), '/leader-root'); + expect(deps.spawnAgentPane).toHaveBeenCalledWith( + expect.stringContaining('/skill:implement WL-BACKLOG-1'), + expect.objectContaining({ cwd: '/leader-root' }), + ); + }, + ); + + it( + 'AC1: falls back when every in-view item is filtered, not only when the view is literally empty', + async () => { + const getHerdrListHead = rankedOwnRoot([eligibleBacklog('WL-FILTERED-BACKLOG')]); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-FILTERED-BACKLOG'); + // The leader's own root was read at all (own-root scope). + expect(getHerdrListHead).toHaveBeenCalledWith('/leader-root', expect.any(Number)); + }, + ); + + it( + 'AC2: reads only the leader own root — another root hidden backlog is never selected', + async () => { + const getHerdrListHead = vi.fn(async (cwd: string, limit = SPRINT) => ({ + ok: true as const, + items: + limit <= SPRINT + ? filteredInView() + : [ + ...filteredInView(), + ...(cwd === '/leader-root' + ? [eligibleBacklog('WL-OWN-BACKLOG')] + : [eligibleBacklog('WL-OTHER-BACKLOG')]), + ], + })); + const deps = makeCoordinationDeps({ + getHerdrListHead, + // The other root's coordination offer is stale/non-dispatchable, so + // nothing from the offer list survives and the fallback must run. + fetchItem: vi.fn().mockResolvedValue({ + ok: true, + info: itemInfo({ id: 'WL-OTHER-ITEM', status: 'in_progress', stage: 'in_progress' }), + }), + }); + + const outcome = await dispatch(deps, [makeEntry('inst-other', 'WL-OTHER-ITEM', '/other-root')]); + + expect(outcome.candidate?.id).toBe('WL-OWN-BACKLOG'); + expect(outcome.candidate?.id).not.toBe('WL-OTHER-BACKLOG'); + // The fallback lookup is never issued against another root. + expect(getHerdrListHead).not.toHaveBeenCalledWith('/other-root', expect.any(Number)); + }, + ); + + it( + 'AC3: preserves canonical ranking order (first ranked eligible wins, not id/insertion order)', + async () => { + // Canonical ranking places WL-RANK-Z ahead of WL-RANK-A. + const getHerdrListHead = rankedOwnRoot([ + eligibleBacklog('WL-RANK-Z', 1000), + eligibleBacklog('WL-RANK-A', 2000), + ]); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-RANK-Z'); + expect(deps.spawnAgentPane).toHaveBeenCalledTimes(1); + }, + ); + + it( + 'AC3: skips a ranked out-of-view item blocked by a safety gate and dispatches the next eligible', + async () => { + const getHerdrListHead = rankedOwnRoot([ + headItem({ + id: 'WL-RANK-GATED', + status: 'open', + stage: 'plan_complete', + priority: 'high', + risk: 'Low', + effort: 'S', + needsProducerReview: true, + sortIndex: 1000, + }), + eligibleBacklog('WL-RANK-ELIGIBLE', 2000), + ]); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-RANK-ELIGIBLE'); + }, + ); + + it( + 'AC9: the fallback window is bounded by DOWNTIME_DISPATCH_EXTEND_MAX', + async () => { + const getHerdrListHead = rankedOwnRoot([eligibleBacklog('WL-BOUNDED')]); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + await dispatch(deps); + + const limits = (getHerdrListHead as ReturnType).mock.calls.map( + (call) => call[1] as number, + ); + expect(limits.length).toBeGreaterThan(0); + // The largest window requested is the sprint view plus the bounded + // extension — never an unbounded backlog read. + expect(Math.max(...limits)).toBe(SPRINT + DOWNTIME_DISPATCH_EXTEND_MAX); + }, + ); + + it( + 'AC9 fail-open: a {ok:false} own-root lookup leaves the original terminal reason unchanged', + async () => { + const getHerdrListHead = vi.fn().mockResolvedValue({ ok: false, error: 'boom' }); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + expect(getHerdrListHead).toHaveBeenCalledWith('/leader-root', expect.any(Number)); + expect(outcome.dispatched).toBe(false); + expect(outcome.reason).toBe('no-candidate'); + expect(deps.spawnAgentPane).not.toHaveBeenCalled(); + }, + ); + + it( + 'AC9 fail-open: a throwing own-root lookup leaves the original terminal reason unchanged', + async () => { + const getHerdrListHead = vi.fn().mockRejectedValue(new Error('boom')); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + expect(getHerdrListHead).toHaveBeenCalledWith('/leader-root', expect.any(Number)); + expect(outcome.dispatched).toBe(false); + expect(outcome.reason).toBe('no-candidate'); + }, + ); + + it( + 'AC9 fail-open: an empty fallback tail leaves the original terminal reason unchanged', + async () => { + const getHerdrListHead = vi.fn().mockResolvedValue({ ok: true, items: [] }); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + expect(getHerdrListHead).toHaveBeenCalledWith('/leader-root', expect.any(Number)); + expect(outcome.dispatched).toBe(false); + expect(outcome.reason).toBe('no-candidate'); + }, + ); + + // ── Leadership step-down surface (WL-0MUZPBRX3007STFK) ─────────────── + // + // `dispatchFromCoordination` does NOT own the leader-election manager (the + // worker loop does, `downtime-worker.ts`), so it cannot call + // `releaseLeadership()` itself. Instead it surfaces a machine-readable + // `ownBacklogDispatch` flag on the outcome — the worker loop steps down + // exactly once per successful own-backlog dispatch, and never for a normal + // in-view dispatch (parent AC4). These tests pin that surface; the + // worker-level `releaseLeadership()` call-count and the next-tick + // non-leader observation are pinned in `downtime-worker.test.ts` / + // `integration-leader-dispatch.test.ts`. + + /** + * Read the step-down surface without a compile-time dependency on the + * (not-yet-declared) field — this test item changes no production code, so + * the flag is read through a narrow structural cast rather than widening + * `DowntimeDispatchOutcome` here. + */ + const hasOwnBacklogFlag = (outcome: { dispatched: boolean }): boolean => + (outcome as { ownBacklogDispatch?: boolean }).ownBacklogDispatch === true; + + it( + 'step-down surface AC1: a successful own-backlog dispatch flags ownBacklogDispatch', + async () => { + const getHerdrListHead = rankedOwnRoot([eligibleBacklog('WL-BACKLOG-STEPDOWN')]); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-BACKLOG-STEPDOWN'); + // The stage-appropriate dispatch completes before the surface is set. + expect(deps.claimItem).toHaveBeenCalled(); + expect(deps.recordDispatch).toHaveBeenCalled(); + expect(deps.spawnAgentPane).toHaveBeenCalled(); + expect(hasOwnBacklogFlag(outcome)).toBe(true); + }, + ); + + it( + 'step-down surface AC2: a normal in-view (offer-list) dispatch does NOT flag ownBacklogDispatch', + async () => { + // An eligible offer in the shared list — the dispatch never reaches the + // own-backlog fallback, so no step-down is surfaced. + const deps = makeCoordinationDeps({ + fetchItem: vi.fn().mockResolvedValue({ + ok: true, + info: itemInfo({ id: 'WL-INVIEW', status: 'open', stage: 'plan_complete', risk: 'Low', effort: 'S' }), + }), + }); + + const outcome = await dispatch(deps, [makeEntry('inst-inview', 'WL-INVIEW', '/leader-root')]); + + expect(outcome.dispatched).toBe(true); + expect(outcome.kind).toBe('implement'); + expect(hasOwnBacklogFlag(outcome)).toBe(false); + // The own-root fallback read never ran. + expect(deps.getHerdrListHead).not.toHaveBeenCalled(); + }, + ); + + it( + 'step-down surface AC3: when the fallback finds no eligible item no step-down is surfaced and the reason is unchanged', + async () => { + // The only out-of-view item is blocked by a safety gate (producer + // review) — the fallback finds nothing to dispatch, so leadership must + // be retained and the pre-existing terminal reason reported. + const gatedTail = headItem({ + id: 'WL-OOV-GATED', + status: 'open', + stage: 'plan_complete', + priority: 'high', + risk: 'Low', + effort: 'S', + needsProducerReview: true, + sortIndex: 1000, + }); + const getHerdrListHead = vi.fn(async (_cwd: string, limit = SPRINT) => ({ + ok: true as const, + items: limit <= SPRINT ? filteredInView() : [...filteredInView(), gatedTail], + })); + const deps = makeCoordinationDeps({ getHerdrListHead }); + + const outcome = await dispatch(deps); + + // The fallback was attempted (post-implementation) — RED before it lands. + expect(getHerdrListHead).toHaveBeenCalledWith('/leader-root', expect.any(Number)); + expect(outcome.dispatched).toBe(false); + expect(outcome.reason).toBe('no-candidate'); + expect(hasOwnBacklogFlag(outcome)).toBe(false); + expect(deps.claimItem).not.toHaveBeenCalled(); + }, + ); +}); diff --git a/packages/herdr/src/dispatcher-anchor.test.ts b/packages/herdr/src/dispatcher-anchor.test.ts index 76f3a5914..5c9022388 100644 --- a/packages/herdr/src/dispatcher-anchor.test.ts +++ b/packages/herdr/src/dispatcher-anchor.test.ts @@ -573,6 +573,48 @@ describe('getItemTabAnchor ACs (AC2/AC5)', () => { try { fs.unlinkSync(lockPath); } catch { /* ignore */ } }); + // Create-vs-reuse signal (WL-0MUYI3K8H007MPKF, AC7): the interactive path + // gates its placeholder root-pane cleanup on a genuine create. + it('AC7: invokes onCreate with the anchor ONLY when the tab is provisioned', async () => { + const onCreate = vi.fn(); + const deps = makeTabDeps({ + createTab: async (ws: string, label: string) => ({ + tabId: `${ws}:t${label}`, + paneId: `${ws}:t${label}:p1`, + }), + }); + const got = await getItemTabAnchor('/repo', deps, 'wC', 'WL-ABC', onCreate); + expect(got).toEqual({ tabId: 'wC:tWL-ABC', paneId: 'wC:tWL-ABC:p1' }); + expect(onCreate).toHaveBeenCalledTimes(1); + expect(onCreate).toHaveBeenCalledWith({ tabId: 'wC:tWL-ABC', paneId: 'wC:tWL-ABC:p1' }); + }); + + it('AC7: does NOT invoke onCreate when REUSING an existing tab', async () => { + const onCreate = vi.fn(); + const deps = makeTabDeps({ + createTab: async () => null, + listTabs: async () => [{ tabId: 'wC:tWL-ABC', label: 'WL-ABC' }], + listPanes: async () => [{ paneId: 'wC:tWL-ABC:p1', tabId: 'wC:tWL-ABC' }], + isPaneAlive: async () => true, + }); + const got = await getItemTabAnchor('/repo', deps, 'wC', 'WL-ABC', onCreate); + expect(got).toEqual({ tabId: 'wC:tWL-ABC', paneId: 'wC:tWL-ABC:p1' }); + expect(onCreate).not.toHaveBeenCalled(); + }); + + it('AC7 fail-open: a throwing onCreate never fails provisioning', async () => { + const deps = makeTabDeps({ + createTab: async (ws: string, label: string) => ({ + tabId: `${ws}:t${label}`, + paneId: `${ws}:t${label}:p1`, + }), + }); + const got = await getItemTabAnchor('/repo', deps, 'wC', 'WL-ABC', () => { + throw new Error('observer boom'); + }); + expect(got).toEqual({ tabId: 'wC:tWL-ABC', paneId: 'wC:tWL-ABC:p1' }); + }); + it('machine dir unresolvable → null', async () => { const fileAsDir = path.join(tmpDir, 'not-a-dir'); fs.writeFileSync(fileAsDir, 'x'); diff --git a/packages/herdr/src/dispatcher-anchor.ts b/packages/herdr/src/dispatcher-anchor.ts index 7ea6860ab..4bac219bc 100644 --- a/packages/herdr/src/dispatcher-anchor.ts +++ b/packages/herdr/src/dispatcher-anchor.ts @@ -1218,12 +1218,21 @@ export async function resolveProjectWorkspace( * unparseable `tab list`/`tab create`/`pane list`, or a matching tab whose * anchor pane cannot be resolved) — the caller fails closed (never a * wrong-project placement). + * + * @param onCreate Optional callback invoked with the freshly-provisioned + * anchor when this call CREATED the tab (as opposed to adopting an existing + * one). Lets the interactive dispatch path gate its fail-safe root-pane + * cleanup on a genuine create-vs-reuse signal without a second `tab list` + * round-trip (WL-0MUYI3K8H007MPKF, parent WL-0MUKZGEQ2007FECS, AC7). A + * throwing callback is swallowed — provisioning must never fail because a + * best-effort observer threw. */ export async function getItemTabAnchor( cwd: string, deps: DispatcherAnchorDeps, workspaceId: string, itemId: string, + onCreate?: (anchor: ItemTabAnchor) => void, ): Promise { if (workspaceId === '' || itemId === '') return null; const dir = getMachineCoordinationDir(); @@ -1259,7 +1268,19 @@ export async function getItemTabAnchor( return null; } if (created === null) return null; - return { tabId: created.tabId, paneId: created.paneId }; + const anchor: ItemTabAnchor = { tabId: created.tabId, paneId: created.paneId }; + // Create-vs-reuse signal (WL-0MUYI3K8H007MPKF): only the provisioning + // branch invokes the observer, so the interactive path can close the + // placeholder root pane after the first dispatch. Best-effort: a throwing + // observer must never fail provisioning. + if (typeof onCreate === 'function') { + try { + onCreate(anchor); + } catch { + // fail-open: an observer error never breaks tab provisioning + } + } + return anchor; } finally { release(); } diff --git a/packages/herdr/src/downtime-worker.fixtures.ts b/packages/herdr/src/downtime-worker.fixtures.ts index 7c19069de..cb454a8db 100644 --- a/packages/herdr/src/downtime-worker.fixtures.ts +++ b/packages/herdr/src/downtime-worker.fixtures.ts @@ -15,13 +15,21 @@ import type { LlamaStatus } from './downtime-worker.js'; export const idleAllSlotsFree: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 4, total_slots: 4, current_model: 'qwen3-8b', + // Per-slot detail is the ONLY trusted availability source under the + // fail-closed contract (WL-0MUXVPXAZ005RESW); an idle fixture therefore + // carries a consistent all-free slot array. + slots: [ + { slot_id: 'slot-1', is_processing: false }, + { slot_id: 'slot-2', is_processing: false }, + { slot_id: 'slot-3', is_processing: false }, + { slot_id: 'slot-4', is_processing: false }, + ], // Normalised owner-session list (WL-0MU88086A0089US4): always present on // the parsed status, empty when the proxy reports no owner. local_owner_session_ids: [], @@ -33,7 +41,16 @@ export const busyLocalLease: LlamaStatus = { ...idleAllSlotsFree, local_lease_ac export const busyServerNotRunning: LlamaStatus = { ...idleAllSlotsFree, llama_server_running: false }; /** available_slots < total_slots → busy with the default N=0 (all slots). */ -export const busyNotAllSlotsFree: LlamaStatus = { ...idleAllSlotsFree, available_slots: 2 }; +export const busyNotAllSlotsFree: LlamaStatus = { + ...idleAllSlotsFree, + available_slots: 2, + slots: [ + { slot_id: 'slot-1', is_processing: true }, + { slot_id: 'slot-2', is_processing: true }, + { slot_id: 'slot-3', is_processing: false }, + { slot_id: 'slot-4', is_processing: false }, + ], +}; /** `total_slots` 0 with everything else idle → ambiguous → busy. */ export const ambiguousZeroTotalSlots: LlamaStatus = { @@ -86,7 +103,6 @@ export const perSlotOneProcessing: LlamaStatus = { */ export const perSlotThreeOfFourFree: LlamaStatus = { ...perSlotOneProcessing, - active_query: true, local_active_query: true, local_lease_active: true, }; diff --git a/packages/herdr/src/downtime-worker.test.ts b/packages/herdr/src/downtime-worker.test.ts index 76c17f860..c9e94ca5d 100644 --- a/packages/herdr/src/downtime-worker.test.ts +++ b/packages/herdr/src/downtime-worker.test.ts @@ -76,6 +76,8 @@ import { buildDowntimePaneArgs, spawnDowntimePane, buildDowntimeSpawnOptions, + scrubRepositoryOverrides, + REPOSITORY_OVERRIDE_ENV_VARS, BLOCKED_QUESTIONS_INSTRUCTION, parseNextItemOutput, parseNextCandidatesOutput, @@ -105,6 +107,7 @@ import { clampDowntimeNonTerminalCooldownMs, clampDowntimeMarkerStaleWindowMs, clampDowntimeMaxAttempts, + resumeStalledWorkIfAble, DEFAULT_DOWNTIME_MAX_ATTEMPTS, DOWNTIME_MAX_ATTEMPTS_MIN, DOWNTIME_MAX_ATTEMPTS_MAX, @@ -203,6 +206,12 @@ import { jsonResponseFixture, type LlamaStatusHttpResponse, } from './downtime-worker.fixtures.js'; +import { + resumeStalledPane, + _resetStalledResumeInFlight, + type ResumeStalledPaneDeps, + type StalledPaneCandidate, +} from './stalled-work.js'; /** Shared deps mock for dispatch tests. */ @@ -279,7 +288,6 @@ describe('downtime worker fixtures', () => { describe('idle detection (isIdleStatus)', () => { const idle: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, @@ -304,7 +312,6 @@ describe('idle detection (isIdleStatus)', () => { // the worker must never dispatch on an unverifiable busy signal. const partial: LlamaStatus = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 4, @@ -313,11 +320,11 @@ describe('idle detection (isIdleStatus)', () => { expect(isIdleStatus(partial, 0)).toBe(false); }); - it('is idle during remote-only traffic when local_active_query=false (global active_query true)', () => { - // Remote provider streams keep the GLOBAL active_query true while the + it('is idle during remote-only traffic when local_active_query=false', () => { + // Remote provider streams keep the global query signal true while the // local model is idle with free slots; the proxy's local_active_query is // the local-only signal (LP-0MSL2ZLLS009RVKR) and must not block dispatch. - expect(isIdleStatus({ ...idle, active_query: true, local_active_query: false }, 0)).toBe(true); + expect(isIdleStatus({ ...idle, local_active_query: false }, 0)).toBe(true); }); it('is busy while a model switch is in progress', () => { @@ -3757,7 +3764,6 @@ describe('endpoint failures and poller', () => { it('derives local_lease_active from the lease fields when the boolean is absent', async () => { const status = parseLlamaStatus({ llama_server_running: true, - active_query: false, model_switch_in_progress: false, available_slots: 4, total_slots: 4, @@ -3815,7 +3821,6 @@ describe('endpoint failures and poller', () => { describe('parseLlamaStatus local_active_query', () => { const base = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, available_slots: 4, total_slots: 4, @@ -3849,7 +3854,6 @@ describe('parseLlamaStatus local_active_query', () => { describe('runtime idle evaluation (evaluateIdle)', () => { const idle: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, @@ -3857,29 +3861,53 @@ describe('runtime idle evaluation (evaluateIdle)', () => { total_slots: 4, }; - it('N=0 (default) requires all slots free', () => { - expect(evaluateIdle(idle, 0)).toBe(true); + function slotsFor(processing: number[], total = 4): LlamaSlot[] { + return Array.from({ length: total }, (_, i) => ({ + slot_id: `s${i + 1}`, + is_processing: processing.includes(i + 1), + })); + } + + it('fails closed when per-slot detail is absent (never admits the count)', () => { + // Fail-closed contract (WL-0MUXVPXAZ005RESW): the untrusted + // `available_slots` count is never a dispatch budget — absent `slots` + // means no trustworthy signal, so even a full count is busy. + expect(evaluateIdle(idle, 0)).toBe(false); expect(evaluateIdle({ ...idle, available_slots: 3 }, 0)).toBe(false); + expect(evaluateIdle({ ...idle, available_slots: 4 }, 2)).toBe(false); + }); + + it('fails closed when per-slot detail is empty or stale', () => { + expect(evaluateIdle({ ...idle, slots: [] }, 0)).toBe(false); + expect( + evaluateIdle( + { ...idle, slots: slotsFor([]), slots_stale: true }, + 0, + ), + ).toBe(false); }); - it('degrades 0 < N < total slots to ALL slots free (fail-closed, no per-slot data)', () => { - // Without per-slot identity (LP-0MSG5TA7Y002GN39), N=2 with 4 slots - // must NOT dispatch on "any 2 free" — it requires all 4 free. - expect(evaluateIdle({ ...idle, available_slots: 2 }, 2)).toBe(false); - expect(evaluateIdle({ ...idle, available_slots: 4 }, 2)).toBe(true); + it('N=0 (default) requires all slots free from the per-slot array', () => { + expect(evaluateIdle({ ...idle, slots: slotsFor([]) }, 0)).toBe(true); + // available_slots still says 4 but the per-slot array shows one busy → busy. + expect( + evaluateIdle({ ...idle, slots: slotsFor([4]) }, 0), + ).toBe(false); }); it('N == total slots behaves like all slots free', () => { - expect(evaluateIdle({ ...idle, available_slots: 4 }, 4)).toBe(true); - expect(evaluateIdle({ ...idle, available_slots: 3 }, 4)).toBe(false); + expect(evaluateIdle({ ...idle, slots: slotsFor([]) }, 4)).toBe(true); + expect(evaluateIdle({ ...idle, slots: slotsFor([4]) }, 4)).toBe(false); }); it('N > total slots can never be idle (never dispatches)', () => { - expect(evaluateIdle({ ...idle, available_slots: 4 }, 5)).toBe(false); + expect(evaluateIdle({ ...idle, slots: slotsFor([]) }, 5)).toBe(false); }); it('ambiguous responses (total_slots 0) are busy', () => { - expect(evaluateIdle({ ...idle, total_slots: 0, available_slots: 0 }, 0)).toBe(false); + expect( + evaluateIdle({ ...idle, total_slots: 0, available_slots: 0 }, 0), + ).toBe(false); }); }); @@ -4480,6 +4508,44 @@ describe('downtime pane spawn (send-to-pi.sh)', () => { expect(options.env.AUDIT_PHASE2_PARALLELISM).toBe('1'); }); + it('buildDowntimeSpawnOptions scrubs repository-override git vars (WL-0MV0TZEWZ003ZXEB)', () => { + const saved = new Map(); + for (const key of REPOSITORY_OVERRIDE_ENV_VARS) { + saved.set(key, process.env[key]); + process.env[key] = `/tmp/leaked/${key}`; + } + try { + const options = buildDowntimeSpawnOptions('/repo'); + for (const key of REPOSITORY_OVERRIDE_ENV_VARS) { + expect(options.env[key]).toBeUndefined(); + } + // The scrub must not discard unrelated forwarding. + expect(options.env.HERDR_RESOLVED_CWD).toBe('/repo'); + } finally { + for (const key of REPOSITORY_OVERRIDE_ENV_VARS) { + const previous = saved.get(key); + if (previous === undefined) delete process.env[key]; + else process.env[key] = previous; + } + } + }); + + it('scrubRepositoryOverrides removes only override vars and never mutates its input', () => { + const input = { + PATH: '/usr/bin', + GIT_DIR: '/leak/.git', + GIT_WORK_TREE: '/leak', + HERDR_ENV: '1', + }; + + const cleaned = scrubRepositoryOverrides(input); + + expect(cleaned).toEqual({ PATH: '/usr/bin', HERDR_ENV: '1' }); + // Input mapping is untouched. + expect(input.GIT_DIR).toBe('/leak/.git'); + expect(input.GIT_WORK_TREE).toBe('/leak'); + }); + // ── Mode-aware PARALLELISM (WL-0MT50S9JW001DHME) ────────────────── it('buildDowntimeSpawnOptions defaults to PARALLELISM=1 when config is absent', () => { @@ -4639,11 +4705,11 @@ describe('downtime worker orchestrator (createDowntimeWorker)', () => { }); it('treats remote-only traffic as idle and dispatches after the threshold (local_active_query=false)', async () => { - // Integration (AC2): a stub status with global active_query=true but - // local_active_query=false (remote streams in flight, local slots free) - // must be treated as idle and dispatch after the idle threshold. + // Integration (AC2): a stub status with local_active_query=false + // (remote streams in flight, local slots free) must be treated as idle + // and dispatch after the idle threshold. const { worker, deps, cfg } = makeWorker({ - status: { ...idleAllSlotsFree, active_query: true, local_active_query: false }, + status: { ...idleAllSlotsFree, local_active_query: false }, }); const start = 1_000_000; vi.setSystemTime(start); @@ -5578,7 +5644,6 @@ describe('three-strike rule on CLI errors (createDowntimeWorker)', () => { describe('parseLlamaStatus per-slot slots array', () => { const base = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, available_slots: 4, total_slots: 4, @@ -5661,7 +5726,6 @@ describe('parseLlamaStatus per-slot slots array', () => { // zero-dispatch regression. Mirror the live payload exactly. const livePayload = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, available_slots: 4, @@ -5785,7 +5849,6 @@ describe('evaluateIdle per-slot mode (0 < N < total with slots present)', () => const busy = (id: string): LlamaSlot => ({ slot_id: id, is_processing: true }); const perSlot = (slots: LlamaSlot[], available: number, total: number): LlamaStatus => ({ llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, @@ -5806,7 +5869,7 @@ describe('evaluateIdle per-slot mode (0 < N < total with slots present)', () => // mode ONLY llama_server_running and model_switch_in_progress remain // global gates — query/lease signals are superseded by per-slot // is_processing (a busy slot's query/lease is the operator's own session). - expect(evaluateIdle({ ...twoFree, active_query: true }, 2)).toBe(true); // relaxed: query does NOT block + expect(evaluateIdle(twoFree, 2)).toBe(true); // relaxed: query does NOT block expect(evaluateIdle({ ...twoFree, local_active_query: true }, 2)).toBe(true); // relaxed expect(evaluateIdle({ ...twoFree, local_lease_active: true }, 2)).toBe(true); // relaxed: lease does NOT block expect(evaluateIdle({ ...twoFree, model_switch_in_progress: true }, 2)).toBe(false); // still global @@ -5840,7 +5903,6 @@ describe('evaluateIdle per-slot mode (0 < N < total with slots present)', () => function perSlotThreeOfFourFreeActiveQuery(): LlamaStatus { return { llama_server_running: true, - active_query: true, model_switch_in_progress: false, local_lease_active: true, available_slots: 3, @@ -5855,11 +5917,11 @@ function perSlotThreeOfFourFreeActiveQuery(): LlamaStatus { } describe('spare-capacity dispatch: relaxed global idle gate (per-slot mode)', () => { - it('per-slot mode: a mid-query (active_query) busy slot does NOT block dispatch into free slots (AC1)', () => { - // 3 of 4 slots free with active_query=true, local_active_query=true, - // local_lease_active=true. In the relaxed global gate (AC1), these - // per-slot query/lease signals are superseded by per-slot is_processing. - // Only llama_server_running and model_switch_in_progress stay global. + it('per-slot mode: a mid-query busy slot does NOT block dispatch into free slots (AC1)', () => { + // 3 of 4 slots free with local_active_query=true, local_lease_active=true. + // In the relaxed global gate (AC1), these per-slot query/lease signals + // are superseded by per-slot is_processing. Only llama_server_running and + // model_switch_in_progress stay global. const status = perSlotThreeOfFourFreeActiveQuery(); // The new relaxed check: active_query/local_active_query/local_lease // are NOT blocking in per-slot mode — 3 free >= N=2. @@ -5869,7 +5931,6 @@ describe('spare-capacity dispatch: relaxed global idle gate (per-slot mode)', () it('per-slot mode: local_lease_active on a busy slot does NOT block dispatch (AC1)', () => { const status: LlamaStatus = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, local_lease_active: true, available_slots: 3, @@ -5888,7 +5949,6 @@ describe('spare-capacity dispatch: relaxed global idle gate (per-slot mode)', () it('per-slot mode: local_active_query on a busy slot does NOT block dispatch (AC2)', () => { const status: LlamaStatus = { llama_server_running: true, - active_query: true, model_switch_in_progress: false, local_active_query: true, local_lease_active: false, @@ -5930,7 +5990,6 @@ describe('spare-capacity dispatch: pre-fix proxy (no local_active_query) fails c // fails closed (busy) on an absent signal — no silent degraded dispatch. const noLocalQuery: LlamaStatus = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 2, @@ -5942,7 +6001,6 @@ describe('spare-capacity dispatch: pre-fix proxy (no local_active_query) fails c // Even with all slots free, absent local_active_query → busy. const allFree: LlamaStatus = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 4, @@ -5954,7 +6012,6 @@ describe('spare-capacity dispatch: pre-fix proxy (no local_active_query) fails c it('pre-fix proxy with N=0 (default): absent local_active_query → busy', () => { const status: LlamaStatus = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 2, @@ -5969,7 +6026,6 @@ describe('spare-capacity dispatch: ambiguous payloads are busy (AC5)', () => { it('missing slot_id fields are treated as busy (fail-closed)', () => { const malformed: LlamaStatus = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 3, @@ -5987,7 +6043,6 @@ describe('spare-capacity dispatch: ambiguous payloads are busy (AC5)', () => { it('total_slots 0 is always busy', () => { const status: LlamaStatus = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 0, @@ -5999,7 +6054,6 @@ describe('spare-capacity dispatch: ambiguous payloads are busy (AC5)', () => { it('N > total can never be idle', () => { const status: LlamaStatus = { llama_server_running: true, - active_query: false, model_switch_in_progress: false, local_lease_active: false, available_slots: 3, @@ -6090,7 +6144,6 @@ describe('downtime worker per-slot routing (createDowntimeWorker)', () => { // while the downtime worker uses the spare capacity. const operatorSessionPayload = { llama_server_running: true, - active_query: true, local_active_query: true, model_switch_in_progress: false, local_lease_active: true, @@ -6138,7 +6191,7 @@ describe('downtime worker per-slot routing (createDowntimeWorker)', () => { // slot-1 goes mid-query while slot-2 stays free; the global query signal // also fires. Spare-capacity relaxation: only slot-1's timer resets. - fetcher.mockResolvedValueOnce(jsonResponseFixture({ ...perSlotTwoFree, active_query: true })); + fetcher.mockResolvedValueOnce(jsonResponseFixture(perSlotTwoFree)); vi.setSystemTime(start + cfg.thresholdMs - 10_000); const mid = await worker.tick(); expect(mid.idle).toBe(true); // free count still ≥ N=2 (slot-1 busy) @@ -6156,7 +6209,6 @@ describe('downtime worker per-slot routing (createDowntimeWorker)', () => { // the audit candidate is skipped and the plan pane dispatches instead. const oneFreePayload = { llama_server_running: true, - active_query: true, model_switch_in_progress: false, local_lease_active: true, available_slots: 1, @@ -6202,7 +6254,6 @@ describe('downtime worker per-slot routing (createDowntimeWorker)', () => { // live payload end-to-end: poll → parse → idle run → threshold → dispatch. const livePayload = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, available_slots: 4, @@ -6293,7 +6344,7 @@ describe('downtime worker per-slot routing (createDowntimeWorker)', () => { // model_switch_in_progress is a GLOBAL gate in per-slot mode (spare- // capacity relaxation, parent WL-0MT32F90V008UAD2 AC2) — it resets - // every slot timer. A per-slot query/lease (active_query) is NOT global + // every slot timer. A per-slot query/lease signal is NOT global // anymore and is covered by the dedicated spare-capacity tests. fetcher.mockResolvedValueOnce(jsonResponseFixture({ ...perSlotAllFree, model_switch_in_progress: true })); vi.setSystemTime(start + 10_000); @@ -6337,24 +6388,25 @@ describe('downtime worker per-slot routing (createDowntimeWorker)', () => { expect(deps.spawnAgentPane).toHaveBeenCalledTimes(1); }); - it('without per-slot data and 0 < N < total, falls back to all-slots-free (never any N slots)', async () => { - // No slots array: evaluateIdle degrades to all-slots-free, so 2 of 4 - // free is BUSY even though N=2 — the worker never dispatches on - // any-N availability without per-slot identity. + it('without usable per-slot data, fails closed (never dispatches on any-N availability)', async () => { + // No usable per-slot detail + 0 < N < total: the worker fails closed + // (WL-0MUXVPXAZ005RESW) — never dispatching on the untrusted count — + // and records the distinct `slots-unavailable` reason. const { worker, deps, cfg, fetcher } = makePerSlotWorker({ requiredFreeSlots: 2, - status: { ...idleAllSlotsFree, available_slots: 2 }, + status: { ...idleAllSlotsFree, slots: undefined, available_slots: 4 }, }); const start = 1_000_000; vi.setSystemTime(start); const first = await worker.tick(); - expect(first.idle).toBe(false); // 2 of 4 free without identity → busy + expect(first.idle).toBe(false); + expect(worker.blockReason).toBe('no-slots'); vi.setSystemTime(start + cfg.thresholdMs); const at = await worker.tick(); expect(at.dispatched).toBe(false); expect(deps.spawnAgentPane).not.toHaveBeenCalled(); - expect(fetcher).toHaveBeenCalled(); // still polling — never any-N dispatch + expect(fetcher).toHaveBeenCalled(); // still polling — just never dispatching }); it('with per-slot data but N=0, falls back to all-slots-free (per-slot mode only when 0 < N < total)', async () => { @@ -8469,7 +8521,6 @@ describe('bounded concurrent dispatch — claim safety regression', () => { describe('parseLlamaStatus: contention + per-slot owner (AC5/AC6)', () => { const base = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, @@ -8641,6 +8692,48 @@ describe('parseHerdrPaneListOutput / countRunningDowntimePanes (AC1)', () => { expect(panes).not.toBeNull(); expect(countRunningDowntimePanes(panes!)).toHaveLength(2); }); + + it('exposes cwd and agent_session on parsed records (WL-0MUYMBMCG000Z4WP AC1)', () => { + const raw = JSON.stringify({ + result: { + panes: [ + { + pane_id: 'w2Y:p87', + label: 'Manually triggered intake - WL-ABC', + agent: 'pi', + agent_status: 'working', + cwd: '/home/u/projects/OtherRoot', + agent_session: { kind: 'path', value: '/tmp/s.jsonl' }, + }, + ], + }, + }); + const panes = parseHerdrPaneListOutput(raw)!; + expect(panes[0].cwd).toBe('/home/u/projects/OtherRoot'); + expect(panes[0].agentSession).toEqual({ kind: 'path', value: '/tmp/s.jsonl' }); + }); + + it('omits cwd / agent_session on missing or malformed fields, without throwing (WL-0MUYMBMCG000Z4WP AC3)', () => { + const raw = JSON.stringify({ + panes: [ + { pane_id: 'p1', cwd: 7, agent_session: ['x'] }, + { pane_id: 'p2' }, + ], + }); + const panes = parseHerdrPaneListOutput(raw)!; + expect(panes).toHaveLength(2); + expect(panes[0].cwd).toBeUndefined(); + expect(panes[0].agentSession).toBeUndefined(); + expect(panes[1].cwd).toBeUndefined(); + expect(panes[1].agentSession).toBeUndefined(); + }); + + it('shares one parse contract: bare arrays and bracketed log prefixes parse (WL-0MUYMBMCG000Z4WP AC3)', () => { + const bare = JSON.stringify([{ pane_id: 'p1', cwd: '/r' }]); + expect(parseHerdrPaneListOutput(bare)![0].cwd).toBe('/r'); + const prefixed = '[herdr] starting\n' + JSON.stringify({ panes: [{ pane_id: 'p1' }] }); + expect(parseHerdrPaneListOutput(prefixed)![0].paneId).toBe('p1'); + }); }); // ── Item-scoped in-flight guard helpers (F3 WL-0MUBVKXQJ000L8EO) ──────── @@ -8883,14 +8976,18 @@ describe('worker tick: the LLM idle check is the sole concurrency limiter (WL-0M vi.useFakeTimers(); try { // Single slot, idle by is_processing, but a live lease is held AND the - // worker has a running pane → the owner gate refuses the dispatch. + // slot is owned by a live pane → the owner gate refuses the dispatch. const { worker, deps } = makeSingleSlotWorker({ status: { ...idleAllSlotsFree, available_slots: 1, total_slots: 1, + local_lease_active: true, local_owner_session_id: 'session-live', local_owner_lease_remaining_seconds: 120, + slots: [ + { slot_id: 'slot-1', is_processing: false, owner_session_id: 'session-live' }, + ], }, runningPanes: () => ({ ok: true, count: 1 }), }); @@ -10712,7 +10809,6 @@ describe('parseLlamaStatus: array local_owner_session_id (WL-0MU88086A0089US4)', function makeBase(): Record { return { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, @@ -10822,7 +10918,6 @@ describe('parseLlamaStatus: local_owner_session_ids normalization (WL-0MU88086A0 function makeBase(): Record { return { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, @@ -10924,7 +11019,6 @@ describe('tick(): owner-lease gate must be per-slot (WL-0MU8807BI008C9ME)', () = const ownedSlot = 'http://localhost:8080'; const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: true, @@ -10963,7 +11057,6 @@ describe('tick(): owner-lease gate must be per-slot (WL-0MU8807BI008C9ME)', () = try { const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: true, @@ -10997,7 +11090,6 @@ describe('tick(): owner-lease gate must be per-slot (WL-0MU8807BI008C9ME)', () = const ownerC = 'http://localhost:8082'; const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: true, @@ -11039,7 +11131,6 @@ describe('tick(): owner-lease gate must be per-slot (WL-0MU8807BI008C9ME)', () = // Only 1 free unowned slot — below audit-tier minimum of 2. const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: true, @@ -11079,7 +11170,6 @@ describe('tick(): owner-lease gate must be per-slot (WL-0MU8807BI008C9ME)', () = try { const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: true, @@ -11124,7 +11214,6 @@ describe('tick(): owner-lease gate must be per-slot (WL-0MU8807BI008C9ME)', () = try { const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: true, @@ -11674,20 +11763,21 @@ describe('downtime no-dispatch decision log (WL-0MU8808ZY0091JIA)', () => { expect(decisionHeaderToken('code-freeze')).toBe('code-freeze'); }); - it('AC1: a slot-owned refusal writes one decision line with reason + slot/owner fields', async () => { - const root = mkdtempSync(join(tmpdir(), 'herdr-decision-slot-')); + it('AC1: a fail-closed refusal writes the distinct slots-unavailable reason + observed fields', async () => { + const root = mkdtempSync(join(tmpdir(), 'herdr-decision-failclosed-')); try { - // Count-based (no `slots`), idle (local_lease_active=false) but the - // global owner session id is present and a downtime pane is alive → - // slotOwned. + // The exact live shape (WL-0MUFP30T2003OX1F): stale/empty per-slot + // detail with a held owner lease. The dispatcher fails closed and + // records the distinct `slots-unavailable` reason (WL-0MUXVPXAZ005RESW). const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, - local_lease_active: false, + local_lease_active: true, available_slots: 1, - total_slots: 1, + total_slots: 3, + slots: [], + slots_stale: true, local_owner_session_id: 'session-live-pane', contention_queue_depth: 0, contention_queued_count: 0, @@ -11699,7 +11789,7 @@ describe('downtime no-dispatch decision log (WL-0MU8808ZY0091JIA)', () => { }); const outcome = await worker.tick(); expect(outcome.dispatched).toBe(false); - expect(worker.blockReason).toBe('slot-owner'); + expect(worker.blockReason).toBe('no-slots'); await vi.waitFor(async () => { const entries = (await readCoordinationLogEntries(root)).filter( (e) => e.kind === 'decision', @@ -11708,11 +11798,11 @@ describe('downtime no-dispatch decision log (WL-0MU8808ZY0091JIA)', () => { expect(entries[0]).toMatchObject({ kind: 'decision', operation: 'no-dispatch', - reason: 'slot-owned', - freeSlots: 1, - totalSlots: 1, + reason: 'slots-unavailable', + freeSlots: 0, + totalSlots: 3, ownerPresent: true, - runningPanes: 1, + runningPanes: null, contentionDepth: 0, }); expect(typeof entries[0].at).toBe('string'); @@ -11727,12 +11817,17 @@ describe('downtime no-dispatch decision log (WL-0MU8808ZY0091JIA)', () => { try { const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, - available_slots: 1, - total_slots: 1, + available_slots: 4, + total_slots: 4, + slots: [ + { slot_id: 'slot-1', is_processing: false }, + { slot_id: 'slot-2', is_processing: false }, + { slot_id: 'slot-3', is_processing: false }, + { slot_id: 'slot-4', is_processing: false }, + ], contention_queue_depth: 3, contention_queued_count: 3, }; @@ -11762,7 +11857,6 @@ describe('downtime no-dispatch decision log (WL-0MU8808ZY0091JIA)', () => { try { const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: false, @@ -11835,21 +11929,22 @@ describe('local dispatch-lease TTL + own-pane leases (WL-0MU8809SZ0022VZG)', () expect(LOCAL_DISPATCH_LEASE_MAX_SECONDS).toBe(1500); }); - it('AC1/AC3: a max-TTL unknown/operator lease on a count-based single slot still refuses dispatch', async () => { + it('AC1/AC3: a max-TTL operator lease on a single owned slot still refuses dispatch', async () => { // The dispatcher must not optimistically assume the lease expired: a lease - // at the proxy MAX on a count-based single-slot setup (an unknown/operator - // owner) keeps blocking while a downtime pane is alive. local_lease_active - // is false so the idle gate passes and the slot-owned gate is the blocker. + // at the proxy MAX on a single-slot setup (an unknown/operator owner) keeps + // the slot owned, so no free unowned slot remains and dispatch is refused. const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, - local_lease_active: false, + local_lease_active: true, available_slots: 1, total_slots: 1, local_owner_session_id: 'operator-session', local_owner_lease_remaining_seconds: LOCAL_DISPATCH_LEASE_MAX_SECONDS, + slots: [ + { slot_id: 'slot-1', is_processing: false, owner_session_id: 'operator-session' }, + ], contention_queue_depth: 0, contention_queued_count: 0, }; @@ -11860,7 +11955,6 @@ describe('local dispatch-lease TTL + own-pane leases (WL-0MU8809SZ0022VZG)', () }); const outcome = await worker.tick(); expect(outcome.dispatched).toBe(false); - expect(worker.blockReason).toBe('slot-owner'); expect(deps.spawnAgentPane).not.toHaveBeenCalled(); }); @@ -11875,7 +11969,6 @@ describe('local dispatch-lease TTL + own-pane leases (WL-0MU8809SZ0022VZG)', () const ownSession = 'dispatched-pane-session'; const status: LlamaStatus = { llama_server_running: true, - active_query: false, local_active_query: false, model_switch_in_progress: false, local_lease_active: true, @@ -11911,14 +12004,13 @@ describe('local dispatch-lease TTL + own-pane leases (WL-0MU8809SZ0022VZG)', () }); }); -// ── Stale/empty slots fallback + count-based spare capacity ─────────── -// (WL-0MUFP30T2003OX1F) +// ── Fail-closed per-slot contract (WL-0MUXVPXAZ005RESW) ────────────── +// Supersedes the count-based fallback introduced by WL-0MUFP30T2003OX1F. -describe('stale/empty slots fallback + count-based spare capacity (WL-0MUFP30T2003OX1F)', () => { +describe('fail-closed on unusable per-slot detail (WL-0MUXVPXAZ005RESW)', () => { function makeRaw(overrides: Record = {}): Record { return { llama_server_running: true, - active_query: true, local_active_query: true, model_switch_in_progress: false, local_lease_active: false, @@ -11969,7 +12061,7 @@ describe('stale/empty slots fallback + count-based spare capacity (WL-0MUFP30T20 return { worker, deps }; } - it('AC4: parses slots_stale and preserves the empty slots array', () => { + it('parses slots_stale and preserves the empty slots array', () => { const status = parseLlamaStatus(makeRaw()); expect(status).not.toBeNull(); expect(status!.slots).toEqual([]); @@ -11979,30 +12071,47 @@ describe('stale/empty slots fallback + count-based spare capacity (WL-0MUFP30T20 expect(status!.local_owner_session_id).toBe('audit-pane-session'); }); - it('AC4: a malformed slots_stale fails closed (null)', () => { + it('a malformed slots_stale fails closed (null)', () => { expect(parseLlamaStatus(makeRaw({ slots_stale: 'yes' }))).toBeNull(); expect(parseLlamaStatus(makeRaw({ slots_stale: 1 }))).toBeNull(); }); - it('AC4: empty/stale slots are NOT per-slot identity — evaluateIdle falls back to the count', () => { - // Active local query → the leased slot is processing, not reserved → 2 free >= N=2. - const active = parseLlamaStatus(makeRaw({ available_slots: 2 }))!; - expect(evaluateIdle(active, 2)).toBe(true); - // Idle-but-leased (no active query) → reserve the leased slot → 1 < 2. - const idleLease = parseLlamaStatus( - makeRaw({ available_slots: 2, local_active_query: false, active_query: false }), + it('evaluateIdle fails closed for absent, empty, and stale per-slot detail', () => { + // The untrusted available_slots count is never a dispatch budget: + // absent, empty, or stale per-slot detail all yield busy. + const absent = parseLlamaStatus(makeRaw({ available_slots: 3, slots: undefined }))!; + expect(absent.slots).toBeUndefined(); + expect(evaluateIdle(absent, 1)).toBe(false); + + const empty = parseLlamaStatus(makeRaw({ available_slots: 3 }))!; + expect(empty.slots).toEqual([]); + expect(evaluateIdle(empty, 1)).toBe(false); + + const stale = parseLlamaStatus( + makeRaw({ available_slots: 3, slots: [{ slot_id: 0, is_processing: false }] }), )!; - expect(evaluateIdle(idleLease, 2)).toBe(false); + expect(stale.slots_stale).toBe(true); + expect(evaluateIdle(stale, 1)).toBe(false); }); - it('AC1: live payload (stale/empty slots, lease, live pane) dispatches into the free slot', async () => { + it('decisionHeaderToken maps the distinct fail-closed reason to no-slots', () => { + expect(decisionHeaderToken('slots-unavailable')).toBe('no-slots'); + expect(decisionHeaderToken('slots-unavailable')).not.toBe( + decisionHeaderToken('slot-owned'), + ); + expect(decisionHeaderToken('slots-unavailable')).not.toBe( + decisionHeaderToken('proxy-contention'), + ); + }); + + it('AC1/AC4: the exact live payload (stale/empty slots, held lease, live pane) does NOT dispatch', async () => { vi.useFakeTimers(); try { // The exact live shape: total=3, available=1, slots:[], slots_stale:true, - // an owner lease held by a dispatched pane. N=1 (single-pane dispatch). + // an owner lease held by a dispatched pane. The dispatcher must fail + // closed and record the distinct `slots-unavailable` reason. const status: LlamaStatus = { llama_server_running: true, - active_query: true, local_active_query: true, model_switch_in_progress: false, local_lease_active: true, @@ -12027,74 +12136,48 @@ describe('stale/empty slots fallback + count-based spare capacity (WL-0MUFP30T20 await worker.tick(); // baseline vi.setSystemTime(start + 5_000); const outcome = await worker.tick(); - expect(outcome.dispatched).toBe(true); - expect(deps.spawnAgentPane).toHaveBeenCalledTimes(1); + expect(outcome.dispatched).toBe(false); + expect(worker.blockReason).toBe('no-slots'); + expect(deps.spawnAgentPane).not.toHaveBeenCalled(); } finally { vi.useRealTimers(); } }); - it('AC2: count-based multi-slot with spare capacity and a held lease dispatches', async () => { + it('never derives the dispatch budget from available_slots (fail-closed budget)', async () => { + // available_slots=3 would previously have admitted a dispatch; with no + // usable per-slot detail the worker refuses regardless of the count. vi.useFakeTimers(); try { - // 3 slots, 2 free, N=2, an active local query holding the lease: the RCA - // window shape. The machine-wide lease must not block the spare capacity. const status: LlamaStatus = { llama_server_running: true, - active_query: true, local_active_query: true, model_switch_in_progress: false, - local_lease_active: true, - available_slots: 2, + local_lease_active: false, + available_slots: 3, total_slots: 3, - local_owner_session_id: 'dispatched-pane-session', - local_owner_session_ids: ['dispatched-pane-session'], - local_owner_lease_remaining_seconds: 840, + slots: [], + slots_stale: true, contention_queue_depth: 0, contention_queued_count: 0, }; const { worker, deps } = makeStaleWorker({ status, - requiredFreeSlots: 2, + requiredFreeSlots: 1, thresholdMs: 1_000, - runningPanes: 1, + runningPanes: 0, }); const start = 2_000_000; vi.setSystemTime(start); await worker.tick(); vi.setSystemTime(start + 5_000); const outcome = await worker.tick(); - expect(outcome.dispatched).toBe(true); - expect(deps.spawnAgentPane).toHaveBeenCalledTimes(1); + expect(outcome.dispatched).toBe(false); + expect(deps.spawnAgentPane).not.toHaveBeenCalled(); } finally { vi.useRealTimers(); } }); - - it('AC3: count-based single-slot with a held lease still refuses', async () => { - const status: LlamaStatus = { - llama_server_running: true, - active_query: false, - local_active_query: false, - model_switch_in_progress: false, - local_lease_active: false, - available_slots: 1, - total_slots: 1, - local_owner_session_id: 'operator-session', - local_owner_session_ids: ['operator-session'], - contention_queue_depth: 0, - contention_queued_count: 0, - }; - const { worker, deps } = makeStaleWorker({ - status, - requiredFreeSlots: 1, - runningPanes: 1, - }); - const outcome = await worker.tick(); - expect(outcome.dispatched).toBe(false); - expect(worker.blockReason).toBe('slot-owner'); - expect(deps.spawnAgentPane).not.toHaveBeenCalled(); - }); }); // ── Per-item/per-kind dispatch-attempt cap (WL-0MUKYEXMK0033MFK) ─────── @@ -12394,3 +12477,627 @@ describe('clampDowntimeMaxAttempts (WL-0MUKYEXMK0033MFK)', () => { expect(clampDowntimeMaxAttempts(4)).toBe(4); }); }); + +// ── Stalled-work scan before new-item dispatch (WL-0MUYMBSA90092QV2) ── +// The dispatcher must, on an idle cycle and BEFORE new-item selection, +// consult the stalled-work scan; a successful in-place resume short-circuits +// the cycle (preference ordering) and every failure degrades to the unchanged +// normal dispatch path (fail-safe). + +describe('stalled-work scan before new-item dispatch (WL-0MUYMBSA90092QV2)', () => { + function stalledCandidate(overrides: Partial = {}): StalledPaneCandidate { + return { + pane: { + paneId: 'pane-1', + label: 'Downtime triggered implement Some item - WL-STALLED', + agent: 'pi', + agentStatus: 'idle', + }, + item: { + id: 'WL-STALLED', + status: 'open', + stage: 'plan_complete', + title: 'Some item', + updatedAt: new Date(Date.now() - 10 * 60 * 1000).toISOString(), + }, + cwd: '/repo', + kind: 'implement', + ...overrides, + }; + } + + const implementHead = (id: string): DowntimeHerdrItem => ({ + id, + title: `Implement ${id}`, + status: 'open', + stage: 'plan_complete', + priority: 'high', + risk: 'Low', + effort: 'S', + sortIndex: 10, + }); + + it('dispatchDowntimeWork: a successful resume short-circuits new-item dispatch', async () => { + const scanStalledPanes = vi.fn().mockResolvedValue([stalledCandidate()]); + const resumeStalledPane = vi.fn().mockResolvedValue({ + resumed: true, + via: 'prompt', + paneId: 'pane-1', + itemId: 'WL-STALLED', + kind: 'implement', + }); + const deps = makeDeps({ + scanStalledPanes, + resumeStalledPane, + getHerdrListHead: vi.fn().mockResolvedValue({ ok: true, items: [implementHead('WL-1')] }), + }); + + const outcome = await dispatchDowntimeWork(deps, { model: 'plan', cwd: '/repo' }); + + expect(outcome).toEqual({ dispatched: false, reason: 'stalled-resume' }); + expect(scanStalledPanes).toHaveBeenCalledTimes(1); + expect(resumeStalledPane).toHaveBeenCalledWith( + expect.objectContaining({ cwd: '/repo', kind: 'implement' }), + expect.objectContaining({ enabled: true, thresholdMs: expect.any(Number) }), + ); + // New-item selection never ran and no pane was spawned. + expect(deps.getHerdrListHead).not.toHaveBeenCalled(); + expect(deps.spawnAgentPane).not.toHaveBeenCalled(); + }); + + it('dispatchDowntimeWork: an unresumable candidate falls through to normal dispatch', async () => { + const resumeStalledPane = vi.fn().mockResolvedValue({ + resumed: false, + reason: 'agent-blocked', + paneId: 'pane-1', + itemId: 'WL-STALLED', + kind: 'implement', + }); + const deps = makeDeps({ + scanStalledPanes: vi.fn().mockResolvedValue([stalledCandidate()]), + resumeStalledPane, + getHerdrListHead: vi.fn().mockResolvedValue({ ok: true, items: [implementHead('WL-1')] }), + }); + + const outcome = await dispatchDowntimeWork(deps, { model: 'plan', cwd: '/repo' }); + + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-1'); + expect(deps.spawnAgentPane).toHaveBeenCalledTimes(1); + }); + + it('dispatchDowntimeWork: a thrown scan degrades to normal dispatch (never a crash)', async () => { + const deps = makeDeps({ + scanStalledPanes: vi.fn().mockRejectedValue(new Error('herdr down')), + resumeStalledPane: vi.fn(), + getHerdrListHead: vi.fn().mockResolvedValue({ ok: true, items: [implementHead('WL-1')] }), + }); + + const outcome = await dispatchDowntimeWork(deps, { model: 'plan', cwd: '/repo' }); + + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-1'); + }); + + it('dispatchDowntimeWork: unwired scan deps degrade to normal dispatch', async () => { + const deps = makeDeps({ + getHerdrListHead: vi.fn().mockResolvedValue({ ok: true, items: [implementHead('WL-1')] }), + }); + + const outcome = await dispatchDowntimeWork(deps, { model: 'plan', cwd: '/repo' }); + + expect(outcome.dispatched).toBe(true); + }); + + it('dispatchDowntimeWork: a disabled scan never consults the scan dep', async () => { + const scanStalledPanes = vi.fn(); + const deps = makeDeps({ + scanStalledPanes, + resumeStalledPane: vi.fn(), + getHerdrListHead: vi.fn().mockResolvedValue({ ok: true, items: [implementHead('WL-1')] }), + }); + + const outcome = await dispatchDowntimeWork(deps, { + model: 'plan', + cwd: '/repo', + stallScanEnabled: false, + }); + + expect(outcome.dispatched).toBe(true); + expect(scanStalledPanes).not.toHaveBeenCalled(); + }); + + it('dispatchFromCoordination: a successful resume short-circuits the offer dispatch', async () => { + const fetchItem = vi.fn(); + const deps = makeDeps({ + scanStalledPanes: vi.fn().mockResolvedValue([stalledCandidate()]), + resumeStalledPane: vi.fn().mockResolvedValue({ + resumed: true, + via: 'prompt', + paneId: 'pane-1', + itemId: 'WL-STALLED', + kind: 'implement', + }), + fetchItem, + }); + + const entry: CoordinationEntry = { + instanceId: 'i1', + workItemId: 'WL-1', + directory: '/repo', + worklogRoot: '/repo', + assignedAt: new Date().toISOString(), + lastUpdated: new Date().toISOString(), + }; + const outcome = await dispatchFromCoordination(deps, [entry], { + model: 'plan', + cwd: '/repo', + coordinationDir: '/coord', + freeSlots: 2, + }); + + expect(outcome).toEqual({ dispatched: false, reason: 'stalled-resume' }); + expect(fetchItem).not.toHaveBeenCalled(); + expect(deps.spawnAgentPane).not.toHaveBeenCalled(); + }); + + it('computeMostImportantItem: a successful resume reports stalledResume and offers nothing', async () => { + const deps = makeDeps({ + scanStalledPanes: vi.fn().mockResolvedValue([stalledCandidate()]), + resumeStalledPane: vi.fn().mockResolvedValue({ + resumed: true, + via: 'prompt', + paneId: 'pane-1', + itemId: 'WL-STALLED', + kind: 'implement', + }), + getHerdrListHead: vi.fn(), + }); + + const result = await computeMostImportantItem( + deps, + '/repo', + Date.now(), + undefined, + undefined, + undefined, + undefined, + true, + 300_000, + ); + + expect(result).toEqual({ ok: true, stalledResume: true }); + expect(deps.getHerdrListHead).not.toHaveBeenCalled(); + }); + + it('computeMostImportantItem: no resume → the normal offer computation runs', async () => { + const deps = makeDeps({ + scanStalledPanes: vi.fn().mockResolvedValue([]), + resumeStalledPane: vi.fn(), + getHerdrListHead: vi.fn().mockResolvedValue({ ok: true, items: [implementHead('WL-1')] }), + }); + + const result = await computeMostImportantItem( + deps, + '/repo', + Date.now(), + undefined, + undefined, + undefined, + undefined, + true, + 300_000, + ); + + expect(result).toMatchObject({ ok: true, candidate: { id: 'WL-1' } }); + expect(deps.getHerdrListHead).toHaveBeenCalledTimes(1); + }); +}); + +// ── Stalled-work cross-root / idempotence / fail-safe integration (WL-0MUYMBUAK007H10J) ── +// End-to-end verification over the REAL `resumeStalledPane` orchestrator (the +// scan dep's candidates are stubbed at the dispatch boundary, so the resume +// path itself — re-classification, the in-process (paneId,itemId) guard, the +// CLI-failure mapping and the fail-open attempt record — is exercised as it +// runs in production). + +describe('stalled-work cross-root, idempotence and fail-safe integration (WL-0MUYMBUAK007H10J)', () => { + const NOW = Date.now(); + + function stalledCandidate(overrides: Partial = {}): StalledPaneCandidate { + return { + pane: { + paneId: 'pane-1', + label: 'Downtime triggered implement Some item - WL-STALLED', + agent: 'pi', + agentStatus: 'idle', + }, + item: { + id: 'WL-STALLED', + status: 'open', + stage: 'plan_complete', + title: 'Some item', + updatedAt: new Date(NOW - 10 * 60 * 1000).toISOString(), + }, + cwd: '/foreign/root', + kind: 'implement', + ...overrides, + }; + } + + const implementHead = (id: string): DowntimeHerdrItem => ({ + id, + title: `Implement ${id}`, + status: 'open', + stage: 'plan_complete', + priority: 'high', + risk: 'Low', + effort: 'S', + sortIndex: 10, + }); + + function realResumeDep(overrides: Partial = {}) { + return (cand: StalledPaneCandidate, opts: { nonTerminalCooldownMs: number; thresholdMs: number }) => + resumeStalledPane( + { + pane: cand.pane, + item: cand.item, + cwd: cand.cwd, + kind: cand.kind, + nonTerminalCooldownMs: opts.nonTerminalCooldownMs, + thresholdMs: opts.thresholdMs, + }, + { + promptAgent: vi.fn().mockResolvedValue({ ok: true }), + startAgent: vi.fn().mockResolvedValue({ ok: true }), + readEntries: async () => [], + recordAttempt: async () => undefined, + markNeedsProducerReview: async () => true, + maxAttempts: 3, + now: () => NOW, + ...overrides, + }, + ); + } + + beforeEach(() => { + _resetStalledResumeInFlight(); + }); + + it('reaper race: a pane that vanishes mid-resume is a neutral skip; dispatch falls through with no duplicate prompt', async () => { + const promptAgent = vi.fn().mockResolvedValue({ ok: false, reason: 'pane-vanished' }); + const recordAttempt = vi.fn().mockResolvedValue(undefined); + const deps = makeDeps({ + scanStalledPanes: vi.fn().mockResolvedValue([stalledCandidate()]), + resumeStalledPane: realResumeDep({ promptAgent, recordAttempt }), + getHerdrListHead: vi.fn().mockResolvedValue({ ok: true, items: [implementHead('WL-1')] }), + }); + + const outcome = await dispatchDowntimeWork(deps, { model: 'plan', cwd: '/leader' }); + + // The vanished pane did not resume and did not block the cycle: the + // normal dispatch path proceeded and opened exactly one NEW pane. + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-1'); + expect(promptAgent).toHaveBeenCalledTimes(1); + expect(recordAttempt).not.toHaveBeenCalled(); + expect(deps.spawnAgentPane).toHaveBeenCalledTimes(1); + }); + + it('a thrown resume is neutral: normal dispatch proceeds, no crash, no duplicate', async () => { + const deps = makeDeps({ + scanStalledPanes: vi.fn().mockResolvedValue([stalledCandidate()]), + resumeStalledPane: vi.fn().mockRejectedValue(new Error('herdr exploded')), + getHerdrListHead: vi.fn().mockResolvedValue({ ok: true, items: [implementHead('WL-1')] }), + }); + + const outcome = await dispatchDowntimeWork(deps, { model: 'plan', cwd: '/leader' }); + + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-1'); + expect(deps.spawnAgentPane).toHaveBeenCalledTimes(1); + }); + + it('prevents a concurrent double-resume of the same pane/item (neutral skip, single prompt)', async () => { + let release!: () => void; + const gate = new Promise((resolve) => { + release = resolve; + }); + const promptAgent = vi.fn().mockImplementation(async () => { + await gate; + return { ok: true }; + }); + const recordAttempt = vi.fn().mockResolvedValue(undefined); + const deps = makeDeps({ + scanStalledPanes: vi.fn().mockResolvedValue([stalledCandidate()]), + resumeStalledPane: realResumeDep({ promptAgent, recordAttempt }), + }); + const opts = { + enabled: true, + frozen: false, + thresholdMs: 5 * 60 * 1000, + nonTerminalCooldownMs: 10 * 60 * 1000, + maxAttempts: 3, + now: NOW, + }; + + const first = resumeStalledWorkIfAble(deps, '/leader', opts); + const second = resumeStalledWorkIfAble(deps, '/leader', opts); + + // The second concurrent cycle is a neutral skip (normal dispatch proceeds). + expect(await second).toBe(false); + expect(promptAgent).toHaveBeenCalledTimes(1); + // Release the first resume: it succeeds, still with a single prompt/record. + release(); + expect(await first).toBe(true); + expect(promptAgent).toHaveBeenCalledTimes(1); + expect(recordAttempt).toHaveBeenCalledTimes(1); + }); + + it('degrades to no stalled resume when the scan/resume seams are unwired', async () => { + const opts = { + enabled: true, + frozen: false, + thresholdMs: 5 * 60 * 1000, + nonTerminalCooldownMs: 10 * 60 * 1000, + maxAttempts: 3, + now: NOW, + }; + const c = stalledCandidate(); + + expect(await resumeStalledWorkIfAble({} as DowntimeWorkerDeps, '/leader', opts)).toBe(false); + // Only the scan wired: no resume attempted. + expect( + await resumeStalledWorkIfAble( + { scanStalledPanes: vi.fn().mockResolvedValue([c]) } as unknown as DowntimeWorkerDeps, + '/leader', + opts, + ), + ).toBe(false); + // Only the resume wired: the scan is never consulted. + const resume = vi.fn(); + expect( + await resumeStalledWorkIfAble( + { resumeStalledPane: resume } as unknown as DowntimeWorkerDeps, + '/leader', + opts, + ), + ).toBe(false); + expect(resume).not.toHaveBeenCalled(); + // A scan that resolves a non-array (unparseable) is a neutral no-resume. + expect( + await resumeStalledWorkIfAble( + { + scanStalledPanes: vi.fn().mockResolvedValue(null), + resumeStalledPane: resume, + } as unknown as DowntimeWorkerDeps, + '/leader', + opts, + ), + ).toBe(false); + expect(resume).not.toHaveBeenCalled(); + }); +}); + +// ── Own-backlog fallback safety gates (WL-0MUZPBRX3007STFK, parent AC7) ── +// +// The own-backlog candidate is a dispatch candidate like any other: every +// existing safety gate must still apply to it as a SEQUENTIAL FILTER, never +// as a fallback ordering. Each test hides an eligible out-of-view item behind +// a gate and asserts the gate is not bypassed — the gated item is never +// claimed, and the fallback continues to the next eligible out-of-view item +// where one exists. The offer-list gate tests already exist above; these pin +// the same gates on the fallback path. +// +// RED WITNESS. The fallback does not exist yet, so every body asserts the +// post-implementation behaviour and is committed with `it.fails` — the suite +// stays GREEN while the assertions are RED (the repo's documented TDD pattern; +// see `docs/dev/downtime-dispatcher-post-fix-verification.md`). The +// implementation child WL-0MUPBSYW004I5D3 flips each `it.fails` to `it` once +// the fallback lands (the RED→GREEN transition is the dependency evidence). +describe('own-backlog fallback safety gates (WL-0MUZPBRX3007STFK)', () => { + const SPRINT = 20; + // 1s marker-stale window so same-stage markers written in the past are + // RELEASED (the dispatched-marker guard) while the attempt-budget filter + // still counts them. + const WINDOW_MS = 1_000; + const ago = (ms: number) => new Date(Date.now() - ms).toISOString(); + const tempDirs: string[] = []; + + afterEach(() => { + for (const dir of tempDirs.splice(0)) rmSync(dir, { recursive: true, force: true }); + }); + + function makeDir(prefix: string): string { + const dir = mkdtempSync(join(tmpdir(), prefix)); + tempDirs.push(dir); + return dir; + } + + /** An eligible non-critical plan_complete out-of-view item (implement tier). */ + const eligibleBacklog = ( + id: string, + overrides: Partial = {}, + ): DowntimeHerdrItem => ({ + id, + title: `Backlog ${id}`, + status: 'open', + stage: 'plan_complete', + priority: 'high', + risk: 'Low', + effort: 'S', + sortIndex: 1000, + ...overrides, + }); + + /** An eligible plan-tier out-of-view item (idea → intake; unaffected by the freeze/review gates). */ + const eligiblePlan = (id: string): DowntimeHerdrItem => ({ + id, + title: `Plan ${id}`, + status: 'open', + stage: 'idea', + priority: 'high', + sortIndex: 3000, + }); + + /** In-view filler that is never dispatchable (the retired `in_review` stage). */ + const inView = (): DowntimeHerdrItem[] => + Array.from({ length: SPRINT }, (_, i) => ({ + id: `WL-VIEW-${i}`, + title: `In-view ${i}`, + status: 'open', + stage: 'in_review', + priority: 'medium', + risk: 'Low', + effort: 'S', + sortIndex: i, + })); + + /** + * Own-root head: the in-view window at/below SPRINT, then the hinted + * out-of-view tail once the dispatcher widens the window. Mirrors the + * canonical ranking path (the same head, just more items included). + */ + const ownRoot = (tail: DowntimeHerdrItem[]) => + vi.fn(async (_cwd: string, limit = SPRINT) => ({ + ok: true as const, + items: limit <= SPRINT ? inView() : [...inView(), ...tail], + })); + + const dispatchFallback = ( + deps: DowntimeWorkerDeps, + coordinationDir: string, + cwd: string, + extra: Record = {}, + ) => + dispatchFromCoordination( + // `fetchItem` is REQUIRED by the coordination leader path (fail-closed + // guard). These fallback tests pass no coordination entries, so it is + // never invoked — this default only satisfies the guard so the + // own-backlog fallback is reachable. + { fetchItem: vi.fn().mockResolvedValue({ ok: false }), ...deps }, + [], + { + model: 'plan', + cwd, + coordinationDir, + browseItemCount: SPRINT, + ...extra, + }, + ); + + it( + 'AC7 code-freeze: an out-of-view implement is held; a plan fallback still dispatches', + async () => { + const coordinationDir = makeDir('dt-oov-freeze-'); + const cwd = makeDir('dt-oov-freeze-root-'); + const deps = makeDeps({ + getHerdrListHead: ownRoot([eligibleBacklog('WL-OOV-IMPL'), eligiblePlan('WL-OOV-PLAN')]), + readCodeFreezeStatus: vi.fn().mockReturnValue('frozen'), + }); + + const outcome = await dispatchFallback(deps, coordinationDir, cwd); + + // The fallback was attempted (post-implementation) — RED before it lands. + expect(deps.getHerdrListHead).toHaveBeenCalledWith(cwd, expect.any(Number)); + // The frozen implement is never claimed; the plan fallback dispatches. + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-OOV-PLAN'); + expect(deps.claimItem).not.toHaveBeenCalledWith('WL-OOV-IMPL', expect.anything(), expect.anything()); + }, + ); + + it( + 'AC7 review-queue hold: a deep queue holds an out-of-view non-critical implement, the plan dispatches', + async () => { + const coordinationDir = makeDir('dt-oov-reviewqueue-'); + const cwd = makeDir('dt-oov-reviewqueue-root-'); + const deps = makeDeps({ + getHerdrListHead: ownRoot([eligibleBacklog('WL-OOV-IMPL'), eligiblePlan('WL-OOV-PLAN')]), + // A completed/in_review queue at/over the sprint threshold = deep. + getReviewQueueCount: vi.fn().mockResolvedValue(SPRINT), + }); + + const outcome = await dispatchFallback(deps, coordinationDir, cwd); + + expect(deps.getHerdrListHead).toHaveBeenCalledWith(cwd, expect.any(Number)); + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-OOV-PLAN'); + expect(deps.claimItem).not.toHaveBeenCalledWith('WL-OOV-IMPL', expect.anything(), expect.anything()); + }, + ); + + it( + 'AC7 attempt budget: a budget-exhausted out-of-view item is filtered, the next eligible dispatches', + async () => { + const coordinationDir = makeDir('dt-oov-budget-'); + const cwd = makeDir('dt-oov-budget-root-'); + // Three same-stage implement markers exhaust the per-item/per-kind cap. + mkdirSync(join(cwd, '.worklog'), { recursive: true }); + writeFileSync( + join(cwd, '.worklog', DOWNTIME_LOG_FILE), + [0, 1, 2] + .map(() => + JSON.stringify({ + itemId: 'WL-OOV-BUDGET', + kind: 'implement', + stage: 'plan_complete', + dispatchedAt: ago(60_000), + }), + ) + .join('\n') + '\n', + 'utf8', + ); + const deps = makeDeps({ + getHerdrListHead: ownRoot([ + eligibleBacklog('WL-OOV-BUDGET', { sortIndex: 1000 }), + eligibleBacklog('WL-OOV-FRESH', { sortIndex: 2000 }), + ]), + markNeedsProducerReview: vi.fn().mockResolvedValue(true), + }); + + const outcome = await dispatchFallback(deps, coordinationDir, cwd, { + maxAttempts: 3, + markerStaleWindowMs: WINDOW_MS, + }); + + expect(deps.getHerdrListHead).toHaveBeenCalledWith(cwd, expect.any(Number)); + // The exhausted item is flagged and skipped; the next eligible dispatches. + expect(outcome.dispatched).toBe(true); + expect(outcome.candidate?.id).toBe('WL-OOV-FRESH'); + expect(deps.markNeedsProducerReview).toHaveBeenCalledWith('WL-OOV-BUDGET', cwd); + expect(deps.claimItem).not.toHaveBeenCalledWith('WL-OOV-BUDGET', expect.anything(), expect.anything()); + }, + ); + + it( + 'AC7 in-flight: a critical out-of-view item with a live working pane is held', + async () => { + const coordinationDir = makeDir('dt-oov-inflight-'); + const cwd = makeDir('dt-oov-inflight-root-'); + const deps = makeDeps({ + getHerdrListHead: ownRoot([eligibleBacklog('WL-OOV-CRIT', { priority: 'critical' })]), + getRunningDowntimePanes: vi.fn().mockResolvedValue({ + ok: true, + count: 1, + paneIds: ['w1:p1'], + records: [ + { + paneId: 'w1:p1', + label: 'Downtime triggered implement Backlog - WL-OOV-CRIT', + agent: 'pi', + agentStatus: 'working', + }, + ], + }), + }); + + const outcome = await dispatchFallback(deps, coordinationDir, cwd); + + expect(deps.getHerdrListHead).toHaveBeenCalledWith(cwd, expect.any(Number)); + expect(outcome.dispatched).toBe(false); + expect(outcome.reason).toBe('in-flight-pane'); + expect(deps.claimItem).not.toHaveBeenCalled(); + }, + ); +}); diff --git a/packages/herdr/src/downtime-worker.ts b/packages/herdr/src/downtime-worker.ts index 11de5b3ba..90edb6ff3 100644 --- a/packages/herdr/src/downtime-worker.ts +++ b/packages/herdr/src/downtime-worker.ts @@ -177,7 +177,13 @@ import { type PaneItemState, } from './pane-lifecycle.js'; import { paneCloseReaperDue } from './pane-close-scheduler.js'; +import { parseHerdrPaneCloseList, type HerdrAgentSession } from './pane-close-herdr.js'; import { buildDowntimePaneTitle, MAX_PANE_TITLE_LENGTH } from './pane-title.js'; +import type { + StalledPaneCandidate, + StalledResumeOutcome, + StalledScanOptions, +} from './stalled-work.js'; import type { DispatcherAnchor, DispatcherPrefixTabAnchor, @@ -264,6 +270,8 @@ export function decisionHeaderToken(reason: string): string { return 'contention'; case 'review-queue-hold': return 'review-queue'; + case 'slots-unavailable': + return 'no-slots'; default: return reason; } @@ -423,6 +431,32 @@ export const DOWNTIME_MARKER_STALE_WINDOW_FLOOR_MS = 60 * 60 * 1000; */ export const DOWNTIME_MARKER_STALE_WINDOW_MAX_MS = 7 * 24 * 60 * 60 * 1000; +/** + * Default for the pre-dispatch stalled-work scan (WL-0MUYMBK6H005PY16): + * enabled. The scan prefers resuming an existing stalled pane over opening a + * new one; it can be turned off without a code change. + */ +export const DEFAULT_DOWNTIME_STALL_SCAN_ENABLED = true; + +/** + * Default stall threshold (WL-0MUYMBK6H005PY16): 5 minutes, aligned with the + * pane-closure reaper's grace period. A pane is a stall candidate only once + * its not-`working` state has persisted at least this long. + */ +export const DEFAULT_DOWNTIME_STALL_THRESHOLD_MS = 5 * 60 * 1000; + +/** + * Hard floor for the stall threshold (60 s): below this the scan would resume + * panes that are merely between tool calls, hijacking active sessions. + */ +export const MIN_DOWNTIME_STALL_THRESHOLD_MS = 60 * 1000; + +/** + * Hard ceiling for the stall threshold (60 min): above this a genuinely + * stalled pane could sit unattended for over an hour, defeating the feature. + */ +export const MAX_DOWNTIME_STALL_THRESHOLD_MS = 60 * 60 * 1000; + /** * Three-strike rule: this many consecutive CLI-error dispatch outcomes * pause the worker entirely (after logging the persistent error). @@ -554,16 +588,6 @@ export type CoordinationTierKind = (typeof COORDINATION_TIER_ORDER)[number]; /** Shape of `GET /llama/local/status` as served by the llama-proxy. */ export interface LlamaStatus { llama_server_running: boolean; - /** - * GLOBAL query activity: any request in flight (local AND remote). - * - * This field is no longer used for idle/busy detection — the worker - * exclusively uses `local_active_query`. Remote provider streams keep this - * true while the local model is idle with free slots, so it cannot be used - * as the busy signal (RCA WL-0MSK9TUCA00206M7). Present in the payload for - * backwards compatibility with older proxy versions. - */ - active_query: boolean; /** * LOCAL-only query activity (served by proxies exposing the * `local_active_queries` counter, LP-0MSL2ZLLS009RVKR). This is the @@ -609,22 +633,24 @@ export interface LlamaStatus { */ contention_queued_count?: number; /** - * Proxy degradation flag (LP-0MSVP7XJ6008QPKX, consumed by - * WL-0MUFP30T2003OX1F): `true` when the served slot COUNTS came from the - * proxy's last-known cache because the fresh `/slots` query failed. When - * true the per-slot `slots` detail cannot be trusted/paired with the - * counts, so the worker must fall back to the count-based path. ABSENT on + * Proxy degradation flag (LP-0MSVP7XJ6008QPKX; fail-closed contract + * WL-0MUXVPXAZ005RESW): `true` when the served slot COUNTS came from the + * proxy's last-known cache because the fresh `/slots` query failed. The + * per-slot `slots` detail then cannot be trusted/paired with the counts, + * so per-slot detail is UNUSABLE and the worker fails closed (no + * dispatch) rather than falling back to the untrusted count. ABSENT on * pre-feature proxies (treated as not stale). */ slots_stale?: boolean; /** * Per-slot identity (LP-0MSG5TA7Y002GN39): served by proxies that expose - * slot-level detail. ABSENT on pre-feature proxies — the worker then - * falls back to the count-based all-slots-free logic. When present, the - * downtime worker tracks idle duration PER SLOT ID so a configured N - * requires the SAME N slots continuously free. An EMPTY array means "no - * per-slot detail available" (WL-0MUFP30T2003OX1F) — never "zero free - * slots" — and likewise falls back to the count-based path. + * slot-level detail, and the ONLY trusted availability source + * (WL-0MUXVPXAZ005RESW). When present and non-stale, the downtime worker + * tracks idle duration PER SLOT ID so a configured N requires the SAME N + * slots continuously free. ABSENT on pre-feature proxies, and an EMPTY + * array means "no per-slot detail available" (WL-0MUFP30T2003OX1F) — + * never "zero free slots"; in both cases the worker fails closed (no + * dispatch) rather than falling back to the count. */ slots?: LlamaSlot[]; } @@ -653,10 +679,10 @@ export interface LlamaSlot { // ── Idle detection (implemented) ────────────────────────────────────── /** - * The FULL global idle checks used by `isIdleStatus` and the count-based - * (non-per-slot) path of `evaluateIdle`: llama-server up, no active local - * query (the sole busy signal), no model switch, no local lease. The - * per-slot branch uses the relaxed `perSlotGlobalIdleChecks` instead + * The FULL global idle checks used by `isIdleStatus` and the all-slots-free + * (`N ≤ 0` / `N ≥ total`) path of `evaluateIdle`: llama-server up, no active + * local query (the sole busy signal), no model switch, no local lease. The + * spare-capacity path uses the relaxed `perSlotGlobalIdleChecks` instead * (spare-capacity dispatch, parent WL-0MT32F90V008UAD2). * * `local_active_query` is the SOLE busy signal for the query gate: @@ -677,13 +703,13 @@ function globalIdleChecks(status: LlamaStatus): boolean { /** * Per-slot-safe global checks (spare-capacity dispatch, parent * WL-0MT32F90V008UAD2): in per-slot identity mode, ONLY llama-server-up and - * no-model-switch remain GLOBAL gates — `active_query` / - * `local_active_query` / `local_lease_active` are superseded by the per-slot + * no-model-switch remain GLOBAL gates — `local_active_query` / + * `local_lease_active` are superseded by the per-slot * `is_processing` signal (the busy slot holding the operator's query/lease IS * the active work the operator wants to run alongside; it must not block - * dispatch into the free slots, LP-0MSG5TA7Y002GN39 Q&A). Used exclusively by - * the per-slot branches of `evaluateIdle` and `createDowntimeWorker.tick()`; - * the count-based path keeps the full `globalIdleChecks` (fail-closed). + * dispatch into the free slots, LP-0MSG5TA7Y002GN39 Q&A). Used by the + * spare-capacity path of `evaluateIdle` and `createDowntimeWorker.tick()`; + * the all-slots-free path keeps the full `globalIdleChecks` (fail-closed). */ function perSlotGlobalIdleChecks(status: LlamaStatus): boolean { if (!status.llama_server_running) return false; @@ -718,13 +744,14 @@ export function countFreeUnownedSlots(slots: LlamaSlot[]): number { } /** - * Usable per-slot identity (WL-0MUFP30T2003OX1F): the `slots` array is only - * trustworthy when it is NON-EMPTY and NOT flagged stale by the proxy. An - * empty array (`slots: []`) means "no per-slot detail available" — NOT "zero - * free slots" — and `slots_stale: true` means the counts came from the - * proxy's last-known cache after a failed /slots query, so the detail cannot - * be paired with them. In both cases the worker must fall back to the - * count-based path; `null` is that signal. + * Usable per-slot identity (WL-0MUFP30T2003OX1F; fail-closed contract + * WL-0MUXVPXAZ005RESW): the `slots` array is only trustworthy when it is + * NON-EMPTY and NOT flagged stale by the proxy. An empty array (`slots: []`) + * means "no per-slot detail available" — NOT "zero free slots" — and + * `slots_stale: true` means the counts came from the proxy's last-known + * cache after a failed /slots query, so the detail cannot be paired with + * them. `null` signals "unusable"; callers MUST fail closed (no dispatch) + * rather than falling back to the untrusted `available_slots` count. */ function usablePerSlotSlots(status: LlamaStatus): LlamaSlot[] | null { if (!Array.isArray(status.slots)) return null; @@ -733,74 +760,6 @@ function usablePerSlotSlots(status: LlamaStatus): LlamaSlot[] | null { return status.slots; } -/** - * Number of local dispatch leases the proxy reports (WL-0MUFP30T2003OX1F): - * the normalised `local_owner_session_ids` list when present, else the legacy - * single `local_owner_session_id` string, else 1 when only a positive lease - * TTL is served. Used both for the owner-lease qualifier and to reserve - * possibly-idle leased slots when the worker only has count-based data. - */ -function ownerLeaseCount(status: LlamaStatus): number { - if (Array.isArray(status.local_owner_session_ids)) { - return status.local_owner_session_ids.length; - } - if ( - typeof status.local_owner_session_id === 'string' && - status.local_owner_session_id.length > 0 - ) { - return 1; - } - if ( - typeof status.local_owner_lease_remaining_seconds === 'number' && - Number.isFinite(status.local_owner_lease_remaining_seconds) && - status.local_owner_lease_remaining_seconds > 0 - ) { - return 1; - } - return 0; -} - -/** - * Slots to reserve for a held lease in the count-based path - * (WL-0MUFP30T2003OX1F). While the lease owner is actively generating - * (`local_active_query === true`) its slot is processing and already excluded - * from `available_slots`, so nothing is reserved; otherwise the owner may be - * idle-but-leased, and one slot per lease is reserved so an idle leased slot - * counted as "available" is never dispatched onto. - */ -function leaseReserve(status: LlamaStatus): number { - return status.local_active_query === true ? 0 : ownerLeaseCount(status); -} - -/** - * Genuinely-free slot count for the count-based path - * (WL-0MUFP30T2003OX1F): the proxy's `available_slots` minus any reserved - * lease slots (never negative). - */ -function countBasedFreeSlots(status: LlamaStatus): number { - const available = Number.isFinite(status.available_slots) ? status.available_slots : 0; - return Math.max(0, available - leaseReserve(status)); -} - -/** - * Count-based spare-capacity decision (WL-0MUFP30T2003OX1F): with a - * multi-slot config (0 < N < total) but NO usable per-slot identity, the - * proxy's `available_slots` count is the only availability signal. A held - * local lease must not block dispatch into the spare slots, so this mirrors - * the per-slot spare-capacity relaxation using the count (after reserving - * possibly-idle leased slots). Fail-closed on ambiguous/missing fields. - */ -function countBasedSpareCapacity(status: LlamaStatus, requiredFreeSlots: number): boolean { - const total = status.total_slots; - const available = status.available_slots; - if (!status.llama_server_running) return false; - if (status.model_switch_in_progress) return false; - if (!Number.isFinite(total) || total <= 0) return false; - if (!Number.isFinite(available) || available < 0) return false; - if (requiredFreeSlots <= 0 || requiredFreeSlots >= total) return false; - return countBasedFreeSlots(status) >= requiredFreeSlots; -} - /** * True when the proxy reports an idle state for the required free-slot * count: @@ -829,66 +788,45 @@ export function isIdleStatus(status: LlamaStatus, requiredFreeSlots: number): bo } /** - * Runtime idle evaluation (F2). Same conditions as `isIdleStatus`, plus the - * fail-closed fallback from planning decision Q7: while the proxy does not - * expose per-slot identity (LP-0MSG5TA7Y002GN39), a configured N with - * 0 < N < total slots degrades to requiring ALL slots free — stricter than - * configured, never "any N slots" without same-slot identity. N > total - * slots never dispatches (carried through `isIdleStatus`). + * Runtime idle evaluation. Fail-closed contract (WL-0MUXVPXAZ005RESW): the + * per-slot `slots` array is the ONLY trusted availability source. When it is + * unusable — absent, empty (`slots: []`), or flagged stale + * (`slots_stale: true`) — `usablePerSlotSlots` returns null and this + * function returns false, so no dispatch can be admitted from the untrusted + * `available_slots` count. * - * Per-slot mode (LP-0MSG5TA7Y002GN39): when the payload serves per-slot - * identity AND 0 < N < total, the free count comes from the slots array - * (which identifies WHICH slots are free) so the same N slots can be - * required; the count-based logic is unchanged for N ≤ 0 / N ≥ total. - * - * Spare-capacity relaxation (parent WL-0MT32F90V008UAD2): in per-slot mode - * the global gate is the per-slot-safe subset (server up + no model switch - * only) — a query/lease tied to a busy slot is the operator's own session - * and must not block dispatch into the free slots (F1 tests AC1/AC2). - * - * Count-based spare-capacity (WL-0MUFP30T2003OX1F): when per-slot identity is - * unusable (stale/empty `slots`) but 0 < N < total AND a local lease is held, - * the same relaxation applies to the proxy's `available_slots` count (after - * reserving possibly-idle leased slots) instead of stalling for the whole - * lease. + * With usable per-slot detail: + * - spare-capacity mode (0 < N < total): the relaxed global gate applies + * (server up + no model switch only) — a query/lease tied to a busy slot + * is the operator's own session and must not block dispatch into the + * free slots (parent WL-0MT32F90V008UAD2). The budget is the per-slot + * free count. + * - all-slots-free mode (N ≤ 0 / N ≥ total): the full `globalIdleChecks` + * apply (unchanged). N ≤ 0 requires every slot free; N > total can never + * be satisfied. */ export function evaluateIdle(status: LlamaStatus, requiredFreeSlots: number): boolean { const total = status.total_slots; if (!Number.isFinite(total) || total <= 0) return false; // ambiguous → busy - // Per-slot mode: usable per-slot identity present AND 0 < N < total. The - // relaxed global gate applies (server up + no model switch); the slot - // requirement is the per-slot free count. + // Fail-closed (WL-0MUXVPXAZ005RESW): without usable per-slot detail there + // is no trustworthy availability signal — never fall back to the count. const perSlotSlots = usablePerSlotSlots(status); - if ( - perSlotSlots !== null && - requiredFreeSlots > 0 && - requiredFreeSlots < total - ) { - if (!perSlotGlobalIdleChecks(status)) return false; - // Fail-closed counting: an entry without an explicit boolean - // `is_processing` is treated as processing (busy), never free. Owned - // slots (live lease) are excluded (AC5, WL-0MTYZXSLN008HZOW). - const free = countFreeUnownedSlots(perSlotSlots); - return free >= requiredFreeSlots; - } + if (perSlotSlots === null) return false; - // Count-based spare-capacity (WL-0MUFP30T2003OX1F): multi-slot config with - // 0 < N < total but no usable per-slot identity, and a held lease. The - // strict all-slots-free fallback below would stall for the whole (adaptive) - // lease; the proxy's count says the capacity is real. - if ( - ownerLeaseCount(status) > 0 && - requiredFreeSlots > 0 && - requiredFreeSlots < total - ) { - return countBasedSpareCapacity(status, requiredFreeSlots); + const spareCapacity = requiredFreeSlots > 0 && requiredFreeSlots < total; + if (spareCapacity) { + if (!perSlotGlobalIdleChecks(status)) return false; + } else if (!globalIdleChecks(status)) { + return false; } - const effective = requiredFreeSlots > 0 && requiredFreeSlots < total - ? total - : requiredFreeSlots; - return isIdleStatus(status, effective); + const required = requiredFreeSlots <= 0 ? total : requiredFreeSlots; + if (required > total) return false; // N > total_slots can never be idle + // Fail-closed counting: an entry without an explicit boolean + // `is_processing` is treated as processing (busy), never free. Owned + // slots (live lease) are excluded (AC5, WL-0MTYZXSLN008HZOW). + return countFreeUnownedSlots(perSlotSlots) >= required; } // ── Idle-duration tracker (implemented — F3) ────────────────────────── @@ -1049,7 +987,7 @@ function parseLocalOwnerSessionIds(value: unknown): string[] { /** * The first (primary) owner session id — kept for compatibility with existing - * call sites (`local_lease_active` derivation, `ownerLeaseHeld`). Returns + * call sites (`local_lease_active` derivation). Returns * `undefined` when no owner session is reported. See * {@link parseLocalOwnerSessionIds}. */ @@ -1074,7 +1012,6 @@ export function parseLlamaStatus(raw: unknown): LlamaStatus | null { const o = raw as Record; if (typeof o.llama_server_running !== 'boolean') return null; - if (typeof o.active_query !== 'boolean') return null; if (typeof o.model_switch_in_progress !== 'boolean') return null; // Local-only signal (LP-0MSL2ZLLS009RVKR): absent on pre-fix proxies. @@ -1186,7 +1123,6 @@ export function parseLlamaStatus(raw: unknown): LlamaStatus | null { return { llama_server_running: o.llama_server_running, - active_query: o.active_query, local_active_query: localActiveQuery, model_switch_in_progress: o.model_switch_in_progress, local_lease_active: localLeaseActive, @@ -1488,6 +1424,8 @@ export type MostImportantItemResult = | { ok: true; nonTerminalCooldownHold: true } /** The only remaining offerable candidates have exhausted their per-item/per-kind attempt budget (WL-0MUKYEXMK0033MFK): nothing offerable NOW, but the backlog is NOT empty — the caller must not pause or drop the entry. */ | { ok: true; attemptBudgetHold: true } + /** A stalled pane was resumed in place this cycle (WL-0MUYMBSA90092QV2): the instance must NOT offer new work — the resumed pane is the preferred work. Neutral: not an error, not a CLI strike, not a no-candidate cooldown. */ + | { ok: true; stalledResume: true } | { ok: false; error?: string }; @@ -1554,6 +1492,10 @@ export interface HerdrPaneRecord { label?: string; agent?: string; agentStatus?: string; + /** Pane working directory (worklog-root hint for cross-root resume). */ + cwd?: string; + /** Raw `agent_session` field (session log path + agent metadata). */ + agentSession?: HerdrAgentSession; } /** @@ -1564,53 +1506,28 @@ export interface HerdrPaneRecord { export const DOWNTIME_PANE_LABEL_PREFIX = 'Downtime'; /** - * Parse `herdr pane list` JSON output into records. Tolerates log lines - * prefixed before the JSON envelope (scan for the first `{`, like - * `parseAgentListOutput`), the `{result:{panes:[…]}}` envelope, and a bare - * array. Returns `null` when no pane array can be found (the caller treats - * this as a failed liveness query → fail-closed). + * Parse `herdr pane list` JSON output into records. Thin adapter over the + * shared {@link parseHerdrPaneCloseList} contract (WL-0MUYMBMCG000Z4WP): the + * reaper and the dispatcher must never drift into two parsers. Tolerates log + * lines/prefixes before the JSON envelope, the `{result:{panes:[…]}}` / + * `{panes:[…]}` shapes and a bare array. Returns `null` when no pane array can + * be found (the caller treats this as a failed liveness query → fail-closed). + * + * Records carry the pane's `cwd` and raw `agent_session` alongside + * `paneId`/`label`/`agent`/`agentStatus` so a stalled pane for an item in + * another worklog root can be resumed against that root. */ export function parseHerdrPaneListOutput(raw: string): HerdrPaneRecord[] | null { - const start = raw.indexOf('{'); - if (start < 0) return null; - let payload: unknown; - try { - payload = JSON.parse(raw.slice(start)); - } catch { - return null; - } - - let panes: unknown = null; - if (payload && typeof payload === 'object') { - const obj = payload as Record; - const result = obj.result; - const resultObj = - result && typeof result === 'object' ? (result as Record) : null; - if (Array.isArray(resultObj?.panes)) { - panes = resultObj.panes; - } else if (Array.isArray(obj.panes)) { - panes = obj.panes; - } - } - if (!Array.isArray(panes)) return null; - - const records: HerdrPaneRecord[] = []; - for (const entry of panes) { - if (!entry || typeof entry !== 'object') continue; - const rec = entry as Record; - const paneId = rec.pane_id ?? rec.paneId; - if (typeof paneId !== 'string' || paneId === '') continue; - const label = typeof rec.label === 'string' ? rec.label : undefined; - const agent = typeof rec.agent === 'string' ? rec.agent : undefined; - const agentStatus = - typeof rec.agent_status === 'string' - ? rec.agent_status - : typeof rec.agentStatus === 'string' - ? rec.agentStatus - : undefined; - records.push({ paneId, label, agent, agentStatus }); - } - return records; + const parsed = parseHerdrPaneCloseList(raw); + if (parsed === null) return null; + return parsed.map((pane) => ({ + paneId: pane.paneId, + label: pane.label, + agent: pane.agent, + agentStatus: pane.agentStatus, + cwd: pane.cwd, + agentSession: pane.agentSession, + })); } /** @@ -2227,6 +2144,30 @@ export interface DowntimeWorkerDeps { * the flag is retried on the next cycle. */ markNeedsProducerReview?(itemId: string, cwd: string): Promise; + /** + * Machine-wide stalled-work scan (WL-0MUYMBSA90092QV2 AC1/AC4): scan every + * `herdr pane list` record for a pane hosting a non-terminal item whose + * agent has stopped working, and return the resumable candidates (each + * carrying its OWN worklog root). Consulted ONCE per idle cycle, + * immediately before new-item selection. OPTIONAL — when absent (or when + * `resumeStalledPane` is absent) the scan degrades to "no stalled resume" + * and normal dispatch proceeds unchanged. Must never throw: an + * unreadable/unparseable scan resolves an empty array. + */ + scanStalledPanes?(cwd: string, opts: StalledScanOptions): Promise; + /** + * Attempt an in-place resume of ONE stalled candidate (WL-0MUYMBSA90092QV2 + * AC2/AC6): prompt a live agent with `continue`, or relaunch pi in the SAME + * pane continuing its session and then submit `continue`. Records a + * dispatch attempt on success so the existing per-item/per-kind cap + * applies. OPTIONAL (see `scanStalledPanes`). Must never throw: every + * failure is a NEUTRAL `{resumed:false}` outcome — never a strike, never a + * crash, never a duplicate pane. + */ + resumeStalledPane?( + candidate: StalledPaneCandidate, + opts: StalledScanOptions, + ): Promise; /** * Record a persistent CLI-error event (three consecutive wl failures). * Must never throw (fail-closed): logging must not crash the worker. @@ -2406,6 +2347,16 @@ export interface DowntimeDispatchOutcome { timeoutMs?: number; /** Work item id being dispatched (set on dispatch-tier failures). */ workItemId?: string; + + /** + * True when this successful dispatch was sourced from the leader's OWN + * backlog beyond the sprint-view window by the fallback (parent + * WL-0MUU4XFU90008E24 AC1/AC4). The worker loop uses this machine-readable + * surface to perform the voluntary `releaseLeadership()` step-down exactly + * once per own-backlog dispatch, and never for a normal in-view dispatch. + * Absent/false on every other outcome. + */ + ownBacklogDispatch?: boolean; } /** @@ -2538,6 +2489,7 @@ export async function fetchExtendedHerdrItems( deps: DowntimeWorkerDeps, cwd: string, current: DowntimeHerdrItem[], + opts: { includeNonCritical?: boolean } = {}, ): Promise { try { const extended = await deps.getHerdrListHead( @@ -2547,8 +2499,13 @@ export async function fetchExtendedHerdrItems( if (!extended.ok) return []; const seen = new Set(current.map((i) => i.id)); // Critical-only escape hatch: a non-critical item beyond the sprint view - // is never dispatchable (WL-0MUNS8X97007C9H9 AC2/AC4). - return extended.items.filter((i) => !seen.has(i.id) && i.priority === 'critical'); + // is never dispatchable (WL-0MUNS8X97007C9H9 AC2/AC4). When + // `includeNonCritical` is true (used by the own-backlog fallback — parent + // WL-0MUU4XFU90008E24), the filter is lifted so any ranked item in the + // extended tail is returned. + return extended.items.filter((i) => + !seen.has(i.id) && (opts.includeNonCritical === true || i.priority === 'critical'), + ); } catch { return []; // fail-open: the extension never breaks a defined outcome } @@ -2648,6 +2605,57 @@ async function flagBudgetExhausted( } } +/** + * Neutral outcome reason when a stalled pane was resumed in place instead of + * dispatching new work (WL-0MUYMBSA90092QV2 AC1/AC3): the cycle short-circuits + * with `dispatched:false` and this reason. NEVER a strike, NEVER a cooldown — + * the next idle tick re-evaluates from scratch. + */ +export const STALLED_RESUME_REASON = 'stalled-resume'; + +/** + * Consult the stalled-work scan ONCE and, if it yields a candidate, attempt to + * resume the first one in place (WL-0MUYMBSA90092QV2 AC1/AC4). Returns true + * when a stalled pane was successfully resumed — the caller MUST then + * short-circuit new-item dispatch for the cycle (preference ordering). + * + * FAIL-SAFE: unwired deps, a disabled scan, a thrown scan, an unreadable pane + * list / unparseable output (the production scan resolves an empty list) and + * every `{resumed:false}` outcome all return false, so the existing dispatch + * path proceeds unchanged — never a crash, never a duplicate. + */ +export async function resumeStalledWorkIfAble( + deps: DowntimeWorkerDeps, + cwd: string, + opts: StalledScanOptions, +): Promise { + // Unwired deps / disabled scan degrade to normal dispatch (AC2/AC3). + if (opts.enabled !== true) return false; + if ( + typeof deps.scanStalledPanes !== 'function' || + typeof deps.resumeStalledPane !== 'function' + ) { + return false; + } + let candidates: StalledPaneCandidate[]; + try { + const scanned = await deps.scanStalledPanes(cwd, opts); + if (!Array.isArray(scanned)) return false; + candidates = scanned; + } catch { + return false; // fail-safe: never crash dispatch on a broken scan + } + for (const candidate of candidates) { + try { + const outcome = await deps.resumeStalledPane(candidate, opts); + if (outcome.resumed) return true; + } catch { + // A thrown resume is neutral — try the next candidate, then fall through. + } + } + return false; +} + async function dispatchFromHerdrList( deps: DowntimeWorkerDeps, items: DowntimeHerdrItem[], @@ -3583,6 +3591,8 @@ export async function computeMostImportantItem( markerStaleWindowMs: number = DEFAULT_DOWNTIME_MARKER_STALE_WINDOW_MS, nonTerminalCooldownMs: number = DEFAULT_DOWNTIME_NON_TERMINAL_COOLDOWN_MS, maxAttempts: number = DEFAULT_DOWNTIME_MAX_ATTEMPTS, + stallScanEnabled: boolean = true, + stallThresholdMs: number = DEFAULT_DOWNTIME_STALL_THRESHOLD_MS, ): Promise { // The check-in (and the worker's no-candidate probe) requires the Herdr // head lookup: without it there is no canonical ranking to offer from — @@ -3593,6 +3603,22 @@ export async function computeMostImportantItem( const freezeStatus = deps.readCodeFreezeStatus(cwd); const frozen = freezeStatus === 'frozen' || freezeStatus === 'ambiguous'; + // ── Stalled-work scan (WL-0MUYMBSA90092QV2 AC4) ── + // Consult the scan FIRST, before any new-item selection: a successfully + // resumed stalled pane is the preferred work, so the instance offers + // NOTHING this cycle (the caller keeps its entry / does not pause). + if (stallScanEnabled) { + const resumed = await resumeStalledWorkIfAble(deps, cwd, { + enabled: stallScanEnabled, + frozen, + thresholdMs: stallThresholdMs, + nonTerminalCooldownMs, + maxAttempts, + now, + }); + if (resumed) return { ok: true, stalledResume: true }; + } + // The dispatch window is the sprint view: the live per-root // `browseItemCount`, clamped exactly as the TUI worklist does // (WL-0MUNS8X97007C9H9 AC1). Passing the effective value means the head — @@ -3871,7 +3897,7 @@ export async function computeMostImportantItem( export async function dispatchFromCoordination( deps: DowntimeWorkerDeps, entries: CoordinationEntry[], - opts: { model: string; cwd: string; coordinationDir: string; freeSlots?: number; leaseTtlMs?: number; browseItemCount?: number; nonTerminalCooldownMs?: number; maxAttempts?: number; spawnConfig?: DowntimeSpawnConfig }, + opts: { model: string; cwd: string; coordinationDir: string; freeSlots?: number; leaseTtlMs?: number; browseItemCount?: number; nonTerminalCooldownMs?: number; maxAttempts?: number; markerStaleWindowMs?: number; stallScanEnabled?: boolean; stallScanThresholdMs?: number; spawnConfig?: DowntimeSpawnConfig }, now: number = Date.now(), ): Promise { // The leader path REQUIRES the item-fetch dep: without it the leader can @@ -3901,6 +3927,24 @@ export async function dispatchFromCoordination( return { dispatched: false, reason: 'no-candidate' }; } + // ── Stalled-work scan (WL-0MUYMBSA90092QV2 AC1/AC3/AC4) ── + // Idle-gated (free slot present) and immediately before new-item + // selection: prefer resuming a stalled pane over opening a new one. A + // successful resume short-circuits the cycle with the neutral + // 'stalled-resume' reason; any failure (unwired deps, unreadable pane + // list, resume failure) falls through to the unchanged path below. + { + const resumed = await resumeStalledWorkIfAble(deps, opts.cwd, { + enabled: opts.stallScanEnabled ?? true, + frozen, + thresholdMs: opts.stallScanThresholdMs ?? DEFAULT_DOWNTIME_STALL_THRESHOLD_MS, + nonTerminalCooldownMs: opts.nonTerminalCooldownMs ?? DEFAULT_DOWNTIME_NON_TERMINAL_COOLDOWN_MS, + maxAttempts: opts.maxAttempts ?? DEFAULT_DOWNTIME_MAX_ATTEMPTS, + now, + }); + if (resumed) return { dispatched: false, reason: STALLED_RESUME_REASON }; + } + // Review-queue depth gate (WL-0MT2UQWOR007CYY9, live-path rewire by // WL-0MTTSWC1X005P4VD): read ONCE per offer root (cached for this // invocation). Each root's queue depth is authoritative for ITS OWN @@ -3941,6 +3985,10 @@ export async function dispatchFromCoordination( }; const cooldownMs = opts.nonTerminalCooldownMs ?? DEFAULT_DOWNTIME_NON_TERMINAL_COOLDOWN_MS; let cooldownHold = false; + // Dispatched-marker staleness window (WL-0MU6UL0RJ008IHGT): forwarded to + // the own-backlog fallback's marker exclusion (the offer path consults the + // marker through `fetchItem` + staleness maps below). + const markerStaleWindowMs = opts.markerStaleWindowMs ?? DEFAULT_DOWNTIME_MARKER_STALE_WINDOW_MS; // Per-item/per-kind attempt-budget hold (WL-0MUKYEXMK0033MFK): neutral hold // — the backlog is not empty, so the caller must not pause or drop the // entry. A flagged set keeps the producer-review flag to at most one write @@ -4148,6 +4196,134 @@ export async function dispatchFromCoordination( }; } + // ── Own-backlog fallback (parent WL-0MUU4XFU90008E24 AC1–AC9) ────── + // Only reached when the coordination offer list yielded no dispatchable + // candidate. The leader MAY then read its OWN backlog beyond the + // sprint-view window: a bounded WINDOW EXTENSION of the SAME canonical + // ranking (`getHerdrListHead`), never a second ranking, and ONLY against + // the leader's own `opts.cwd` (never another instance's root — AC2). Every + // existing safety gate is applied as a sequential filter (AC7): + // classify (producer review / freshness / caps), dispatched-marker + // exclusion, attempt budget, non-terminal cooldown, code-freeze, + // audit slot minimum, in-flight (critical) and the review-queue hold. + // The window is bounded by `DOWNTIME_DISPATCH_EXTEND_MAX` (AC9). + // Fail-open: a `{ok:false}`/thrown/empty lookup resolves `[]`, leaving the + // original terminal reason unchanged and never panicking (AC9). A + // successful own-backlog dispatch is flagged `ownBacklogDispatch` so the + // worker loop steps down leadership exactly once (AC4); a normal in-view + // dispatch never reaches this block. + { + const sprintWindow = resolveSprintViewWindow(opts.browseItemCount); + // Fail-open (AC9): a thrown own-root lookup must leave the original + // terminal reason unchanged and never panic. + let ownHead: DowntimeHerdrListResult | null = null; + try { + ownHead = await deps.getHerdrListHead(opts.cwd, sprintWindow); + } catch { + ownHead = null; + } + if (ownHead !== null && ownHead.ok) { + const ownTail = await fetchExtendedHerdrItems(deps, opts.cwd, ownHead.items, { + includeNonCritical: true, + }); + if (ownTail.length > 0) { + const ownEntries = await cooldownEntriesForRoot(opts.cwd); + const ownMarkerMaps: Record> = { + audit: _dispatchedMarkers(ownEntries, 'audit'), + implement: _dispatchedMarkers(ownEntries, 'implement'), + plan: _dispatchedMarkers(ownEntries, 'plan'), + intake: _dispatchedMarkers(ownEntries, 'intake'), + 'risk-effort': _dispatchedMarkers(ownEntries, 'risk-effort'), + }; + const ownInFlight = await inFlightForRoot(opts.cwd); + const flaggedForOwnBudget = new Set(); + for (const item of ownTail) { + // Sequential filters — the same contract as the offer path. + const kind = classifyItemForDispatch(item, now); + if (kind === null) continue; + const marker = ownMarkerMaps[kind]?.get(item.id); + if ( + marker !== undefined && + _markerStillExcludes( + marker, + item.stage, + now, + markerStaleWindowMs, + kind === 'audit' || kind === 'implement' ? 'id-guard' : 'stage-guard', + ) + ) { + continue; + } + if (isAttemptBudgetExhausted(ownEntries, item.id, kind, item.stage, maxAttempts)) { + if (!flaggedForOwnBudget.has(item.id)) { + flaggedForOwnBudget.add(item.id); + await flagBudgetExhausted(deps, item.id, opts.cwd); + } + budgetHold = true; + continue; + } + if (_isNonTerminalCooldownActive(ownEntries, item.id, kind, item.stage, cooldownMs, now)) { + cooldownHold = true; + continue; + } + if (frozen && (kind === 'audit' || kind === 'implement' || kind === 'risk-effort')) continue; + if (kind === 'audit' && !auditEligible) continue; + if (item.priority === 'critical') { + const guard = evaluateCriticalFirstGuard( + item.id, + ownInFlight, + marker?.dispatchedAt, + now, + markerStaleWindowMs, + ); + if (!guard.escalate) { + inFlightHold = true; + continue; + } + } + if (kind === 'implement') { + const gate = await reviewGateForRoot(opts.cwd); + if (isImplementHeldByReviewGate(kind, item, gate)) { + reviewHold = true; + continue; + } + } + const ownOutcome = await dispatchClaimedTier( + deps, + kind, + { + id: item.id, + title: item.title, + stage: kind === 'audit' ? 'audit' : ((item.stage as DowntimeStage) ?? 'idea'), + status: item.status, + priority: item.priority, + sortIndex: item.sortIndex, + }, + { + model: opts.model, + cwd: opts.cwd, + selectionPath: 'own-backlog', + selectionReason: 'own-backlog-fallback', + spawnConfig: opts.spawnConfig, + }, + ); + if (ownOutcome.dispatched) { + return { ...ownOutcome, ownBacklogDispatch: true }; + } + // A lost CAS race / marker-recovery rollback applies to one + // candidate: try the next ranked own-backlog item. + if (ownOutcome.reason === 'claim-failed' || ownOutcome.reason === 'claim-rolled-back') { + continue; + } + // Any other terminal outcome (wl-error / spawn-failed / + // anchor-unavailable / marker-write-failed) is returned unchanged + // so the caller's strike/skip semantics are preserved. + return ownOutcome; + } + } + } + } + // No offer survived the dispatch-time filters (or a freeze skip with no // plan/intake offers). The caller decides cooldown vs. resume via the // same reason semantics as the legacy dispatcher — in the worker tick, @@ -4198,17 +4374,26 @@ export async function runCoordinationCheckIn( now: number = Date.now(), ): Promise<{ offered: string | null; updated: boolean }> { const current = getEntry(coordinator.coordinationDir, coordinator.instanceId); - const result = await computeMostImportantItem(deps, coordinator.cwd, now, coordinator.browseItemCount, coordinator.markerStaleWindowMs, coordinator.nonTerminalCooldownMs, coordinator.maxAttempts); + // NOTE (WL-0MUYMBSA90092QV2 AC3): the check-in is NOT idle-gated (it runs + // before the proxy poll, for leader and non-leader alike), so the + // stalled-work scan is deliberately DISABLED here — resuming a pane is a + // dispatch-like side effect that must only run when the dispatcher is + // idle-gated and would otherwise dispatch. The idle-gated paths + // (`dispatchDowntimeWork` / `dispatchFromCoordination` / the no-candidate + // probe below) perform the scan+resume. + const result = await computeMostImportantItem(deps, coordinator.cwd, now, coordinator.browseItemCount, coordinator.markerStaleWindowMs, coordinator.nonTerminalCooldownMs, coordinator.maxAttempts, false); if (!result.ok) { // wl/CLI errors — keep the existing entry (fail-open), retry next check-in. return { offered: current?.workItemId ?? null, updated: false }; } if (!('candidate' in result)) { - // Genuinely nothing dispatchable (`noCandidate`) OR the deep-queue hold - // left nothing offerable (`reviewQueueHold`, WL-0MTTSWC1X005P4VD) — - // either way the own entry is removed (no dead offers; the owner - // re-offers once work becomes offerable). The caller distinguishes the - // two via the probe path (reviewQueueHold must never pause). + // Genuinely nothing dispatchable (`noCandidate`), the deep-queue hold + // left nothing offerable (`reviewQueueHold`, WL-0MTTSWC1X005P4VD), or a + // stalled pane was resumed in place (`stalledResume`, + // WL-0MUYMBSA90092QV2) — either way the own entry is removed (no dead + // offers; the owner re-offers once work becomes offerable). The caller + // distinguishes the reasons via the probe path (reviewQueueHold and + // stalledResume must never pause). const removed = removeEntry(coordinator.coordinationDir, coordinator.instanceId) !== null; // Audit trail (WL-0MSXHAE290067VAL): log the emptied check-in. void appendCoordinationLogEntry(coordinator.cwd, { @@ -4410,6 +4595,19 @@ export async function dispatchDowntimeWork( * falls back to `DEFAULT_DOWNTIME_MAX_ATTEMPTS` (3). */ maxAttempts?: number; + /** + * Stalled-work scan enable (WL-0MUYMBSA90092QV2 AC2/AC3): read live from + * `downtimeStallScanEnabled`. Optional — absent defaults to enabled; the + * scan is `inert` unless `scanStalledPanes`/`resumeStalledPane` deps are + * wired, so an unwired caller is unaffected. + */ + stallScanEnabled?: boolean; + /** + * Stalled-work stall threshold ms (WL-0MUYMBSA90092QV2): read live from + * `downtimeStallThresholdMs` (clamped). Optional — absent falls back to + * `DEFAULT_DOWNTIME_STALL_THRESHOLD_MS` (5 min). + */ + stallScanThresholdMs?: number; /** Mode-aware Phase 2 parallelism inputs (WL-0MT50S9JW001DHME). */ spawnConfig?: DowntimeSpawnConfig; }, @@ -4482,6 +4680,23 @@ export async function dispatchDowntimeWork( const freezeStatus = deps.readCodeFreezeStatus(opts.cwd); const frozen = freezeStatus === 'frozen' || freezeStatus === 'ambiguous'; + // ── Stalled-work scan (WL-0MUYMBSA90092QV2 AC1/AC3/AC4) ── + // Idle-gated (a pane slot is free) and immediately before new-item + // selection: prefer resuming a stalled pane over opening a new one. A + // successful resume short-circuits the cycle with the neutral + // 'stalled-resume' reason; any failure (unwired deps, unreadable pane + // list, resume failure) falls through to the unchanged path below. + if (panesEligible) { + const resumed = await resumeStalledWorkIfAble(deps, opts.cwd, { + enabled: opts.stallScanEnabled ?? true, + frozen, + thresholdMs: opts.stallScanThresholdMs ?? DEFAULT_DOWNTIME_STALL_THRESHOLD_MS, + nonTerminalCooldownMs: opts.nonTerminalCooldownMs ?? DEFAULT_DOWNTIME_NON_TERMINAL_COOLDOWN_MS, + maxAttempts: opts.maxAttempts ?? DEFAULT_DOWNTIME_MAX_ATTEMPTS, + }); + if (resumed) return { dispatched: false, reason: STALLED_RESUME_REASON }; + } + if (!frozen && panesEligible) { const duePrompt = await deps.getDueScheduledPrompt(opts.cwd); if (duePrompt !== null) { @@ -5040,13 +5255,47 @@ export function buildDowntimeSpawnOptions( // no audit-skill change needed. Interactive (non-downtime) panes are // unaffected. env: { - ...process.env, + ...scrubRepositoryOverrides(process.env), HERDR_RESOLVED_CWD: cwd, AUDIT_PHASE2_PARALLELISM: parallelism, }, }; } +/** + * Git environment variables that override the repository a `git` command + * operates on. Git honours these over `cwd`, so a value leaked into a + * spawned pane can redirect a `git` command into a live checkout + * (SA-0MUIZSXEY008NGR8 / WL-0MV0TZEWZ003ZXEB). Mirrors the canonical list in + * `skill/shared/git_sandbox.py::REPOSITORY_OVERRIDE_ENV_VARS`. + */ +export const REPOSITORY_OVERRIDE_ENV_VARS = [ + 'GIT_DIR', + 'GIT_WORK_TREE', + 'GIT_CONFIG', + 'GIT_CONFIG_GLOBAL', + 'GIT_CONFIG_SYSTEM', + 'GIT_CONFIG_NOSYSTEM', + 'GIT_OBJECT_DIRECTORY', + 'GIT_ALTERNATE_OBJECT_DIRECTORIES', +] as const; + +/** + * Return a copy of *env* with every repository-override git variable removed. + * The input mapping is never mutated. Used by + * {@link buildDowntimeSpawnOptions} so a launcher-side leak cannot be + * forwarded to the spawned pane and its descendants (test subprocesses). + */ +export function scrubRepositoryOverrides( + env: NodeJS.ProcessEnv, +): NodeJS.ProcessEnv { + const cleaned: NodeJS.ProcessEnv = { ...env }; + for (const name of REPOSITORY_OVERRIDE_ENV_VARS) { + delete cleaned[name]; + } + return cleaned; +} + /** Default spawn: detached, stdio ignored, resolved cwd forwarded. */ export const defaultDowntimeSpawn: DowntimeSpawn = ( scriptPath, @@ -5337,6 +5586,16 @@ export interface DowntimeWorkerConfig { * `DEFAULT_DOWNTIME_MARKER_STALE_WINDOW_MS` (24 h). */ markerStaleWindowMs?: number; + /** + * Stalled-work scan enable (WL-0MUYMBSA90092QV2): read live on every tick. + * Optional for backward compat — defaults to enabled. + */ + stallScanEnabled?: boolean; + /** + * Stalled-work stall threshold, ms (WL-0MUYMBSA90092QV2): read live on + * every tick. Optional — defaults to `DEFAULT_DOWNTIME_STALL_THRESHOLD_MS`. + */ + stallScanThresholdMs?: number; /** * Pane-lifecycle monitor cadence, ms (WL-0MU308WSF0002JWN). Optional — * defaults to `DOWNTIME_PANE_LIFECYCLE_INTERVAL_MS` (30 s). Tests may @@ -6029,8 +6288,28 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker // dispatch, parent WL-0MT32F90V008UAD2). A slot reporting processing // resets only its own timer. const perSlotSlots = usablePerSlotSlots(status); + // Fail-closed precondition (WL-0MUXVPXAZ005RESW): unusable per-slot + // detail — absent, empty (`slots: []`), or stale (`slots_stale: true`) + // — means there is NO trustworthy availability signal. Refuse to + // dispatch and record the distinct `slots-unavailable` reason so the + // stop is observable in the decision log and header (never fall back + // to the count-based path). + if (perSlotSlots === null) { + tracker.record(false); + perSlotTracker.record([]); // fail-closed: reset every slot timer + void recordDecision(cfg.cwd, 'slots-unavailable', { + freeSlots: 0, + totalSlots: status.total_slots, + ownerPresent: status.local_lease_active === true, + runningPanes: null, + contentionDepth: + typeof status.contention_queue_depth === 'number' + ? status.contention_queue_depth + : 0, + }); + return { polled: true, dispatched: false, idle: false }; + } const perSlotMode = - perSlotSlots !== null && cfg.requiredFreeSlots > 0 && cfg.requiredFreeSlots < status.total_slots; @@ -6084,20 +6363,16 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker // ── Dispatch (parent WL-0MST3OJ8S0001ROL AC4, F5 WL-0MTII48OV008P2QU AC4) ── // Single machine-wide slot budget: ONE leader poll → ONE freeSlots - // snapshot (per-slot free count when `slots` served, else - // `available_slots`), forwarded to the sole dispatch call. No - // per-worklog duplication — total concurrently dispatched never exceeds - // the shared budget (WL-0MT50LKAK001EF5Q single cap source). Per-tier - // minimums (WL-0MT32F90V008UAD2 AC3): audit needs ≥2 (parent + Phase 2 - // child), single-pane tiers need ≥1; idle-duration gate (configured N) - // is unchanged and shared. - // Count-based: the proxy's own free-slot count (historical behaviour). - // The lease reserve in `countBasedSpareCapacity` is a GATE only — the - // reported budget stays the proxy figure so the strict single-slot path - // and its decision log are unchanged. - const freeSlots = perSlotSlots - ? countFreeUnownedSlots(perSlotSlots) - : status.available_slots; + // snapshot derived from the per-slot array — the ONLY trusted + // availability source (fail-closed contract, WL-0MUXVPXAZ005RESW) — + // forwarded to the sole dispatch call. No per-worklog duplication — + // total concurrently dispatched never exceeds the shared budget + // (WL-0MT50LKAK001EF5Q single cap source). Per-tier minimums + // (WL-0MT32F90V008UAD2 AC3): audit needs ≥2 (parent + Phase 2 child), + // single-pane tiers need ≥1; idle-duration gate (configured N) is + // unchanged and shared. The untrusted `available_slots` count is NEVER + // used as a dispatch budget. + const freeSlots = countFreeUnownedSlots(perSlotSlots); // Mode-aware Phase 2 parallelism inputs (WL-0MT50S9JW001DHME): the // genuine free-slot snapshot (never the total budget), the current @@ -6115,21 +6390,21 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker concurrentDispatchCap: cfg.concurrentDispatchCap ?? DISPATCH_PIPELINE_SINGLE_FLIGHT, }; - // ── Running-pane liveness (owner-lease qualifier only) ── + // ── Running-pane liveness (decision-log telemetry only) ── // Count dispatched downtime panes that are STILL ALIVE (machine-wide, // across roots and leader/instance restarts). // // NOTE (WL-0MU2EP6JL006A1U3): this count is deliberately NOT a dispatch // cap. Dispatched panes stay open until an operator closes them, so // gating on it allows exactly one dispatch per pane-close (the overnight - // stall). It is consulted ONLY to qualify the owner-lease gate below - // ("does this worker have a pane of its own?"). Capacity is limited by + // stall). With the fail-closed contract (WL-0MUXVPXAZ005RESW) the + // ownership gate is derived from the trusted per-slot array, so this + // count is reported in the decision log only. Capacity is limited by // the local LLM idle / free-slot check alone. // - // Fail-open: a failed query yields `null` → no known worker pane → the - // qualifier is inactive and dispatch proceeds (the proxy's own lease - // signal still gates slot capacity). A `herdr pane list` hiccup must - // never silently stop the dispatcher. + // Fail-open: a failed query yields `null` → no known worker pane is + // reported (the proxy's own lease signal still gates slot capacity). A + // `herdr pane list` hiccup must never silently stop the dispatcher. let runningPanes: number | null = null; if (typeof opts.deps.getRunningDowntimePanes === 'function') { try { @@ -6141,46 +6416,18 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker } // ── Owner-lease + contention gates (AC2/AC5/AC6) ── - // `slotOwned`: a non-null global Local Proxy owner lease means the - // target slot is owned by a live pane — only dispatch into a truly - // unowned slot. When per-slot identity is served, ownership is - // already excluded from `freeSlots` above; the global owner is the - // fallback signal for count-based (single-slot) setups. + // `slotOwned`: no free UNOWNED slot remains (every slot is processing + // or lease-owned), so a new pane must not be dispatched. Under the + // fail-closed contract (WL-0MUXVPXAZ005RESW) the budget and the + // ownership qualifier both come from the SAME per-slot array (the only + // trusted availability source), so the two gates can never disagree. // - // Per-slot gate (WL-0MU8807BI008C9ME): in per-slot mode, the owner - // lease must NOT block dispatch into free unowned slots. The gate - // fires only when countFreeUnownedSlots === 0 (all slots owned or - // busy). In count-based mode the original conservative gate is - // preserved: an owned sole slot blocks dispatch when runningPanes > 0. - // - // Per-slot mode reuses the SAME predicate as the idle-capacity gate - // above and `evaluateIdle` (`slots` served AND 0 < N < total). Sharing - // one definition keeps the ownership gate from ever being stricter - // than the capacity gate that admitted the tick — a divergent check - // would re-introduce exactly the stall this RCA fixes. - // - // `slotOwned` is qualified on a known running-pane count > 0 so an - // OPERATOR lease on a spare-capacity multi-slot setup does not block - // dispatch into the free slots when the worker has no running pane - // of its own. The lease is NOT short-lived: the proxy holds an - // ADAPTIVE lease up to `LOCAL_DISPATCH_LEASE_MAX_SECONDS` (default - // 1500 s, cross-repo `llm-manager`), so an idle-but-open pane CAN block - // the count-based path for its whole run — the historical "~180 s" - // assumption was wrong (WL-0MU8809SZ0022VZG). The per-slot path - // (WL-0MU8807BI008C9ME) is what keeps a dispatched pane's own lease - // from blocking the OTHER free unowned slots. - const ownerLeaseHeld = ownerLeaseCount(status) > 0; - // Count-based spare-capacity (WL-0MUFP30T2003OX1F): when per-slot - // identity is unusable and 0 < N < total, a held lease no longer blocks - // dispatch into the proxy-reported spare slots. This mirrors the - // `evaluateIdle` relaxation exactly, so the idle gate and the ownership - // gate can never disagree. - const countBasedSpare = - !perSlotMode && countBasedSpareCapacity(status, cfg.requiredFreeSlots); - const slotOwned = - perSlotMode && perSlotSlots !== null - ? countFreeUnownedSlots(perSlotSlots) === 0 - : ownerLeaseHeld && (runningPanes ?? 0) > 0 && !countBasedSpare; + // Per-slot gate (WL-0MU8807BI008C9ME): a busy or owned slot does NOT + // block dispatch into the free unowned slots — `slotOwned` fires only + // when countFreeUnownedSlots === 0. `ownerPresent` is retained purely + // for the decision log (derived from `local_lease_active`). + const ownerPresent = status.local_lease_active === true; + const slotOwned = countFreeUnownedSlots(perSlotSlots) === 0; // LIVE depth only (WL-0MU1DWXO600153OI): `contention_queued_count` is // cumulative telemetry and must never gate dispatch. const contentionQueueDepth = @@ -6202,7 +6449,7 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker void recordDecision(cfg.cwd, 'slot-owned', { freeSlots, totalSlots: status.total_slots, - ownerPresent: ownerLeaseHeld, + ownerPresent, runningPanes, contentionDepth: contentionQueueDepth, }); @@ -6212,7 +6459,7 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker void recordDecision(cfg.cwd, 'proxy-contention', { freeSlots, totalSlots: status.total_slots, - ownerPresent: ownerLeaseHeld, + ownerPresent, runningPanes, contentionDepth: contentionQueueDepth, }); @@ -6246,6 +6493,12 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker browseItemCount: cfg.browseItemCount, nonTerminalCooldownMs: cfg.nonTerminalCooldownMs, maxAttempts: cfg.maxAttempts, + // Dispatched success-marker staleness window + // (WL-0MU6UL0RJ008IHGT): forwarded to the own-backlog + // fallback's marker exclusion. + markerStaleWindowMs: cfg.markerStaleWindowMs, + stallScanEnabled: cfg.stallScanEnabled, + stallScanThresholdMs: cfg.stallScanThresholdMs, leaseTtlMs: opts.leaseTtlSeconds ? opts.leaseTtlSeconds * 1000 : DEFAULT_LEASE_TTL_SECONDS * 1000, @@ -6268,6 +6521,10 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker nonTerminalCooldownMs: cfg.nonTerminalCooldownMs, // Per-item/per-kind dispatch-attempt cap (WL-0MUKYEXMK0033MFK). maxAttempts: cfg.maxAttempts, + // Stalled-work scan before new-item selection + // (WL-0MUYMBSA90092QV2). + stallScanEnabled: cfg.stallScanEnabled, + stallScanThresholdMs: cfg.stallScanThresholdMs, spawnConfig, }); // Record the refusal reason (WL-0MU8808ZY0091JIA): actionable @@ -6279,7 +6536,7 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker void recordDecision(cfg.cwd, outcome.reason, { freeSlots, totalSlots: status.total_slots, - ownerPresent: ownerLeaseHeld, + ownerPresent, runningPanes, contentionDepth: contentionQueueDepth, }); @@ -6292,6 +6549,23 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker // dispatch (AC5). tracker.record(false); perSlotTracker.record([]); + // Voluntary leadership hand-off after an own-backlog fallback + // dispatch (parent WL-0MUU4XFU90008E24 AC4/AC5/A3): the fallback + // reached beyond the sprint-view window into the leader's own + // backlog, so the instance relinquishes leadership so another + // instance can win the next election. The claim / marker / spawn + // already completed inside `dispatchFromCoordination` before this + // point, so the step-down never rolls back a pending dispatch and + // the dispatched pane is not orphaned. Exactly once per own-backlog + // dispatch (the outcome is consumed here), and never for a normal + // in-view dispatch. Setting `leaderState = false` prevents an + // immediate self re-election in this tick (the election block runs + // at the start of the next tick). Legacy mode (`leaderManager === + // null`) is unchanged. + if (outcome.ownBacklogDispatch === true && leaderManager !== null) { + leaderManager.releaseLeadership(); + leaderState = false; + } } else if (outcome.reason === 'no-candidate') { // WL-0MTEZ4XZJ006Y9U7 (AC1): in coordination mode the shared file // is an OFFER LIST, not the backlog — after a dispatch the leader @@ -6312,7 +6586,7 @@ export function createDowntimeWorker(opts: DowntimeWorkerConfig): DowntimeWorker // and reset the idle tracker (fresh full idle period required // after the pause). if (opts.coordinationDir && typeof opts.deps.fetchItem === 'function') { - const probe = await computeMostImportantItem(opts.deps, cfg.cwd, Date.now(), cfg.browseItemCount, cfg.markerStaleWindowMs, cfg.nonTerminalCooldownMs, cfg.maxAttempts); + const probe = await computeMostImportantItem(opts.deps, cfg.cwd, Date.now(), cfg.browseItemCount, cfg.markerStaleWindowMs, cfg.nonTerminalCooldownMs, cfg.maxAttempts, cfg.stallScanEnabled, cfg.stallScanThresholdMs); if (probe.ok && 'noCandidate' in probe && probe.noCandidate) { errorStrikes = 0; // the CLI answered — it is healthy cooldownUntil = Date.now() + cfg.noCandidateCooldownMs; @@ -7415,6 +7689,21 @@ export function clampDowntimeMaxAttempts(value: number): number { ); } +/** + * Clamp the stalled-work scan threshold (WL-0MUYMBK6H005PY16): reject + * negative/non-finite (fall back to the 5-minute default) and clamp to + * [MIN_DOWNTIME_STALL_THRESHOLD_MS, MAX_DOWNTIME_STALL_THRESHOLD_MS] + * (60 s – 60 min) so the scan can neither resume panes that are merely + * between tool calls nor leave genuinely stalled panes unattended for hours. + */ +export function clampDowntimeStallThresholdMs(value: number): number { + if (!Number.isFinite(value) || value < 0) return DEFAULT_DOWNTIME_STALL_THRESHOLD_MS; + return Math.min( + Math.max(Math.round(value), MIN_DOWNTIME_STALL_THRESHOLD_MS), + MAX_DOWNTIME_STALL_THRESHOLD_MS, + ); +} + // ── Round-robin helpers (WL-0MTJE0FXC006WAOX) ────────────────────────── /** Internal tier-group entry: a CoordinationEntry paired with its dispatch info. */ diff --git a/packages/herdr/src/fetcher-timeout.test.ts b/packages/herdr/src/fetcher-timeout.test.ts index 081662659..0db7a0492 100644 --- a/packages/herdr/src/fetcher-timeout.test.ts +++ b/packages/herdr/src/fetcher-timeout.test.ts @@ -191,7 +191,7 @@ describe('fetcher timeout defaults (WL-0MSJNJXX2001NMHS)', () => { }); setExecFileAsync(mockFn as any); - await fetchItemsByStage('in_progress'); + await fetchItemsByStage('plan_complete'); const opts = mockFn.mock.calls[0][2] as { timeout?: number } | undefined; expect(opts).toBeDefined(); diff --git a/packages/herdr/src/fetcher.ts b/packages/herdr/src/fetcher.ts index 4cc965686..aac97e785 100644 --- a/packages/herdr/src/fetcher.ts +++ b/packages/herdr/src/fetcher.ts @@ -12,6 +12,8 @@ import { promisify } from 'node:util'; import { join } from 'node:path'; import { selectWorkItems } from './smart-selection.js'; import { regroupWorkItems } from './grouping.js'; +import { STAGES } from './worklist.js'; +import type { Stage } from './worklist.js'; import type { AgentState } from './agent-tracker.js'; import { isAuditFresh, type ParentAuditState } from '@worklog/shared/icons'; @@ -637,17 +639,24 @@ const STAGE_STATUS: Record = { /** * Fetch work items filtered by stage (via `wl list`). - * Status per stage: - * - `in_review` (WL-0MSKCRX730052IIW): `completed`, `in-progress`, `open` — - * in_review items carry `completed`/`in-progress` status per the project + * + * Guards against invalid/unknown stages: returns an empty list without + * issuing a CLI call when the requested stage is not in the canonical + * `STAGES` list (WL-0MUY1CRS7001UTQJ). This prevents misleading empty + * worklists from being populated for removed stages such as `in_progress` + * and `completed`. + * + * Status per valid stage: + * - `in_review` (WL-0MSKCRX730052IIW): `completed`, `in-progress`, `open` + * — in_review items carry `completed`/`in-progress` status per the project * workflow. * - `plan_complete` / `intake_complete` (WL-0MUIB7D30009KG00): `open`, * `in-progress` — actively-worked items retain their pipeline stage and * flip `status` to `in-progress` (WL-0MTOHS5B4001Y9FX removed the * in_progress stage). - * - All other stages: `open` only (WL-0MSDT8X1V003206G): items with status - * `blocked`, `in-progress`, or `completed` are excluded even when their - * stage matches. + * - `idea` / `done`: `open` only — items with status `blocked`, `in-progress`, + * or `completed` are excluded even when their stage matches. + * * Root-only (WL-0MS964SIA0057ABR): stage-filtered top-level lists hide * child items; children remain reachable via expand (wl list --parent). * Results are regrouped priority-first (WL-0MSOPHLD1000EWNN): priority @@ -656,6 +665,9 @@ const STAGE_STATUS: Record = { * priority buckets + id tie-break apply. */ export async function fetchItemsByStage(stage: string): Promise { + if (!STAGES.includes(stage as Stage)) { + return []; + } const status = STAGE_STATUS[stage] ?? 'open'; const output = await runWl(['list', '--status', status, '--stage', stage, '--root-only']); const payload = extractJson(output); diff --git a/packages/herdr/src/grouping.test.ts b/packages/herdr/src/grouping.test.ts index 183cefa3e..95389ae42 100644 --- a/packages/herdr/src/grouping.test.ts +++ b/packages/herdr/src/grouping.test.ts @@ -113,6 +113,36 @@ describe('grouping.ts — duplicated algorithm mirrors core spec', () => { const sorted = items.slice().sort(compareGroupableItems); expect(sorted.map(i => i.id)).toEqual(['P-high', 'P-med', 'I-high', 'I-low']); }); + + it('places actively-worked items (status in-progress) in Group N (WL-0MUY1CSQG007TCYX)', () => { + const items: GroupableItem[] = [ + { id: 'WL-active', status: 'in-progress', stage: 'plan_complete', filePaths: ['src/active.ts'], priority: 'medium' }, + { id: 'WL-idea', stage: 'idea', filePaths: ['src/idea.ts'], priority: 'medium' }, + ]; + const groups = assignItemGroups(items, 3); + expect(groups.get('WL-active')!.groupLabel).toBe('Group 1'); + expect(groups.get('WL-idea')!.groupLabel).toBe('Idea'); + expect(groups.get('WL-active')!.group).toBeLessThan(groups.get('WL-idea')!.group); + }); + + it('sorts actively-worked items before plan_complete and intake_complete within a group', () => { + const items: GroupableItem[] = [ + { id: 'WL-intake', stage: 'intake_complete', filePaths: [], priority: 'high' }, + { id: 'WL-plan', stage: 'plan_complete', filePaths: [], priority: 'high' }, + { id: 'WL-active', status: 'in-progress', stage: 'plan_complete', filePaths: [], priority: 'low' }, + ]; + const sorted = items.slice().sort(compareGroupableItems); + // Actively-worked first, even at lower priority than the plan/intake rows. + expect(sorted.map(i => i.id)).toEqual(['WL-active', 'WL-plan', 'WL-intake']); + }); + + it('treats the removed in_progress stage as Other (fail-soft legacy rows)', () => { + const groups = assignItemGroups( + [{ id: 'WL-legacy', stage: 'in_progress', filePaths: ['src/legacy.ts'], priority: 'medium' }], + 3, + ); + expect(groups.get('WL-legacy')!.groupLabel).toBe('Other'); + }); }); describe('regroupWorkItems — merged-list regression (WL-0MSAK8YLB0025EGW)', () => { @@ -209,6 +239,18 @@ describe('regroupWorkItems — merged-list regression (WL-0MSAK8YLB0025EGW)', () expect(order.indexOf('WL-I-low')).toBeLessThan(order.indexOf('WL-idea')); expect(order.indexOf('WL-idea')).toBeLessThan(order.indexOf('WL-R')); }); + + it('treats status=in-progress as Group N and orders it first (WL-0MUY1CSQG007TCYX)', () => { + const merged: WorkItem[] = [ + makeItem('WL-plan', { stage: 'plan_complete', priority: 'high' }), + makeItem('WL-active', { stage: 'plan_complete', priority: 'low', status: 'in-progress' }), + makeItem('WL-idea', { stage: 'idea', priority: 'low' }), + ]; + const regrouped = regroupWorkItems(merged, 3); + expect(regrouped.find(i => i.id === 'WL-active')!.groupLabel).toBe('Group 1'); + expect(regrouped.find(i => i.id === 'WL-plan')!.groupLabel).toBe('Group 1'); + expect(regrouped[0].id).toBe('WL-active'); + }); }); describe('extractFilePaths (duplicated)', () => { diff --git a/packages/herdr/src/grouping.ts b/packages/herdr/src/grouping.ts index 397cd2aa3..d06cdc809 100644 --- a/packages/herdr/src/grouping.ts +++ b/packages/herdr/src/grouping.ts @@ -115,6 +115,10 @@ function isFilePath(candidate: string): boolean { export interface GroupableItem { id: string; stage?: string; + /** Lifecycle status (`wl` status vocabulary). Used to detect actively-worked + * items (`status === 'in-progress'`), which replaced the removed + * `in_progress` stage (WL-0MUY1CSQG007TCYX). */ + status?: string; filePaths: string[]; priority?: string; // Optional audit fields — used for in_review bucket sort @@ -207,8 +211,9 @@ export function groupItemsByFilePaths( * Group display order (most actionable first): * - **Critical Group N** — critical items partitioned by file-path conflicts * (per-category label counter, no single all-inclusive "Critical" group). - * - **Group N** — non-critical `in_progress` + `plan_complete` + `intake_complete` - * items partitioned by file-path conflicts (no stage prefix in the label). + * - **Group N** — non-critical actively-worked (`status === 'in-progress'`) + * + `plan_complete` + `intake_complete` items partitioned by file-path + * conflicts (no stage prefix in the label). * - **Idea** — single group for all `idea` items. * - **Other** — single group for all remaining non-critical items (safety net * for unknown/custom stages, `done`, etc.). @@ -249,11 +254,13 @@ export function assignItemGroups( nextGroup += count; } - // 1. Group N (in_progress + plan_complete + intake_complete). + // 1. Group N (actively worked + plan_complete + intake_complete). + // `in_progress` is no longer a CLI stage; "actively worked" is derived from + // `status === 'in-progress'` (WL-0MUY1CSQG007TCYX). const groupNItems = items.filter( item => item.priority !== 'critical' && - (item.stage === 'in_progress' || item.stage === 'plan_complete' || item.stage === 'intake_complete'), + (item.status === 'in-progress' || item.stage === 'plan_complete' || item.stage === 'intake_complete'), ); if (groupNItems.length > 0) { const groupNGroups = groupItemsByFilePaths(groupNItems, maxFilePathGroups); @@ -276,11 +283,14 @@ export function assignItemGroups( } // 3. Other. + // Safety net for remaining non-critical items (unknown/legacy stages such + // as the removed `in_progress`, `done`, etc.). Excludes actively-worked + // items because they belong to Group N via `status === 'in-progress'`. const otherItems = items.filter( item => item.priority !== 'critical' && + item.status !== 'in-progress' && item.stage !== 'in_review' && - item.stage !== 'in_progress' && item.stage !== 'plan_complete' && item.stage !== 'intake_complete' && item.stage !== 'idea', @@ -318,17 +328,28 @@ function buildLabelCounter(fileGroups: Map): Map // ── Within-group ordering ───────────────────────────────────────────── /** - * Stage sub-order within a group: in_progress first (actively being worked), - * then plan_complete, then intake_complete, then all remaining stages. + * Stage sub-order within a group: actively-worked items first + * (derived from `status === 'in-progress'`), then plan_complete, then + * intake_complete, then all remaining stages. * No headings are rendered between sub-groups. */ const WITHIN_GROUP_STAGE_ORDER: Record = { - in_progress: 0, plan_complete: 1, intake_complete: 2, }; const REMAINING_STAGE_ORDER = 3; +/** + * Within-group sub-order for an item. Actively-worked items + * (`status === 'in-progress'`) sort before every stage sub-group; the + * `in_progress` stage no longer exists in the CLI vocabulary + * (WL-0MUY1CSQG007TCYX). All other items fall back to the stage sub-order. + */ +function withinGroupStageOrder(item: GroupableItem): number { + if (item.status === 'in-progress') return 0; + return WITHIN_GROUP_STAGE_ORDER[item.stage ?? ''] ?? REMAINING_STAGE_ORDER; +} + /** * Priority order for within-group sorting: high → medium → low. * Unknown priorities are treated as medium. @@ -429,9 +450,9 @@ export function compareInReviewItems(a: GroupableItem & { /** * Compare two items for within-group display order: - * stage sub-sort (in_progress → plan_complete → intake_complete → remaining - * stages), then priority (high → medium → low), then id as a deterministic - * tie-break. + * stage sub-sort (actively worked → plan_complete → intake_complete → + * remaining stages), then priority (high → medium → low), then id as a + * deterministic tie-break. * * When BOTH items have stage === 'in_review', the 6-bucket predicate * (see `compareInReviewItems`) takes priority over the default stage @@ -449,8 +470,8 @@ export function compareGroupableItems(a: GroupableItem, b: GroupableItem): numbe if (bucketCmp !== 0) return bucketCmp; } - const stageA = WITHIN_GROUP_STAGE_ORDER[a.stage ?? ''] ?? REMAINING_STAGE_ORDER; - const stageB = WITHIN_GROUP_STAGE_ORDER[b.stage ?? ''] ?? REMAINING_STAGE_ORDER; + const stageA = withinGroupStageOrder(a); + const stageB = withinGroupStageOrder(b); if (stageA !== stageB) return stageA - stageB; const prioA = WITHIN_GROUP_PRIORITY_ORDER[a.priority ?? ''] ?? DEFAULT_PRIORITY_ORDER; const prioB = WITHIN_GROUP_PRIORITY_ORDER[b.priority ?? ''] ?? DEFAULT_PRIORITY_ORDER; @@ -472,6 +493,8 @@ export function compareGroupableItems(a: GroupableItem, b: GroupableItem): numbe export function regroupWorkItems ({ id: item.id, stage: item.stage, + status: item.status, priority: item.priority, filePaths: extractFilePaths(item.description ?? ''), needsProducerReview: item.needsProducerReview, @@ -502,13 +526,13 @@ export function regroupWorkItems { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** Nearest-rank percentile of a numeric sample. */ +function percentile(samples: number[], p: number): number { + if (samples.length === 0) return 0; + const sorted = [...samples].sort((a, b) => a - b); + const index = Math.min(sorted.length - 1, Math.floor(p * sorted.length)); + return sorted[index]; +} + +describe('hydrate UI responsiveness (WL-0MUY1CZES0099A04)', () => { + it('keeps event-loop delay under 100 ms p95 during a worst-case hydrate', async () => { + const panes = makeWorstCasePanes(60); + const items = makeWorstCaseItems(25); + const deps: HydratorDeps = { + // Deliberately slow seams: each CLI-shaped lookup yields the event + // loop, as the real async `execFile` spawns must. + listInProgressItems: async () => { + await delay(4); + return items; + }, + listActivePanes: async () => { + await delay(4); + return panes; + }, + listOutboundDeps: async () => { + await delay(4); + return []; + }, + applyDemotion: async () => { + await delay(4); + return true; + }, + maxItemsPerTick: 25, + log: () => {}, + }; + + const intervalMs = 5; + const delays: number[] = []; + let last = Date.now(); + const sampler = setInterval(() => { + const tick = Date.now(); + delays.push(Math.max(0, tick - last - intervalMs)); + last = tick; + }, intervalMs); + + let result; + try { + result = await runHydrationOnce(deps, 'wCurrent'); + } finally { + clearInterval(sampler); + } + + expect(result.ok).toBe(true); + expect(result.demoted).toBe(25); + // Command-input lag proxy: the event loop was never starved for long. + expect(percentile(delays, 0.95)).toBeLessThan(100); + }); + + it('bounds the run and names a timed-out step instead of hanging', async () => { + // A hung `wl`/`herdr` seam must not wedge the hydrate: the per-step + // budget aborts the run, names the offending step, and keeps the + // wall-clock bound well under the 20 s scheduler watchdog. + const deps: HydratorDeps = { + listInProgressItems: () => new Promise(() => {}), + listActivePanes: async () => [], + listOutboundDeps: async () => [], + applyDemotion: async () => true, + stepTimeoutMs: 50, + log: () => {}, + }; + + const started = Date.now(); + const result = await runHydrationOnce(deps); + const elapsedMs = Date.now() - started; + + expect(result.ok).toBe(false); + expect(result.reason ?? '').toContain('timeout'); + expect(result.reason ?? '').toContain('in-progress-list'); + expect(elapsedMs).toBeLessThan(1_000); + expect(result.demoted).toBe(0); + }); + + it('defers candidates beyond the per-tick cap without touching them', async () => { + const items = makeWorstCaseItems(25); + const touched: string[] = []; + const deps: HydratorDeps = { + listInProgressItems: async () => items, + listActivePanes: async () => [] as HydratorPane[], + listOutboundDeps: async (id: string) => { + touched.push(id); + return []; + }, + applyDemotion: async () => true, + maxItemsPerTick: 10, + log: () => {}, + }; + + const result = await runHydrationOnce(deps, 'wCurrent'); + + expect(result.demoted).toBe(10); + expect(touched).toHaveLength(10); + }); +}); diff --git a/packages/herdr/src/hydrator.fixtures.ts b/packages/herdr/src/hydrator.fixtures.ts new file mode 100644 index 000000000..0330e73a4 --- /dev/null +++ b/packages/herdr/src/hydrator.fixtures.ts @@ -0,0 +1,108 @@ +/** + * packages/herdr/src/hydrator.fixtures.ts — shared fixtures for the hydrator + * responsiveness contract suite (WL-0MUY1CX9900258E6). + * + * The contract is written test-first, so the fixtures are deliberately small, + * deterministic and free of real timers: + * + * • `hydratorExecRouter()` installs a recording exec seam: it answers every + * hydrator CLI shape with minimal valid JSON and records the exact + * `(bin, args, options)` of every spawn, so tests can assert the stdio + * and argument contract. + * • `makeWorstCasePanes()` / `makeWorstCaseItems()` build the load profile + * the fix must survive (50+ panes, 20+ in-progress items). + * + * No `vitest` import: these are plain builders consumed by the test file (and + * by the implementation children's tests), so they stay usable outside a + * vitest runtime. + */ + +import type { HydratorItem, HydratorPane } from './hydrator.js'; + +// ── Recording exec seam ─────────────────────────────────────────────── + +/** One recorded CLI spawn. */ +export interface RecordedSpawn { + bin: string; + args: string[]; + options: Record; +} + +/** Injectable exec seam shape used by the hydrator production deps. */ +export type HydratorExec = ( + bin: string, + args: string[], + options?: unknown, +) => Promise<{ stdout: string; stderr: string }>; + +export interface HydratorExecHarness { + /** Every spawn, in order. */ + calls: RecordedSpawn[]; + /** The recording implementation to install via `setExecFileAsync()`. */ + exec: HydratorExec; +} + +/** + * Build a recording exec seam that answers the four hydrator CLI shapes with + * minimal valid JSON and records every call. + */ +export function hydratorExecRouter(): HydratorExecHarness { + const calls: RecordedSpawn[] = []; + + const exec: HydratorExec = async (bin, args, options) => { + calls.push({ + bin, + args, + options: (options ?? {}) as Record, + }); + + if (bin === 'herdr' && args.includes('pane')) { + return { stdout: JSON.stringify({ result: { panes: [] } }), stderr: '' }; + } + if (args.includes('dep')) { + return { stdout: JSON.stringify({ outbound: [] }), stderr: '' }; + } + if (args.includes('update')) { + return { stdout: JSON.stringify({ success: true }), stderr: '' }; + } + if (args.includes('list')) { + return { stdout: JSON.stringify({ workItems: [] }), stderr: '' }; + } + return { stdout: '{}', stderr: '' }; + }; + + return { calls, exec }; +} + +// ── Worst-case load fixtures ────────────────────────────────────────── + +/** Zero-pad `n` so the generated work-item IDs match WORK_ITEM_ID_REGEX. */ +function padded(n: number, width: number): string { + return String(n).padStart(width, '0'); +} + +/** + * Worst-case pane volume: `count` panes (default 60) split across the current + * workspace (`wCurrent`, even indices) and another workspace, each carrying a + * work-item ID in its label as a real pane title would. + */ +export function makeWorstCasePanes(count = 60): HydratorPane[] { + return Array.from({ length: count }, (_, i) => ({ + pane_id: `pane-${i}`, + label: `implement WL-PANE${padded(i, 7)}`, + workspace_id: i % 2 === 0 ? 'wCurrent' : 'wOther', + })); +} + +/** + * Worst-case in-progress volume: `count` (default 25) pane-less candidates, + * all at `plan_complete` so every one is eligible for demotion. + */ +export function makeWorstCaseItems(count = 25): HydratorItem[] { + return Array.from({ length: count }, (_, i) => ({ + id: `WL-ITEM${padded(i, 7)}`, + title: `Stuck item ${i}`, + status: 'in-progress', + stage: 'plan_complete', + })); +} diff --git a/packages/herdr/src/hydrator.test.ts b/packages/herdr/src/hydrator.test.ts index b4a14bb50..4447a07ec 100644 --- a/packages/herdr/src/hydrator.test.ts +++ b/packages/herdr/src/hydrator.test.ts @@ -7,8 +7,15 @@ * and the test asserts observable behaviour (which items were released and * to what status/stage), plus the fail-open paths. */ -import { describe, expect, it, vi } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { resetExecFileAsync, setExecFileAsync } from './fetcher.js'; import { + hydratorExecRouter, + makeWorstCaseItems, + makeWorstCasePanes, +} from './hydrator.fixtures.js'; +import { + ALLOWED_STAGES_BY_STATUS, collectPaneWorkItemIds, compatibleStage, createHydratorRunner, @@ -18,6 +25,8 @@ import { isActiveBlocker, isActivePaneMatch, runHydrationOnce, + type HydrationResult, + type HydratorDepTarget, type HydratorDeps, type HydratorItem, type HydratorPane, @@ -31,12 +40,28 @@ interface AppliedDemotion { stage: string; } -function makeDeps(overrides: Partial = {}): { - deps: HydratorDeps; +/** + * The injected-deps contract for the responsiveness fix. The clock, per-step + * timeout and per-tick cap are part of the target contract and are added to + * `HydratorDeps` by the implementation children (WL-0MUY1CXSV008F8TZ, + * WL-0MUY1CYV6008BQWX). Declaring them here lets the test seams carry them + * without changing the production type until the implementation lands. + */ +interface ContractDeps extends HydratorDeps { + /** Injectable monotonic clock (ms). */ + now?: () => number; + /** Per-step timeout (ms); must be below the scheduler watchdog. */ + stepTimeoutMs?: number; + /** Maximum candidates processed per hydrate tick. */ + maxItemsPerTick?: number; +} + +function makeDeps(overrides: Partial = {}): { + deps: ContractDeps; applied: AppliedDemotion[]; } { const applied: AppliedDemotion[] = []; - const deps: HydratorDeps = { + const deps: ContractDeps = { listInProgressItems: async () => [], listActivePanes: async () => [], listOutboundDeps: async () => [], @@ -140,6 +165,21 @@ describe('compatibleStage', () => { it('repairs an invalid stage (blocked cannot sit at in_progress)', () => { expect(compatibleStage('in_progress', 'blocked')).toBe('plan_complete'); }); + + it('never allows the removed in_progress stage for any status (WL-0MUY1CSQG007TCYX)', () => { + for (const status of Object.keys(ALLOWED_STAGES_BY_STATUS)) { + expect(ALLOWED_STAGES_BY_STATUS[status]).not.toContain('in_progress'); + } + // The legacy stage is therefore always repaired to a CLI-valid stage. + expect(compatibleStage('in_progress', 'open')).toBe('plan_complete'); + expect(compatibleStage('in_progress', 'in-progress')).toBe('plan_complete'); + expect(compatibleStage('in_progress', 'blocked')).toBe('plan_complete'); + }); + + it('keeps the CLI-valid stages allowed for open and in-progress', () => { + expect(ALLOWED_STAGES_BY_STATUS.open).toEqual(['idea', 'intake_complete', 'plan_complete']); + expect(ALLOWED_STAGES_BY_STATUS['in-progress']).toEqual(['intake_complete', 'plan_complete']); + }); }); describe('decideDemotion', () => { @@ -354,3 +394,287 @@ describe('createProductionHydratorDeps', () => { expect(typeof deps.applyDemotion).toBe('function'); }); }); + +// ── Responsiveness-fix contract (WL-0MUY1CX9900258E6) ───────────────── +// +// Test-first contract for the hydrate-responsiveness fix. The assertions in +// this section are RED until the implementation children land: +// • WL-0MUY1CXSV008F8TZ — per-step timing and bounded timeouts (AC2, AC3) +// • WL-0MUY1CYCB003DTKO — isolate child-process stdio (AC1) +// • WL-0MUY1CYV6008BQWX — trim per-tick work (AC4, AC5) +// +// Contract pinned here (implement to this): +// • every hydrator CLI spawn passes `{ stdin: 'ignore', stderr: 'pipe' }`; +// • `runHydrationOnce` passes the workspace id to `listActivePanes`, which +// the production deps translate to `herdr pane list [--workspace ]`; +// • each step runs under a per-step timeout (`HydratorDeps.stepTimeoutMs`, +// default below the 20 s watchdog); a timeout aborts the run with +// `ok: false` and a `reason` containing the step label; +// • each step logs `...