Skip to content

fix(workflow-executor): hand back the runs the executor lets go instead of leaving them locked - #1945

Merged
Scra3 merged 2 commits into
feature/prd-1183-runtime-automation-pollerfrom
feature/prd-1387-executor-releases-the-runs-it-lets-go
Oct 1, 2026
Merged

Scra3 merged 2 commits into
feature/prd-1183-runtime-automation-pollerfrom
feature/prd-1387-executor-releases-the-runs-it-lets-go

Conversation

@Scra3

@Scra3 Scra3 commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

fixes PRD-1387

Targets the integration branch feature/prd-1183-runtime-automation-poller (#1906). Server side: ForestAdmin/forestadmin-server#8550 (POST /release-run), to deploy before #1906 is released; without it this PR degrades to today's behaviour (see Compatibility).

Problem

Since forestadmin-server#8536 (PRD-1305) a claimed run stays locked for 7 minutes. When the executor lets a claimed run go on purpose, the orchestrator does not know, and the run stays loading until the reaper frees it 7 to 7.5 minutes later: spinner then "No progress since…" for the operator, 409 on a re-trigger, and one of the inbox's 20 slots held for an automated run. It happens on every executor deploy or restart, for every run in the middle of an automatic chain.

Change

The executor hands the run back with WorkflowPort.releaseRun(runId, lockedAt), which posts /api/workflow-orchestrator/release-run with the lockedAt it received. The orchestrator only releases the run under that exact lock, so a run another executor claimed since is never touched.

  • AvailableRunDispatch carries lockedAt, copied from the run the orchestrator sent (pending-run, available-run, chained update-step).
  • The runner hands the next step back when it does not run it:
    • graceful stop in the middle of a chain, including once a drain timeout already marked the runner stopped while the step was running (before, that chain went on to the next step against closed stores);
    • chain depth cap reached;
    • non-progressing next step of the same run. A step of another run is left to its lock: another chain or instance may be running it.
  • A chained update-step answer the adapter cannot use (fails to parse, or no step left to run) is handed back before returning null.
  • Runs a poll brings back after stop() began are handed back instead of left to expire. Runs already in flight on this executor are not.
  • stop() also waits for the poll in progress, inside the same drain timeout, so those hand backs go out before the process exits. The drain timeout log says pollInProgress.
  • The request body is ServerReleaseRunRequest in server-types.ts, mirroring the server's ReleaseRunRequest.
  • releaseRun never throws: one request, no retry (it has to fit in the 30 s drain). A 404 is warned once then logged at debug; any other failure is warned. The lock expiring stays the safety net.

Compatibility

  • Orchestrator without the route: 404, one warn per process, the run is freed after 7 minutes as today.
  • Nothing else in the chain or poll changes; the docs in the package CLAUDE.md follow.
  • Contract amendment on PRD-1177.

Tests

  • runner.test.ts: hand back on graceful stop (and stop() waits for it), after a drain timeout, at the depth cap, on a non-progressing step of the same run; nothing handed back for another run's step, a completed chain or a chain that runs every step; poll after stop() hands back each claim but not one already in flight, stop() waits for it, and a poll that never answers is still bounded by the drain timeout; shutdown completes when a hand back rejects.
  • forest-server-workflow-port.test.ts: dispatches carry lockedAt (or null); the exact request; no call without a lock; 404 warned once then debug; 500 warned and not retried; unparseable or finished chained answer handed back, null answer not.
  • Mutation-checked, one at a time: removing each hand back, handing back runs already in flight, handing back another run's step, not waiting for the poll or awaiting it outside the drain timeout, dropping lockedAt in the adapter, dropping the chained-answer hand back, rethrowing, retrying, warning on every 404, and yielding only on draining each fail at least one test.
  • Package suite: 2036 passed.

🤖 Generated with Claude Code

…ad of leaving them locked

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@linear-code

linear-code Bot commented Sep 30, 2026

Copy link
Copy Markdown

PRD-1387

@qltysh

qltysh Bot commented Sep 30, 2026

Copy link
Copy Markdown

Qlty


Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (2)

RatingFile% DiffUncovered Line #s
Coverage rating: A Coverage rating: A
packages/workflow-executor/src/runner.ts100.0%
Coverage rating: A Coverage rating: A
.../workflow-executor/src/adapters/forest-server-workflow-port.ts100.0%
Total100.0%
🚦 See full report on Qlty Cloud »

🛟 Help
  • Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

  • Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

  • File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

    • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

@hercemer42 hercemer42 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spec (PRD-1387): conforms. Every let-go case the ticket and its PRD-1177 amendment list is handed back.

Comment thread packages/workflow-executor/src/runner.ts
Comment thread packages/workflow-executor/test/runner.test.ts Outdated
…re closes with the runner's logger

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Scra3
Scra3 merged commit 22d814c into feature/prd-1183-runtime-automation-poller Oct 1, 2026
37 checks passed
@Scra3
Scra3 deleted the feature/prd-1387-executor-releases-the-runs-it-lets-go branch October 1, 2026 18:17
Scra3 added a commit that referenced this pull request Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants