feat: retry and redrive dead effects and broadcasts

[cardmagic]

Ports cardmagic/solid-objects-ruby#78, which closes #73 there.

Why

DeadLetterManager covered message dead letters only. A dead effect or
broadcast reached status = 'dead' and nothing brought it back, so the recovery
path was an operator writing an UPDATE against a runtime table. Retry was also
one row at a time, and an incident produces dead rows in the hundreds.

The kind rides on the receiver

runtime.deadLetters              // messages, unchanged
runtime.deadLetters.effects
runtime.deadLetters.broadcasts

retry returns a dead row to pending with a zero attempt count, no claim, and
immediate availability, and keeps its id so a handler that deduplicates on the
effect id still sees the same key. It acts only on a dead row, so a second press
cannot double-enqueue and cannot take a row away from a worker that holds it.

Each scope authorizes under its own resource name: effect_dead_letters,
broadcast_dead_letters, redrives.

Redrive

const task = await runtime.deadLetters.effects.redrive({
  actorType: "payments",
  failedAfter: new Date(Date.now() - 6 * 60 * 60 * 1000),
  limit: 5_000,
  authorizationContext,
})
await task.cancel({ authorizationContext })

await runtime.redrives.find(task.id, { authorizationContext })
await runtime.redrives.all({ status: "running", authorizationContext })

Idempotency is a database constraint: the scope and filter digest go in a unique
active_scope column that holds the digest while the task runs and NULL once
it finishes, so the index is total rather than partial and two processes that
start the same redrive share one task.

runtime.run() advances one bounded batch per pass through a RedriveScheduler
component, each batch its own short transaction. A pass that moved rows pauses
redriveBatchPauseMilliseconds; an idle pass waits the idle polling interval,
so an empty table is not polled every few milliseconds.

A defect this port found, fixed in both runtimes

A redrive read its scope fresh on every pass, so a row it moved that failed
again landed straight back in it. With a handler that is still broken and no
limit, the task would move the same rows forever and never finish.

Ruby did not show it, because that test drives the redrive with no effect worker
running. Here the workers run beside it, so the churn was immediate: the
supervised test hung until it timed out.

A pass now takes only rows that were already dead when the task started. That
needs a failure stamp, so schema version 11 adds failed_at_ms to effects and
broadcasts, stamps it on the dead transition, and backfills existing dead rows
from their availability stamp. The same bound is now in
the Ruby branch,
keyed on updated_at.

Audit

Every retry and every redrive transition writes one row to
solid_objects_administration_events. The identity comes from the authorization
context through a new administrationIdentity option, defaulting to its
String form and bounded to 255 bytes. A refused caller writes nothing; a read
writes nothing.

The event id carries the timestamp and a counter, so a reader orders by it and
sees insertion order within a process. Two events can share a millisecond, and
across processes the stamp is all any log can offer.

Schema

Version 11: two tables and one column on each of effects and broadcasts. No
change to statuses, because dead and pending already exist.

Tests

27 new tests across two files, each watched failing first.

test/dead-letter-scopes.test.ts: a dead effect returns to pending and runs
again, it keeps its id, retrying a pending effect changes nothing, a dead
broadcast returns and delivers, runtime.deadLetters is unchanged, each scope
reads only its own kind, an unauthorized caller is refused, and each scope names
its own resource.

test/redrive.test.ts: bounded batches, the limit, running idempotency, a
separate task per scope and per filter set, a new task after the first finishes,
cancel leaving moved rows moved, both filters, reading tasks back, remaining
counted at read time, a row that died after the task started is not moved, a
running runtime draining a redrive with no caller driving it, refusal, and the
audit rows for retry, finish, cancel, identity, refusal, and reads.

Validation

Gate	Result
vitest run	477 passed, 32 skipped
PostgreSQL 18	52 passed
MySQL 8.4	39 passed, 7 skipped
Cloudflare Workers	51 passed
Browser, Playwright Chromium	9 passed
Prettier, 3 check scripts, 4 tsc projects	clean

BIGINT rather than INTEGER for failed_at_ms outside SQLite: the first
version overflowed a 32-bit column with a millisecond epoch, which PostgreSQL
and MySQL both caught.

Parity

docs/parity.md records the two differences. This runtime filters on the new
failed_at_ms stamp, while Ruby filters on the updated_at column Active
Record already maintains. The Durable Objects engine keeps its own message and
outbox tables inside each object, so these scopes cover the SQL backends here
and its deadLetters call is unchanged.

What this does not do

Automatic redrive on a schedule, which the issue puts out of scope, and the
dashboard UI for the new scopes.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issue was introduced since the previous review, and all previous findings are resolved.

Summary

Adds durable retry and redrive support for dead effects and broadcasts.

• Introduces kind-specific dead-letter scopes with authorization, retry, filtering, and audit records.
• Adds durable, idempotent redrive tasks processed in bounded transactional batches.
• Adds schema version 11, including redrive and administration-event tables plus failure timestamps.
• Integrates redrive scheduling into the runtime and documents configuration, operations, dashboard scope, and runtime parity.
• The latest revision adds concise dashboard documentation for the new APIs and audit records.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Operator[Operator] -->|retry or redrive| Auth[Administration authorization]
  Auth -->|retry| Scope[Effect or broadcast dead-letter scope]
  Scope -->|dead to pending| Worker[Effect or broadcast worker]
  Auth -->|start| Task[Durable redrive task]
  Task --> Scheduler[Redrive scheduler]
  Scheduler -->|bounded transaction| Scope
  Auth --> Audit[Administration event]
  Task --> Audit

Loading

_{Reviews (6) · Last reviewed commit: "docs: point the dashboard at the new sco..."}

[cardmagic]

@greptileai review

[cardmagic]

@greptileai review

[cardmagic]

@greptileai review

[cardmagic]

@greptileai review

[cardmagic]

@greptileai review