Skip to content

Fix fleet recovery and add Python qualification fixtures - #103

Open
forhappy wants to merge 22 commits into
mainfrom
codex/python-reliability-replay
Open

forhappy wants to merge 22 commits into
mainfrom
codex/python-reliability-replay

Conversation

@forhappy

@forhappy forhappy commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Cell fetches could retry full peers indefinitely, while a zero-sized or evicted route cache discarded the placement sample needed to exclude a refused peer. Carry that sample through dispatch, preserve a monotonic rejection watermark, and bound retries by the original deadline and cancellation. Preserve Overloaded/unknown through the guest bridge; ambiguous transport outcomes stop without replay.

Retaining shutdown checked the node-log seal before syncing resident cells into the per-cell layout. Sync before the seal check so the last node can restart without its stopped witnesses, retaining deadline, authority and coverage guards. A two-node regression covers bundle pacing and reload.

Remove an SDK-owned Python reference cycle: Context eagerly held Storage, which held the context. Even stateless requests retained contexts and native bridges until cyclic collection. Create storage handles on access; handles still retain their originating context and enforce scope and expiration. Release successful eager invocations before further admission. Tests disable cyclic GC so collection cannot hide retention on completion or capacity rejection.

Completed Raft requests release payloads and capacity before publishing replies. Deferred peer steps retain both ordinary and inbound permits until execution drains. The single-slot startup fixture retries only read-only Overloaded status observations within three seconds because background inspection shares that slot. The cancellation fixture now polls cancellation before releasing held SQL. Original startup, cancellation, outcome and durability assertions remain strict.

Add the Python S3 counter qualification fixture with the same native host, load generator and recovery scenarios. Linux qualification checks pinned runtime images, loader provisioning, binary hashes and raw shutdown evidence. The load harness records disjoint host/client CPU affinity and supports a pinned reference revision on the same runner, preserving every arrival and refusal. The Node package fixture owns an independent execution domain instead of colliding with the bootstrap singleton. Register the PydanticAI example in the catalog and cover lost acknowledgments in model/tool checkpoints.

Completed Python invocations now mark managed cleanup as finished, avoiding a redundant native completion wakeup while preserving cancellation wakeups for unfinished or directly expired tasks. Deterministic scheduler time isolates receipt-accounting tests from CI scheduling delays; a separate test verifies genuinely late arrivals are rejected before transport. Native recovery fixtures capture the last settlement state, exact acknowledgment response and binding-adoption failure context without changing deadlines, retries or assertions.

The Python native sender now captures the original event-loop and invocation accessors before application imports, while checking their current values on every operation. Rebinding a module attribute can no longer substitute caller identity. GC traversal covers the captured references; the regression fails against the previous implementation.

Validation:

  • The caller-accessor change passes 15 library tests, the full bridge and isolated bootstrap suites, 10 runner tests and strict affected-crate Clippy on both CPython 3.14.5 ABIs. Formatting, layout, dependency policy and documentation checks pass.
  • The controlled Linux caller-accessor comparison passes on both ABIs with five repetitions, independently verified against 3,382,500 raw arrivals, 30 joined echo hosts and 20 churn/recovery cases. At 15k offered requests/s, CPU per completed handler falls about 10.3%, completion rate improves 5.6–6.2%, and p99 falls 10.4–12.5% against e5057bf on the same runner. The candidate still completes only 71–73% of native throughput with p99 approximately 11 times native; all capacity drops remain in the report. Absolute comparisons between ABI runners are invalid.
  • The candidate Linux image and S3 recovery matrix passes on both ABIs at 9312ed4. Independent verification checks image/source/binary identities, acknowledged-write totals, and 16 joined shutdown reports plus 21 recovery records for each native/Python runtime on each ABI runner. Recovery reads encountered HTTP 500 responses and were retried; mutations were not replayed. This is functional recovery evidence with shared host resources, not uninterrupted availability or production capacity qualification.
  • Both CPython 3.14.5 ABIs pass the full bridge suite, 15 library tests, 10 runner tests and strict Clippy for the completion-wakeup change. Coverage includes cancellation, direct expiration, scope isolation, fencing, alarms, owner termination and disk deletion, restart, handoff and complete joined shutdown. Formatting, layout, documentation and dependency checks pass.
  • All 209 performance-tool tests pass (one existing skip), including the reproduced scheduler-time sensitivity regression. All 16 native clock/recovery tests and affected strict Clippy pass with the additional failure capture.
  • The controlled Linux wakeup comparison passes on both ABIs with five repetitions each. Independent verification covers 3,382,500 raw arrivals, source/binary identities, disjoint host/client CPU affinity, 30 joined echo hosts and 20 churn/recovery cases. Interpreter CPU per completed handler falls at all offered rates: approximately 8–24% on the standard runner and 5–27% on the free-threaded runner. Each ABI used a different VM, so absolute ABI comparisons are invalid.
  • Native performance parity is not achieved. At 15k offered requests/s the free-threaded candidate completes about 9,678 requests/s versus 14,984 for native, with p99 13.98 ms versus 0.99 ms; roughly 35% of arrivals are dropped at the client inflight limit. The standard candidate sustains the offered rate but p99 remains 2.05 ms versus 0.54 ms for native. Successful workflow completion is not a capacity/SLO acceptance gate.
  • The storage reference-cycle regression fails before the fix and retains zero of 10,000 completed contexts afterward with cyclic collection disabled, on both ABIs. The earlier controlled comparison establishes prompt release, not a consistent throughput benefit.
  • All 11 PydanticAI tests pass on both ABIs; the real free-threaded HTTP smoke covers pending-alarm restart, receipt replay, concurrent isolation, cancellation and joined shutdown. Both ABIs passed the Linux image and S3 recovery run, with 16 complete joined reports and 21 recovery records per ABI.
  • Native fixes passed 496 library tests, 20 Raft operations tests, 17 execution tests (two existing process fixtures ignored), and 15 runtime tests (one existing process fixture ignored). The deterministic reply-wakeup regression fails before the admission fix; the startup-contention fixture passes 100 fresh replays.

Qualification remains open:

  • Previous full SDK CI is not green. PR jobs at 31521e2 failed in the now-corrected receipt-accounting fixture and cold follower binding adoption. The standard push job also failed initial settlement and two positive acknowledgment assertions. Twenty isolated Linux cold-recovery replays and ten local replays passed without reproducing those native failures. The new failure capture is diagnostic evidence support, not a claimed production fix. Full SDK runs at e5057bf were still in progress when this update was published; they are not counted as passing. Publishing the new head restarts full qualification under workflow concurrency rules. Updated-head CI must finish before merge.
  • The 16-cell durable comparison remains failed after LTX proof timeouts and ambiguous/rejected lease renewals. Its failed primary required cleanup termination. Recovery from a preserved copy with local owner state removed retained all 1,300 acknowledged writes and all 10 writes with unknown outcomes, without replaying mutations, then joined cleanly. This proves the observed retained state, not uninterrupted availability or a complete performance comparison.
  • An earlier durable comparison also remains failed on a native control. Recovery retained all 51 acknowledged writes; one unknown write was absent from recovered bucket state and was not retried. Passing repetitions do not explain the original durability/lease failure or earlier lifecycle/cancellation failures.
  • Native parity, sustained production capacity and broad production readiness remain unproven. Generated benchmark and failure evidence stays outside Git; no plans/evidence files are added.

@forhappy forhappy changed the title Preserve native recovery evidence and add isolated reliability replay Bound peer capacity retries and preserve native recovery evidence Oct 23, 2026
@forhappy forhappy changed the title Bound peer capacity retries and preserve native recovery evidence Fix peer capacity retries with bounded routing caches Oct 23, 2026
@forhappy forhappy changed the title Fix peer capacity retries with bounded routing caches Fix capacity retries and retaining fleet restarts Oct 23, 2026
@forhappy forhappy changed the title Fix capacity retries and retaining fleet restarts Fix fleet recovery and add Python qualification fixtures Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant