Skip to content

🧪 test: runner: Capture cancellation phases near the deadline #475

Open

Description

@pascalandy

Two public20,000-file cancellation calls at f96f257 had unexplained elapsed gaps: TERM7 took13719ms with271ms cleanup, and TERM8 took14983ms with273ms cleanup. Both kept the handler result. The original run saved no force-stop, cleanup-end or host-exit stamps, so their location cannot be recovered.

Twelve instrumented calls at7314668f measured13288-13315ms. One later40-call attempt at9766bd85 measured13303-13377ms with0 calls above13600ms. Nineteen calls were fully inside a deliberate just-test load window, one overlapped its end and20 ran afterward. All40 canonical state guards passed. The historical gaps remain unexplained, not reproduced. This does not establish a measurement artifact or a runtime cause.

The metric signal_ms ends after the Verify wrapper records its transcript. It is not host-exit timing. cleanup_ms ends before cleanup.json and cleanup.done are written. The17ms difference between14983ms and15s is therefore a lower bound on host margin. The missing phases leave a near-cutoff risk that another occurrence could localize.

Scope to refine:

  • Record correlated phase timestamps in the stress driver and host answer path, without secret values, arguments or raw user payloads
  • Separate signal reference, reservation sweep, worker exit, cleanup start/end, stats writes, handler answer, host exit and wrapper transcript tail
  • Preserve the15s outer bound and nominal2s reservation; add no extension-specific exception
  • Check the observer's own overhead and distinguish captured events from pidfd observations or inferred times
  • Keep the existing public state/answer/survivor controls and make a future late call diagnosable from its saved evidence

This follow-up does not block the #468 fix. Before that fix a resistant worker lost handler data; the observed fixed20k calls keep it. Chef accepted the residual timing uncertainty after the bounded attempt. The oversized mini100k recovery limit remains separate in #473. The agent k10 failure during deliberate load is context only; no agent repair is requested unless it recurs unloaded.

Related: #468 and PR #471. Complete capture/commands/quotes: #471 (comment). Complete phase analysis and failed-load logs: #471 (comment).

by GPT-6.1-Sol via Codex

Activity

  1. added
    1-needs-triageScope, approach, or direction needs evaluation
    2-type:taskSupporting work such as maintenance, documentation, refactoring, or verification
    3-pty:p2Normal planned work; default when no priority is established
    on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    1-needs-triageScope, approach, or direction needs evaluation2-type:taskSupporting work such as maintenance, documentation, refactoring, or verification3-pty:p2Normal planned work; default when no priority is established

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions