Skip to content

🚑 fix: runner: Report staging left at the cancellation deadline #473

Open

Description

@pascalandy

The public host can reach its cancellation deadline while the extension is still deleting a large private staging tree. On mini, a 100,000-file tree remained after both SIGINT and SIGTERM. The answer kept the signal error and exit, but omitted the handler's stage field, so it did not identify the remaining tree.

This is the measured larger-tree follow-up requested during #468, outside the batch. Chef explicitly treats it as nonblocking for #468's 20,000-file acceptance. Do not extend or remove the outer bound as an untested workaround.

Observed on mini (Darwin), head 66a7ed509d8f7cb1e72ddad020eeb9ed6e2e18ed, build ncly v0.26.1-0.20261104022449-66a7ed509d8f, through the real public ncly hello staged path with a resistant worker and detached grandchild. The nominal reservation is the final 2 s of a 15 s cancellation budget.

CLI-exit observation SIGINT SIGTERM
Signal to exit 15028 ms 15011 ms
Exit and code 130 INTERRUPTED 143 TERMINATED
Remaining staging entries 61849 62201
cleanup.done absent absent
Completed delete/cleanup measurements unavailable unavailable
Worker and grandchild still running neither neither
Input and existing output bytes unchanged unchanged
Handler stage field in answer absent absent

The handler never wrote cleanup.json, so there is no completed deletion or cleanup duration to quote. The measurement records the incomplete cleanup at CLI exit, before harness cleanup. Receipt ok:true means both observations were collected and the disposable remote folder was cleaned. The observer correctly records observations_complete:true, passed:false.

The separate strict gate passed ten SIGINT plus ten SIGTERM calls with 20,000 files. Maximum deletion was 811 ms, maximum cleanup 912 ms, and minimum reservation margin 1088 ms. A separate 100-file calibration passed both signals with deletion 7 ms and cleanup 108 ms.

Complete strict receipt and twenty-run quote. Complete calibration and 100,000-file receipts, exact commands, observer diff and quotes. These comments preserve the source and commands when the temporary folders are removed. The scratch observer snapshots state before the canonical guards and continues to the second signal after a failure. It changes no product file, fixture or cancellation budget.

The actual signal answers were:

{"ok":false,"contract_version":1,"errors":[{"code":"INTERRUPTED","message":"Stopped by Ctrl-C. Finished work is kept.","hint":"Check the results before you run anything again."}]}
{"ok":false,"contract_version":1,"errors":[{"code":"TERMINATED","message":"Stopped by SIGTERM. Finished work is kept.","hint":"Check the results before you run anything again."}]}

Next: decide how bounded cancellation reports and recovers an incomplete private staging cleanup. Keep the outer bound and descendant-stop behavior. A repair must cover both signals with a real public host and preserve input and existing output. Retain the passing 20,000-file and cooperative/direct controls. Avoid a video-convert-specific exception.

by GPT-6.1-Sol via Codex

Activity

  1. added
    1-needs-triageScope, approach, or direction needs evaluation
    2-type:bugExisting behavior is broken
    3-pty:p2Normal planned work; default when no priority is established
    on Nov 4, 2026
  2. pascalandy commented on Nov 4, 2026

    @pascalandy
    OwnerAuthor

    Note

    🤖 GPT-6.1-Sol responding on behalf of pascalandy

    Fresh mini (Darwin) probes at f96f257 repeat the observed finite limit. The strict20k gate passes20/20, max deletion848ms and cleanup949ms. The separate100k observations leave61558 entries after SIGINT/130 at15015ms and63448 after SIGTERM/143 at15014ms. In both cases, input/output are unchanged, worker and grandchild are stopped, cleanup.done is absent and the host answer lacks handler data. No completed deletion/cleanup duration exists.

    The observer was calibrated on a fresh100-file run. Its exit0 means observations complete and remote cleanup succeeded; passed remains false for100k. This does not change the accepted non-blocking scope of this issue.

    Strict fleet receipt/command/full quote, calibration, 100k fleet receipt/command/full observations. All were read back after publication.

    by GPT-6.1-Sol via Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    1-needs-triageScope, approach, or direction needs evaluation2-type:bugExisting behavior is broken3-pty:p2Normal planned work; default when no priority is established

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions