[Enrique Saurez (esaurez)]

Part of #619
Refs #535

Step G3 of #619: any guest init or agent can run each program as one execution, with its launch, bounded and backpressure-aware output, input, ending and outcome, without NVX's control protocol or an async runtime. NVX's managed agent becomes its NVXC adapter.

What changed

• nvx::guest::exec

  • Pipes::new(input) creates the program's standard output and error pipes, and its input pipe if asked. The execution's ends are nonblocking, with open file descriptions of their own, and the program's ends (ProgramEnds) stay blocking, for the caller's spawn::Command.

  • Execution::new(spawned, pipes, options) drives the program that G1's Command spawned. Options chooses what ending the program kills, the program alone or its process_group(), plus a cgroup(&ExecutionCgroup, Scope); an optional timeout; and who collects the program's status (caller_reaps()).

  • A failed start: new opens the program's pidfd by the program's ID, so nothing may collect the program's status before new returns, as its docs and caller_reaps()' now say. If opening the pidfd fails, new kills the program by its ID and then, with process_group(), the group that it led, unless ESRCH says that the status is gone already, so that a program that joined another group since ends too. It then closes the pipes and the launch report, and kills the cgroup through the directory that its ExecutionCgroup holds (nvx-rs: add caller-owned guest cgroup and child-status APIs #655), so that a failure for want of descriptors still leaves one for that kill. The status stays the caller's to collect.

  • The caller drives it. next_event(buffer) takes one step without blocking and returns an Event, or None once every descriptor would block. descriptors() lists what to poll, each with its Source: the launch report, an output stream, or the program's pidfd. deadline() gives the timeout. Executions share the caller's loop or reactor and never wait for each other.

  • Events: Launch, as G1's LaunchReport::wait returns it; Output(stream, bytes); Discarded; Closed; ReadFailed; KillFailed(target, error); and finally Ended(Outcome), with the program's status, the Cause if the execution ended it, and the discarded byte counts.

  • Flow control: grant(stream, bytes) is the seam. The execution reads a stream only within its allowance (usize::MAX never runs out), into the buffer that the caller lends, and alternates between the two streams one read at a time. A stalled sink stalls only its own execution, and the program blocks once its pipe fills. discard(stream) reads the rest into the same buffer and drops it, so that the buffer bounds every read, as the C agent's drain did with its 32 KiB output buffer. nvx-rs: exec v2 with credit-based streaming #623's credit engine extends this seam: it grants each stream what the peer credits.

  • Input: write_input writes what the pipe takes now and never raises SIGPIPE, whatever the process's disposition. It blocks the signal in the writing thread and takes back the one that a failed write left pending, unless the thread itself already had one pending, which absorbs the write's and stays. A SIGPIPE pending for the whole process doesn't absorb it, so it stays beside the write's, which the write takes back. sigpending reports both queues at once, so when it shows a SIGPIPE, the write reads the thread's own from the SigPnd line of /proc/thread-self/status, into a stack buffer. close_input sends EOF, and input() lends the writer for polling. Once the program has ended, the execution closes the input, so that a descendant that still reads it sees EOF.

  • Ending: cancel(), terminate(), the timeout, and disconnect(), which also discards both streams. The first cause while the program is unreaped becomes the outcome's cause. Each closes the input and sends SIGKILL:

    • to the program, through its pidfd, which the kernel refuses with ESRCH once a collector has taken the program's status, so that checking and signalling are one step;
    • then, with process_group(), to the group that the program leads, through the same pidfd with PIDFD_SIGNAL_PROCESS_GROUP, which names that group whatever later takes its ID. A kernel before Linux 6.9 refuses the flag, and the execution signals the group by its ID instead, right after the program's own signal found it unreaped;
    • to the cgroup, through the directory that its ExecutionCgroup holds, so that a kill never reaches a later cgroup at the same path.

    Only ESRCH tells that the program is gone. A signal that the kernel refuses otherwise, for example with EPERM, is one KillFailed event: the program still counts as unreaped, so the cause stands, the timeout fires once, and the execution goes on with the program's output and status.

    Once the program has ended with its output still open, the execution kills the cgroup again. Ending the program doesn't end its streams: the caller takes or discards what remains, so normal completion loses no byte.

  • Status: by default, the execution collects its program's status itself, by its pidfd, and never another child's.

    • A caller that collects every child's status itself, such as a PID 1 with a waitpid(-1) loop, hands the program's status over with observe (G2).
    • With Options::caller_reaps(), the caller's collector is the program's only one: the execution never waits for the program, and doesn't list its pidfd.
    • In the default mode, if another collector takes the status first, next_event fails once with ECHILD. The program has ended, so the execution closes its input and kills the cgroup if the output is still open, as once it has the status. It then stops collecting and signalling, and waits for observe.

    Dropping an unfinished execution kills its program without waiting.

• The managed agent as the NVXC adapter. managed_agent/exec.rs now drives each workload as an Execution. The agent keeps:

  • the NVXC framing, the null device as the workload's input, admission, identity selection, and mode-specific cleanup;
  • the 1 MiB output cap, checked for each read of up to 32 KiB;
  • BUSY for other requests, cancellation, timeout, and reset handling, with a cancellation taking precedence over a timeout, the timeout over the cap, and the cap over the workload's own status;
  • its 25 ms poll and its diagnostics, including exec-cgroup-kill.

  The direct agent's kill of descendants that hold the output open is now the execution's cgroup kill, through cgroup::options. Its waitpid(-1) loop hands the workload's status over with observe, from the agent's one thread, in the default mode. A workload status that the agent can't collect, which nvx-rs: add caller-owned guest cgroup and child-status APIs #655 handles once the agent has ended the workload for a cause, now also ends a request without one, with EXIT 125 failed and the exec-wait diagnostic, logged once per request, even while a descendant holds the workload's output, which the cgroup kill then ends.

• Internals: sys gains write_without_sigpipe, kill_process and kill_through_pidfd. readable_or_hung_up polls any number of descriptors, and kill_process_group returns its result. managed_agent::cgroup::terminate goes: its last caller, the agent's launch-failure path, now only reaps the workload that Execution::new has ended.

• Test harness: tests/managed_lifecycle.rs now waits until its copy of the guest console has connected before it drives the guest, and each connection attempt gives up after 100 ms.

  • On a fast WHP host here, the whole managed lifecycle with this change ran in about 0.33 s of OpenVMM's life. Windows retries a refused loopback connection for about two seconds, so the copy missed the console every time, and the test failed in capture_console after the lifecycle had passed.
  • With nvx-rs: add caller-owned guest cgroup and child-status APIs #655's agent, the run lasted about 0.57 s, and a retried connection still caught it.
  • The scenario checks nothing new or less.

• Docs: the crate README's module table and guest-binaries section, lib.rs, the guest and managed-agent module docs, and doc/project-structure.md describe the driver.

Behavior

The NVXC frames that the agent sends change in three ways, and no fixture depends on any of them:

• Interleaving: in each poll turn, the agent used to read all that standard output held before standard error. It now alternates reads of up to 32 KiB between them. Each stream keeps its byte order, the cap still counts both, and a stream that never pauses no longer delays the other.
• Faster wakeups: the agent's loop also wakes when the workload exits or its launch report arrives, not only at its next 25 ms poll.
• A lost status without a cause: if the kernel collects the status of a workload that the agent didn't end, as when SIGCHLD is ignored, the request ends with EXIT 125 failed and the exec-wait diagnostic, even while a descendant holds the workload's output, which the cgroup kill then ends. A cancellation or a timeout that comes after the agent learned of the lost status, while the output drains, changes that outcome no more than it would a known status; a cause from before stands. nvx-rs: add caller-owned guest cgroup and child-status APIs #655's agent kept waiting for that status until a cancellation or a reset ended the request.

There are also two new failure points before the workload runs:

• If the workload's pidfd can't be opened, Execution::new ends the workload and its cgroup. The agent then logs an exec-launch diagnostic, reaps the workload, and answers launch-failed.
• The pipes are made nonblocking before the spawn. If that fails, the launch fails before any workload exists.

Tests

• guest::exec, 29 tests on real programs:
  • Basics: launch, output and status; a failed launch stage, reported before status 125; and an execution without output that ends through the launch report's hangup and the pidfd alone.
  • Concurrency: 16 executions in one loop, each with its own input and output.
  • A stalled sink: a 256 KiB writer without an allowance blocks while another execution finishes, then every byte arrives once it is granted.
  • Cancellation of a stalled program; the caller then discards the rest. Every discarded read fits the buffer that the test lends, as every output event does.
  • Input: stdin EOF after 300,000 bytes through cat. With the default SIGPIPE disposition, each in a process of its own, a program that closes its input, and four threads that write at once, both get EPIPE without SIGPIPE. The program's end closes its input, which a background descendant still reads.
  • Ending: the timeout; a timeout whose signals a test-only hook has the kernel refuse with EPERM, which fires once, reports each failed kill once, and still delivers the program's output and status; disconnection; causes ignored once the program's status is known; observe; a dropped execution kills its program; and a cancellation, without a cgroup, ends a descendant left in the program's process group.
  • Status:
    • An unrelated child's status is left alone.
    • With caller_reaps(), the execution takes every other step but leaves the status for the caller's collector.
    • Another collector takes the status first, after which the caller cancels the execution, waits for it, or does neither. Collecting fails with ECHILD once at most, and the deadline goes. Cancellation and disconnection signal nothing: a descendant left in the program's process group survives them.
    • new signals nothing by ID for a program that was reaped before it started.
  • Fairness: with 8 KiB waiting in each stream, 1 KiB reads alternate strictly between them.
  • Cgroups (fake hierarchies):
    • the second kill once the program ends with a descendant holding its output, and the same kill, with the input closed, once another collector took the program's status, which the ECHILD of next_event tells;
    • a failed kill, reported;
    • a kill that never reaches a cgroup recreated at the same path;
    • a start that runs out of descriptors at the pidfd still kills the program and the cgroup. It runs under a lowered RLIMIT_NOFILE, in a process of its own.
  • A failed start without a cgroup: the test runs its own executable again as a program that joins the test's process group, which leaves the group that it led without a process. A start that then runs out of descriptors still kills it.
  • Pipes: only the execution's ends are nonblocking.
• guest::sys::signal: under SIG_DFL, in a process of its own, a write without a reader fails with EPIPE, leaves no SIGPIPE pending, and restores the mask. In a thread that blocks SIGPIPE, the write takes its own signal back, and a signal that was pending for the thread before the write stays. In a forked single-threaded child, which keeps a SIGPIPE sent to the whole process pending, the write takes its own back and leaves exactly that one.
• guest::sys::process: a SIGKILL through a pidfd succeeds for an ended but unreaped child and fails with ESRCH once its status is collected. A group signal through a leader's pidfd still reaches a member of its group once the leader's status was collected, which is the interleaving of a collection between the program's signal and the group's (on a kernel before Linux 6.9, the test only checks the member's end).
• managed_agent::dispatch, through the direct agent's whole request path:
  • Output-cap parity with the C agent: a workload that writes 1.5 MiB gets stdout frames of at most 32 KiB, totaling more than 1 MiB less 32 KiB and at most 1 MiB, then EXIT 125 output-limit. It never reaches the marker that it would write afterwards.
  • Requests while a workload runs: a PING gets BUSY, and a cancel of another request changes nothing. The workload's own cancel ends it with EXIT 137 cancelled, and a 100 ms timeout ends the next one with EXIT 124 timeout.
  • A lost status: nvx-rs: add caller-owned guest cgroup and child-status APIs #655's test, which ignores SIGCHLD in a process of its own, now also runs a workload without a timeout after the timed-out one. That workload exits on its own, but only once its first output frame has shown that the agent opened its pidfd: it blocks on a FIFO that the test closes then, as a status that the kernel collects before Execution::new would answer launch-failed. It ends with EXIT 125 failed, and each request logs exec-wait once. In a second test, a workload exits on its own while a descendant holds its output open, both behind FIFOs that the test controls. Once the agent's cgroup kill shows that it learned of the lost status, the test cancels the first such workload, and lets the 1 s timeout of the second pass, before it lets the descendant end. Both still end with EXIT 125 failed.
• Mutation checks: each of 39 mutations makes one of these tests fail:
  • in the agent: drop the kill at the cap, shift the cap by one chunk, or cancel or time out without a kill; keep waiting for a lost status, miss one that next_event or the wait after a cause reports, send no exit for one without a cause, or let a later cancellation or timeout override it;
  • output and input: skip the cgroup kill once the program ends, ignore the allowance, discard into a buffer of the execution's own, stop alternating the streams, or leave the input open on ending or at the program's end;
  • timeout and status: never fire the timeout, count a program whose signal the kernel refused as reaped, which fires the timeout again at every step, or ignore an observed status;
  • a failed start: leave the cgroup, kill it before closing descriptors, signal nothing, or signal only the group that the program led;
  • collection: signal a reaped program on a kill or a failed start, leave a collector's status unnoted when a kill finds it, keep collecting after ECHILD from next_event or wait, leave the program unended then, keep the deadline after it, or ignore caller_reaps();
  • signals: spare the process group, signal it without the group flag, or send the group's signal to the program alone in sys;
  • SIGPIPE: take back a signal that was pending before the write, leave the write's own pending in a thread that blocks the signal, count one pending for the process as the thread's, or read the process's pending set instead of the thread's.

Sizes

I built the guest binaries and the Alpine initramfs natively on Linux, the same way at this PR's base and with this change. The base is dev at 068f8c8, which this PR is rebased on, built as root in a worktree of its own. The binaries use the guest profile, which strips them.

Artifact	Before	After	Change
nvx-managed-agent	483,808 B	487,904 B	+4,096 B
initramfs.cpio.gz (Alpine)	8,211,701 B	8,216,512 B	+4,811 B

The other guest binaries keep their sizes. Uncompressed, the initramfs grows by the agent's 4,096 B alone.

Adapter contract

Yes, additively: the guest configuration gains one public module, nvx::guest::exec. No existing public item, persisted format or wire format changes, and the runtime adapter builds unchanged in every configuration (C0).

Validation

• WHP, Windows 11: python scripts\nvx.py test-microvm --backend whp passes every scenario of the default suite (43 runs), and so does a run with --scenario managed-exec-config --scenario sandbox-lifecycle --scenario managed-lifecycle. Both use the Alpine initramfs built from this change, with the pinned OpenVMM and kernel.
• KVM, nested on the same machine: the machine's nested KVM can't boot the built-in CPU profile that --cpu-profile auto selects. With a local wrapper that starts OpenVMM with --cpu-profile host instead, the same three managed scenarios pass with --backend kvm, which covers the direct and sandbox agents.
• Windows 11 and Linux (Ubuntu, x86-64), in nvx-rs with NVX_BUNDLE=skip:
  • cargo fmt --all --check
  • cargo clippy --locked --all-targets --all-features -- -D warnings
  • cargo clippy --locked --lib --no-default-features -- -D warnings
  • cargo test --locked --all-features, then cargo test --locked
  • the synthetic-release bundled test
  • cargo doc --locked --no-deps --all-features, with RUSTDOCFLAGS=-D warnings
  • cargo +1.89 check --locked --all-targets --all-features
• Linux only:
  • cargo check --locked --all-targets --all-features --target aarch64-apple-darwin
  • python3 scripts/nvx.py check-guest-binaries --toolchain 1.95.0
  • build-guest-binaries, then test-guest-binaries
  • the nvx-rs tests unprivileged, as CI runs them, and as root, which the guest::cgroup tests need to change the real hierarchy
  • the guest::exec, guest::sys::signal and guest::sys::process tests, pinned to two CPUs, 20 times unprivileged and 10 times as root
  • the managed_agent::dispatch tests, pinned to two CPUs, 30 times unprivileged and 15 times as root
• The runtime adapter's configurations, as check-nvx-rs runs them: protocol-only and host-only on both OSes, and guest-only on Linux.
• aci_edge_sandboxes, on both OSes: its fmt, Clippy, test, doc and MSRV steps, and test-agent-bundling. On Linux, its macOS check too.
• python scripts/nvx.py verify and python scripts/check_markdown_links.py.

Everything passes except one environmental failure on Linux as root, which dev has too, and which #739 tracks: the CLI golden test's caller-owned share scenarios refuse their temporary share because UID 0 owns it, as --mount-owner caller requires. The unprivileged run passes.

I didn't run MSHV, or the KVM suite beyond the managed scenarios. CI runs the KVM, MSHV and WHP matrix.

[Copilot]

🟡 Changes recommended

Exit handling can prevent completion indefinitely, and constructor failure can leave cgroup descendants running.

2 open issues

• Close stdin when recording final program status · New
• Clean up configured cgroup when pidfd opening fails · New

What changed in this PR

Adds a reusable guest execution driver and makes the managed agent its NVXC adapter, building on the guest spawn, cgroup, and child-status APIs.

Changes:

• Introduces caller-driven execution with output flow control, stdin, cancellation, and timeouts.
• Moves managed workload execution onto the shared driver.
• Adds execution tests, synchronizes console capture, and updates documentation.

File	Description
nvx-rs/​tests/​managed_lifecycle.rs	Waits for console connection before driving the guest.
nvx-rs/​src/​lib.rs	Documents the execution API.
nvx-rs/​src/​guest/​sys/​signal.rs	Adds SIGPIPE-safe writes and tests.
nvx-rs/​src/​guest/​sys/​process.rs	Adds process killing and returns group-kill errors.
nvx-rs/​src/​guest/​sys/​poll.rs	Supports variable descriptor lists.
nvx-rs/​src/​guest/​sys/​mod.rs	Exports new syscall helpers.
nvx-rs/​src/​guest/​sys/​fd.rs	Adds directory-relative opens and descriptor borrowing.
nvx-rs/​src/​guest/​spawn.rs	Exposes launch-report polling internally.
nvx-rs/​src/​guest/​mod.rs	Exports and documents the execution module.
nvx-rs/​src/​guest/​managed_agent/​tty.rs	Removes the obsolete nonblocking helper.
nvx-rs/​src/​guest/​managed_agent/​mod.rs	Documents the adapter architecture.
nvx-rs/​src/​guest/​managed_agent/​exec.rs	Drives workloads through the shared execution API.
nvx-rs/​src/​guest/​managed_agent/​dispatch.rs	Tests output limits, cancellation, and timeout behavior.
nvx-rs/​src/​guest/​managed_agent/​cgroup.rs	Supplies execution options and forwards observed statuses.
nvx-rs/​src/​guest/​exec/​tests.rs	Tests execution lifecycle, concurrency, and flow control.
nvx-rs/​src/​guest/​exec.rs	Implements the public execution driver.
nvx-rs/​src/​guest/​cgroup.rs	Pins cgroup directories for stable kill targets.
nvx-rs/​README.md	Describes the driver and managed-agent integration.
doc/​project-structure.md	Updates the guest-library overview.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Concurrent signalling, pending SIGPIPE handling, and lost-status cleanup retain correctness issues.

3 open issues

• Numeric-ID signaling can kill a reused process after collection · New
• Lost child status can leave output pipes open and hang execution · New
• Pending signal handling can miss thread-directed SIGPIPE · New

2 resolved since last review

• Close stdin when recording final program status
• Clean up configured cgroup when pidfd opening fails

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Denied termination signals can leave an expired timeout repeatedly emitting failures, starving the caller’s execution loop.

1 open issue

• Deadline remains active after signal denial, causing busy-spin · New

3 resolved since last review

• Lost child status can leave output pipes open and hang execution
• Numeric-ID signaling can kill a reused process after collection
• Pending signal handling can miss thread-directed SIGPIPE

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The signal-handling and process-lifecycle changes require final human review and cross-backend validation.

1 open issue

• Immediate exit races pidfd acquisition, causing flaky launch-failed test · New

1 resolved since last review

• Deadline remains active after signal denial, causing busy-spin

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The process-control API and agent lifecycle rewrite require human review of signalling, status ownership, and cleanup interactions.

No open issues 🎉

1 resolved since last review

• Immediate exit races pidfd acquisition, causing flaky launch-failed test

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Lost-status outcome handling needs correction, and the signalling and lifecycle changes warrant final human review.

1 open issue

• Prevent timeout or cancellation overrides after workload status is lost · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Kernel-sensitive signalling, status ownership, and cgroup cleanup warrant final human review and full backend CI validation.

No open issues 🎉

1 resolved since last review

• Prevent timeout or cancellation overrides after workload status is lost

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Failed-start cleanup can leave a workload running after it changes process groups, and the broader signal and lifecycle changes still warrant human review.

1 open issue

• Signal the process itself when pidfd acquisition fails · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The interacting signal-delivery, status-ownership, and descendant-cleanup paths require final human review despite extensive regression coverage.

No open issues 🎉

1 resolved since last review

• Signal the process itself when pidfd acquisition fails

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The public process-lifecycle, pidfd-signalling, and thread-local SIGPIPE changes warrant final human review despite no additional confirmed blockers.

No open issues 🎉

_{Review effort: Balanced}