[Enrique Saurez (esaurez)]

Part of #619
Refs #535

Step G2 of #619: any guest init or agent can place, account, kill and settle its executions in cgroups that it names, and collect exactly its own children's statuses, without NVX's control protocol, its PID 1, or an async runtime. NVX's managed agent becomes one caller.

What changed

• nvx::guest::cgroup
  • Layout::new(agent, workload): the caller names its own cgroup and the workload cgroup, instead of the fixed nvx-exec. A relative path, a .. component, or a workload cgroup that is or holds the agent's is EINVAL, so ending a workload never ends the agent.
  • ExecutionCgroup: the handle of one execution's cgroup, either the workload cgroup itself (open(&layout, None)), for a caller that runs one execution at a time, or a child cgroup of it (Some(name)).
    • It holds its cgroups' directories open (the execution's, and the workload cgroup's for a child cgroup), and each operation opens its interface files in them.
    • So a handle kept after the caller removed its cgroup and prepared another at the same path reaches nothing: its kill, occupancy check and placements fail with ENOENT, and its remove finds the cgroup gone. The new cgroup is left alone.
    • open therefore needs the cgroup to exist.
    • Clones share the directories, and dropping the last one closes them. It neither kills nor removes the cgroup, which the caller does with kill, settle (with its status collector) and remove, whose failures it sees.
  • prepare(&layout, name, resources) creates the workload cgroup if needed, and an execution's own cgroup exclusively (EEXIST if it exists). It enables pids and each resource's controller with one cgroup.subtree_control write per level, then writes the caller's resource values, such as ("pids.max", "64"), an empty one as a newline, so that the kernel applies or refuses it. A resource file that is not a controller's interface file is refused before anything changes.
  • placement() returns G1's spawn::Cgroup for the held directory, which a spawned child joins before its program runs; place(id) moves a running process.
  • is_occupied(scope) reads pids.current, which counts zombies too; kill(scope) writes cgroup.kill; and settle(scope, timeout, reap) waits until the scope is empty, calling the caller's status collector between its checks. It never collects a status itself.
  • Scope::Execution covers the execution's cgroup and Scope::Workload the whole workload subtree, which is how the direct mode settles today.
  • remove() refuses an occupied cgroup (EBUSY), so that a removal never hides a status that the caller has yet to collect. A prepare that fails once it created an execution's own cgroup removes that cgroup again, so its name can be prepared anew.
    • The caller places no process in a cgroup while it removes it, with place or with a spawned child whose launch has yet to report. The kernel removes a cgroup that only zombies occupy, so a process that joined after the check and ended before the removal would escape it. No in-process lock can cover a child's placement, which the child makes itself after fork, without being held across every spawn, so this is the caller's part of the contract.
  • Removals take turns. A removal names its cgroup by its path, so the process's removals take turns with its preparations of executions' own cgroups, through one process-wide lock.
    • In its turn, a removal checks through its held directory that its cgroup is still there, then removes the cgroup at its path. Cgroup v2 never renames a cgroup, so that path still names it. A cgroup that the caller prepared at the path of a removed one stays, even when clones of one handle remove the cgroup at once.
    • A preparation holds its turn from creating its cgroup until it returns, so its handle holds the cgroup that it created, and a failure removes that cgroup alone.
    • An execution that runs in the workload cgroup itself, as the direct agent's does, takes no turn. The kernel already serializes cgroup creation and removal, so the lock adds no meaningful contention.
• nvx::guest::process: Child gives each child one status collector, the caller.
  • Child::with_pidfd(id) waits with waitid(P_PIDFD) and lends its pidfd for polling. Child::new(id) waits with waitid(P_PID). Neither waits for an arbitrary child.
  • A caller that must collect every status, such as a PID 1 that adopts orphans, keeps its own waitpid(-1) loop and hands each status to its Child with observe(id, status).
  • Statuses are std::process::ExitStatus, real-time signals included. The waits use libc's waitid, because nix rejects real-time signals when it decodes a status.
  • Only an exit or a terminating signal ends a child. A stop doesn't, including one that waitid reports to the caller as the child's tracer, whatever the options, or one that a collector with WUNTRACED hands to observe.
• The managed agent as a caller
  • The direct agent runs in the hierarchy's root and its workloads in nvx-exec. It prepares, kills and settles nvx-exec through an ExecutionCgroup over Scope::Workload, within the same 2 s, with its waitpid(-1) loop as the collector. The exec-cgroup-kill diagnostic stays.
  • The workload's status lives in a process::Child. The agent's loop hands it over with observe, and the launch-failure and termination paths wait for that child alone.
  • Those waits no longer drop a failure. They report it as the new exec-wait diagnostic. Once the agent has ended the workload for a cancellation, a timeout, the output cap or a lost session, the outcome is that cause, so such a failure ends the loop, which used to wait for the status forever.
  • The sandbox agent places each launcher in container through the same handles, and still ends only the workload's process group.
  • Admission, mode-specific cleanup, diagnostics and the agent's test fixtures are unchanged. Its cgroup, dispatch and loopback tests pass as they are.
• Internals: sys::open_at, and spawn::Cgroup::open_in, crate-private, which opens cgroup.procs in a held directory.
• Docs: the crate README's module table, the guest module docs, and doc/project-structure.md describe the two libraries.

Behavior

Two edge cases change in the agent:

• A traced workload's stop (direct agent): a workload that makes the agent its tracer, with PTRACE_TRACEME, and then stops, was reported as ended with status 125 (failed) as soon as it stopped. Now its stop doesn't end it: like any other stopped workload, it runs until it exits, its timeout, a cancellation or a lost session ends it.
• A lost status: if the agent can't collect the status of a workload that it ended for a cause, as when the kernel collects it because SIGCHLD is ignored, the request now ends with that cause and the exec-wait diagnostic. It used to stay unanswered.

Everything else that the agent reports is unchanged.

Tests

• guest::process: named waits, by process ID and by pidfd, for an exit, SIGTERM and real-time signal 40, leave an unrelated child's status in place. Also pidfd readiness, observe, which ignores stops and continuations, ECHILD through a pidfd after another collector took the status, IDs that no process can have, and a traced child whose stop ends neither a non-blocking nor a blocking wait.
• guest::cgroup, on fake hierarchies: layouts and names, and open of a missing cgroup (ENOENT) or a file (ENOTDIR); combined controller enabling; refused resources that change nothing; exclusive named executions; a failed preparation that leaves no cgroup behind; an empty resource value that still reaches its file; scoped occupancy and kill; settling (collector calls, a failing collector, ETIMEDOUT, EPROTO); placement; removal; and cgroup_operations_leave_an_unrelated_child_s_status.
  • a_handle_reaches_its_own_cgroups_alone_once_another_takes_a_path: once its cgroup is replaced at the same path, an old handle and its clone fail to kill, check, place and make placements, and their removal succeeds. The new cgroup's files stay untouched, and the old handle's workload scope still works.
• guest::cgroup, on a real cgroup v2 hierarchy, as root where the hierarchy lets the test create a cgroup (it returns early otherwise): in descendants_that_hold_pipes_are_found_and_settled, a descendant that leaves the session with setsid and holds the execution's output pipe is found, killed and settled while a sibling execution keeps running; settling the workload scope then ends the sibling too. A pids.max value that the kernel refuses, an empty one included, leaves no cgroup behind. Once the first execution's cgroup is removed and another is prepared at its path, its old handle's kill and occupancy check fail with ENOENT, and its removal leaves the new cgroup.
  • a_removal_leaves_a_cgroup_prepared_at_its_path_by_a_concurrent_one pauses one clone's removal after its check, while another clone removes the cgroup and prepares another at its path. a_failed_preparation_removes_its_own_cgroup_alone pauses a preparation once it created its cgroup, while another handle opens that cgroup, removes it and prepares another at its path, and then fails the preparation with a value that the kernel refuses. In both, the cgroup prepared at the path stays. Each racer's setup, such as that open, takes place within the pause, which waits for it however late the racer's thread starts.
• managed_agent::dispatch: a_workload_status_that_the_agent_cannot_collect_still_ends_its_request ignores SIGCHLD in a process of its own, so that the kernel collects the workload's status. A workload with a 100 ms timeout still ends with EXIT 124 timeout and the exec-wait diagnostic, where the agent used to hang.
• tests/managed_lifecycle.rs: a workload whose detached setsid sleep 30 descendant holds its output ends at once with its own status, and the next workload runs.
• Mutation checks: each of 10 mutations makes one of these tests fail. They kill, check occupancy, place or make placements through the cgroup's path; let remove miss a removed cgroup; let a removal or a preparation skip its turn, or a removal take its turn only after its check; keep the agent's loop waiting for a lost status; or drop its diagnostic.

Sizes

I built the guest binaries and the Alpine initramfs natively on Linux, the same way at this PR's base, dev at b47a9f8, and with this change. The binaries use the guest profile, which strips them.

Artifact	Before	After	Change
nvx-managed-agent	471,520 B	483,808 B	+12,288 B
initramfs.cpio.gz (Alpine)	8,199,693 B	8,204,375 B	+4,682 B

The other guest binaries keep their sizes.

Adapter contract

Yes, additively: the guest configuration gains two public modules, nvx::guest::cgroup and nvx::guest::process. No existing item, persisted format or wire format changes. The one behavior change, for a direct workload that makes the agent its tracer and stops, is described above, and needs no migration. The runtime adapter builds unchanged in every configuration (C0). The public-API check (#630) isn't on dev yet: if it lands first, this PR adds the snapshot update.

Validation

• WHP, Windows 11: python scripts\nvx.py test-microvm --backend whp, the default suite, then --scenario managed-exec-config --scenario sandbox-lifecycle, with the Alpine initramfs built from this change and the pinned OpenVMM and kernel. Every scenario passes, including managed-lifecycle with its new case.
• Windows 11 and Linux (Ubuntu, x86-64), in nvx-rs with NVX_BUNDLE=skip:
  • cargo fmt --all --check
  • cargo clippy --locked --all-targets --all-features -- -D warnings
  • cargo clippy --locked --lib --no-default-features -- -D warnings
  • cargo test --locked --all-features, then cargo test --locked
  • the synthetic-release bundled test
  • cargo doc --locked --no-deps --all-features, with RUSTDOCFLAGS=-D warnings
  • cargo +1.89 check --locked --all-targets --all-features
• Linux only:
  • cargo check --locked --all-targets --all-features --target aarch64-apple-darwin
  • python3 scripts/nvx.py check-guest-binaries --toolchain 1.95.0
  • build-guest-binaries, then test-guest-binaries
  • the nvx-rs tests both as root, which runs the real-cgroup test, and unprivileged, as CI runs them
• The runtime adapter's configurations, as check-nvx-rs runs them: protocol-only and host-only on both OSes, and guest-only on Linux.
• aci_edge_sandboxes, on both OSes: its fmt, Clippy, test, doc and MSRV steps, and test-agent-bundling. On Linux, its macOS check too.
• python scripts/nvx.py verify and python scripts/check_markdown_links.py.

Everything passes except one environmental failure on Linux as root: the CLI golden test's caller-owned-share scenario refuses its temporary share because UID 0 owns it, as --mount-owner caller requires. The base fails the same way there, and the unprivileged run passes.

KVM, nested on the same Windows 11 machine: this machine's nested KVM can't boot the built-in CPU profile that --cpu-profile auto selects (E_PROFILE_UNSUPPORTED). With a local wrapper that starts OpenVMM with --cpu-profile host instead, test-microvm --backend kvm --scenario managed-lifecycle --scenario managed-exec-config --scenario sandbox-lifecycle passes, which covers the direct and sandbox agents. I didn't run the rest of the KVM suite, which depends on the built-in profile, or MSHV. CI runs the KVM, MSHV and WHP matrix once this targets dev.

[Copilot]

🟡 Changes recommended

Child-status handling, launch-failure reporting, and privileged-test reliability need correction.

4 open issues

• Skip hierarchy tests when the cgroup mount is not writable · New
• Synchronize on the second child's successful launch · New
• Ignore nonterminal wait statuses when caching child state · New
• Prevent SIGPIPE from masking launch failure status · New

What changed in this PR

Advances the guest-runtime library extraction in #619 with caller-owned cgroups and child-status collection, building on #629’s spawn API.

Changes:

• Adds public cgroup and named-child status APIs.
• Moves managed-agent launch and cleanup onto these libraries.
• Adds lifecycle tests and documents the public interfaces.

File	Description
nvx-rs/​tests/​managed_lifecycle.rs	Tests detached-descendant cleanup.
nvx-rs/​src/​lib.rs	Documents guest libraries.
nvx-rs/​src/​guest/​sys/​spawn.rs	Implements prepared fork/exec operations.
nvx-rs/​src/​guest/​sys/​process.rs	Adds named waits and pidfd support.
nvx-rs/​src/​guest/​sys/​mod.rs	Exposes internal syscall helpers.
nvx-rs/​src/​guest/​sys/​fd.rs	Supports descriptor ownership transfer.
nvx-rs/​src/​guest/​spawn.rs	Provides the public spawn API.
nvx-rs/​src/​guest/​process.rs	Adds caller-owned child-status handles.
nvx-rs/​src/​guest/​mod.rs	Exports guest libraries.
nvx-rs/​src/​guest/​managed_agent/​seam.rs	Adapts launch seams and test helpers.
nvx-rs/​src/​guest/​managed_agent/​mod.rs	Documents library integration.
nvx-rs/​src/​guest/​managed_agent/​loopback.rs	Exercises real launch operations.
nvx-rs/​src/​guest/​managed_agent/​launch.rs	Builds workload launch commands.
nvx-rs/​src/​guest/​managed_agent/​exec.rs	Uses child handles and launch reports.
nvx-rs/​src/​guest/​managed_agent/​dispatch.rs	Adds launch-failure and directory tests.
nvx-rs/​src/​guest/​managed_agent/​cgroup.rs	Adapts cleanup to public cgroup handles.
nvx-rs/​src/​guest/​cgroup.rs	Adds cgroup operations and tests.
nvx-rs/​README.md	Documents the new APIs.
doc/​project-structure.md	Updates the guest-library description.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Recursive process termination and child-status ownership warrant final human review and backend CI confirmation.

No open issues 🎉

4 resolved since last review

• Prevent SIGPIPE from masking launch failure status
• Ignore nonterminal wait statuses when caching child state
• Synchronize on the second child's successful launch
• Skip hierarchy tests when the cgroup mount is not writable

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Kernel-dependent process collection and cgroup termination changes need final human review and completed cross-backend CI validation.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Kernel-facing process collection and cgroup lifecycle changes warrant final human review and backend CI confirmation.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Public process-control and cleanup guarantees need human review, and empty resource-value handling remains unresolved.

1 open issue

• Empty resource values are silently ignored · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Privileged cgroup teardown and PID 1 status collection warrant human review of lifecycle behavior across backends.

No open issues 🎉

1 resolved since last review

• Empty resource values are silently ignored

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Public containment and child-status ownership contracts need human review alongside confirmation of the remaining backend validation.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Concurrent removal through cloned handles can delete a replacement cgroup at a reused path.

1 open issue

• Serialize cgroup removal to prevent deleting a replacement · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Concurrency-sensitive cgroup lifecycle and kernel child-status handling warrant final human review alongside backend CI results.

No open issues 🎉

1 resolved since last review

• Serialize cgroup removal to prevent deleting a replacement

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Kernel-dependent cgroup lifetime, concurrent cleanup, and child-status semantics warrant final human review.

No open issues 🎉

_{Review effort: Balanced}