nvx-rs: add the VMM supervisor with a spawn allowlist, stop, and reattach

nvx-rs/Cargo.toml

@@ -99,7 +99,7 @@ sha2 = { version = "0.10", optional = true }
 serde_json = "1.0.120"
 tempfile = "3.10"
 
-# The job objects and process security descriptors of the process supervision tests.
+# The job objects, process security descriptors, and handles of the process supervision tests.
 [target.'cfg(windows)'.dev-dependencies]
 windows-sys = { version = "0.61", features = [
     "Win32_Foundation",

nvx-rs/README.md

@@ -61,7 +61,8 @@ nvx-rs/
 | `nvx::host::store` | `SandboxStore`: the runtime records of one sandbox in a private directory with one lock, which every host that supervises the sandbox shares. The `RuntimeRecord` names the running VM: the `ProcessRef` of its VMM, its control endpoint, and the `RuntimeIdentity` that it was launched with, the identities of its OpenVMM, kernel, and initramfs from install or staging metadata and its boot and control contract versions, which decide whether another build of the host may reattach to it. The `LaunchJournal` of a launch holds a marker written before the spawn, the VMM's `ProcessRef` right after it, and an inherited claim on the VMM's log, which proves that a launch that never published its endpoint has exited, so that recovery ends an interrupted launch and signals only a VMM that it holds through a verified `Process`. Records are JSON in one versioned format, and records in any other format fail rather than being migrated |
 | `nvx::host::instance` | `Instance`: a VM in a directory of its own. It starts OpenVMM with a fresh capability and waits until the guest is ready and has mounted the mapped paths, reconciles recorded state, recovers interrupted launches, runs workloads and pumps their output, and stops the VM with a forced fallback |
 | `nvx::host::nvxs` | `HostConnection`: the host's role of the NVXS session over a platform transport, on which `nvx::host::session` and other host applications run their protocols. It attaches with the launch capability, waiting while the broker asks it to, sends application bytes in as many DATA records as they take, and returns each DATA payload from the guest, with deadlines; a receive that times out keeps a partial record |
-| `nvx::host::process` | The detached launch of OpenVMM, and verified process references. A `ProcessRef` names one process by its ID, its start time, and the `BootId` of the host boot during which it started. `Process::open` checks a reference and holds its process through a pidfd on Linux or a process handle on Windows, whether or not the caller started it, so that waits sleep until the process exits and termination never reaches a process that reuses its ID. A reference from an earlier boot names an exited process, and a process whose identity this user cannot check, such as another user's, makes `Process::open` fail rather than report an exit. Linux hosts without pidfds poll the reference instead, and cannot terminate the process |
+| `nvx::host::process` | Verified process references, and the launch of OpenVMM for the CLI and `nvx::host::instance`, through `nvx::host::supervisor`. A `ProcessRef` names one process by its ID, its start time, and the `BootId` of the host boot during which it started. `Process::open` checks a reference and holds its process through a pidfd on Linux or a process handle on Windows, whether or not the caller started it, so that waits sleep until the process exits and termination never reaches a process that reuses its ID. A reference from an earlier boot names an exited process, and a process whose identity this user cannot check, such as another user's, makes `Process::open` fail rather than report an exit. Linux hosts without pidfds poll the reference instead, and cannot terminate the process |
+| `nvx::host::supervisor` | The supervisor of a sandbox's VMM, beneath the CLI, `Instance`, and external orchestrators, which keep the configuration, the authorization, and the application protocol. `launch` runs the store's launch steps under the sandbox's lock: it writes the journal's marker, spawns OpenVMM with the claimed log, records its `ProcessRef` right after, and returns a `Starting` launch that the caller commits once the VMM serves its control endpoint, or abandons. A launch that fails never leaves an unrecorded VMM running. `VmmCommand::inherit` lists the descriptors (Linux) or handles (Windows) that the VMM inherits under their numbers, besides its standard streams, and it inherits nothing else of the caller's: on Linux, they must be close-on-exec, a flag that only the VMM's own process clears; on Windows, they join the VMM's explicit handle list. `Vmm::stop` runs one graceful step of the caller's and waits for the VMM's exit event until a deadline, and takes a bounded forced step through the VMM's verified identity only on request. `reattach` recovers an interrupted launch, and holds the recorded VMM again after the orchestrator restarts within the same host boot: `Live`, `Mismatched` with another runtime identity, `Exited`, or `Indeterminate` |
 | `nvx::host::session`, `platform`, `report` | The authenticated control session, which carries NVX's control protocol, `NVXC`, one frame per DATA record of a `HostConnection`; Linux and Windows primitives; and OpenVMM's outcome report |
 | `nvx::host::sandbox` | The launch profile of `nvx.py sandbox`, ported from the one that `scripts/nvx_tools/sandbox.py` held: `SandboxLaunch`, with the EROFS distro `SandboxLayer`, the one layer that a sandbox takes, the ext4 scratch disk, the entrypoint, arguments, hostname, workload identity, and resource limits, and its `--microvm-sandbox-block` and `--microvm-workload-identity` arguments and kernel command line, within the 1024-byte budget that OpenVMM leaves it. `mount`: `SandboxMount` live shares of host directories and files, their guest targets and the reserved ones, their access policies and owner, the aggregate of several shares, and the checks of their host paths. `options`: the checks that `nvx.py sandbox` and `nvx.py run` apply to their options: the attribution of `--mount-deny`, `--mount-allow`, and `--mount-write` to shares, the systemd policy, the targets of `nvx.py run` shares, and the managed exec environment; see [Python parity](#python-parity) |
 | `nvx::testing::fake_openvmm` | A fake OpenVMM that serves the control console and runs scripted workloads without a VM, for direct launches and for the managed and one-shot sandboxes of `nvx-rs sandbox` and `nvx.py sandbox` |
@@ -343,7 +344,7 @@ in sandbox mode.
   code page unless `PYTHONIOENCODING` or UTF-8 mode says otherwise.
 - `start` refuses a control endpoint that another process serves, which it
   treats as one that has not appeared yet, and, on Windows, starts OpenVMM
-  without a console window, as `nvx::host::process` starts it.
+  without a console window, as `nvx::host::supervisor` starts it.
 - `exec` refuses more than 1 MiB of workload output, which the guest agent
   never forwards.
 
@@ -538,6 +539,11 @@ broker that relays records as OpenVMM's does, with messages of an application
 framing of its own that span records. The store test crashes orchestrator
 processes at each step of a launch, recovers the VMM stand-ins that they leave
 behind, and loads records that another build wrote, in `tests/fixtures/store/`.
+The process-supervision test crashes orchestrator processes at each step of a
+launch through the supervisor, reattaches to the VMM stand-ins that they leave
+behind and stops them, and checks that a VMM inherits only its standard streams
+and the descriptors or handles that it is allowed while another thread spawns
+processes.
 On Linux, a loopback test drives
 the host `ControlSession` through a broker shim into the real managed agent's
 session, dispatch, and execution engine, and `nvx.py` also lints and tests the

nvx-rs/src/host/instance.rs

@@ -965,6 +965,7 @@ mod tests {
     use std::thread;
 
     use super::*;
+    use crate::host::test_support::released;
 
     /// A process ID that fits every platform's type and that no process has.
     const GONE_PID: u32 = 0x7fff_fff0;
@@ -1137,7 +1138,11 @@ mod tests {
         let contender = OpenOptions::new().write(true).open(&path).unwrap();
         assert!(contender.try_lock().is_err());
         drop(guard);
-        contender.try_lock().unwrap();
+        assert!(released(|| match contender.try_lock() {
+            Ok(()) => true,
+            Err(std::fs::TryLockError::WouldBlock) => false,
+            Err(error) => panic!("{error}"),
+        }));
         drop(contender);
         // A waiter takes the lock once the holder releases it.
         let guard = lock(&path).unwrap();

nvx-rs/src/host/mod.rs

@@ -14,14 +14,17 @@
 //! - [`store`] keeps the runtime records of a sandbox in a private directory with one lock: the
 //!   runtime record of its VM, with the VM's runtime identity, and the journal of its launch,
 //!   through which an interrupted launch is recovered without signaling an unverified process.
+//! - [`supervisor`] launches a sandbox's VMM through the store's journal, passing it only the
+//!   descriptors or handles that the caller allows, stops it through the caller's graceful step,
+//!   with a bounded forced step on request, and reattaches to it after the orchestrator restarts
+//!   within the same host boot.
 //! - [`session`] speaks NVX's control protocol, `NVXC`, over [`nvxs`], which runs the host's role
 //!   of the NVXS session for it and for any other application protocol, over a platform
 //!   transport. [`platform`] holds the primitives of Linux and Windows hosts. Other hosts compile
 //!   but report NVX as unavailable.
-//! - [`process`] launches OpenVMM processes. Its verified process references name a process by
-//!   its ID, start time, and host boot, and hold it through a pidfd or a process handle, so that
-//!   waits sleep until it exits and termination reaches only it, whether or not the caller
-//!   started it.
+//! - [`process`] names processes by verified references: their ID, start time, and host boot,
+//!   through which a pidfd or a process handle holds them, so that waits sleep until a process
+//!   exits and termination reaches only it, whether or not the caller started it.
 //! - [`sandbox`] is the launch profile of `nvx.py sandbox`: a container over read-only EROFS
 //!   layers and a private ext4 scratch disk, with live shares, and the checks of its options.
 //! - `cli` (feature `cli`) is the `nvx-rs` command-line interface, which `nvx.py` runs for the
@@ -51,6 +54,9 @@ pub mod report;
 pub mod sandbox;
 pub mod session;
 pub mod store;
+pub mod supervisor;
 #[cfg(unix)]
 #[allow(unsafe_code)]
 mod sys;
+#[cfg(test)]
+mod test_support;

nvx-rs/src/host/platform/linux.rs

@@ -1,6 +1,6 @@
 use std::fs::{self, DirBuilder, File, OpenOptions, Permissions, TryLockError};
 use std::io::{self, Read};
-use std::os::fd::AsRawFd;
+use std::os::fd::{AsFd, AsRawFd, RawFd};
 use std::os::unix::fs::{DirBuilderExt, MetadataExt, OpenOptionsExt, PermissionsExt};
 use std::os::unix::net::UnixStream;
 use std::path::Path;
@@ -9,6 +9,7 @@ use std::thread;
 use std::time::{Duration, Instant};
 
 use super::Transport;
+use crate::host::supervisor::Inherited;
 use crate::host::sys::{self, Errno, PidFd};
 use crate::hypervisor::Hypervisor;
 
@@ -240,10 +241,47 @@ pub(crate) fn boot_id() -> io::Result<[u8; 16]> {
     })
 }
 
-/// Starts the command in a new session so it outlives the caller and its terminal, and with no
-/// descriptor of the caller beyond the standard streams that the command sets up.
-pub(crate) fn detach(command: &mut Command, _breakaway_from_job: bool) {
-    sys::detach(command);
+/// Starts the command in a new session so it outlives the caller and its terminal, with the
+/// descriptors `inherit` under their numbers in the caller, and no other descriptor of the caller
+/// beyond the standard streams that the command sets up.
+///
+/// Only the new process clears the close-on-exec flag of the descriptors in `inherit`, so no
+/// process that another thread spawns meanwhile inherits them. Fails with
+/// [`io::ErrorKind::InvalidInput`] for a descriptor of a standard stream, 0 to 2, which the
+/// command's own standard streams replace in the new process, and for a descriptor that is not
+/// close-on-exec in the caller, which any process that the caller spawns would inherit.
+pub(crate) fn detach(command: &mut Command, inherit: &[Inherited<'_>]) -> io::Result<()> {
+    use nix::fcntl::{FcntlArg, FdFlag, fcntl};
+
+    let mut descriptors: Vec<RawFd> = Vec::with_capacity(inherit.len());
+    for inherited in inherit {
+        let descriptor = inherited.as_fd().as_raw_fd();
+        if descriptor <= sys::LAST_STANDARD_STREAM {
+            return Err(io::Error::new(
+                io::ErrorKind::InvalidInput,
+                format!(
+                    "descriptor {descriptor} cannot be inherited: the VMM's own standard streams \
+                     replace descriptors 0 to {}",
+                    sys::LAST_STANDARD_STREAM
+                ),
+            ));
+        }
+        let flags = FdFlag::from_bits_retain(fcntl(inherited.as_fd(), FcntlArg::F_GETFD)?);
+        if !flags.contains(FdFlag::FD_CLOEXEC) {
+            return Err(io::Error::new(
+                io::ErrorKind::InvalidInput,
+                format!(
+                    "descriptor {descriptor} cannot be inherited by the VMM alone: it is not \
+                     close-on-exec, so any process that this process spawns inherits it"
+                ),
+            ));
+        }
+        if !descriptors.contains(&descriptor) {
+            descriptors.push(descriptor);
+        }
+    }
+    sys::detach(command, descriptors);
+    Ok(())
 }
 
 /// Checks that the hypervisor device is present and accessible.

nvx-rs/src/host/platform/other.rs

@@ -5,6 +5,7 @@ use std::process::Command;
 use std::time::{Duration, Instant};
 
 use super::Transport;
+use crate::host::supervisor::Inherited;
 use crate::hypervisor::Hypervisor;
 
 #[cfg(all(feature = "cli", unix))]
@@ -105,7 +106,15 @@ pub(crate) fn private_file_problem(_file: &File) -> io::Result<Option<String>> {
     Err(unsupported())
 }
 
-pub(crate) fn detach(_command: &mut Command, _breakaway_from_job: bool) {}
+/// Leaves the command as it is, unless it should inherit descriptors: this host cannot restrict
+/// what the command inherits, and fails with [`io::ErrorKind::Unsupported`] instead.
+pub(crate) fn detach(_command: &mut Command, inherit: &[Inherited<'_>]) -> io::Result<()> {
+    if inherit.is_empty() {
+        Ok(())
+    } else {
+        Err(unsupported())
+    }
+}
 
 pub(crate) fn probe_hypervisor(_hypervisor: Hypervisor) -> Result<(), String> {
     Err(unsupported().to_string())