Repository navigation
nvx-rs: enter guest executions with namespaces, a view, and an identity - #777
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Default ACLs can still prevent non-root mapping access, and single-component relative runtime sources fail to bind.
2 open issues
What changed in this PR
Adds reusable guest execution entry to nvx-rs, advancing #621’s layered-root support. Guest agents can launch programs with isolated namespaces, a restricted filesystem view, and a typed identity; the runtime adapter exercises these root profiles.
Changes:
- Adds execution-entry APIs and launch stages, including session creation and cancellable barriers.
- Separates private time records from public status and updates mapping creation permissions.
- Adds root-profile probes, VM assertions, and supporting documentation.
| File | Description |
|---|---|
| tests/runtime-adapter/tests/vm_layered.rs | Adds root-profile VM assertions. |
| tests/runtime-adapter/src/worker/entries.rs | Implements execution-entry probes. |
| tests/runtime-adapter/src/worker.rs | Dispatches entry probes. |
| tests/runtime-adapter/src/lib.rs | Defines helper command names. |
| tests/runtime-adapter/src/bin/runtime-adapter-guest.rs | Handles entry helper commands. |
| tests/runtime-adapter/README.md | Documents probes and VM coverage. |
| tests/runtime-adapter/Cargo.toml | Enables required syscall features. |
| nvx-rs/src/lib.rs | Documents the entry library. |
| nvx-rs/src/guest/time/system.rs | Moves private records outside public status. |
| nvx-rs/src/guest/sys/spawn.rs | Integrates entry stages and barriers. |
| nvx-rs/src/guest/sys/rooted.rs | Adds umask-independent creation helpers. |
| nvx-rs/src/guest/sys/mod.rs | Exposes new internal syscall helpers. |
| nvx-rs/src/guest/sys/fd.rs | Adds permission and socket-pair helpers. |
| nvx-rs/src/guest/sys/entry.rs | Implements namespace, mount, and identity operations. |
| nvx-rs/src/guest/spawn.rs | Exposes sessions, barriers, and execution entry. |
| nvx-rs/src/guest/root/tests.rs | Checks share-target creation permissions. |
| nvx-rs/src/guest/root/kernel.rs | Updates fake-kernel creation operations. |
| nvx-rs/src/guest/root.rs | Documents target permission behavior. |
| nvx-rs/src/guest/mod.rs | Exports the entry module. |
| nvx-rs/src/guest/mappings/tests.rs | Checks mapping permissions under restrictive umasks. |
| nvx-rs/src/guest/mappings/privileged.rs | Adds non-root mapping-access coverage. |
| nvx-rs/src/guest/mappings/kernel.rs | Adds a dedicated file-creation operation. |
| nvx-rs/src/guest/mappings.rs | Uses updated creation helpers. |
| nvx-rs/src/guest/managed_agent/mappings.rs | Tests mapping permissions under a restrictive umask. |
| nvx-rs/src/guest/managed_agent/launch.rs | Documents revised cgroup placement order. |
| nvx-rs/src/guest/entry.rs | Adds typed entry configuration and runtime binds. |
| nvx-rs/README.md | Documents entry and expanded spawn APIs. |
Review effort: Balanced
3711ac7 to
a48857c
Compare
db5ceba to
5dccd78
Compare
5dccd78 to
298c828
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Raw namespace creation, privileged mount operations, descriptor handling, and credential transitions establish a security boundary that warrants final human review.
No open issues 🎉
2 resolved since last review
Review effort: Balanced
|
A final automated review of 298c828 found two defects, which three new commits fix:
Both heads were validated again on Linux/MSHV and Windows/WHP, as the description lists; #778 is rebased onto 3b9d9af with its commit unchanged. |
|
A second final automated review, of 3b9d9af, found one more defect (medium), which 9049f18 fixes: An entry's child could panic on a denied Both heads were validated again on Linux/MSHV and Windows/WHP; #778 is rebased onto 9049f18, with its commit changed only in one re-export's context line. |
A runtime may bind the time-status directory, /run/nvx/time, read-only into a workload's root. Its lock files and private records are owned by user ID 0, so a workload of user ID 0 could open them through that bind even without capabilities, and hold the exclusive lock of state updates or of the suppression record. That would block the time daemon's discipline and violation reports, and a restore's repair. Keep the state and suppression locks, the suppression record, and the restore record in /run/nvx, beside the port-window lock, so that the status directory holds only what any process may read: the published state, the daemon's process identifier, and their temporary files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
nvx::guest::entry lets any guest init or agent have a program that it spawns enter an execution, from typed inputs: a root, which is a mount, an identity, a hostname, a machine ID mode, a publication state, a time status source, and the device nodes of /dev. The child starts with clone3 in new mount, PID, and UTS namespaces, so that the program is PID 1 of its namespace with no process between it and its caller. It makes its mounts private, sets the hostname, and moves the root over /, never entering it by chroot alone, after it makes the root's mounts nosuid and nodev. It mounts a private procfs, a read-only sysfs, and a /dev tmpfs with only the allowed nodes, its own devpts, and an shm, so that every other mount is nodev. For user ID 0, the kernel's risky procfs entries and /sys/firmware are read-only or masked. It then takes its identity: the supplementary groups, the group and user IDs, and only the capabilities of an allowlist of 13, which only user ID 0 may hold, and of which a workload with published listeners holds neither the binding nor the raw-socket one, applied to every set, with no_new_privs. Every other descriptor of its caller closes as its program runs, and it leads a new session without a controlling terminal. In existing-account mode, the entry reads the identity's passwd and group entries below the root's descriptor, with bounds, and requires one passwd entry with the requested primary group, an existing group, and an absolute home; numeric mode takes the IDs as given. Nothing creates an account. A generated machine ID is one random ID that the caller writes once for the boot, readable by every execution whatever the umask, and the image's own is left alone. Entry::bind_runtime binds a generated ID over /etc/machine-id, and the time status at /run/nvx/time, read-only into the root once, through the mappings module's single bind, in the caller's mount namespace, of which each execution's starts as a copy, so that every later execution sees them; a symbolic link at /etc/machine-id gives way to an empty file first. Each step is a launch stage of nvx::guest::spawn::Command, whose child still makes only system calls over storage prepared before it starts, and reports the stage that failed. A command can now also lead a new session, and hold its child at a barrier until the caller has placed it. The barrier is a connected socket pair, so that dropping it cancels the launch at once by shutting down the caller's end, even while other children that have not run their programs yet hold copies of it. The child now joins its cgroup before its other stages, as an entry must while it still holds the privilege to. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
The adapter's worker gains an entries probe, which enters its assembled root through nvx::guest::entry and nvx::guest::spawn, as an adapter's root profiles do. The worker copies its own executable into the root and runs the executions in a helper that enters a mount namespace of its own first, as NVX's managed agent does, so that the runtime's binds of a generated machine ID and of the time status end with it before the worker tears the root down. Each execution joins the workload cgroup, receives a descriptor of the guest's own root that its caller leaks without close-on-exec, and runs the staged executable, which reports from inside the root what the execution sees and may do. The root_profiles_enter_the_root scenario checks that user ID 0 with each allowed capability alone holds only that capability, and that only its own attempt succeeds, that root with none and a numeric identity without an account may do none of it, and that the entry refuses the capabilities outside its allowlist, the binding and raw-socket ones while listeners are published, capabilities for another user ID, and an identity without an account in existing-account mode. Every execution is PID 1 of its namespaces with only its standard streams, cannot write /proc/sys or lower the unprivileged port start, and stays in its root after a nested chroot escape. With published listeners, which both publication modes reach the entry as, a root workload can bind no privileged source port and open no raw socket, so it cannot forge a listener announcement. The executions see the root's own machine ID, then a generated one and the time status that the guest's time daemon publishes, read-only, in which a root workload finds no lock of the daemon's, and which leave no mount in the worker's namespace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
A runtime file that a relative path of one component names, such as a machine ID that MachineId::generate made at "machine-id", has an empty parent, which Entry::bind_runtime opened as the source root of its bind, and failed with ENOENT. The source root of such a file is now the current directory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
A child that waits at its barrier holds a copy of every descriptor of the test process until it runs its program, such as the end of another test's pipe, which then stays open for as long. Two tests held a child there for tens of milliseconds while other tests ran. They now run in processes of their own, as the test of an ended child's barrier does. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
A child that waits at its barrier was forked with copies of every descriptor of its caller, close-on-exec ones included, and kept them until it ran its program, which it does only once released. Among them are the launch report's writer and the output pipes of any child that the caller started or prepared meanwhile, so an execution that waits for its pipes to hang up could not end while another child waited. Before it waits, the child now closes every descriptor that it does not use, with close_range over the gaps between the sorted numbers that PreparedChild::new records: its stream sources, root, cgroup file, barrier reader, report writer, and inherited descriptors. Its copy of the barrier's writer is among them, which an explicit close dropped before. So the program of a child with a barrier no longer receives descriptors that its caller opened without close-on-exec, which an entry's program never received either. The barrier tests no longer need processes of their own, and a new test checks that another child's pipes hang up while a child waits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
An entry's child only made its other descriptors close on exec, so the descriptors that its caller leaked and those that its view's stages opened, such as the O_PATH ones below procfs, stayed open until its program ran. Its working directory, which chdir resolved before then, could name one of them through /proc/self/fd, or a working directory or root through another magic link, and so place the program outside its root, or in the writable /proc/sys below its read-only bind, where the program stayed after exec. The child now closes every descriptor that it does not use when it places its descriptors, rather than marking it, and, once its stages have run, every descriptor but its program's and its launch report's. It then opens its working directory, / unless its command names another, with openat2 and RESOLVE_NO_MAGICLINKS, and enters it with fchdir, so that neither a magic link nor a symbolic link to one resolves. Either failure is the working-directory stage's. The privileged entry test checks that / and an ordinary directory are entered, and that the links of a leaked descriptor, of each descriptor up to 63, of the child's working directory and root, and of the working directory of its namespace's first process, and a symbolic link to one of them, each fail the stage before the program runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
The adapter's entries probe now reports each execution's working directory, the top of the root unless the execution names another, and runs working-directory cases with the root profile of every allowed capability: its own case directory, which it enters, and, through magic links, the directory of the descriptor that its caller leaked, its own working directory and root, the working directory of its namespace's first process, a symbolic link to its working directory, and the link of each descriptor up to 63, whose distinct outcomes it reports. The layered VM scenario checks that every execution without a working directory runs at the top of its root, that the case directory is entered, and that each magic link fails the working-directory stage before the program runs: with ENOENT for the leaked descriptor, which the child closed, and ELOOP otherwise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
An entry's child cleared its umask before it mounted its view, and restored it after, with nix's umask, which panics when the system call fails, as a seccomp policy can make it. The child that clone3 starts must not panic: the panic would neither report the failed stage nor end the child cleanly, and could unwind into its caller's code. The child now makes the system call itself. A failure to clear the umask is the proc stage's, and one to restore it the devices stage's, so the launch reports it and runs nothing. A test denies umask with a seccomp filter, in a process of its own, and checks that a simulated entry then fails its proc stage with EPERM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e06da46b-cc03-40f6-84e4-199397657333
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
9049f18 to
d2bbbc4
Compare

Part of #621 (A3.4).
Adds
nvx::guest::entry, through which any guest init or agent has a program that it spawns enter an execution, and runs root profiles of it in the runtime adapter's layered VMs, each with its negative tests.The entry. An
Entrytakes typed inputs: the root, which is a mount, an identity, a hostname, a machine ID mode, the publication state, the time status's source, and the device nodes of/dev. Its steps are new launch stages ofnvx::guest::spawn::Command, whoseLaunchStagestays non-exhaustive, so the child still makes only system calls over storage prepared before it starts, and reports the stage that failed:clone3into new mount, PID, and UTS namespaces, so that the program is PID 1 of its namespace with no process between it and its caller; recursively private mounts, and the hostname;nosuidandnodev, then the root moved over/, never entered bychrootalone; a private procfs, a read-only sysfs, and a/devtmpfs with only the allowed nodes, its own devpts, and an shm, so that every other mount isnodev; for user ID 0, the kernel's risky procfs entries and/sys/firmwareread-only or masked;CAP_NET_BIND_SERVICEnorCAP_NET_RAW, applied to every set, withno_new_privs;/unless the command names another, entered through no magic link, so that neither a/proclink to a descriptor, a working directory, or a root, nor a symbolic link to one, places the program outside its root.In existing-account mode, the entry reads the identity's passwd and group entries beneath the root's descriptor, with bounds, and requires one passwd entry with the requested primary group, an existing group, and an absolute home; numeric mode takes the IDs as given. Nothing creates an account. A generated machine ID is one random ID per boot, readable by every execution whatever the umask; image mode leaves the image's own.
Entry::bind_runtimebinds a generated ID over/etc/machine-id, and the time status at/run/nvx/time, read-only, once, through the mappings module's single bind.A command can now also lead a new session, and hold its child at a barrier until the caller has placed it. The barrier is a connected socket pair whose drop shuts it down, so that dropping it cancels the launch at once, even while another child that has not run its program yet holds a copy of it. A child that waits at its barrier first closes every descriptor that it doesn't use, so that it holds no copy of another child's pipes or launch report while it waits. The child now joins its cgroup before its other stages, as an entry must while it still holds the privilege to.
Time status. The guest's time tools now keep their locks and private records in
/run/nvxrather than in/run/nvx/time, so that the status directory that the entry binds holds only what any process may read: an execution of user ID 0 could otherwise hold the time daemon's locks.Root profiles. The runtime adapter's guest agent enters its layered root with root profiles and checks, in its VMs: each allowlisted capability, a refused one, and user ID 0 with none; a refused write to
/proc/sysas user ID 0; nestedchrootand leaked-descriptor escapes, and working directories reached through magic links, which fail the launch; in both publication states, that a root workload can neither lower the unprivileged port threshold nor forge a listener announcement, from a privileged source port or through a raw socket; numeric mode; the image's own machine ID; and that the time status shows only its public files. NVX's own adapter keeps a non-root identity without capabilities.Stripped sizes, built on Linux, from dev:
nvx-managed-agentgrows from 496,160 to 512,544 bytes, the Alpine initramfs from 8,223,214 to 8,231,379, the Ubuntu initramfs from 38,780,724 to 38,788,882, and the Ubuntu distro layer from 59,887,616 to 59,899,904; the other eight guest binaries keep their sizes.Validation
At this branch's head, 9049f18, and at 991eb62, the head of #778, which adds NVX's managed agent switch on top of it:
nvx-rs, withNVX_BUNDLE=skip,cargo fmt --all --check,cargo clippy --locked --all-targets --all-features -- -D warnings,cargo clippy --locked --lib --no-default-features -- -D warnings,cargo test --locked --all-features,cargo test --locked, the bundled-artifacts test,cargo docwithRUSTDOCFLAGS=-D warnings,cargo +1.89 check, and the macOS check;tests/runtime-adapterin its default,host, and muslguestconfigurations; theaci_edge_sandboxeschecks;nvx.py check-guest-binaries,build-guest-binariesandtest-guest-binaries, andtest-agent-bundling; the quality gates (ruff, pyright, shellcheck, shfmt, the Python tests, the markdown links, andnvx.py verify). Thenvxlibrary tests also ran as root, with the ignored privileged mapping tests and the privileged entry test, which checks the working directories that magic links name too, and 20 times each as root and as a normal user.nvx.py test-microvm --backend mshvwith every default scenario butcpu-profile-fallback, the public managed sandbox acceptance, the Ubuntu and Azure Linux guest smoke tests, the Ubuntu sandbox steps, theaci_edge_sandboxeslifecycle test, and the runtime adapter's VM scenarios, includingroot_profiles_enter_the_root. As root, the caller-owned live-share step refuses its root-owned share, and five read-write mapping tests ofaci_edge_sandboxescannot write to root-owned host directories; the five pass as a normal user on WHP.nvx-rs, runtime adapter (default andhost), andaci_edge_sandboxeschecks,test-agent-bundling, and the Python tests; at 991eb62, the CI microVM job's steps, with the full default suite, and the runtime adapter's VM scenarios.Left to CI: KVM; the debug kernel; the caller-owned live-share step, which checks a runner without the privilege to change its user ID, on Linux only;
cpu-profile-fallback, since the MSHV host's CPU has no built-in profile that OpenVMM can present, so the local runs used--cpu-profile host; the musl MSRV check of the runtime adapter's guest agent; and the Azure Linux image, which the local runs took from the development release.