You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
nvx-rs: add the sandbox store, launch journal, and runtime identity - #628
Step H2 of #620: one store of a sandbox's runtime records, nvx::host::store, for the supervisor (H3), the CLI and host::instance (H4), and external orchestrators, with a launch journal that never loses a VMM whose launch crashed midway.
What changed
SandboxStore: one directory per sandbox, private to the current user, since its records decide which process recovery signals. On Unix it has mode 0700, which OpenVMM also requires of its control socket's directory, and its files mode 0600, which the store sets again through each new directory and file, since a umask can clear even the user's own permission to read or write them (a lock file left without that permission is given its mode again before it is opened); on Windows, a directory that the store creates gets a protected access control list that allows the user alone, which its files inherit, whatever its parent allows. An existing directory must already be private: it must belong to the user (or, on Windows, to SYSTEM or the Administrators group), allow no one else, and on Windows pass entries on to the files created in it, and create and open refuse any other rather than change it, since others could have placed records in it or still hold its files open. Since a Windows file's own list decides who may open it, the store also refuses a lock, record, or capability whose list allows anyone else. The directories above it must be ones that only the user or an administrator may change. One lock, sandbox.lock, which lock(deadline) takes, serializes lifecycle transitions. Every transition (begin_launch, recover, and clear_runtime) borrows the resulting LockedStore exclusively, and a Launch keeps it borrowed until it commits or is abandoned. A relative directory is resolved to an absolute one, so that the recorded paths and endpoint name the same files from any working directory. The store names the capability, control endpoint, log, console, and outcome paths. Records are replaced atomically, so readers that hold no lock see whole records.
RuntimeIdentity: ArtifactIdentity tokens for OpenVMM, the kernel, and the initramfs, such as sha256: and the digest that a release's SHA256SUMS lists, which the caller takes from install or staging metadata (nothing here reads or hashes an artifact), plus bootContract (MICROVM_ABI_VERSION), controlContract (CONTROL_CONTRACT_REVISION), and controlSession (CONTROL_SESSION_PROTOCOL_VERSION). RuntimeRecord::check_identity returns Matches, or Mismatched with the IdentityParts that differ, for H3's reattach to report.
LaunchJournal (launch.json), in two states:
Prepared: the marker that LockedStore::begin_launch writes before anything is spawned, with the launch ID, endpoint, runtime identity, and log claim;
Spawned: after Launch::record_process records the VMM's ProcessRef right after the spawn, and returns the child held as a Process.
Launch::commit writes the runtime record, which is the commit point, then removes the journal. Launch::abandon ends a failed launch as recovery does, and dropping a Launch leaves its journal, as a crash would.
The log claim: every launch creates a fresh log, and the VMM inherits the claim with its standard output and error. On Linux it is an exclusive flock lock of the log's open file description, which only its last close releases. On Windows it is the inherited handle itself, since an open that shares nothing fails while any process holds the log. The journal records the log's file identity. A released claim proves that a launch that never recorded its VMM or published its endpoint has exited.
LockedStore::recover ends an interrupted launch, and signals only a VMM that it holds through a verified Process:
a crash before the spawn leaves the claim released, so the launch has exited;
after a crash between the spawn and the identity, the process that serves the journal's endpoint is held with Process::open, found to serve it still once held, which a reused process ID cannot pass, and terminated. On Windows the endpoint is a pipe whose name each launch draws at random, which names the VMM by itself, so none of the process's objects are examined; on Linux, where the socket path is the same for every launch of the sandbox, the process must also hold the claimed log as its standard output (/proc/<pid>/fd/1). Until a process serves the endpoint, recovery is Indeterminate and keeps the journal; once the VMM exits, the released claim proves it;
after a crash before the endpoint, the recorded ProcessRef decides through Process::open, so a reused process ID, or a reference of an earlier boot, is never signaled. Within the current boot, a VMM that Process::open cannot hold, because no process matches the reference or the process that has its ID cannot be examined, such as another user's, counts as exited only once the claim is released, since until then a process of the launch may still run.
Once nothing of the launch runs, recovery removes the capability, the socket, and the journal last, and keeps the log.
Format: one versioned format, FORMAT = 4, which follows the runtime formats of the CLI (1) and host::instance (3). A record in any other format fails before its fields are read, with a message that names its format; nothing is migrated, so switching to the store takes one stop of every sandbox. Unknown fields are refused. Configuration records stay with the caller.
Platform primitives for the store, private to the crate, on Linux and Windows: the creation and check of a private store directory (the owner and mode on Linux; the owner and access control list on Windows), the identity of an open file, the claim and its check, and whether the process that serves a launch's endpoint is its VMM. Other hosts compile and report them unsupported.
Tests: nvx-rs/tests/store.rs runs the test executable as an orchestrator that crashes at each launch step (before the spawn, between the spawn and the identity, and before the endpoint) and as a VMM stand-in, and checks that recovery never signals a process that the journal does not name. With the testing feature, the fake OpenVMM serves the endpoint for the endpoint-and-log verification. Fixtures in tests/fixtures/store/ hold a runtime record and journals that another build wrote, with a matching and a mismatched runtime identity, and corrupt, unknown-field, and unsupported records: the CLI's format 1, host::instance's format 3, and a future format 5. These are the host parts of C6 and C7 of Shared image-backed sandbox runtime #535. Unit tests cover the identities, the JSON forms, the lock, the privacy of the store's directory and files (including a Windows parent that allows every user, a Windows directory that passes nothing on to its files, records whose own list allows others, a setgid parent, umasks of 0400, 0200, and 0777, and a lock file left with mode 0000 on Linux, and directories that other users may access or own), the launch bookkeeping, and the recovery of a recorded VMM that cannot be held.
Not in this PR: the supervisor's reattach and stop (H3), and the delegation of the CLI and host::instance to the store (H4). The CLI and host::instance keep their current state formats until H4.
Windows 11, x64:python scripts\nvx.py verify and python scripts\check_markdown_links.py. In nvx-rs, with NVX_BUNDLE=skip: cargo fmt --all --check, cargo clippy --locked --all-targets --all-features -- -D warnings, cargo clippy --locked --lib --no-default-features -- -D warnings, cargo test --locked --all-features, cargo test --locked, cargo doc --locked --no-deps --all-features with RUSTDOCFLAGS=-D warnings, and cargo +1.89 check --locked --all-targets --all-features. The same steps for the aci_edge_sandboxes workspace, and python scripts\nvx.py test-agent-bundling --toolchain 1.95.0, as check-aci-edge-sandboxes runs them.
Linux (Ubuntu on WSL 2): the same steps for both crates, plus cargo check --locked --all-targets --all-features --target aarch64-apple-darwin, as an unprivileged user: as root, the recorded --mount-owner caller scenarios of tests/cli.rs refuse a root-owned share.
The runtime adapter (tests/runtime-adapter), on both: the steps of check-nvx-rs for its protocol-only and host-only configurations (cargo fmt --all --check, then Clippy with -D warnings, the build, the tests, and cargo +1.89 check --locked --all-targets), and, on Linux, for its static x86_64-unknown-linux-musl guest-only configuration too.
No real-VM runs: the CLI and host::instance lifecycle is unchanged.
Left to CI: the guest-binary checks and the KVM, MSHV, and WHP matrix.
Files that OpenVMM itself writes for the caller, its report and a captured console, are not the store's; keeping them readable under a restrictive umask is tracked in #651.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A umask clearing owner-write permission leaves the lifecycle lock unusable after its first acquisition, blocking subsequent operations and recovery.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Umask 0200 creates sandbox.lock without owner-write permission
nvx-rs/src/host/store/mod.rs:344
For an unprivileged caller with umask 0200, this creates sandbox.lock with mode 0400. The initial write-only open and lock succeed, but subsequent lock() calls fail with PermissionDenied, preventing lifecycle operations and recovery by a successor. Unlike files::create_new, this path never restores mode 0600. Set the mode through the opened descriptor, and extend the umask subprocess test beyond 0400 to cover clearing owner-write permission.
Every supervisor of a sandbox needs the same runtime records: which
VMM runs the sandbox's VM, where its control endpoint is, and which
runtime it was launched with. The CLI and host::instance each keep
their own, in different formats with different locks, and neither
records a launch before its spawn, so a caller that crashes between
spawning OpenVMM and recording its process ID leaves the VMM running
unrecorded.
host::store is one store for them: a private directory per sandbox,
mode 0700 with files of mode 0600 on Unix, with one lock, which every
lifecycle transition borrows exclusively. The store resolves its
directory to an absolute path, so that the paths and the control
endpoint that it records name the same files from any working
directory. Its runtime record holds the VMM's ProcessRef, with its
process ID, start time, and host boot, the control endpoint, and the
runtime identity: the identities of the OpenVMM executable, kernel,
and initramfs, which the caller takes from install or staging
metadata rather than hashing them at launch, and the boot and control
contract versions. check_identity compares a record with the runtime
that the host would launch now, so that a later build of the host may
reattach under the same runtime identity, and reports any other as
mismatched. The store also names the capability, endpoint, log,
console, and outcome paths.
A launch writes its journal before it spawns anything: a marker with
the launch's endpoint, runtime identity, and claim on a fresh log,
then the VMM's ProcessRef right after the spawn. Committing replaces
the journal with the runtime record. The VMM inherits the log claim
with its standard output and error: an exclusive flock lock of the
log's open file description on Linux, and the inherited handle itself
on Windows, where an open that shares nothing fails while any process
holds the log. A released claim proves that a launch that never
recorded its VMM or published its endpoint has exited.
recover ends an interrupted launch and signals only a VMM that it
holds through a verified Process. A crash before the spawn leaves the
claim released. After a crash between the spawn and the identity,
the process that serves the journal's endpoint is held, found to
serve it still, and terminated: on Windows the endpoint is a pipe
whose name each launch draws at random, and on Linux, where the
socket path is the same for every launch, the process must also hold
the claimed log as its standard output. Until a process serves the
endpoint, recovery is indeterminate and keeps the journal. After a
crash before the endpoint, Process::open holds the recorded VMM only
while it has the recorded identity, so neither a process that reuses
the ID nor any process of an earlier boot is signaled; within the
current boot, a VMM that it cannot hold, because no process matches
the reference or the process that has its ID cannot be examined,
counts as exited only once the claim is released.
Records are JSON in format 4, after the runtime formats of the CLI
and host::instance, and a record in any other format fails with a
message that names its format rather than being migrated.
Configuration stays with the caller. The CLI and host::instance keep
their own state until they delegate to the store.
tests/store.rs runs orchestrator processes that crash at each launch
step against VMM stand-ins, on Linux and Windows, from a relative
store path, and loads records that another build wrote: a matching
and a mismatched runtime identity, a journal of each step, and
corrupt and unsupported records.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3de46675-b5c7-4dab-90a8-e03b876def2c
A store's records decide which process recovery signals, so another
user who could write to a store's directory could make recovery
terminate an unrelated process of the store's user. The store relied
on its directory being private, but did not make it so: on Unix,
create restricted the directory to mode 0700 without checking its
owner, on Windows the directory inherited the access control list of
its parent, which is private only under a user's profile, and open
checked nothing.
create now makes a new directory private to the current user: mode
0700 on Unix, and on Windows a protected access control list that
allows the user alone, which the store's files inherit, whatever the
parent allows. An existing directory must belong to the user, and no
one else may write to it: on Unix, one that others may only read is
restricted to the user, and on Windows, one that allows anyone else
is refused, since a handle opened before keeps its access. On
Windows, SYSTEM and the Administrators group may also own or access a
directory that the store did not create, as under a user's profile.
open refuses any directory that is not private, and both refuse a
link in place of the directory.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3de46675-b5c7-4dab-90a8-e03b876def2c
create restricted an existing Unix directory that others could only
read to mode 0700. Restricting a directory does not revoke the
descriptors that others already hold: one that another user opened on
a journal that anyone may write to, in a directory that others could
search, keeps writing to it, and recovery would trust the journal that
it changed. create now refuses an existing directory that is not
already private, on Unix as on Windows, rather than restricting it,
so that the store only ever adopts a directory that others could not
access.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3de46675-b5c7-4dab-90a8-e03b876def2c
On Windows, a file's own access control list decides who may open it,
whatever its directory allows, and a file created in a directory that
passes no entries on to its files takes its creator's default list,
which may allow anyone. An adopted directory that allowed its user
alone, without passing anything on, could thus leave the journal and
the capability open to other users. A store's directory must now pass
entries on to the files created in it, and the store refuses a record
or a capability whose own list allows anyone else, since others could
have written it.
On Linux, a new directory takes the setgid bit of its parent, which a
store's directory has kept since the store stopped setting the mode of
the directories that it creates, while OpenVMM requires the directory
of its control socket to have mode 0700 exactly. The store sets the
mode of a directory that it creates to 0700, and refuses an existing
one with any other mode.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3de46675-b5c7-4dab-90a8-e03b876def2c
The store refused a record or a capability whose own Windows access
control list allowed other users, but not its lock file, which it
opens whatever list the file has. Another user who could open an
existing lock file could hold its lock and block every lifecycle
transition of the sandbox. lock now refuses a lock file that is not
private to the current user, as the store does a record.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3de46675-b5c7-4dab-90a8-e03b876def2c
Answers the review of the sandbox store. The form of artifact identities
and control contract revisions moves into a private Token type that both
use, with one message. A SHA-256 digest's 64 digits and a launch
identity's 16 random bytes and 32 digits become named constants.
RuntimeIdentity::differences and IdentityCheck::Mismatched now return
IdentityPart values, whose names are the earlier strings, rather than
the strings themselves. Recovery waits for a terminated VMM for the
KILL_TIMEOUT of process supervision rather than a copy of it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3de46675-b5c7-4dab-90a8-e03b876def2c
A umask can clear even the current user's permission to read what the
user creates. On Linux, the store then failed to reopen its new
directory before setting its mode, and wrote records, a capability, and
a log that it, its successors, and OpenVMM could not read.
The directory is now opened only to locate it, with O_PATH, which needs
no permission on it, and its mode is set through that descriptor's link
in /proc. Each new file gets mode 0600 again through its descriptor, and
the lock file opens for writing only on Unix, which flock and fstat
allow, so a lock file that such a umask created still opens. Existing
directories stay unchanged.
a_restrictive_umask_leaves_the_store_usable creates a store and launches
twice in a process whose umask is 0400.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3de46675-b5c7-4dab-90a8-e03b876def2c
Answers the review of the sandbox store. The lock file was created
with the mode that the umask left it, so a umask that clears the
current user's permission to write, such as 0200 or 0777, made every
later lock of the sandbox fail.
The call that creates the lock file now sets its mode to 0600 again
through the new file, as the store's other files are. A lock file of
the current user that it can no longer open for writing, which another
call that creates it leaves until it sets the mode, or leaves for good
if it crashes first, is given its mode again before it is opened.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3de46675-b5c7-4dab-90a8-e03b876def2c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pedro Henrique Penna (ppenna)
assigned
Enrique Saurez (esaurez) and unassigned
Pedro Henrique Penna (ppenna)
Step H2 of #620: one store of a sandbox's runtime records,
nvx::host::store, for the supervisor (H3), the CLI andhost::instance(H4), and external orchestrators, with a launch journal that never loses a VMM whose launch crashed midway.What changed
SandboxStore: one directory per sandbox, private to the current user, since its records decide which process recovery signals. On Unix it has mode 0700, which OpenVMM also requires of its control socket's directory, and its files mode 0600, which the store sets again through each new directory and file, since a umask can clear even the user's own permission to read or write them (a lock file left without that permission is given its mode again before it is opened); on Windows, a directory that the store creates gets a protected access control list that allows the user alone, which its files inherit, whatever its parent allows. An existing directory must already be private: it must belong to the user (or, on Windows, to SYSTEM or the Administrators group), allow no one else, and on Windows pass entries on to the files created in it, andcreateandopenrefuse any other rather than change it, since others could have placed records in it or still hold its files open. Since a Windows file's own list decides who may open it, the store also refuses a lock, record, or capability whose list allows anyone else. The directories above it must be ones that only the user or an administrator may change. One lock,sandbox.lock, whichlock(deadline)takes, serializes lifecycle transitions. Every transition (begin_launch,recover, andclear_runtime) borrows the resultingLockedStoreexclusively, and aLaunchkeeps it borrowed until it commits or is abandoned. A relative directory is resolved to an absolute one, so that the recorded paths and endpoint name the same files from any working directory. The store names the capability, control endpoint, log, console, and outcome paths. Records are replaced atomically, so readers that hold no lock see whole records.RuntimeRecord(runtime.json): the launch ID, the VMM'sProcessReffrom nvx-rs: add verified process references with event-driven exit waits #625 (pid,startTime,boot), the control endpoint, and the runtime identity.RuntimeIdentity:ArtifactIdentitytokens for OpenVMM, the kernel, and the initramfs, such assha256:and the digest that a release'sSHA256SUMSlists, which the caller takes from install or staging metadata (nothing here reads or hashes an artifact), plusbootContract(MICROVM_ABI_VERSION),controlContract(CONTROL_CONTRACT_REVISION), andcontrolSession(CONTROL_SESSION_PROTOCOL_VERSION).RuntimeRecord::check_identityreturnsMatches, orMismatchedwith theIdentityParts that differ, for H3's reattach to report.LaunchJournal(launch.json), in two states:Prepared: the marker thatLockedStore::begin_launchwrites before anything is spawned, with the launch ID, endpoint, runtime identity, and log claim;Spawned: afterLaunch::record_processrecords the VMM'sProcessRefright after the spawn, and returns the child held as aProcess.Launch::commitwrites the runtime record, which is the commit point, then removes the journal.Launch::abandonends a failed launch as recovery does, and dropping aLaunchleaves its journal, as a crash would.The log claim: every launch creates a fresh log, and the VMM inherits the claim with its standard output and error. On Linux it is an exclusive
flocklock of the log's open file description, which only its last close releases. On Windows it is the inherited handle itself, since an open that shares nothing fails while any process holds the log. The journal records the log's file identity. A released claim proves that a launch that never recorded its VMM or published its endpoint has exited.LockedStore::recoverends an interrupted launch, and signals only a VMM that it holds through a verifiedProcess:Process::open, found to serve it still once held, which a reused process ID cannot pass, and terminated. On Windows the endpoint is a pipe whose name each launch draws at random, which names the VMM by itself, so none of the process's objects are examined; on Linux, where the socket path is the same for every launch of the sandbox, the process must also hold the claimed log as its standard output (/proc/<pid>/fd/1). Until a process serves the endpoint, recovery isIndeterminateand keeps the journal; once the VMM exits, the released claim proves it;ProcessRefdecides throughProcess::open, so a reused process ID, or a reference of an earlier boot, is never signaled. Within the current boot, a VMM thatProcess::opencannot hold, because no process matches the reference or the process that has its ID cannot be examined, such as another user's, counts as exited only once the claim is released, since until then a process of the launch may still run.Once nothing of the launch runs, recovery removes the capability, the socket, and the journal last, and keeps the log.
Format: one versioned format,
FORMAT = 4, which follows the runtime formats of the CLI (1) andhost::instance(3). A record in any other format fails before its fields are read, with a message that names its format; nothing is migrated, so switching to the store takes one stop of every sandbox. Unknown fields are refused. Configuration records stay with the caller.Platform primitives for the store, private to the crate, on Linux and Windows: the creation and check of a private store directory (the owner and mode on Linux; the owner and access control list on Windows), the identity of an open file, the claim and its check, and whether the process that serves a launch's endpoint is its VMM. Other hosts compile and report them unsupported.
Tests:
nvx-rs/tests/store.rsruns the test executable as an orchestrator that crashes at each launch step (before the spawn, between the spawn and the identity, and before the endpoint) and as a VMM stand-in, and checks that recovery never signals a process that the journal does not name. With thetestingfeature, the fake OpenVMM serves the endpoint for the endpoint-and-log verification. Fixtures intests/fixtures/store/hold a runtime record and journals that another build wrote, with a matching and a mismatched runtime identity, and corrupt, unknown-field, and unsupported records: the CLI's format 1,host::instance's format 3, and a future format 5. These are the host parts of C6 and C7 of Shared image-backed sandbox runtime #535. Unit tests cover the identities, the JSON forms, the lock, the privacy of the store's directory and files (including a Windows parent that allows every user, a Windows directory that passes nothing on to its files, records whose own list allows others, a setgid parent, umasks of 0400, 0200, and 0777, and a lock file left with mode 0000 on Linux, and directories that other users may access or own), the launch bookkeeping, and the recovery of a recorded VMM that cannot be held.Not in this PR: the supervisor's reattach and stop (H3), and the delegation of the CLI and
host::instanceto the store (H4). The CLI andhost::instancekeep their current state formats until H4.Part of #620. Refs #535.
Validation
python scripts\nvx.py verifyandpython scripts\check_markdown_links.py. Innvx-rs, withNVX_BUNDLE=skip:cargo fmt --all --check,cargo clippy --locked --all-targets --all-features -- -D warnings,cargo clippy --locked --lib --no-default-features -- -D warnings,cargo test --locked --all-features,cargo test --locked,cargo doc --locked --no-deps --all-featureswithRUSTDOCFLAGS=-D warnings, andcargo +1.89 check --locked --all-targets --all-features. The same steps for theaci_edge_sandboxesworkspace, andpython scripts\nvx.py test-agent-bundling --toolchain 1.95.0, ascheck-aci-edge-sandboxesruns them.cargo check --locked --all-targets --all-features --target aarch64-apple-darwin, as an unprivileged user: as root, the recorded--mount-owner callerscenarios oftests/cli.rsrefuse a root-owned share.tests/runtime-adapter), on both: the steps ofcheck-nvx-rsfor its protocol-only and host-only configurations (cargo fmt --all --check, then Clippy with-D warnings, the build, the tests, andcargo +1.89 check --locked --all-targets), and, on Linux, for its staticx86_64-unknown-linux-muslguest-only configuration too.devafter nvx-rs: decode the MAPS entries once for both payload decoders #616 and nvx-rs: convert OSErrors without a filename to CLI exits in one place #617 merged, and again with the review fixes.host::instancelifecycle is unchanged.Files that OpenVMM itself writes for the caller, its report and a captured console, are not the store's; keeping them readable under a restrictive umask is tracked in #651.