Skip to content

nvx-rs: assemble a guest root over a tmpfs or a scratch in an overlay mode - #775

Merged
Enrique Saurez (esaurez) merged 3 commits into
devfrom
esaurez/a3-3-gpt-tmpfs-overlay-20261021
Oct 23, 2026
Merged

Enrique Saurez (esaurez) merged 3 commits into
devfrom
esaurez/a3-3-gpt-tmpfs-overlay-20261021

Conversation

@esaurez

@esaurez Enrique Saurez (esaurez) commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

A guest root's assembly took an EROFS image and an ext4 scratch, and mounted the overlay with one fixed option set. This PR lets the caller of nvx::guest::root::assemble name what it assembles, with typed inputs, and the assembly probes nothing:

  • The image: Image::Erofs, with the UUID that the caller expects, if any. The enum is non-exhaustive, for the GPT format that the next PR of A3.3 adds.
  • The upper: Upper::Scratch, a whole-device ext4 scratch, mounted rw,nosuid,nodev, with the caller's Journal policy, which allows or forbids a journal, checked in the superblock's feature flags before the scratch is mounted; or Upper::Tmpfs, a tmpfs at the kernel's default size, mounted nosuid,nodev, for a VM without a scratch device. The upper and work directories are separate directories of the upper, created 0755 whatever the umask.
  • The overlay mode: OverlayMode::TrustedLowers, the default, mounts the overlay with metacopy=on,xino=on, and OverlayMode::UntrustedLowers, for lowers that may carry untrusted overlay xattrs, with metacopy=off,redirect_dir=nofollow. Each mode has one option set: when the kernel refuses it, the assembly fails, with no fallback to another. The merged root is mounted nosuid,nodev.

The kernel has fallbacks of its own, which don't fail the mount: it mounts an overlay read-only when it can't create the work directory, and turns metacopy off and redirect_dir to nofollow when the upper can't hold the overlay's xattrs. So once the overlay is mounted, the assembly reads its options in the kernel's mount table, where an option at its module default doesn't show, and refuses and unmounts an overlay that isn't writable or lacks its mode's options.

The assembly checks everything else before it mounts anything:

  • each device's access mode, from sysfs, against its role's, read-only for the image and writable for the scratch, before it flushes the device, so that a device in the wrong role is refused;
  • each device's expected size, the image's UUID, and the scratch's journal;
  • the runtime directory, which a mount that a failed teardown left refuses, at the image's, the root's, or the aggregate share's mount point, or at either upper's, whichever upper the new root mounts;
  • the length of the overlay's options. mount(2) passes a file system one page of options, and the kernel cuts longer ones short, which would drop the last lower directories or the mode's options. The assembly refuses options longer than MOUNT_OPTIONS_LIMIT, 4,095 bytes, instead.

The Root reports what it mounted: its Lower layers, its UpperKind, and its OverlayMode, whose options() are the overlay's. Its teardown unmounts the live shares and mappings in reverse order, the overlay, the layers from the top down, and then the upper, and reports every unmount that fails.

The runtime adapter's layered VMs (tests/runtime-adapter) now assemble the EROFS image over a scratch or a tmpfs, in the overlay mode and with the journal policy that the host names, and probe copy-up, the rename of a lower directory, a full upper, and the guest's disks. The map probe finds the targets that it creates on the overlay in the upper of either kind.

Part of #621. Refs #622 and #535.

Validation

On this branch's head, locally:

  • python scripts/nvx.py verify.
  • nvx-rs, on Linux with NVX_BUNDLE=skip: cargo clippy --locked --all-targets --all-features -- -D warnings, cargo clippy --locked --lib --no-default-features -- -D warnings, cargo test --locked --all-features (1,103 passed), cargo test --locked (508 passed), cargo doc --locked --no-deps --all-features with RUSTDOCFLAGS=-D warnings, cargo +1.89 check --locked --all-targets --all-features, and check-nvx-rs's macOS check and bundling test; cargo fmt --all --check on Windows.
  • Guest binaries: check-guest-binaries, build-guest-binaries and test-guest-binaries (34 passed). The stripped guest binaries are byte-identical before and after this change, and so is the Alpine initramfs: 8,222,829 bytes, 19,707,392 uncompressed.
  • tests/runtime-adapter: Clippy, build, tests and the MSRV check in the protocol-only, host and guest configurations on Linux, the guest configuration's MSRV check for the GNU target, and fmt, Clippy, build, tests and the MSRV check in the protocol-only and host configurations on Windows.
  • Real VMs, the adapter's ignored tests, cargo test --locked --features host --test vm_layered --test vm_restart -- --ignored: 12 layered scenarios, this PR's 10 and the 2 of the root's mappings, and 2 restart tests pass on MSHV and WHP. They show, per mode: TrustedLowers mounts with redirect_dir=on,uuid=on,xino=on,metacopy=on in effect, copies up a file's metadata alone, and renames a lower directory; UntrustedLowers shows only uuid=on, copies up the data too, and refuses the rename with EXDEV. The tmpfs upper is rw,nosuid,nodev with mode=755 and no size=, with no scratch disk in the guest; a full scratch and a full tmpfs, half of the guest's memory, refuse writes with ENOSPC until the file that filled them is gone.
  • NVX's own layered scenarios still pass, unchanged: python scripts/nvx.py test-microvm --backend <mshv|whp> --processors 1 with managed-exec-config, sandbox-lifecycle, sandbox-blocks, scratch-snapshot, snapshot-tiers and managed-lifecycle, on MSHV and WHP, with the Alpine initramfs built from this branch.

MSHV ran as root and with --cpu-profile host, since its test host can't present a built-in CPU profile. KVM and the full backend matrix are left to CI.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged mount assembly and kernel-dependent validation require human review, with an unresolved stale-mount check when switching upper kinds.

1 open issue
What changed in this PR

Extends NVX’s guest root library with typed image, upper, and overlay-mode inputs, advancing the shared layered sandbox runtime in #621.

Changes:

  • Supports EROFS roots over ext4 scratch or tmpfs, with caller-selected journal and overlay policies.
  • Validates device roles, mount-option length, and effective overlay options; exposes mounted layers and upper kind.
  • Extends the runtime adapter’s configuration, probes, documentation, and tests for the new combinations.
File Description
tests/​runtime-adapter/​tests/​vm_layered.rs Adds upper/mode combinations and full-upper scenarios.
tests/​runtime-adapter/​tests/​layered_launch.rs Tests rejection of invalid policy options.
tests/​runtime-adapter/​tests/​guest_session.rs Tests RAM assembly and conflicting configuration.
tests/​runtime-adapter/​src/​worker/​probes.rs Adds copy-up, rename, capacity, and disk probes.
tests/​runtime-adapter/​src/​worker.rs Passes typed root settings to guest assembly.
tests/​runtime-adapter/​src/​root.rs Encodes and validates overlay and journal policies.
tests/​runtime-adapter/​src/​bin/​runtime-adapter-host.rs Adds overlay-mode and journal-policy arguments.
tests/​runtime-adapter/​README.md Documents configuration, probes, and scenarios.
tests/​runtime-adapter/​Cargo.toml Enables filesystem calls for capacity probes.
nvx-rs/​src/​guest/​root/​tests.rs Covers upper modes, validation, and teardown failures.
nvx-rs/​src/​guest/​root/​kernel.rs Reads effective mount options and extends the fake kernel.
nvx-rs/​src/​guest/​root/​block.rs Checks device access and ext4 journal flags.
nvx-rs/​src/​guest/​root.rs Implements typed assembly, tmpfs uppers, and overlay validation.
nvx-rs/​README.md Updates the guest root API description.

Review effort: Balanced

Comment thread nvx-rs/src/guest/root.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged mount lifecycle changes and security-sensitive, kernel-dependent overlay checks warrant human review alongside the remaining backend CI results.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged mount-policy enforcement and teardown changes warrant human sign-off with the KVM and full backend CI results.

No open issues 🎉

Review effort: Balanced

Base automatically changed from esaurez/adapter-layered-vms-20261021 to dev October 23, 2026 03:00
Copilot AI balanced review requested due to automatic review settings October 23, 2026 03:16
@esaurez
Enrique Saurez (esaurez) force-pushed the esaurez/a3-3-gpt-tmpfs-overlay-20261021 branch from 24ecfbd to 5e8ec70 Compare October 23, 2026 03:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Security-sensitive overlay enforcement and privileged mount lifecycle changes warrant human review and confirmation of the full backend CI results.

No open issues 🎉

Review effort: Balanced

@esaurez
Enrique Saurez (esaurez) marked this pull request as ready for review October 23, 2026 03:22
Enrique Saurez and others added 3 commits October 22, 2026 21:32
… mode

A guest root's assembly took an EROFS image and an ext4 scratch, and
mounted the overlay with its one option set, metacopy=on,xino=on. The
caller now names what it assembles with typed inputs, and nothing is
probed:

- the image, an EROFS file system with the UUID that the caller
  expects, if any, as an Image that later formats extend;
- the upper: a whole-device ext4 scratch, with a journal policy that
  the assembly checks in the scratch's superblock feature flags before
  it mounts it, or a tmpfs at the kernel's default size, mounted
  nosuid,nodev, for a VM without a scratch device;
- the overlay mode, whose one option set the overlay is mounted with:
  metacopy=on,xino=on by default, or metacopy=off,redirect_dir=nofollow
  for lowers that may carry untrusted overlay xattrs. The kernel's
  refusal of the set fails the assembly, with no fallback to another.
  Nor does the assembly accept the kernel's own fallbacks: without
  failing the mount, the kernel mounts an overlay read-only when it
  can't create its work directory, and drops metacopy=on when the upper
  can't hold the overlay's xattrs. The assembly reads the overlay's
  options in the kernel's mount table, where an option at its default,
  which a parameter of the overlay module holds, doesn't show, and
  refuses an overlay that isn't writable or lacks the mode's options.

The assembly also checks each device's access mode against its role's,
read-only for the distro and writable for the scratch, so that a device
in the wrong role is refused before it is flushed. It checks every
device, and the length of the overlay's options, before it mounts
anything: mount(2) passes a file system one page of options, and the
kernel cuts longer ones short, so options longer than 4,095 bytes are
refused rather than mounted without their last options.

The root reports what it mounted, its layers, the kind of its upper,
and the overlay mode, and its teardown unmounts the live shares and
mappings, the overlay, the layers from the top down, and then the
upper, reporting every unmount that fails. The runtime adapter's
worker names its EROFS image over its scratch with the new inputs, in
the default mode.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 30bab0d7-7c3e-4120-9add-6c9cc1dc9e7e
The runtime adapter's worker assembled only an EROFS image over an ext4
scratch, in the default overlay mode. It now also assembles one over a
tmpfs in RAM, in the overlay mode that the host names, so that the guest
root's uppers and overlay modes get real-kernel coverage:

- The root configuration takes two optional fields, overlay-mode and
  scratch-journal, which the host program's launch sets from
  --overlay-mode and --scratch-journal. Without them, the guest takes
  nvx::guest::root's defaults. The worker refuses a configuration that
  configures a scratch for an upper in RAM, and still refuses a GPT
  image at the layout stage.
- The copy-up probe changes the mode of a lower file that the scenario
  names, and reports whether the file's upper copy holds its data, which
  a metadata-only copy-up leaves in the layer. The rename probe renames a
  lower directory that the scenario names, which the untrusted-lowers
  mode refuses with EXDEV.
- The fill probe fills the upper through the root until a write fails,
  removes the file, and writes again. The devices probe lists the
  guest's disks, and the mounts probe also reports a tmpfs upper. The
  map probe finds the targets that it created on the overlay in the
  upper of either kind.
- New scenarios cover EROFS over a scratch and over RAM in both modes,
  a full scratch, a full tmpfs, which the kernel sizes at half of the
  guest's memory, and a scratch with a journal that the launch forbids,
  which fails the assembly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 30bab0d7-7c3e-4120-9add-6c9cc1dc9e7e
An assembly checked only the mount point of the upper that it mounts.
A root whose teardown failed to unmount its upper, say a scratch that a
process held open, left that upper mounted, and the next root, over a
tmpfs, passed the check and later reported a clean teardown with the
scratch still mounted; and the other way around.

The assembly now also checks the other upper's mount point, which may
be missing, as it does the aggregate share's, and refuses the runtime
directory when a file system, or a bind mount of a directory, is
mounted there. Tests cover both transitions after a failed unmount of
the upper.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 30bab0d7-7c3e-4120-9add-6c9cc1dc9e7e
Copilot AI balanced review requested due to automatic review settings October 23, 2026 04:49
@esaurez
Enrique Saurez (esaurez) force-pushed the esaurez/a3-3-gpt-tmpfs-overlay-20261021 branch from 5e8ec70 to 28524ce Compare October 23, 2026 04:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Security-sensitive mount lifecycle changes and kernel-dependent overlay validation warrant human review alongside the full backend CI results.

No open issues 🎉

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants