diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 59bfe7693..ec21354f8 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -95,7 +95,7 @@ nvx-rs/ | `nvx::guest::exec` | `Execution`: drives one program that the caller spawned with the program's ends of its `Pipes`, without blocking, from the caller's own loop or reactor, and reports each step as an `Event`: the launch report, the output of each stream, kill failures, and the `Outcome`. It reads a stream only as far as the caller's `grant` allows, so a stalled sink stalls its own execution alone, writes the program's standard input without `SIGPIPE`, and ends the program on a cancellation, a timeout, or a disconnection, with `SIGKILL` through the program's pidfd to the program and, if the caller asks, its process group, while no collector has taken its status, and, through the opened directory of a cgroup that the caller names, to that cgroup. It closes the program's input once the program has ended. It waits for its program alone, by a pidfd, or takes the status from the caller's collector with `observe`; with `Options::caller_reaps`, that collector is the program's only one. NVX's managed agent is one caller | | `nvx::guest::time` | x86_64: `TimeAbi`, the steps of NVX time ABI v1 as typed calls in the caller's process, for any guest init or agent: the cold-boot time step and, separately, the start of the time daemon in a child process that the caller supervises, the VM generation ID, a `CaptureRequest` that repairs a `Restore`, and a `RestoreFinish`. A call takes the permission for OpenVMM's ports on the calling thread for each port transaction alone, and fails with `TimeError::PortRelease` if the thread may still hold some of it, which a process that it started would inherit, and from which `nvx::guest::spawn` starts none. No call waits for a child that it did not start. `nvx-time` runs its steps through them | | `nvx::guest::reseed` | `Seed::read` and `reseed`: a restore's fresh entropy, which starts with its `GenerationId`, and the reseed of the kernel's random number generator with it, in the caller's process. `nvx-reseed` runs them | -| `nvx::guest::root` | The operations of a layered root, for a guest init or agent, whether or not it is PID 1. `bootstrap`: the agent's and the workload's sibling cgroups of a `Layout`, with every process of their parent moved into the agent's, the agent's `memory.low`, and the workload cgroup prepared through `ExecutionCgroup::prepare`, with its `memory.max` and `pids.max`, and user namespaces kept unavailable. `assemble`: the `Assembly`'s block devices, found by their MMIO windows, checked against their expected sizes and flushed, and the `Root` that it mounts, at a runtime directory, a canonical path that no symbolic link leads through, that it holds exclusively until its teardown, where nothing is mounted already: a read-only EROFS image, whose expected `nvx::uuid::Uuid` it checks in its superblock, under an overlay with `metacopy=on,xino=on`, mounted `nosuid,nodev`, whose upper and work directories are on an ext4 scratch mounted `nosuid,nodev` too. `Root::mount_share` and `Root::mount_aggregate`: virtio-fs live shares at targets in the root that it checks, against the targets that `protocol::share` reserves among others, with one aggregate per root, which it attaches through its `mappings::Mapper` beneath the overlay, with NVX's container root's reserved targets: a share's device mounted out of the root, then moved onto its target, and a copy of each child's entry of the aggregate. `Root::map`: a caller's host mappings, through the same mapper. A share, a child, or a mapping that fails latches the root. `Root::teardown`: the shares and mappings in reverse order, then an aggregate's own mount, the overlay, the image, and the scratch, with every failed unmount reported. Each failure is a typed `Error`. The operations collect no child's status, change neither the caller's root nor its credentials, start no async runtime, and never power the VM off | +| `nvx::guest::root` | The operations of a layered root, for a guest init or agent, whether or not it is PID 1. `bootstrap`: the agent's and the workload's sibling cgroups of a `Layout`, with every process of their parent moved into the agent's, the agent's `memory.low`, and the workload cgroup prepared through `ExecutionCgroup::prepare`, with its `memory.max` and `pids.max`, and user namespaces kept unavailable. `assemble`: the `Assembly`'s block devices, found by their MMIO windows, checked against their roles' access modes and their expected sizes, and flushed, and the `Root` that it mounts, at a runtime directory, a canonical path that no symbolic link leads through, that it holds exclusively until its teardown, where nothing is mounted already, once it has checked everything: the read-only layer of the `Image` that the caller names, which it never probes, an EROFS image whose expected `nvx::uuid::Uuid` it checks in its superblock, under an overlay mounted `nosuid,nodev` with the one option set of the caller's `OverlayMode`, `metacopy=on,xino=on` or, for lowers that may carry untrusted overlay xattrs, `metacopy=off,redirect_dir=nofollow`, with no fallback, whose upper and work directories are on the `Upper`: an ext4 scratch mounted `nosuid,nodev`, whose superblock it checks for a journal that the caller's `Journal` policy forbids, or a tmpfs mounted `nosuid,nodev` at the kernel's default size. It refuses overlay options longer than `MOUNT_OPTIONS_LIMIT`, the most that a mount passes, and an overlay that the kernel's mount table shows read-only or without the mode's options in effect, since the kernel mounts one so, rather than fail, when its upper can't hold its work directory or its extended attributes. The `Root` reports its `Lower` layers, its `UpperKind`, and its `OverlayMode`. `Root::mount_share` and `Root::mount_aggregate`: virtio-fs live shares at targets in the root that it checks, against the targets that `protocol::share` reserves among others, with one aggregate per root, which it attaches through its `mappings::Mapper` beneath the overlay, with NVX's container root's reserved targets: a share's device mounted out of the root, then moved onto its target, and a copy of each child's entry of the aggregate. `Root::map`: a caller's host mappings, through the same mapper. A share, a child, or a mapping that fails latches the root. `Root::teardown`: the shares and mappings in reverse order, then an aggregate's own mount, the overlay, the layers from the top down, and the upper, with every failed unmount reported. Each failure is a typed `Error`. The operations collect no child's status, change neither the caller's root nor its credentials, start no async runtime, and never power the VM off | | `nvx::guest::gpt` | `read_disk`: the partition table of a GPT disk, read and checked with `nvx::gpt`'s `Disk::read` through the disk's block device, which the caller opened, at the logical block size and size that `BLKSSZGET` and `BLKGETSIZE64` report, with the caller's `Expected` values, such as the disk GUID that a boot parameter names. It only reads the device, at explicit offsets, so it never moves the file's position. It neither compares the partitions that the kernel created with the table nor asks the kernel to reread the table. Each failure is a typed `DeviceError` | Lowering functions take the names by which messages refer to their inputs, such diff --git a/nvx-rs/src/guest/root.rs b/nvx-rs/src/guest/root.rs index 19961bcb8..52e396269 100644 --- a/nvx-rs/src/guest/root.rs +++ b/nvx-rs/src/guest/root.rs @@ -6,17 +6,19 @@ //! agent's, applies the agent's and the workload's resource values, and keeps user namespaces //! unavailable. //! - [`assemble`] finds the block devices of an [`Assembly`] by their MMIO windows, checks and -//! flushes them, and mounts the root: a read-only EROFS image, the `distro` layer, under an -//! overlay whose upper and work directories are on an ext4 scratch, and which is mounted -//! `nosuid,nodev`, as the scratch is. It returns the [`Root`] that owns the mounts and holds -//! the [runtime directory](#runtime-directory). +//! flushes them, and mounts the root: an overlay, mounted `nosuid,nodev` with the option set of +//! the caller's [`OverlayMode`], which it checks the kernel applied, of the read-only layers of +//! the `distro` image, in the [`Image`] format that the caller names, which the assembly never +//! probes, over an upper, an ext4 scratch or a tmpfs. It returns the [`Root`] that owns the +//! mounts, holds the [runtime directory](#runtime-directory), and reports what it mounted: the +//! [`Lower`] layers, the [`UpperKind`], and the [`OverlayMode`]. //! - [`Root::mount_share`] and [`Root::mount_aggregate`] mount the VMM's live shares at targets in //! the root, and [`Root::map`] applies a caller's host mappings there, through the root's //! [`Mapper`], as [live shares and mappings](#live-shares-and-mappings) describes. //! - [`Root::teardown`] unmounts, in order, the live shares and mappings in reverse order, the -//! mount of an aggregate live share, the overlay, the layer, and the scratch. It attempts each, -//! whatever failed before, and reports every unmount that failed. A root that is dropped tears -//! down the same way, but ignores failures. +//! mount of an aggregate live share, the overlay, the layers from the top down, and the upper. +//! It attempts each, whatever failed before, and reports every unmount that failed. A root +//! that is dropped tears down the same way, but ignores failures. //! //! A failure is an [`Error`] that names what failed. The operations need the guest's root user, //! in the guest's initial mount and cgroup namespaces. They never collect a child's status, @@ -29,7 +31,8 @@ //! which holds: //! //! - `layers/distro`, the EROFS image; -//! - `scratch`, the scratch, with the overlay's `upper` and `work` directories; +//! - `scratch`, an ext4 scratch, or `tmpfs`, a tmpfs, with the overlay's `upper` and `work` +//! directories; //! - `rootfs`, the overlay, which is the root; //! - `shares`, an aggregate live share, whose root lists every child, out of the workload's //! reach; @@ -43,10 +46,15 @@ //! lead through a directory that one of the root's mounts hides later. A root also holds the //! exclusive lock of its runtime directory, from the start of its assembly until it is torn down: //! an assembly refuses a runtime directory that another root holds, in this process or another, -//! and one where a file system, or a bind mount of a directory, is mounted already at one of these -//! mount points, such as one that a failed teardown left; a live share refuses a `staging` where -//! one is mounted already. The lock is advisory: it keeps out the roots of these operations, not -//! other mounts. +//! and one where a file system, or a bind mount of a directory, is mounted already at any of these +//! mount points but `staging`, whether or not the new root mounts there, such as an upper of +//! either kind that a failed teardown left; a live share refuses a `staging` where one is mounted +//! already. The lock is advisory: it keeps out the roots of these operations, not other mounts. +//! +//! The overlay's options name its layers' and its upper's directories by these paths, and +//! `mount(2)` passes a file system at most [`MOUNT_OPTIONS_LIMIT`] bytes of options. An assembly +//! refuses longer options, which a long runtime directory makes, before it mounts anything, +//! rather than let the kernel cut them short. //! //! # Live shares and mappings //! @@ -90,7 +98,8 @@ //! //! use nvx::guest::cgroup::Layout; //! use nvx::guest::mappings::Access; -//! use nvx::guest::root::{self, Assembly, Device, Resources}; +//! use nvx::guest::root::{self, Assembly, Device, Image, OverlayMode, Resources, Upper}; +//! use nvx::protocol::root::DISTRO_MMIO_BASE; //! //! # fn main() -> Result<(), Box> { //! let layout = Layout::new("/sys/fs/cgroup/agent", "/sys/fs/cgroup/container")?; @@ -101,11 +110,13 @@ //! let bootstrap = root::bootstrap(&layout, &resources)?; //! let mut root = root::assemble(&Assembly { //! runtime: "/run/nvx".into(), -//! distro: Device { mmio: 0xd000_0000, sectors: None }, -//! distro_uuid: None, -//! scratch: Device { mmio: 0xd000_1000, sectors: None }, +//! distro: Device { mmio: DISTRO_MMIO_BASE, sectors: None }, +//! image: Image::Erofs { uuid: Some("8d2f3a4e-1c5b-4e7a-9f60-2b3c4d5e6f70".parse()?) }, +//! upper: Upper::Tmpfs, +//! overlay: OverlayMode::UntrustedLowers, //! timeout: Duration::from_secs(30), //! })?; +//! println!("{} layer over a {:?} upper", root.lowers().len(), root.upper()); //! root.mount_share("workspace", Path::new("/workspace"), Access::ReadWrite)?; //! // Run the workload in `bootstrap.workload()`, in the root at `root.path()`; then: //! root.teardown()?; @@ -140,8 +151,10 @@ pub use self::share::{ShareChild, ShareKind, ShareRefusal}; /// The EROFS image's mount point, below the runtime directory. const DISTRO_DIRECTORY: &str = "layers/distro"; -/// The scratch's mount point, below the runtime directory. +/// The ext4 scratch's mount point, below the runtime directory. const SCRATCH_DIRECTORY: &str = "scratch"; +/// The tmpfs upper's mount point, below the runtime directory. +const TMPFS_DIRECTORY: &str = "tmpfs"; /// The overlay's mount point, the root, below the runtime directory. const ROOT_DIRECTORY: &str = "rootfs"; /// The mount point of an aggregate live share, below the runtime directory and outside the root, @@ -150,20 +163,28 @@ const AGGREGATE_DIRECTORY: &str = "shares"; /// The mount point of a live share before it moves to its target, below the runtime directory /// and outside the root, which exists once a root has mounted a share. const STAGING_DIRECTORY: &str = "staging"; -/// The overlay's upper directory, below the scratch. +/// The overlay's upper directory, below the upper's mount point. const UPPER_DIRECTORY: &str = "upper"; -/// The overlay's work directory, below the scratch. +/// The overlay's work directory, below the upper's mount point. const WORK_DIRECTORY: &str = "work"; -/// The mode of the scratch's `upper` and `work` directories that an assembly creates, whatever +/// The mode of the upper's `upper` and `work` directories that an assembly creates, whatever /// the process's umask, as the shell init's `mkdir -p` created them under PID 1's: the overlay's /// root directory takes the upper's mode, and a workload that is not root must enter its root. const SCRATCH_DIRECTORY_MODE: u32 = 0o755; -/// The overlay's options other than its directories: metadata-only copy-up, and inode numbers -/// that stay unique across the layers. -const OVERLAY_OPTIONS: &[u8] = b"metacopy=on,xino=on"; +/// The tmpfs upper's options: the mode of its root directory, which an ext4 scratch's root has +/// too, rather than tmpfs's world-writable default. Its size is the kernel's default. +const TMPFS_OPTIONS: &[u8] = b"mode=0755"; /// The characters that a runtime directory may not hold: the separators and escape of the /// overlay's options, and the C string's terminator. const RUNTIME_RESERVED: &[u8] = b",:\\=\0"; +/// The directory of the overlay module's parameters, below sysfs, which hold the defaults of the +/// overlay's options. +const OVERLAY_PARAMETERS: &str = "module/overlay/parameters"; + +/// The most bytes of options that `mount(2)` passes a file system whole: one 4 KiB page, the +/// smallest into which Linux copies them, less the NUL that ends them. Linux cuts longer options +/// short, so [`assemble`] refuses an overlay whose options are longer. +pub const MOUNT_OPTIONS_LIMIT: usize = 4095; /// The role of a block device in a layered root. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] @@ -175,6 +196,14 @@ pub enum Role { Scratch, } +impl Role { + /// Returns whether the role's device is read-only: the distro's is, and the scratch's is + /// writable. + const fn read_only(self) -> bool { + matches!(self, Self::Distro) + } +} + impl fmt::Display for Role { /// Writes `distro` or `scratch`. fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { @@ -203,8 +232,10 @@ pub enum DeviceStep { Size, /// Flushing its buffers. Flush, - /// Reading its EROFS superblock. + /// Reading its EROFS or ext4 superblock. Superblock, + /// Reading its access mode: read-only or writable. + Access, } impl fmt::Display for DeviceStep { @@ -214,6 +245,7 @@ impl fmt::Display for DeviceStep { Self::Size => "read the size of", Self::Flush => "flush the buffers of", Self::Superblock => "read the superblock of", + Self::Access => "read the access mode of", }) } } @@ -226,38 +258,139 @@ pub struct Assembly { /// [`MAX_PATH_BYTES`](super::mappings::MAX_PATH_BYTES) bytes, that no symbolic link leads /// through, and without `,`, `:`, `\`, `=`, or NUL, which the overlay's options reserve. pub runtime: PathBuf, - /// The device of the EROFS image. + /// The distro device, which holds the image. pub distro: Device, - /// The UUID that the EROFS image's superblock must hold, if the caller expects one. - pub distro_uuid: Option, - /// The device of the ext4 scratch. - pub scratch: Device, + /// The image's format, and what the caller expects of it. + pub image: Image, + /// The upper, which holds the overlay's upper and work directories. + pub upper: Upper, + /// The overlay's mode, which selects its option set. + pub overlay: OverlayMode, /// How long to wait for each device to appear. pub timeout: Duration, } +/// The image on the distro device, in the format that the caller names: an assembly never probes +/// the device for another. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub enum Image { + /// An EROFS file system: the overlay's one lower layer, which an assembly mounts read-only at + /// `layers/distro`. + Erofs { + /// The UUID that the file system's superblock must hold, if the caller expects one. + uuid: Option, + }, +} + +/// The file system that holds the overlay's upper and work directories, two separate +/// directories on it. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Upper { + /// A whole-device ext4 file system on the scratch device, which an assembly mounts + /// `rw,nosuid,nodev` at `scratch`, and whose directories persist across boots. + Scratch { + /// The scratch device. + device: Device, + /// Whether the file system may have a journal. + journal: Journal, + }, + /// A tmpfs at the kernel's default size, which an assembly mounts `nosuid,nodev` at `tmpfs`, + /// for a VM that has no scratch device. + Tmpfs, +} + +impl Upper { + /// Returns the upper's kind. + pub const fn kind(self) -> UpperKind { + match self { + Self::Scratch { .. } => UpperKind::Scratch, + Self::Tmpfs => UpperKind::Tmpfs, + } + } +} + +/// The kind of a root's upper, which [`Root::upper`] reports. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum UpperKind { + /// An ext4 scratch, on the scratch device. + Scratch, + /// A tmpfs. + Tmpfs, +} + +/// Whether an ext4 scratch may have a journal, which [`assemble`] checks in the feature flags of +/// its superblock before it mounts it. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Journal { + /// The scratch may have a journal or none, so the assembly reads nothing to check it. + Allowed, + /// The scratch may not have a journal: the assembly refuses a scratch whose ext4 superblock + /// has the `has_journal` feature, and one that holds no ext4 superblock. + Forbidden, +} + +/// The overlay's mode, which selects its one option set. The kernel's refusal of the set fails +/// the assembly, which never falls back to another set, and so does an overlay that the kernel +/// mounts without the set in effect. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum OverlayMode { + /// `metacopy=on,xino=on`, the default, for lowers whose overlay xattrs the caller trusts: a + /// change of a lower file's metadata copies up its metadata alone, renaming a lower directory + /// records a redirect, and inode numbers stay unique across the layers. + #[default] + TrustedLowers, + /// `metacopy=off,redirect_dir=nofollow`, for lowers that may carry untrusted overlay xattrs: + /// the overlay neither creates metacopy and redirect xattrs nor follows a lower's, so a + /// copy-up copies a file's data too, and renaming a lower directory fails with `EXDEV`. + UntrustedLowers, +} + +impl OverlayMode { + /// Returns the mode's option set, which follows the overlay's directories in its options. + pub const fn options(self) -> &'static str { + match self { + Self::TrustedLowers => "metacopy=on,xino=on", + Self::UntrustedLowers => "metacopy=off,redirect_dir=nofollow", + } + } +} + /// Assembles the layered root of `assembly`, as the [module](self) describes, and returns the -/// root that owns its mounts. In order, it: +/// root that owns its mounts. It checks everything before it mounts anything. In order, it: /// /// 1. checks the runtime directory's path; /// 2. creates the runtime directory, as `mkdir -p` does, and refuses it if a symbolic link leads /// through it; -/// 3. creates its `layers/distro`, `scratch`, and `rootfs` directories, a component at a time, and -/// refuses a component that exists as anything but a directory, a symbolic link included; -/// takes the runtime directory's exclusive lock, which the root holds from then on, and -/// refuses the runtime directory if another root holds it, or if one of the three, or -/// `shares`, where a root mounts an aggregate live share, is a mount point already, whether of -/// a file system or of a bind mount of a directory; -/// 4. finds the distro device, polling for it every 50 ms until `timeout` passes, checks its -/// size if the caller expects one, and flushes its buffers; -/// 5. checks the expected UUID, if any, in the device's EROFS superblock, and reads nothing else; -/// 6. mounts the device's EROFS image read-only at `layers/distro`; -/// 7. finds, checks, and flushes the scratch device the same way; -/// 8. mounts its ext4 file system at `scratch` with `nosuid` and `nodev`; -/// 9. creates the scratch's `upper` and `work` directories unless they exist, with mode 0755 -/// whatever the process's umask, since the overlay's root directory takes the upper's mode; -/// 10. and mounts the overlay at `rootfs`, `nosuid` and `nodev`, with the distro layer as its -/// lower directory, and `metacopy=on,xino=on`. +/// 3. creates the mount points of the EROFS image, `layers/distro`, of the upper, `scratch` or +/// `tmpfs`, and of the root, `rootfs`, a component at a time, and refuses a component that +/// exists as anything but a directory, a symbolic link included; takes the runtime directory's +/// exclusive lock, which the root holds from then on, and refuses the runtime directory if +/// another root holds it, or if one of these, the other upper's, or `shares`, where a root +/// mounts an aggregate live share, is a mount point already, whether of a file system or of a +/// bind mount of a directory; +/// 4. finds the distro device, polling for it every 50 ms until `timeout` passes, checks that it +/// is read-only and, if the caller expects a size, of that size, and flushes its buffers; +/// 5. checks the expected UUID, if any, in the EROFS image's superblock, and reads nothing else; +/// 6. for an ext4 scratch, finds, checks, and flushes the scratch device as step 4 does, except +/// that it must be writable, and, if the caller forbids a journal, checks that its superblock +/// lacks the `has_journal` feature; +/// 7. checks that the overlay's options fit in [`MOUNT_OPTIONS_LIMIT`] bytes; +/// 8. mounts the EROFS image read-only at `layers/distro`; +/// 9. mounts the upper with `nosuid` and `nodev`: the scratch's ext4 file system, or a tmpfs +/// whose root directory has mode 0755; +/// 10. creates the upper's `upper` and `work` directories unless they exist, with mode 0755 +/// whatever the process's umask, since the overlay's root directory takes the upper's mode; +/// 11. mounts the overlay at `rootfs`, `nosuid` and `nodev`, with the layers as its lower +/// directories, the top first, and the option set of the overlay mode; +/// 12. and checks, in the kernel's mount table, that the overlay is writable and has each option +/// of the set in effect, where an option that the table doesn't show has its default, which +/// a parameter of the overlay module holds. Without failing the mount, the kernel mounts an +/// overlay read-only when it can't create its work directory, and drops metadata copy-up and +/// redirects when the upper can't hold the overlay's extended attributes. /// /// A failure unmounts what the assembly mounted, as [`Root::teardown`] does, and reports both. pub fn assemble(assembly: &Assembly) -> Result { @@ -286,8 +419,10 @@ fn assemble_with( runtime: assembly.runtime.clone(), path: assembly.runtime.join(ROOT_DIRECTORY), overlay: None, - distro: None, - scratch: None, + lowers: Vec::new(), + upper: None, + upper_kind: assembly.upper.kind(), + overlay_mode: assembly.overlay, shares: Vec::new(), mapper: None, mapped: Vec::new(), @@ -303,8 +438,36 @@ fn assemble_with( } } -/// The handle of an assembled layered root, which owns its mounts: those of the assembly, and -/// those of its live shares and mappings. +/// A read-only layer of an assembled root, one of the overlay's lower directories. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Lower { + /// The layer's mount. + mount: Mount, + /// The node of the layer's block device. + device: PathBuf, + /// The layer's mount point. + path: PathBuf, +} + +impl Lower { + /// Returns the layer's mount: [`Mount::Distro`] for an EROFS image. + pub fn mount(&self) -> &Mount { + &self.mount + } + + /// Returns the node of the layer's block device: the distro device that holds an EROFS image. + pub fn device(&self) -> &Path { + &self.device + } + + /// Returns the layer's mount point, below the runtime directory. + pub fn path(&self) -> &Path { + &self.path + } +} + +/// The handle of an assembled layered root, which owns its mounts, those of the assembly and +/// those of its live shares and mappings, and reports what the assembly mounted. /// /// [`teardown`](Self::teardown) unmounts them and reports each failure. Dropping the root /// unmounts them the same way, and ignores failures. Until then, the root holds the lock of its @@ -318,10 +481,14 @@ pub struct Root { path: PathBuf, /// The overlay's mount point, while it is mounted. overlay: Option, - /// The EROFS image's mount point, while it is mounted. - distro: Option, - /// The scratch's mount point, while it is mounted. - scratch: Option, + /// The mounted layers, in the order of their mounts: the base first. + lowers: Vec, + /// The upper's mount and mount point, while it is mounted. + upper: Option<(Mount, PathBuf)>, + /// The upper's kind. + upper_kind: UpperKind, + /// The overlay's mode. + overlay_mode: OverlayMode, /// The mounts of live shares that the root unmounts by path, and their mount points, in mount /// order: an aggregate's, and a share's that stays at `staging` because its unmount failed. shares: Vec<(Mount, PathBuf)>, @@ -354,10 +521,26 @@ impl Root { self.latched } + /// Returns the overlay's layers, the base first: the EROFS image's one layer. + pub fn lowers(&self) -> &[Lower] { + &self.lowers + } + + /// Returns the kind of the root's upper. + pub fn upper(&self) -> UpperKind { + self.upper_kind + } + + /// Returns the overlay's mode, whose option set, [`OverlayMode::options`], the assembly found + /// in effect. + pub fn overlay(&self) -> OverlayMode { + self.overlay_mode + } + /// Unmounts the root's mounts, in order: a live share or a mapping that a rename moved and /// that its mapper failed to detach, which the mapper detaches, the live shares and mappings - /// in reverse order, the mount of an aggregate live share, the overlay, the EROFS image, and - /// the scratch. + /// in reverse order, the mount of an aggregate live share, the overlay, the layers from the + /// top down, and the upper. /// /// Each unmount is attempted, whatever failed before, without detaching a busy mount but the /// moved one, and every one that failed is reported, with the mount that it leaves in place. @@ -384,27 +567,27 @@ impl Root { }); } } - let distro = create_below(&runtime, DISTRO_DIRECTORY)?; - let scratch = create_below(&runtime, SCRATCH_DIRECTORY)?; - create_below(&runtime, ROOT_DIRECTORY)?; + let distro_point = create_below(&runtime, DISTRO_DIRECTORY)?; + let upper_point = create_below( + &runtime, + match assembly.upper.kind() { + UpperKind::Scratch => SCRATCH_DIRECTORY, + UpperKind::Tmpfs => TMPFS_DIRECTORY, + }, + )?; + let root_point = create_below(&runtime, ROOT_DIRECTORY)?; self.reservation = Some(reserve(&runtime)?); - let shares = runtime.join(AGGREGATE_DIRECTORY); - for directory in [&distro, &scratch, &self.path, &shares] { - match self.kernel.is_mount_point(directory) { - Ok(false) => {} - Ok(true) => { - return Err(Error::Occupied { - path: directory.clone(), - }); - } - Err(error) if error.kind() == io::ErrorKind::NotFound && *directory == shares => {} - Err(error) => { - return Err(Error::MountPoint { - path: directory.clone(), - error, - }); - } - } + for point in [&distro_point, &upper_point, &root_point] { + self.check_unoccupied(point, false)?; + } + // A root whose teardown failed may have left mounted an upper of the other kind, or an + // aggregate share, which this root mounts only later. + let other_upper = match assembly.upper.kind() { + UpperKind::Scratch => TMPFS_DIRECTORY, + UpperKind::Tmpfs => SCRATCH_DIRECTORY, + }; + for point in [other_upper, AGGREGATE_DIRECTORY] { + self.check_unoccupied(&runtime.join(point), true)?; } let device = block::prepare( self.kernel.as_mut(), @@ -413,80 +596,154 @@ impl Root { assembly.distro, assembly.timeout, )?; - if let Some(expected) = assembly.distro_uuid { - match block::erofs_uuid(&device) { - Ok(Some(actual)) if actual == expected => {} - Ok(Some(actual)) => return Err(Error::Uuid { expected, actual }), - Ok(None) => return Err(Error::NotErofs { device }), - Err(error) => { - return Err(Error::Device { - role: Role::Distro, - step: DeviceStep::Superblock, - device, - error, - }); + let Image::Erofs { uuid } = &assembly.image; + check_erofs_uuid(&device, *uuid)?; + let lowers = vec![Lower { + mount: Mount::Distro, + device, + path: distro_point, + }]; + let scratch = match assembly.upper { + Upper::Scratch { device, journal } => { + let device = block::prepare( + self.kernel.as_mut(), + paths, + Role::Scratch, + device, + assembly.timeout, + )?; + if journal == Journal::Forbidden { + check_no_journal(&device)?; } + Some(device) } + Upper::Tmpfs => None, + }; + let upper = upper_point.join(UPPER_DIRECTORY); + let work = upper_point.join(WORK_DIRECTORY); + let options = overlay_options(&lowers, &upper, &work, assembly.overlay)?; + for lower in lowers { + self.mount_at( + &lower.mount, + Some(lower.device.as_os_str().as_bytes()), + &lower.path, + Some(b"erofs"), + MountFlags::MS_RDONLY, + None, + )?; + self.lowers.push(lower); } - let source = device.as_os_str().as_bytes(); - self.mount_at( - &Mount::Distro, - Some(source), - &distro, - Some(b"erofs"), - MountFlags::MS_RDONLY, - None, - )?; - self.distro = Some(distro.clone()); - let device = block::prepare( - self.kernel.as_mut(), - paths, - Role::Scratch, - assembly.scratch, - assembly.timeout, - )?; - self.mount_at( - &Mount::Scratch, - Some(device.as_os_str().as_bytes()), - &scratch, - Some(b"ext4"), - MountFlags::MS_NOSUID | MountFlags::MS_NODEV, - None, - )?; - self.scratch = Some(scratch.clone()); - let upper = scratch.join(UPPER_DIRECTORY); - let work = scratch.join(WORK_DIRECTORY); + // Neither set-ID programs nor device nodes of the upper take effect. + let flags = MountFlags::MS_NOSUID | MountFlags::MS_NODEV; + let mount = match scratch { + Some(device) => { + self.mount_at( + &Mount::Scratch, + Some(device.as_os_str().as_bytes()), + &upper_point, + Some(b"ext4"), + flags, + None, + )?; + Mount::Scratch + } + None => { + self.mount_at( + &Mount::Tmpfs, + Some(b"tmpfs"), + &upper_point, + Some(b"tmpfs"), + flags, + Some(TMPFS_OPTIONS), + )?; + Mount::Tmpfs + } + }; + self.upper = Some((mount, upper_point)); for directory in [&upper, &work] { create_scratch_directory(directory).map_err(|error| Error::MountPoint { path: directory.clone(), error, })?; } - let options = [ - b"lowerdir=".as_slice(), - distro.as_os_str().as_bytes(), - b",upperdir=", - upper.as_os_str().as_bytes(), - b",workdir=", - work.as_os_str().as_bytes(), - b",", - OVERLAY_OPTIONS, - ] - .concat(); let root = self.path.clone(); - // Neither set-ID programs nor device nodes of the image or the upper take effect. + // Neither set-ID programs nor device nodes of the layers or the upper take effect. self.mount_at( &Mount::Overlay, Some(b"overlay"), &root, Some(b"overlay"), - MountFlags::MS_NOSUID | MountFlags::MS_NODEV, + flags, Some(&options), )?; - self.overlay = Some(root); + self.overlay = Some(root.clone()); + self.check_overlay(paths, &root, assembly.overlay) + } + + /// Checks that the kernel mounted the overlay at `root` as the assembly asked: writable, and + /// with each option of `mode`'s set in effect. The kernel's mount table shows only those of + /// the overlay's options that differ from their defaults, so an option that it doesn't show + /// has the default that a parameter of the overlay module holds. + fn check_overlay( + &mut self, + paths: &Paths, + root: &Path, + mode: OverlayMode, + ) -> Result<(), Error> { + let options = self + .kernel + .mount_options(root) + .map_err(|error| Error::OverlayCheck { + path: root.to_owned(), + error, + })?; + let mut options = options.split(','); + let access = options.next().unwrap_or_default(); + if access != "rw" { + return Err(Error::OverlayFallback { + requested: "rw".to_owned(), + effective: access.to_owned(), + }); + } + let shown: Vec<(&str, &str)> = options + .filter_map(|option| option.split_once('=')) + .collect(); + for option in mode.options().split(',') { + let (key, value) = option.split_once('=').unwrap_or((option, "")); + let effective = match shown.iter().find(|(shown, _)| *shown == key) { + Some(&(_, effective)) => Some(effective), + None => overlay_default(paths, key)?, + }; + if effective != Some(value) { + return Err(Error::OverlayFallback { + requested: option.to_owned(), + effective: effective.map_or_else( + || format!("no {key}"), + |effective| format!("{key}={effective}"), + ), + }); + } + } Ok(()) } + /// Refuses the directory `path` where a file system, or a bind mount of a directory, is + /// mounted already, and one that can't be checked, unless `may_be_missing` and it doesn't + /// exist. + fn check_unoccupied(&mut self, path: &Path, may_be_missing: bool) -> Result<(), Error> { + match self.kernel.is_mount_point(path) { + Ok(false) => Ok(()), + Ok(true) => Err(Error::Occupied { + path: path.to_owned(), + }), + Err(error) if may_be_missing && error.kind() == io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(Error::MountPoint { + path: path.to_owned(), + error, + }), + } + } + /// Mounts `source` at `target` with `filesystem`, `flags`, and the file system's own options /// `data`, and reports a failure as one to mount `mount`. fn mount_at( @@ -539,21 +796,15 @@ impl Root { }); } } - let assembly = [ - (Mount::Overlay, self.overlay.take()), - (Mount::Distro, self.distro.take()), - (Mount::Scratch, self.scratch.take()), - ]; - let mounts: Vec<(Mount, PathBuf)> = self - .shares - .drain(..) - .rev() - .chain( - assembly - .into_iter() - .filter_map(|(mount, path)| Some((mount, path?))), - ) - .collect(); + let mut mounts: Vec<(Mount, PathBuf)> = self.shares.drain(..).rev().collect(); + mounts.extend(self.overlay.take().map(|path| (Mount::Overlay, path))); + mounts.extend( + self.lowers + .drain(..) + .rev() + .map(|lower| (lower.mount, lower.path)), + ); + mounts.extend(self.upper.take()); for (mount, path) in mounts { let unmounted = sys::path_c_string(&path).and_then(|target| self.kernel.unmount(&target)); @@ -587,8 +838,10 @@ impl fmt::Debug for Root { .field("runtime", &self.runtime) .field("path", &self.path) .field("overlay", &self.overlay) - .field("distro", &self.distro) - .field("scratch", &self.scratch) + .field("overlay_mode", &self.overlay_mode) + .field("lowers", &self.lowers) + .field("upper", &self.upper) + .field("upper_kind", &self.upper_kind) .field("shares", &self.shares) .field("mapped", &self.mapped) .field("latched", &self.latched) @@ -596,6 +849,105 @@ impl fmt::Debug for Root { } } +/// Checks the EROFS image on the distro device `device` for the UUID `expected`, if the caller +/// expects one, in its superblock, and reads nothing else. +fn check_erofs_uuid(device: &Path, expected: Option) -> Result<(), Error> { + let Some(expected) = expected else { + return Ok(()); + }; + match block::erofs_uuid(device) { + Ok(Some(actual)) if actual == expected => Ok(()), + Ok(Some(actual)) => Err(Error::Uuid { expected, actual }), + Ok(None) => Err(Error::NotErofs { + device: device.to_owned(), + }), + Err(error) => Err(Error::Device { + role: Role::Distro, + step: DeviceStep::Superblock, + device: device.to_owned(), + error, + }), + } +} + +/// Checks that the ext4 file system on the scratch device `device` has no journal, in its +/// superblock's feature flags, and reads nothing else. +fn check_no_journal(device: &Path) -> Result<(), Error> { + match block::ext4_has_journal(device) { + Ok(Some(false)) => Ok(()), + Ok(Some(true)) => Err(Error::Journal { + device: device.to_owned(), + }), + Ok(None) => Err(Error::NotExt4 { + device: device.to_owned(), + }), + Err(error) => Err(Error::Device { + role: Role::Scratch, + step: DeviceStep::Superblock, + device: device.to_owned(), + error, + }), + } +} + +/// Returns the overlay's options: its lower directories, the mount points of `lowers` from the +/// top down, its `upper` and `work` directories, and the option set of `mode`, or fails if they +/// are longer than [`MOUNT_OPTIONS_LIMIT`]. +fn overlay_options( + lowers: &[Lower], + upper: &Path, + work: &Path, + mode: OverlayMode, +) -> Result, Error> { + let mut options = b"lowerdir=".to_vec(); + for (index, lower) in lowers.iter().rev().enumerate() { + if index > 0 { + options.push(b':'); + } + options.extend_from_slice(lower.path.as_os_str().as_bytes()); + } + for (name, directory) in [(&b",upperdir="[..], upper), (b",workdir=", work)] { + options.extend_from_slice(name); + options.extend_from_slice(directory.as_os_str().as_bytes()); + } + options.push(b','); + options.extend_from_slice(mode.options().as_bytes()); + if options.len() > MOUNT_OPTIONS_LIMIT { + return Err(Error::OverlayOptions { + length: options.len(), + }); + } + Ok(options) +} + +/// Returns the default of the overlay's option `key`, which a parameter of the overlay module +/// holds, as the kernel shows the option's value, or `None` for an option without one. +fn overlay_default(paths: &Paths, key: &str) -> Result, Error> { + let parameter = |name: &str| { + let path = paths.sys.join(OVERLAY_PARAMETERS).join(name); + match fs::read(&path) { + Ok(value) if value == b"Y\n" => Ok(true), + Ok(value) if value == b"N\n" => Ok(false), + Ok(_) => Err(Error::OverlayCheck { + path, + error: Errno::EPROTO.into(), + }), + Err(error) => Err(Error::OverlayCheck { path, error }), + } + }; + let default = match key { + "metacopy" if parameter("metacopy")? => "on", + "metacopy" => "off", + "xino" if parameter("xino_auto")? => "auto", + "xino" => "off", + "redirect_dir" if parameter("redirect_dir")? => "on", + "redirect_dir" if parameter("redirect_always_follow")? => "follow", + "redirect_dir" => "nofollow", + _ => return Ok(None), + }; + Ok(Some(default)) +} + /// Takes the exclusive lock of the runtime directory `runtime`, without waiting for another root /// that holds it. fn reserve(runtime: &Path) -> Result { @@ -658,8 +1010,10 @@ fn create_scratch_directory(path: &Path) -> io::Result<()> { pub enum Mount { /// The EROFS image, at `layers/distro` in the runtime directory. Distro, - /// The scratch, at `scratch` in the runtime directory. + /// The ext4 scratch, at `scratch` in the runtime directory. Scratch, + /// The tmpfs upper, at `tmpfs` in the runtime directory. + Tmpfs, /// The overlay, which is the root, at `rootfs` in the runtime directory. Overlay, /// A live share, at its target in the root. @@ -693,6 +1047,7 @@ impl fmt::Display for Mount { match self { Self::Distro => formatter.write_str("the distro layer"), Self::Scratch => formatter.write_str("the scratch"), + Self::Tmpfs => formatter.write_str("the tmpfs upper"), Self::Overlay => formatter.write_str("the overlay"), Self::Share { tag, target } => { write!(formatter, "the live share {tag} at {}", target.display()) @@ -741,8 +1096,8 @@ pub enum Error { /// The runtime directory. path: PathBuf, }, - /// A file system, or a bind mount of a directory, is mounted already where the assembly - /// mounts, such as one that a failed teardown left. + /// A file system, or a bind mount of a directory, is mounted already at one of the runtime + /// directory's mount points, such as one that a failed teardown left. Occupied { /// The mount point. path: PathBuf, @@ -774,6 +1129,16 @@ pub enum Error { /// The device's size, in 512-byte sectors. actual: u64, }, + /// A device's access mode is not its role's: the distro device must be read-only, and the + /// scratch device writable. + Access { + /// The device's role. + role: Role, + /// The device's node. + device: PathBuf, + /// Whether the device is read-only. + read_only: bool, + }, /// A step on a device failed. Device { /// The device's role. @@ -797,6 +1162,41 @@ pub enum Error { /// The image's UUID. actual: Uuid, }, + /// The scratch device holds no ext4 file system, so it has no journal feature to check. + NotExt4 { + /// The device's node. + device: PathBuf, + }, + /// The scratch's ext4 file system has a journal, which the caller forbids. + Journal { + /// The device's node. + device: PathBuf, + }, + /// The overlay's options are longer than [`MOUNT_OPTIONS_LIMIT`], as the paths of a long + /// runtime directory make them. + OverlayOptions { + /// The options' length, in bytes. + length: usize, + }, + /// The kernel mounted the overlay, but not as the assembly asked: read-only, or without an + /// option of the overlay mode's set in effect. The kernel does so, without failing the mount, + /// when it can't create the overlay's work directory, or when the upper can't hold the + /// overlay's extended attributes. + OverlayFallback { + /// What the assembly asked for: `rw`, or an option of the overlay mode's set, such as + /// `metacopy=on`. + requested: String, + /// What the kernel applied in its place, such as `ro` or `metacopy=off`. + effective: String, + }, + /// The overlay's effective options could not be read: its options in the kernel's mount + /// table, or the default of one of them, which a parameter of the overlay module holds. + OverlayCheck { + /// The overlay's mount point, or the parameter's file. + path: PathBuf, + /// The error. + error: io::Error, + }, /// A mount failed. Mount { /// The mount that failed. @@ -859,6 +1259,22 @@ impl fmt::Display for Error { "the {role} device {} has {actual} sectors, expected {expected}", device.display() ), + Self::Access { + role, + device, + read_only, + } => { + let (actual, expected) = if *read_only { + ("read-only", "writable") + } else { + ("writable", "read-only") + }; + write!( + formatter, + "the {role} device {} is {actual}, expected {expected}", + device.display() + ) + } Self::Device { role, step, @@ -878,6 +1294,33 @@ impl fmt::Display for Error { formatter, "the distro layer's UUID is {actual}, expected {expected}" ), + Self::NotExt4 { device } => write!( + formatter, + "the scratch device {} holds no ext4 file system", + device.display() + ), + Self::Journal { device } => write!( + formatter, + "the scratch device {} has an ext4 journal, which the assembly forbids", + device.display() + ), + Self::OverlayOptions { length } => write!( + formatter, + "the overlay's options take {length} bytes, more than the \ + {MOUNT_OPTIONS_LIMIT} that a mount passes" + ), + Self::OverlayFallback { + requested, + effective, + } => write!( + formatter, + "the kernel mounted the overlay with {effective} in place of {requested}" + ), + Self::OverlayCheck { path, error } => write!( + formatter, + "failed to read the overlay's effective options at {}: {error}", + path.display() + ), Self::Mount { mount, error } => write!(formatter, "failed to mount {mount}: {error}"), Self::Mapping { mount, error } => write!(formatter, "failed to mount {mount}: {error}"), Self::Share(refusal) => refusal.fmt(formatter), diff --git a/nvx-rs/src/guest/root/block.rs b/nvx-rs/src/guest/root/block.rs index 5d3309deb..ace489a45 100644 --- a/nvx-rs/src/guest/root/block.rs +++ b/nvx-rs/src/guest/root/block.rs @@ -29,11 +29,22 @@ const EROFS_SUPERBLOCK: u64 = 1024; const EROFS_MAGIC: u32 = 0xE0F5_E1E2; /// The range of the file system's UUID in the EROFS superblock. const EROFS_UUID: std::ops::Range = 48..64; +/// The byte offset of the ext4 superblock on its device. +const EXT4_SUPERBLOCK: u64 = 1024; +/// The range of the ext4 superblock's magic number, a little-endian `u16`. +const EXT4_MAGIC_RANGE: std::ops::Range = 0x38..0x3A; +/// The ext4 superblock's magic number. +const EXT4_MAGIC: u16 = 0xEF53; +/// The range of the ext4 superblock's compatible feature flags, a little-endian `u32`. +const EXT4_COMPAT_RANGE: std::ops::Range = 0x5C..0x60; +/// The compatible feature flag of an ext4 file system that has a journal, `has_journal`. +const EXT4_HAS_JOURNAL: u32 = 0x0004; /// Finds the block device of `role` that the VMM attached at `device`'s MMIO window, polling -/// until it appears for as long as `timeout` allows, checks its size if the caller expects one, -/// and flushes its buffers, so that later reads come from the device. Returns the path of its -/// node. +/// until it appears for as long as `timeout` allows, checks that its access mode is the role's, +/// read-only for the distro and writable for the scratch, checks its size if the caller expects +/// one, and flushes its buffers, so that later reads come from the device. Returns the path of +/// its node. pub(super) fn prepare( kernel: &mut dyn Kernel, paths: &Paths, @@ -42,9 +53,28 @@ pub(super) fn prepare( timeout: Duration, ) -> Result { let (name, path) = resolve(kernel, paths, role, device.mmio, timeout)?; + let attributes = paths.sys.join("class/block").join(&name); + let read_only = fs::read(attributes.join("ro")) + .and_then(|text| match parse_count(&text)? { + 0 => Ok(false), + 1 => Ok(true), + _ => Err(sys::Errno::EPROTO.into()), + }) + .map_err(|error| Error::Device { + role, + step: DeviceStep::Access, + device: path.clone(), + error, + })?; + if read_only != role.read_only() { + return Err(Error::Access { + role, + device: path, + read_only, + }); + } if let Some(expected) = device.sectors { - let size = paths.sys.join("class/block").join(&name).join("size"); - let actual = fs::read(&size) + let actual = fs::read(attributes.join("size")) .and_then(|text| parse_count(&text)) .map_err(|error| Error::Device { role, @@ -187,6 +217,32 @@ pub(super) fn erofs_uuid(device: &Path) -> io::Result> { ))) } +/// Reads whether the ext4 file system on the block device `device` has a journal, from its +/// superblock's compatible feature flags, and reads nothing else, or returns `None` if the device +/// holds no ext4 superblock. +pub(super) fn ext4_has_journal(device: &Path) -> io::Result> { + let mut superblock = [0; EXT4_COMPAT_RANGE.end]; + match File::open(device)?.read_exact_at(&mut superblock, EXT4_SUPERBLOCK) { + Ok(()) => {} + Err(error) if error.kind() == io::ErrorKind::UnexpectedEof => return Ok(None), + Err(error) => return Err(error), + } + let magic = u16::from_le_bytes( + superblock[EXT4_MAGIC_RANGE] + .try_into() + .expect("the magic number is two bytes"), + ); + if magic != EXT4_MAGIC { + return Ok(None); + } + let compatible = u32::from_le_bytes( + superblock[EXT4_COMPAT_RANGE] + .try_into() + .expect("the feature flags are four bytes"), + ); + Ok(Some(compatible & EXT4_HAS_JOURNAL != 0)) +} + #[cfg(test)] mod tests { use super::*; @@ -223,4 +279,31 @@ mod tests { io::ErrorKind::NotFound ); } + + #[test] + fn the_superblock_tells_whether_an_ext4_file_system_has_a_journal() { + let temporary = tempfile::tempdir().unwrap(); + let device = temporary.path().join("device"); + let mut image = vec![0u8; 2048]; + image[1024 + 0x38..1024 + 0x3A].copy_from_slice(&EXT4_MAGIC.to_le_bytes()); + // Other compatible features, such as `dir_index`, are not a journal. + image[1024 + 0x5C..1024 + 0x60].copy_from_slice(&0x0020_u32.to_le_bytes()); + fs::write(&device, &image).unwrap(); + assert_eq!(ext4_has_journal(&device).unwrap(), Some(false)); + image[1024 + 0x5C] |= 0x04; + fs::write(&device, &image).unwrap(); + assert_eq!(ext4_has_journal(&device).unwrap(), Some(true)); + // Another magic number, or a device too short for the feature flags, holds no ext4. + image[1024 + 0x38] ^= 1; + fs::write(&device, &image).unwrap(); + assert_eq!(ext4_has_journal(&device).unwrap(), None); + fs::write(&device, &image[..1024 + 0x5E]).unwrap(); + assert_eq!(ext4_has_journal(&device).unwrap(), None); + assert_eq!( + ext4_has_journal(&temporary.path().join("missing")) + .unwrap_err() + .kind(), + io::ErrorKind::NotFound + ); + } } diff --git a/nvx-rs/src/guest/root/kernel.rs b/nvx-rs/src/guest/root/kernel.rs index cd589f3de..54a0b8698 100644 --- a/nvx-rs/src/guest/root/kernel.rs +++ b/nvx-rs/src/guest/root/kernel.rs @@ -3,24 +3,31 @@ //! The operations read the kernel's text files and walk directories through the standard //! library, at the paths of a [`Paths`], so that a test relocates them to a temporary directory. //! What a test cannot do in a temporary directory, such as a mount, an unmount, or a flush of a -//! block device's buffers, and the writes to the kernel's control files and the check for a -//! mount point, whose effects and answers a test simulates, go through a [`Kernel`], which also -//! performs the mapper's operations for the root's live shares and mappings. +//! block device's buffers, and the writes to the kernel's control files, the check for a mount +//! point, and the options of a mount, whose effects and answers a test simulates, go through a +//! [`Kernel`], which also performs the mapper's operations for the root's live shares and +//! mappings. use std::ffi::CStr; +use std::fs; use std::io; use std::path::{Path, PathBuf}; use std::time::Duration; use crate::guest::mappings::Kernel as MappingKernel; +use crate::guest::parse::parse_decimal; use crate::guest::sys::{self, Errno, Mode, MountFlags, OpenFlags}; pub(crate) use crate::guest::mappings::SystemKernel; +/// The kernel's table of the calling process's mounts. +const MOUNT_TABLE: &str = "/proc/self/mountinfo"; + /// The kernel's operations that the root operations perform, other than reads: mounts and /// unmounts, flushes of a block device's buffers, writes of a control file, and the sleeps -/// between two polls for a block device; the check for a mount point, which depends on them; -/// and the mapper's operations, through which the root mounts its live shares and mappings. +/// between two polls for a block device; the check for a mount point and the options of a +/// mount, which depend on them; and the mapper's operations, through which the root mounts +/// its live shares and mappings. /// /// [`SystemKernel`] performs them. The tests' fake records the operations, simulates the effect /// of a cgroup's membership file and its own mounts, and fails the operations that a test @@ -52,6 +59,11 @@ pub(super) trait Kernel: MappingKernel + Send + Sync { /// of the parent's own file system is bound. fn is_mount_point(&mut self, path: &Path) -> io::Result; + /// Returns the options of the file system mounted at the mount point `path`, as the kernel's + /// mount table shows them: `ro` or `rw`, then those of the file system's own options that + /// differ from their defaults. + fn mount_options(&mut self, path: &Path) -> io::Result; + /// Sleeps for `duration`. fn sleep(&mut self, duration: Duration); } @@ -98,11 +110,33 @@ impl Kernel for SystemKernel { Ok(mount != sys::mount_id(&sys::path_c_string(parent)?)?) } + fn mount_options(&mut self, path: &Path) -> io::Result { + let mount = sys::mount_id(&sys::path_c_string(path)?)?; + let table = fs::read(MOUNT_TABLE)?; + let options = super_options(&table, mount).ok_or(Errno::ENOENT)?; + Ok(String::from_utf8_lossy(options).into_owned()) + } + fn sleep(&mut self, duration: Duration) { std::thread::sleep(duration); } } +/// Returns the options of the file system of the mount whose ID is `mount`, the last field of its +/// line in the mount table `table`, or `None` if the table has no such line. +/// +/// A line's fields are separated by spaces, which the kernel escapes within a field: the mount's +/// ID first, and its file system's type, source, and options after a field `-`. +pub(super) fn super_options(table: &[u8], mount: u64) -> Option<&[u8]> { + table.split(|&byte| byte == b'\n').find_map(|line| { + let mut fields = line.split(|&byte| byte == b' '); + if parse_decimal(fields.next()?)? != mount { + return None; + } + fields.skip_while(|field| *field != b"-").nth(3) + }) +} + /// The directories of the kernel's interfaces that the root operations read. #[derive(Clone, Debug, PartialEq, Eq)] pub(super) struct Paths { @@ -183,6 +217,8 @@ mod fake { MoveMount, /// A detachment of a tree, which its mount point names. Detach, + /// A read of a mount's options, which its mount point names. + Options, } /// One operation that the fake kernel recorded, with its paths and options as text. @@ -272,6 +308,9 @@ mod fake { trees: Vec, /// The last unique mount ID that the fake kernel gave, if any. last_mount: Option, + /// The options that the kernel shows for the mounts at some mount points, in place of + /// their own. + shown: Vec<(PathBuf, String)>, /// The sleeps so far. sleeps: usize, /// What happens after each sleep, given the number of sleeps so far. @@ -329,6 +368,10 @@ mod fake { /// the status of a file tells the fake kernel's mount of its path. A tree of `open_tree` is a /// memory file, which stands for the copy of a file's mount, or for that mount itself. Its /// clones share what they record, so a test keeps one while a root owns another. + /// + /// It shows a mount's options as `ro` or `rw`, as its flags have `MS_RDONLY` or not, then the + /// options that the mount passed, unless a test chose others, as for a kernel that applied + /// other options than those passed, or that shows options at their defaults. #[derive(Clone, Default)] pub(crate) struct FakeKernel(Arc>); @@ -359,6 +402,11 @@ mod fake { self.with(|state| state.unmovable.push((process, writes))); } + /// Makes the kernel show `options` as the options of the mounts at `path` from now on. + pub(in crate::guest::root) fn show(&self, path: impl Into, options: &str) { + self.with(|state| state.shown.push((path.into(), options.to_owned()))); + } + /// Runs `action` after each sleep, with the number of sleeps so far. pub(crate) fn on_sleep(&self, action: impl FnMut(usize) + Send + 'static) { self.with(|state| state.on_sleep = Some(Box::new(action))); @@ -691,6 +739,53 @@ mod fake { Ok(self.with(|state| state.mounted.iter().any(|(point, _)| point == path))) } + fn mount_options(&mut self, path: &Path) -> std::io::Result { + self.with(|state| { + let failure = state + .failures + .iter() + .find(|(failing, failing_path, _)| { + *failing == Operation::Options && failing_path == path + }) + .map(|&(_, _, error)| error); + if let Some(error) = failure { + return Err(error.into()); + } + if !state.mounted.iter().any(|(point, _)| point == path) { + return Err(Errno::ENOENT.into()); + } + if let Some((_, options)) = + state.shown.iter().rev().find(|(point, _)| point == path) + { + return Ok(options.clone()); + } + let (flags, data) = state + .calls + .iter() + .rev() + .find_map(|call| match call { + Call::Mount { + target, + flags, + data, + .. + } if Path::new(target) == path + && !flags.contains(MountFlags::MS_REMOUNT) => + { + Some((*flags, data.clone())) + } + _ => None, + }) + .ok_or(Errno::ENOENT)?; + let access = if flags.contains(MountFlags::MS_RDONLY) { + "ro" + } else { + "rw" + }; + Ok(data.map_or_else(|| access.to_owned(), |data| format!("{access},{data}"))) + }) + } + fn sleep(&mut self, _duration: Duration) { let (sleeps, mut action) = self.with(|state| { state.sleeps += 1; diff --git a/nvx-rs/src/guest/root/tests.rs b/nvx-rs/src/guest/root/tests.rs index d419b3769..442b83d03 100644 --- a/nvx-rs/src/guest/root/tests.rs +++ b/nvx-rs/src/guest/root/tests.rs @@ -9,7 +9,7 @@ use std::path::{Path, PathBuf}; use std::sync::{Arc, Mutex}; use std::time::Duration; -use super::kernel::{Call, FakeKernel, Operation, Paths}; +use super::kernel::{Call, FakeKernel, Operation, Paths, super_options}; use super::*; use crate::guest::cgroup::Layout; use crate::guest::mappings::{Access, SourceRoot}; @@ -21,7 +21,6 @@ const DISTRO: u64 = 0xd000_0000; const SCRATCH: u64 = 0x1_d000_1000; /// The UUID of the test's EROFS image. const UUID: Uuid = Uuid::from_bytes([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]); - /// A guest in a temporary directory, and the fake kernel that the operations get clones of. struct Guest { /// The temporary directory, which the guest holds only so that dropping it removes the @@ -66,6 +65,15 @@ impl Guest { SCRATCH + 0x1ff, ), ); + // The defaults of the overlay's options, as the guest's kernel sets them: all off. + for parameter in [ + "metacopy", + "redirect_dir", + "redirect_always_follow", + "xino_auto", + ] { + write(&paths.sys.join(OVERLAY_PARAMETERS).join(parameter), "N\n"); + } Self { _directory: directory, base, @@ -84,7 +92,8 @@ impl Guest { self.path(relative).display().to_string() } - /// Returns an assembly of the guest's devices, in its runtime directory, `run/nvx`. + /// Returns an assembly of the guest's devices, in its runtime directory, `run/nvx`: an EROFS + /// image over an ext4 scratch that may have a journal, with the default overlay mode. fn assembly(&self) -> Assembly { Assembly { runtime: self.path("run/nvx"), @@ -92,11 +101,9 @@ impl Guest { mmio: DISTRO, sectors: None, }, - distro_uuid: None, - scratch: Device { - mmio: SCRATCH, - sectors: None, - }, + image: Image::Erofs { uuid: None }, + upper: scratch_upper(Journal::Allowed), + overlay: OverlayMode::TrustedLowers, timeout: Duration::from_millis(100), } } @@ -105,6 +112,55 @@ impl Guest { fn assemble(&self, assembly: &Assembly) -> Result { assemble_with(Box::new(self.kernel.clone()), &self.paths, assembly) } + + /// Sets the overlay module's parameter `name`, the default of one of the overlay's options: + /// on, or off. + fn set_overlay_default(&self, name: &str, on: bool) { + let parameter = self.paths.sys.join(OVERLAY_PARAMETERS).join(name); + write(¶meter, if on { "Y\n" } else { "N\n" }); + } + + /// Makes the kernel show, as the options of the overlay of an assembly of the guest's EROFS + /// image over `upper`, its `access` and its directories, then `options`, if any. + fn show_overlay(&self, access: &str, upper: UpperKind, options: &str) { + let runtime = self.text("run/nvx"); + let point = upper_point(upper); + let mut shown = format!( + "{access},lowerdir={runtime}/layers/distro,upperdir={runtime}/{point}/upper,\ + workdir={runtime}/{point}/work" + ); + if !options.is_empty() { + shown = format!("{shown},{options}"); + } + self.kernel.show(self.path("run/nvx/rootfs"), &shown); + } +} + +/// Returns the mount point of an upper of `kind`, below the runtime directory. +fn upper_point(kind: UpperKind) -> &'static str { + match kind { + UpperKind::Scratch => "scratch", + UpperKind::Tmpfs => "tmpfs", + } +} + +/// Returns the upper on the guest's scratch device, with `journal`. +fn scratch_upper(journal: Journal) -> Upper { + Upper::Scratch { + device: Device { + mmio: SCRATCH, + sectors: None, + }, + journal, + } +} + +/// Returns the scratch device of `assembly`, whose upper must be an ext4 scratch. +fn scratch(assembly: &mut Assembly) -> &mut Device { + match &mut assembly.upper { + Upper::Scratch { device, .. } => device, + upper => panic!("{upper:?}"), + } } /// Writes `contents` to `path`, after creating its parent directories. @@ -114,14 +170,15 @@ fn write(path: &Path, contents: impl AsRef<[u8]>) { } /// Attaches the block device `name`, of `sectors` sectors, below the MMIO window `index` of the -/// guest in `root`: its sysfs directory, its size, and its node. +/// guest in `root`: its sysfs directory, its size and access mode, and its node. fn attach(root: &Path, index: usize, name: &str, sectors: u64) { probe(root, index, name, sectors); write(&root.join("dev").join(name), ""); } -/// Creates the sysfs directory and size of the block device `name` below the MMIO window -/// `index` of the guest in `root`, without its node. +/// Creates the sysfs directory, size, and access mode of the block device `name` below the MMIO +/// window `index` of the guest in `root`, without its node. The device is read-only at window 0, +/// where the tests attach the distro device, and writable elsewhere, as at the scratch's window 1. fn probe(root: &Path, index: usize, name: &str, sectors: u64) { let window = format!("sys/devices/virtio-mmio-cmdline/virtio-mmio.{index}"); fs::create_dir_all( @@ -129,12 +186,33 @@ fn probe(root: &Path, index: usize, name: &str, sectors: u64) { .join(format!("virtio{index}/block/{name}")), ) .unwrap(); + set_read_only(root, name, index == 0); write( &root.join("sys/class/block").join(name).join("size"), format!("{sectors}\n"), ); } +/// Sets the access mode of the block device `name` of the guest in `root`: read-only, or +/// writable. +fn set_read_only(root: &Path, name: &str, read_only: bool) { + write( + &root.join("sys/class/block").join(name).join("ro"), + if read_only { "1\n" } else { "0\n" }, + ); +} + +/// Returns the first 2 KiB of an ext4 file system, whose superblock has the `has_journal` +/// feature if `journal`. +fn ext4_image(journal: bool) -> Vec { + let mut image = vec![0; 2048]; + image[1024 + 0x38..1024 + 0x3A].copy_from_slice(&0xEF53_u16.to_le_bytes()); + if journal { + image[1024 + 0x5C] = 0x04; + } + image +} + /// Returns an EROFS image's first 4 KiB, whose superblock holds `uuid`. fn erofs_image(uuid: Uuid) -> Vec { let mut image = vec![0; 4096]; @@ -174,10 +252,12 @@ fn assembly_mounts_the_image_and_the_scratch_under_the_overlay() { "lowerdir={runtime}/layers/distro,upperdir={runtime}/scratch/upper,\ workdir={runtime}/scratch/work,metacopy=on,xino=on" ); + // Both devices are checked and flushed before anything is mounted. assert_eq!( guest.kernel.calls(), [ Call::Flush(guest.text("dev/vda")), + Call::Flush(guest.text("dev/vdb")), mount_call( Some(&guest.text("dev/vda")), &format!("{runtime}/layers/distro"), @@ -185,7 +265,6 @@ fn assembly_mounts_the_image_and_the_scratch_under_the_overlay() { MountFlags::MS_RDONLY, None, ), - Call::Flush(guest.text("dev/vdb")), mount_call( Some(&guest.text("dev/vdb")), &format!("{runtime}/scratch"), @@ -210,6 +289,15 @@ fn assembly_mounts_the_image_and_the_scratch_under_the_overlay() { } assert_eq!(root.path(), guest.path("run/nvx/rootfs")); assert_eq!(root.runtime(), guest.path("run/nvx")); + // The root reports what it mounted: the image's one layer, the scratch, and the mode. + let [lower] = root.lowers() else { + panic!("{:?}", root.lowers()); + }; + assert_eq!(lower.mount(), &Mount::Distro); + assert_eq!(lower.device(), guest.path("dev/vda")); + assert_eq!(lower.path(), guest.path("run/nvx/layers/distro")); + assert_eq!(root.upper(), UpperKind::Scratch); + assert_eq!(root.overlay(), OverlayMode::TrustedLowers); root.teardown().unwrap(); } @@ -249,11 +337,9 @@ fn devices_are_found_when_they_appear_and_missing_ones_time_out() { error.to_string(), "the scratch device at MMIO 0x1d0001000 did not appear" ); - // The distro layer, which was mounted, is unmounted again. - assert_eq!( - guest.kernel.unmounts(), - [guest.text("run/nvx/layers/distro")] - ); + // The assembly checks the scratch device before it mounts anything. + assert!(guest.kernel.mounts().is_empty()); + assert!(guest.kernel.unmounts().is_empty()); } #[test] @@ -284,7 +370,7 @@ fn a_device_of_another_size_is_refused_before_it_is_flushed() { assert!(guest.kernel.calls().is_empty()); assembly.distro.sectors = Some(4096); - assembly.scratch.sectors = Some(8192); + scratch(&mut assembly).sectors = Some(8192); guest.assemble(&assembly).unwrap().teardown().unwrap(); } @@ -293,7 +379,7 @@ fn the_expected_uuid_is_checked_in_the_superblock_before_the_mount() { let guest = Guest::new(); fs::write(guest.path("dev/vda"), erofs_image(UUID)).unwrap(); let mut assembly = guest.assembly(); - assembly.distro_uuid = Some(UUID); + assembly.image = Image::Erofs { uuid: Some(UUID) }; guest.assemble(&assembly).unwrap().teardown().unwrap(); let guest = Guest::new(); @@ -302,7 +388,9 @@ fn the_expected_uuid_is_checked_in_the_superblock_before_the_mount() { bytes[15] = 0xff; let expected = Uuid::from_bytes(bytes); assembly = guest.assembly(); - assembly.distro_uuid = Some(expected); + assembly.image = Image::Erofs { + uuid: Some(expected), + }; let error = guest.assemble(&assembly).unwrap_err(); assert!( matches!(error.error(), Error::Uuid { expected: e, actual: UUID } if *e == expected), @@ -536,6 +624,16 @@ fn new_upper_and_work_directories_have_mode_0755_whatever_the_umask() { .unwrap(); assert_eq!(mode(&upper), 0o700); assert_eq!(mode(&guest.path("run/nvx/scratch/work")), 0o755); + + // So are a tmpfs upper's, which are always new. + let guest = Guest::new(); + let assembly = Assembly { + upper: Upper::Tmpfs, + ..guest.assembly() + }; + guest.assemble(&assembly).unwrap().teardown().unwrap(); + assert_eq!(mode(&guest.path("run/nvx/tmpfs/upper")), 0o755); + assert_eq!(mode(&guest.path("run/nvx/tmpfs/work")), 0o755); } #[test] @@ -683,6 +781,36 @@ fn a_runtime_directory_where_a_file_system_is_mounted_is_refused() { "{error:?}" ); assert_eq!(guest.kernel.mounts().len(), mounts); + + // So does an upper that a failed unmount left, whichever upper the next root mounts. + for (first, second) in [ + (scratch_upper(Journal::Allowed), Upper::Tmpfs), + (Upper::Tmpfs, scratch_upper(Journal::Allowed)), + ] { + let guest = Guest::new(); + let left = guest.path(&format!("run/nvx/{}", upper_point(first.kind()))); + guest + .kernel + .fail(Operation::Unmount, left.clone(), Errno::EBUSY); + let assembly = Assembly { + upper: first, + ..guest.assembly() + }; + guest.assemble(&assembly).unwrap().teardown().unwrap_err(); + let mounts = guest.kernel.mounts().len(); + let calls = guest.kernel.calls(); + let assembly = Assembly { + upper: second, + ..guest.assembly() + }; + let error = guest.assemble(&assembly).unwrap_err(); + assert!( + matches!(error.error(), Error::Occupied { path } if *path == left), + "{error:?}" + ); + assert_eq!(guest.kernel.mounts().len(), mounts); + assert_eq!(guest.kernel.calls(), calls); + } } #[test] @@ -697,6 +825,37 @@ fn the_kernel_finds_its_mount_points() { assert_eq!(error.kind(), io::ErrorKind::NotFound); } +#[test] +fn the_kernel_shows_a_mount_s_options() { + let directory = tempfile::tempdir().unwrap(); + let mut kernel = SystemKernel; + let options = kernel.mount_options(Path::new("/proc")).unwrap(); + let access = options.split(',').next().unwrap(); + assert!(access == "rw" || access == "ro", "{options}"); + let error = kernel + .mount_options(&directory.path().join("missing")) + .unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::NotFound); +} + +#[test] +fn a_mount_s_options_are_the_last_field_of_its_line_in_the_mount_table() { + let table = b"21 1 0:20 / /proc rw,nosuid,nodev,noexec,relatime shared:12 - proc proc rw\n\ + 35 21 8:1 / /run/a\\040b/rootfs rw,nosuid,nodev - overlay overlay \ + rw,lowerdir=/a\\040b:/c,upperdir=/u,workdir=/w,metacopy=on\n\ + 36 35 0:5 / /mnt master:1 propagate_from:2 - tmpfs a\\040b ro,size=4k\n\ + 37 35 0:6 / /x rw\n"; + assert_eq!(super_options(table, 21), Some(&b"rw"[..])); + assert_eq!( + super_options(table, 35), + Some(&b"rw,lowerdir=/a\\040b:/c,upperdir=/u,workdir=/w,metacopy=on"[..]) + ); + assert_eq!(super_options(table, 36), Some(&b"ro,size=4k"[..])); + // A mount without a line, and a line without the separator. + assert_eq!(super_options(table, 3), None); + assert_eq!(super_options(table, 37), None); +} + /// The variable that gives the bind-mount test's inner run the runtime directory where the outer /// run bound a directory at the distro layer's mount point. const BOUND_RUNTIME: &str = "NVX_ROOT_TEST_BOUND_RUNTIME"; @@ -763,6 +922,579 @@ fn a_runtime_directory_where_a_directory_is_bound_is_refused() { ); } +/// Returns the overlay's options for the layers at `lowers` below the runtime directory +/// `runtime`, the top first, over the upper mounted at `upper` below it, with `mode`. +fn overlay_data(runtime: &str, lowers: &[&str], upper: &str, mode: OverlayMode) -> String { + let lowers: Vec = lowers + .iter() + .map(|lower| format!("{runtime}/{lower}")) + .collect(); + format!( + "lowerdir={},upperdir={runtime}/{upper}/upper,workdir={runtime}/{upper}/work,{}", + lowers.join(":"), + mode.options() + ) +} + +/// Returns the paths of `names` below the runtime directory `runtime`, as the fake kernel +/// records them. +fn below(runtime: &str, names: &[&str]) -> Vec { + names + .iter() + .map(|name| format!("{runtime}/{name}")) + .collect() +} + +#[test] +fn both_overlay_modes_assemble_on_both_uppers_with_their_own_option_sets() { + for upper in [scratch_upper(Journal::Allowed), Upper::Tmpfs] { + for mode in [OverlayMode::TrustedLowers, OverlayMode::UntrustedLowers] { + let case = format!("{upper:?}, {mode:?}"); + let guest = Guest::new(); + let assembly = Assembly { + upper, + overlay: mode, + ..guest.assembly() + }; + let root = guest.assemble(&assembly).unwrap(); + let runtime = guest.text("run/nvx"); + let flags = MountFlags::MS_NOSUID | MountFlags::MS_NODEV; + let (point, upper_mount) = match upper.kind() { + UpperKind::Scratch => ( + "scratch", + mount_call( + Some(&guest.text("dev/vdb")), + &format!("{runtime}/scratch"), + Some("ext4"), + flags, + None, + ), + ), + UpperKind::Tmpfs => ( + "tmpfs", + mount_call( + Some("tmpfs"), + &format!("{runtime}/tmpfs"), + Some("tmpfs"), + flags, + Some("mode=0755"), + ), + ), + }; + // The upper, then the overlay, with the mode's one option set, and the flags + // `nosuid,nodev`. + let overlay = mount_call( + Some("overlay"), + &format!("{runtime}/rootfs"), + Some("overlay"), + flags, + Some(&overlay_data(&runtime, &["layers/distro"], point, mode)), + ); + let calls = guest.kernel.calls(); + assert_eq!(calls[calls.len() - 2..], [upper_mount, overlay], "{case}"); + // A tmpfs upper looks for no scratch device. + let scratch_flushed = calls.contains(&Call::Flush(guest.text("dev/vdb"))); + assert_eq!( + scratch_flushed, + upper.kind() == UpperKind::Scratch, + "{case}" + ); + assert_eq!(root.upper(), upper.kind(), "{case}"); + assert_eq!(root.overlay(), mode, "{case}"); + for directory in ["upper", "work"] { + let path = guest.path(&format!("run/nvx/{point}/{directory}")); + assert!(path.is_dir(), "{case}: {directory}"); + } + root.teardown().unwrap(); + assert_eq!( + guest.kernel.unmounts(), + below(&runtime, &["rootfs", "layers/distro", point]), + "{case}" + ); + } + } +} + +#[test] +fn a_tmpfs_upper_needs_no_scratch_device() { + let guest = Guest::bare(); + attach(&guest.base, 0, "vda", 4096); + let assembly = Assembly { + upper: Upper::Tmpfs, + ..guest.assembly() + }; + let root = guest.assemble(&assembly).unwrap(); + // The assembly waited for no other device, and made no mount point for a scratch. + assert_eq!(guest.kernel.sleeps(), 0); + assert!(!guest.path("run/nvx/scratch").exists()); + root.teardown().unwrap(); + assert_eq!( + guest.kernel.unmounts(), + below( + &guest.text("run/nvx"), + &["rootfs", "layers/distro", "tmpfs"] + ) + ); +} + +#[test] +fn a_refused_option_set_fails_without_a_fallback() { + for mode in [OverlayMode::TrustedLowers, OverlayMode::UntrustedLowers] { + for upper in [scratch_upper(Journal::Allowed), Upper::Tmpfs] { + let guest = Guest::new(); + guest.kernel.fail( + Operation::Mount, + guest.path("run/nvx/rootfs"), + Errno::EINVAL, + ); + let assembly = Assembly { + upper, + overlay: mode, + ..guest.assembly() + }; + let error = guest.assemble(&assembly).unwrap_err(); + assert!( + matches!( + error.error(), + Error::Mount { mount: Mount::Overlay, error } + if error.raw_os_error() == Some(Errno::EINVAL as i32) + ), + "{error:?}" + ); + assert!(error.cleanup().is_none()); + // The overlay was mounted once, with the mode's own option set. + let overlays: Vec = guest + .kernel + .calls() + .into_iter() + .filter(|call| matches!(call, Call::Mount { target, .. } if target.ends_with("/rootfs"))) + .collect(); + let [ + Call::Mount { + data: Some(data), .. + }, + ] = overlays.as_slice() + else { + panic!("{overlays:?}"); + }; + assert!(data.ends_with(&format!(",{}", mode.options())), "{data}"); + // The layer and the upper are unmounted again. + let point = match upper.kind() { + UpperKind::Scratch => "scratch", + UpperKind::Tmpfs => "tmpfs", + }; + assert_eq!( + guest.kernel.unmounts(), + below(&guest.text("run/nvx"), &["layers/distro", point]) + ); + } + } +} + +#[test] +fn an_overlay_that_the_kernel_mounts_without_its_mode_in_effect_is_refused_and_unmounted() { + // Without failing the mount, the kernel drops `metacopy=on` for an upper that can't hold the + // overlay's extended attributes. Its mount table then shows no `metacopy`, which is off by + // default. + for upper in [scratch_upper(Journal::Allowed), Upper::Tmpfs] { + let guest = Guest::new(); + guest.show_overlay("rw", upper.kind(), "xino=on"); + let assembly = Assembly { + upper, + ..guest.assembly() + }; + let error = guest.assemble(&assembly).unwrap_err(); + assert!( + matches!(error.error(), Error::OverlayFallback { requested, effective } + if requested == "metacopy=on" && effective == "metacopy=off"), + "{error:?}" + ); + assert_eq!( + error.error().to_string(), + "the kernel mounted the overlay with metacopy=off in place of metacopy=on" + ); + assert!(error.cleanup().is_none()); + assert_eq!( + guest.kernel.unmounts(), + below( + &guest.text("run/nvx"), + &["rootfs", "layers/distro", upper_point(upper.kind())] + ) + ); + } + + // It mounts an overlay read-only when it can't create the overlay's work directory, as on a + // full upper. + for mode in [OverlayMode::TrustedLowers, OverlayMode::UntrustedLowers] { + let guest = Guest::new(); + guest.show_overlay("ro", UpperKind::Scratch, mode.options()); + let assembly = Assembly { + overlay: mode, + ..guest.assembly() + }; + let error = guest.assemble(&assembly).unwrap_err(); + assert!( + matches!(error.error(), Error::OverlayFallback { requested, effective } + if requested == "rw" && effective == "ro"), + "{error:?}" + ); + assert_eq!( + guest.kernel.unmounts(), + below( + &guest.text("run/nvx"), + &["rootfs", "layers/distro", "scratch"] + ) + ); + } +} + +#[test] +fn options_that_the_kernel_does_not_show_have_their_defaults() { + // The kernel's mount table shows no option at its default, as for each option of the + // untrusted lowers under the guest's kernel, whose defaults are off. + let untrusted = |guest: &Guest| Assembly { + upper: Upper::Tmpfs, + overlay: OverlayMode::UntrustedLowers, + ..guest.assembly() + }; + let guest = Guest::new(); + guest.show_overlay("rw", UpperKind::Tmpfs, ""); + let root = guest.assemble(&untrusted(&guest)).unwrap(); + assert_eq!(root.overlay(), OverlayMode::UntrustedLowers); + root.teardown().unwrap(); + + // Under a kernel that copies up metadata alone by default, the table shows the trusted + // lowers' `metacopy=on` no more, and must show the untrusted lowers' `metacopy=off`. + let guest = Guest::new(); + guest.set_overlay_default("metacopy", true); + guest.show_overlay("rw", UpperKind::Scratch, "redirect_dir=on,xino=on"); + guest + .assemble(&guest.assembly()) + .unwrap() + .teardown() + .unwrap(); + let guest = Guest::new(); + guest.set_overlay_default("metacopy", true); + guest.show_overlay("rw", UpperKind::Tmpfs, ""); + let error = guest.assemble(&untrusted(&guest)).unwrap_err(); + assert!( + matches!(error.error(), Error::OverlayFallback { requested, effective } + if requested == "metacopy=off" && effective == "metacopy=on"), + "{error:?}" + ); + + // Under a kernel that creates or follows redirects by default, the table must show + // `redirect_dir=nofollow`. + for (parameter, effective) in [("redirect_dir", "on"), ("redirect_always_follow", "follow")] { + let guest = Guest::new(); + guest.set_overlay_default(parameter, true); + guest.show_overlay("rw", UpperKind::Tmpfs, ""); + let error = guest.assemble(&untrusted(&guest)).unwrap_err(); + assert!( + matches!(error.error(), Error::OverlayFallback { requested, effective: actual } + if requested == "redirect_dir=nofollow" + && *actual == format!("redirect_dir={effective}")), + "{parameter}: {error:?}" + ); + let guest = Guest::new(); + guest.set_overlay_default(parameter, true); + guest.show_overlay("rw", UpperKind::Tmpfs, "redirect_dir=nofollow"); + guest + .assemble(&untrusted(&guest)) + .unwrap() + .teardown() + .unwrap(); + } +} + +#[test] +fn effective_options_that_cannot_be_read_fail_the_assembly() { + // The overlay's options in the kernel's mount table. + let guest = Guest::new(); + let rootfs = guest.path("run/nvx/rootfs"); + guest + .kernel + .fail(Operation::Options, rootfs.clone(), Errno::EIO); + let error = guest.assemble(&guest.assembly()).unwrap_err(); + assert!( + matches!(error.error(), Error::OverlayCheck { path, error } + if *path == rootfs && error.raw_os_error() == Some(Errno::EIO as i32)), + "{error:?}" + ); + assert_eq!( + error.error().to_string(), + format!( + "failed to read the overlay's effective options at {}: {}", + rootfs.display(), + os_error(Errno::EIO) + ) + ); + assert!(error.cleanup().is_none()); + assert_eq!( + guest.kernel.unmounts(), + below( + &guest.text("run/nvx"), + &["rootfs", "layers/distro", "scratch"] + ) + ); + + // The default of an option that the table doesn't show: missing, or neither `Y` nor `N`. + for contents in [None, Some("1\n")] { + let guest = Guest::new(); + let parameter = guest.paths.sys.join(OVERLAY_PARAMETERS).join("metacopy"); + match contents { + Some(contents) => write(¶meter, contents), + None => fs::remove_file(¶meter).unwrap(), + } + guest.show_overlay("rw", UpperKind::Tmpfs, ""); + let assembly = Assembly { + upper: Upper::Tmpfs, + overlay: OverlayMode::UntrustedLowers, + ..guest.assembly() + }; + let error = guest.assemble(&assembly).unwrap_err(); + let expected = match contents { + Some(_) => Some(Errno::EPROTO as i32), + None => Some(Errno::ENOENT as i32), + }; + assert!( + matches!(error.error(), Error::OverlayCheck { path, error } + if *path == parameter && error.raw_os_error() == expected), + "{error:?}" + ); + assert_eq!(guest.kernel.unmounts().len(), 3); + } +} + +#[test] +fn overlay_options_that_a_mount_cannot_pass_are_refused_before_any_mount() { + // "lowerdir=/L,upperdir=/u,workdir=/w,metacopy=on,xino=on" takes 53 bytes and the length of + // the lower's name L: options of the limit's length pass, and longer ones do not. + let options = |name: usize| { + let lower = Lower { + mount: Mount::Distro, + device: PathBuf::new(), + path: format!("/{}", "l".repeat(name)).into(), + }; + let mode = OverlayMode::TrustedLowers; + overlay_options(&[lower], Path::new("/u"), Path::new("/w"), mode) + }; + assert_eq!( + options(MOUNT_OPTIONS_LIMIT - 53).unwrap().len(), + MOUNT_OPTIONS_LIMIT + ); + assert!(matches!( + options(MOUNT_OPTIONS_LIMIT - 52), + Err(Error::OverlayOptions { length: 4096 }) + )); + + // A runtime directory long enough that the options, which name it three times, take more + // than a mount passes is refused before any mount. + let guest = Guest::new(); + let component = "d".repeat(250); + let runtime = guest.path(&format!("run/{}/nvx", [component.as_str(); 6].join("/"))); + let assembly = Assembly { + runtime: runtime.clone(), + upper: Upper::Tmpfs, + ..guest.assembly() + }; + let error = guest.assemble(&assembly).unwrap_err(); + let runtime = runtime.display().to_string(); + let mode = OverlayMode::TrustedLowers; + let length = overlay_data(&runtime, &["layers/distro"], "tmpfs", mode).len(); + assert!(length > MOUNT_OPTIONS_LIMIT, "{length}"); + assert!( + matches!(error.error(), Error::OverlayOptions { length: actual } if *actual == length), + "{error:?}" + ); + assert_eq!( + error.to_string(), + format!( + "the overlay's options take {length} bytes, more than the 4095 that a mount passes" + ) + ); + assert!(guest.kernel.mounts().is_empty()); +} + +#[test] +fn the_journal_policy_is_checked_in_the_scratch_s_superblock_before_any_mount() { + // A scratch without a journal mounts whatever the policy, and one with a journal only where + // the policy allows one. + let cases = [ + (false, Journal::Allowed, true), + (false, Journal::Forbidden, true), + (true, Journal::Allowed, true), + (true, Journal::Forbidden, false), + ]; + for (journal, policy, mounts) in cases { + let guest = Guest::new(); + fs::write(guest.path("dev/vdb"), ext4_image(journal)).unwrap(); + let assembly = Assembly { + upper: scratch_upper(policy), + ..guest.assembly() + }; + let result = guest.assemble(&assembly); + if mounts { + result.unwrap().teardown().unwrap(); + continue; + } + let error = result.unwrap_err(); + assert!( + matches!(error.error(), Error::Journal { device } if *device == guest.path("dev/vdb")), + "{error:?}" + ); + assert_eq!( + error.to_string(), + format!( + "the scratch device {} has an ext4 journal, which the assembly forbids", + guest.text("dev/vdb") + ) + ); + assert!(guest.kernel.mounts().is_empty()); + } + + // A scratch that holds no ext4 file system has no journal feature to check. + let guest = Guest::new(); + fs::write(guest.path("dev/vdb"), vec![0; 2048]).unwrap(); + let assembly = Assembly { + upper: scratch_upper(Journal::Forbidden), + ..guest.assembly() + }; + let error = guest.assemble(&assembly).unwrap_err(); + assert!( + matches!(error.error(), Error::NotExt4 { device } if *device == guest.path("dev/vdb")), + "{error:?}" + ); + assert_eq!( + error.to_string(), + format!( + "the scratch device {} holds no ext4 file system", + guest.text("dev/vdb") + ) + ); + assert!(guest.kernel.mounts().is_empty()); +} + +#[test] +fn a_device_whose_access_mode_is_not_its_role_s_is_refused_before_it_is_flushed() { + // A writable distro device. + let guest = Guest::new(); + set_read_only(&guest.base, "vda", false); + let error = guest + .assemble(&guest.assembly()) + .unwrap_err() + .into_parts() + .0; + assert!( + matches!(&error, Error::Access { role: Role::Distro, device, read_only: false } + if *device == guest.path("dev/vda")), + "{error:?}" + ); + assert_eq!( + error.to_string(), + format!( + "the distro device {} is writable, expected read-only", + guest.text("dev/vda") + ) + ); + assert!(guest.kernel.calls().is_empty()); + + // A read-only scratch device, checked after the distro device. + let guest = Guest::new(); + set_read_only(&guest.base, "vdb", true); + let error = guest + .assemble(&guest.assembly()) + .unwrap_err() + .into_parts() + .0; + assert!( + matches!(&error, Error::Access { role: Role::Scratch, device, read_only: true } + if *device == guest.path("dev/vdb")), + "{error:?}" + ); + assert_eq!( + error.to_string(), + format!( + "the scratch device {} is read-only, expected writable", + guest.text("dev/vdb") + ) + ); + assert_eq!(guest.kernel.calls(), [Call::Flush(guest.text("dev/vda"))]); + + // The devices in each other's roles: the distro's role names the scratch's window. + let guest = Guest::new(); + let mut assembly = guest.assembly(); + assembly.distro.mmio = SCRATCH; + scratch(&mut assembly).mmio = DISTRO; + let error = guest.assemble(&assembly).unwrap_err().into_parts().0; + assert!( + matches!(&error, Error::Access { role: Role::Distro, device, read_only: false } + if *device == guest.path("dev/vdb")), + "{error:?}" + ); + assert!(guest.kernel.calls().is_empty()); + + // An access mode in another form than the kernel's. + let guest = Guest::new(); + write(&guest.path("sys/class/block/vda/ro"), "2\n"); + let error = guest + .assemble(&guest.assembly()) + .unwrap_err() + .into_parts() + .0; + assert!( + matches!(&error, Error::Device { role: Role::Distro, step: DeviceStep::Access, error, .. } + if error.raw_os_error() == Some(Errno::EPROTO as i32)), + "{error:?}" + ); + assert!(guest.kernel.calls().is_empty()); +} + +#[test] +fn a_tmpfs_upper_is_unmounted_last_and_every_failure_reported() { + let guest = Guest::new(); + let assembly = Assembly { + upper: Upper::Tmpfs, + overlay: OverlayMode::UntrustedLowers, + ..guest.assembly() + }; + let root = guest.assemble(&assembly).unwrap(); + for name in ["run/nvx/layers/distro", "run/nvx/tmpfs"] { + guest + .kernel + .fail(Operation::Unmount, guest.path(name), Errno::EBUSY); + } + let error = root.teardown().unwrap_err(); + assert_eq!( + guest.kernel.unmounts(), + below( + &guest.text("run/nvx"), + &["rootfs", "layers/distro", "tmpfs"] + ) + ); + let failed: Vec<(&Mount, &Path)> = error + .failures() + .iter() + .map(|failure| (failure.mount(), failure.path())) + .collect(); + let distro = guest.path("run/nvx/layers/distro"); + let tmpfs = guest.path("run/nvx/tmpfs"); + assert_eq!( + failed, + [ + (&Mount::Distro, distro.as_path()), + (&Mount::Tmpfs, tmpfs.as_path()) + ] + ); + assert_eq!( + error.to_string(), + format!( + "failed to unmount the distro layer: {busy}; failed to unmount the tmpfs upper: {busy}", + busy = os_error(Errno::EBUSY) + ) + ); +} + /// Returns a child of an aggregate share. fn child(name: &str, target: &str, access: Access, kind: ShareKind) -> ShareChild { ShareChild { diff --git a/tests/runtime-adapter/Cargo.toml b/tests/runtime-adapter/Cargo.toml index ebe01b523..775d781e9 100644 --- a/tests/runtime-adapter/Cargo.toml +++ b/tests/runtime-adapter/Cargo.toml @@ -26,11 +26,13 @@ nvx = { path = "../../nvx-rs", default-features = false } # its plan. The host side of `nvx` depends on the same release. serde_json = { version = "1.0.120", optional = true } -# The guest agent's PID 1 collects the status of every child and ends the unrelated one, whose -# system calls `nvx` leaves to the caller, and the worker's mapping probes mount a procfs in the -# root. The guest side of `nvx` depends on the same release, with these features among its own. +# The guest agent's PID 1 collects the status of every child and ends the unrelated one, its +# probes read how much space a file system has, and its mapping probes mount a procfs in the +# root: system calls that `nvx` leaves to the caller. The guest side of `nvx` depends on the +# same release, with these features among its own. [target.'cfg(target_os = "linux")'.dependencies] nix = { version = "0.31.3", default-features = false, features = [ + "fs", "mount", "process", "signal", diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index b179a8c71..596a7f144 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -90,20 +90,24 @@ zeros: | `workload-pids-max` | The workload cgroup's `pids.max` | no | | `distro-sectors` | The size of the image's block device, in 512-byte sectors | no | | `scratch-sectors` | The size of the scratch block device, in 512-byte sectors | no | +| `overlay-mode` | The overlay's mode: `trusted-lowers` or `untrusted-lowers` | no | +| `scratch-journal` | Whether an ext4 scratch may have a journal: `allowed` or `forbidden` | no | The paths are absolute and canonical guest paths of at most 4,095 bytes, without a NUL or a newline: not `/`, and without an empty, `.`, or `..` component, or a trailing `/`. The runtime directory holds no `,`, `:`, `\`, or `=`, which the overlay's options reserve, and the two cgroups are siblings, -whose parent is not `/`. The optional numbers are positive. The host program -and the guest agent ship together, so the guest refuses every other -configuration, one with an unknown field included, rather than ignore what it -does not know. `RootConfig::default()` holds the values that the host program -sends unless its caller names others: the runtime directory -`/run/runtime-adapter`, the cgroups `/sys/fs/cgroup/agent` and +whose parent is not `/`. The optional numbers are positive. Without an +overlay mode or a journal policy, the guest takes the defaults of +`nvx::guest::root`: an overlay that trusts its lowers, and a scratch that may +have a journal. The host program and the guest agent ship together, so the +guest refuses every other configuration, one with an unknown field included, +rather than ignore what it does not know. `RootConfig::default()` holds the +values that the host program sends unless its caller names others: the runtime +directory `/run/runtime-adapter`, the cgroups `/sys/fs/cgroup/agent` and `/sys/fs/cgroup/workload`, a wait of 30 seconds, an agent's `memory.low` of 16 MiB, as NVX's own guest reserves, and a workload's `memory.max` of 128 MiB -and `pids.max` of 128. +and `pids.max` of 128, with neither an overlay mode nor a journal policy. A stage of the root that fails is a `root::RootFailure`: the stage's code, one byte, then the problem, as UTF-8 text. The guest's worker, and with it the VM, @@ -130,12 +134,18 @@ key at most once. A key holds lowercase ASCII letters, digits, `.`, and `-`, and a value escapes `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`. -The worker runs the probes of `worker::probes`, none of which takes arguments: +The worker runs the probes of `worker::probes`. FILE and DIRECTORY are +relative paths in the root, of normal components, and the other probes take no +arguments: | Probe | Facts | | --- | --- | -| `mounts` | For the image's mount, `distro`, the scratch's, `scratch`, and the overlay's, `root`: the number of mounts at its mount point, `NAME.count`, and of the topmost, its `NAME.type`, `NAME.source`, `NAME.options`, and `NAME.super`, the file system's own options | -| `contents` | The root's `os-release.id` and `root.mode`. It writes `.runtime-adapter-probe` at the top of the root, and reports what the root, `write.root`, the upper, `write.upper`, and the image, `write.lower`, hold at that name, and whether the image refuses a new file, `lower.write` | +| `mounts` | For the mount of each layer, by the name of its mount point, `distro` for an EROFS image, the scratch's, `scratch`, the tmpfs's, `tmpfs`, and the overlay's, `root`: the number of mounts at its mount point, `NAME.count`, and of the topmost, its `NAME.type`, `NAME.source`, `NAME.options`, and `NAME.super`, the file system's own options | +| `contents` | The root's `os-release.id` and `root.mode`. It writes `.runtime-adapter-probe` at the top of the root, and reports what the root, `write.root`, the upper, `write.upper`, and the top layer, `write.lower`, hold at that name, and whether the top layer refuses a new file, `lower.write` | +| `copy-up FILE` | It sets the mode of FILE, a regular file of the layers, to the mode that it has, which copies the file up, and reports its size in the topmost layer that holds it, `lower.size`, and in the upper, `upper.size`; whether the upper's copy holds the file's data, `upper.data`: `copied`, `none` once the overlay copied up the file's metadata alone, or `different`; and whether the root still shows the layer's data, `root.data`: `intact` or `different` | +| `rename DIRECTORY` | It renames DIRECTORY, a directory of the layers, to `.runtime-adapter-renamed` at the top of the root, and back if it can: `outcome` is `allowed`, `refused` once the overlay refused it with `EXDEV`, or `failed: PROBLEM` | +| `fill` | It writes a file in the root until the upper is full, then removes it, and writes a file there again: the upper's size, `total`, and the bytes available in it before, `available`; the bytes that it wrote, `written`; the error that ended the writes, `error`: `ENOSPC` once the upper was full, or the problem; what the second file holds, `after`; and the guest's memory, `memory`, in bytes | +| `devices` | The guest's disks, the block devices that a device backs: `disks`, each as `NAME:SECTORS:ACCESS`, its size in 512-byte sectors and its access mode, `ro` or `rw`, in the order of their names, separated by spaces | | `cgroups` | The file system at the cgroups' parent, `parent.type`, and its `parent.subtree-control`; whether each cgroup exists, `agent.exists` and `workload.exists`; `agent.memory.low`, `workload.memory.max`, and `workload.pids.max`; the number of processes in each, `agent.processes` and `workload.processes`; and `user-namespaces`, the guest's limit of user namespaces, or `unavailable` without them | | `processes` | For PID 1, `pid1`, the worker, `worker`, and PID 1's unrelated child, `sibling`: `NAME.pid`, and while the process exists, `NAME.state`, `NAME.ppid`, `NAME.root`, `NAME.cgroup`, `NAME.command`, and `NAME.namespaces`: `same` if it shares PID 1's, or the ones that differ | | `hold` | It opens the root, and keeps it open until the worker ends, so that the teardown finds the root busy: `held`, the path that it holds | @@ -238,17 +248,18 @@ tty with `nvx::guest::terminal`, as `serve_terminal` does, and then: `RootFailed`, confirms the stop with `Stopped`, and ends. Before the root configuration, the worker accepts only the configuration, or a -`Stop`, which ends it without a root. It assembles an EROFS image over an ext4 -scratch, and refuses the other layouts that the boot tokens can name at the -layout stage. It ends with 0 once the host stopped it, after a teardown that -succeeded, with the stage's status, from 11 to 15, once a stage failed, and -with 1 once its session failed or the broker closed the tty, each but the -first after a diagnostic on standard error. Before it exits, it closes what its -probes hold, then unmounts a root that it still holds, as -`nvx::guest::root::Root` does when it drops, ignoring failures. A teardown that -failed leaves the root mounted until the VM ends. The worker runs the root's -operations through `worker::RootOperations`, so that tests run it against a -fake root. +`Stop`, which ends it without a root. It assembles an EROFS image, over an ext4 +scratch or a tmpfs in RAM, in the overlay mode of the configuration, and +refuses a GPT image at the layout stage, as it refuses a configuration that +configures a scratch for an upper in RAM. It ends with 0 once the host stopped +it, after a teardown that succeeded, with the stage's status, from 11 to 15, +once a stage failed, and with 1 once its session failed or the broker closed +the tty, each but the first after a diagnostic on standard error. Before it +exits, it closes what its probes hold, then unmounts a root that it still +holds, as `nvx::guest::root::Root` does when it drops, ignoring failures. A +teardown that failed leaves the root mounted until the VM ends. The worker +runs the root's operations through `worker::RootOperations`, so that tests run +it against a fake root. ## Host program @@ -354,8 +365,9 @@ launch capability on standard input, and the outcome report. The boot tokens carry only what NVX's boot contract defines. The launch sends the rest in the [root configuration](#root-configurations), once it attaches: the defaults, with the runtime directory that the caller names, as it is, so -that the guest checks it, and the size of each image in sectors, if it is a -whole number of them, which the guest checks against its block device. +that the guest checks it, the size of each image in sectors, if it is a whole +number of them, which the guest checks against its block device, and the +overlay mode and the scratch's journal policy that the caller names, if any. The launch records what it launched in the sandbox's store, beside NVX's runtime records, in `runtime-adapter.json`, before it spawns the VMM: @@ -389,17 +401,20 @@ launch sends, whether or not the guest accepts it. ends its VM. LAYERED, - `--distro FORMAT:PATH [--distro-identity UUID] --upper (scratch:PATH | ram) [--runtime-dir DIR]`, + `--distro FORMAT:PATH [--distro-identity UUID] --upper (scratch:PATH | ram) [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY]`, gives the VM a [layered root](#layered-roots). Its image is the file PATH, an EROFS file system, `erofs`, or a GPT disk, `gpt`, of which the guest expects the identity UUID, if given. Its upper is the ext4 scratch image PATH, or RAM. The guest assembles the root below its runtime directory DIR, - `/run/runtime-adapter` by default. Once attached, the launch sends the root - configuration, and exchanges the `Ping` and the `Echo` once the guest - assembled the root. If the guest reports that a stage of the root failed - instead, the launch waits for the VM to end, abandons the launch, and checks - that the VM ended with the stage's status. The worker assembles only an - EROFS image over a scratch, and fails the other layouts at the layout stage. + `/run/runtime-adapter` by default, in the overlay mode MODE, + `trusted-lowers` or `untrusted-lowers`, and checks a scratch's journal + against the policy POLICY, `allowed` or `forbidden`, which only a scratch + takes, with the guest's defaults unless given. Once attached, the launch + sends the root configuration, and exchanges the `Ping` and the `Echo` once + the guest assembled the root. If the guest reports that a stage of the root + failed instead, the launch waits for the VM to end, abandons the launch, and + checks that the VM ended with the stage's status. The worker assembles only + an EROFS image, and fails a GPT image at the layout stage. - `reattach STORE` takes the sandbox's lock, and reattaches to its VM, which another process launched, and which it holds, although the VM is not its child, through its verified process identity. It reads the capability from @@ -462,8 +477,8 @@ byte, the headers that the framing refuses before their body arrives, and the root configurations, failures, probes, and facts that decode, and those that don't. In the guest-only configuration, the unit tests of `worker::probes` cover what the probes read: the mount table with its escapes, the kernel's -records of a process, and `os-release`, and the probes that the worker -refuses. +records of a process, `os-release`, and the guest's memory, and the probes and +arguments that the worker refuses. `tests/guest_session.rs` checks, in the guest-only configuration, that the guest agent serves the control session with the adapter's own framing. A test @@ -518,12 +533,13 @@ byte value, and that `init` fails at once on a path that is no tty. `tests/guest_session.rs` also runs the worker against a fake root, behind its test broker: a root that the worker assembles, probes, and tears down on a -stop, across a reset of the session; the failure of each stage, which the -worker reports before it ends with the stage's status, without the operations -of the later stages; a teardown that fails; before the root, the configuration -and the `Stop` that it accepts, and the messages that it refuses; and a second -configuration and a broker that closes the stream, which leave the root that -it assembled to its operations. Through the binary, it checks that +stop, across a reset of the session; an image over RAM, in the overlay mode of +the configuration; the failure of each stage, which the worker reports before +it ends with the stage's status, without the operations of the later stages; +a teardown that fails; before the root, the configuration and the `Stop` that +it accepts, and the messages that it refuses; and a second configuration and a +broker that closes the stream, which leave the root that it assembled to its +operations. Through the binary, it checks that `runtime-adapter-guest init TTY layered`, which is not PID 1 there, starts the worker, which fails at the layout stage on a kernel command line without a layered root, and ends with its status, and checks `worker`, `idle`, and the @@ -581,7 +597,13 @@ own: | Scenario | Root | Expects | | --- | --- | --- | -| `erofs_over_scratch` | NVX's Ubuntu EROFS image, with its own identity, over a copy of the scratch template | A root that the worker assembled: the image, read-only, under the overlay, whose upper and work directories are on the scratch, each mounted once below the runtime directory; the merged root's contents, with a file that reaches the upper and not the image; the agent's and the workload's cgroups, with their resource values; a worker that is PID 1's child, and PID 1's unrelated child, untouched; and a teardown that unmounts the scratch cleanly | +| `erofs_over_scratch` | NVX's Ubuntu EROFS image, with its own identity, over a copy of the scratch template | A root that the worker assembled: the image, read-only, under the overlay, whose upper and work directories are on the scratch, each mounted once below the runtime directory, with the options of the default mode, which trusts its lowers; the merged root's contents, with a file that reaches the upper and not the image; a copy-up of a file's metadata alone, which leaves its data in the image, and the rename of a lower directory; the image's and the scratch's disks; the agent's and the workload's cgroups, with their resource values; a worker that is PID 1's child, and PID 1's unrelated child, untouched; and a teardown that unmounts the scratch cleanly | +| `erofs_over_scratch_with_untrusted_lowers` | The image over the scratch, in the untrusted-lowers mode, with a journal that the launch allows | The root, whose overlay has none of the trusting mode's options, copies up a file's data too, and refuses the rename of a lower directory with `EXDEV` | +| `erofs_over_ram` | The image over a tmpfs in RAM | The root, whose upper and work directories are on a tmpfs of the kernel's default size, whose root directory has mode 0755, with no scratch mounted, and no disk but the image's | +| `erofs_over_ram_with_untrusted_lowers` | The image over a tmpfs, in the untrusted-lowers mode | The root over the tmpfs, which copies up a file's data too, and refuses the rename | +| `full_scratch_refuses_writes` | The image over the scratch | A file that fills the scratch, through the root, until a write fails with `ENOSPC`, and a write that succeeds once the file is gone | +| `full_ram_upper_refuses_writes` | The image over a tmpfs | A file that fills the tmpfs, half of the guest's memory, until a write fails with `ENOSPC`, and a write that succeeds once the file is gone | +| `forbidden_journal_fails_the_assembly` | The image over the scratch, whose journal the launch forbids | The assembly's failure, which the launch reports, and a VM that ends with 14 before the guest mounts the scratch | | `other_identity_fails_the_assembly` | The image, of which the launch expects another identity | The assembly's failure, which the launch reports, and a VM that ends with 14 before the guest mounts the scratch | | `invalid_configuration_fails_its_stage` | A relative runtime directory | The configuration's failure, which the launch reports, and a VM that ends with 11 | | `busy_root_fails_the_teardown` | The image over the scratch, which the `hold` probe keeps open | The teardown's failure, which the stop reports, and a VM that ends with 15, leaving the scratch mounted | @@ -589,11 +611,12 @@ own: | `a_refused_mapping_latches_the_root` | The image over the scratch, in which the `map-refused` probe plants a symbolic link | A mapping through the link that the root refuses, nothing mounted or created, and a root that refuses every later mapping, then tears down cleanly | A row names the root, the image, the identity that the launch expects of it, -the upper, and the runtime directory, and what the scenario expects: a root -that the guest assembles, with a check that probes it, a teardown that fails -after a check, or the stage that fails, with its problem. Each scenario checks -the launch's record, and whether the VM left the scratch's journal to recover, -which tells whether the guest unmounted the scratch. +the upper, the runtime directory, the overlay mode, and the scratch's journal +policy, and what the scenario expects: a root that the guest assembles, with +a check that probes it, a teardown that fails after a check, or the stage that +fails, with its problem. Each scenario checks the launch's record, and, over a +scratch, whether the VM left the scratch's journal to recover, which tells +whether the guest unmounted the scratch. The scenarios use the fixtures of NVX's own layered scenarios, from NVX's artifacts: the Ubuntu EROFS image, `ubuntu-distro.erofs`, its manifest, which diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 4ab596f3a..51349eda6 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -12,14 +12,18 @@ //! `whp`, or else on the one that `nvx::Hypervisor::from_env_or_default` selects. //! //! LAYERED, `--distro FORMAT:PATH [--distro-identity UUID] --upper (scratch:PATH | ram) -//! [--runtime-dir DIR]`, gives the VM a layered root, which the guest agent's worker assembles; -//! see `runtime_adapter::layered` and `runtime_adapter::worker`. Its image is the raw file PATH, -//! an EROFS filesystem, `erofs`, or a GPT disk, `gpt`, of which the guest expects the identity -//! UUID, if given. Its upper is the ext4 scratch image PATH, or RAM. The guest assembles the -//! root below its runtime directory DIR, `/run/runtime-adapter` by default. Once attached, the -//! launch sends the root configuration, and exchanges the `Ping` and the `Echo` once the guest -//! assembled the root. If the guest reports that a stage of the root failed instead, the launch -//! waits for the VM to end, abandons it, and checks that the VM ended with the stage's status. +//! [--runtime-dir DIR] [--overlay-mode MODE] [--scratch-journal POLICY]`, gives the VM a layered +//! root, which the guest agent's worker assembles; see `runtime_adapter::layered` and +//! `runtime_adapter::worker`. Its image is the raw file PATH, an EROFS filesystem, `erofs`, or a +//! GPT disk, `gpt`, of which the guest expects the identity UUID, if given. Its upper is the ext4 +//! scratch image PATH, or RAM. The guest assembles the root below its runtime directory DIR, +//! `/run/runtime-adapter` by default, in the overlay mode MODE, `trusted-lowers` or +//! `untrusted-lowers`, and checks a scratch's journal against the policy POLICY, `allowed` or +//! `forbidden`, with the guest's defaults, `trusted-lowers` and `allowed`, unless given. Once +//! attached, the launch sends the root configuration, and exchanges the `Ping` and the `Echo` +//! once the guest assembled the root. If the guest reports that a stage of the root failed +//! instead, the launch waits for the VM to end, abandons it, and checks that the VM ended with +//! the stage's status. //! - `runtime-adapter-host reattach STORE` reattaches to the VM of STORE, which an earlier process //! launched and which it holds although it is not its child, attaches a new session to it, in //! the session's next epoch, and exchanges a `Ping`, leaving the VM running. @@ -85,7 +89,7 @@ use runtime_adapter::host::Client; use runtime_adapter::initramfs; use runtime_adapter::layered::{LaunchRecord, LayeredLaunch}; use runtime_adapter::probe::{Probe, ProbeReply, escape}; -use runtime_adapter::root::{RootFailure, RootReply}; +use runtime_adapter::root::{JournalPolicy, OverlayMode, RootFailure, RootReply}; use runtime_adapter::runtime::{GUEST_BUILD, Runtime}; /// How long a command waits for the sandbox's lock. @@ -114,7 +118,8 @@ const USAGE: &str = concat!( " runtime-adapter-host launch STORE --guest-agent PATH [--hypervisor NAME]", " [--guest-build BUILD]\n", " [--distro FORMAT:PATH [--distro-identity UUID]", - " --upper (scratch:PATH | ram) [--runtime-dir DIR]]\n", + " --upper (scratch:PATH | ram) [--runtime-dir DIR]\n", + " [--overlay-mode MODE] [--scratch-journal POLICY]]\n", " runtime-adapter-host (reattach | stop) STORE [--guest-build BUILD]\n", " runtime-adapter-host probe STORE PROBE... [--guest-build BUILD]", ); @@ -166,6 +171,10 @@ struct Layered { scratch: Option, /// The guest's runtime directory, unless the default. runtime: Option, + /// The overlay's mode, unless the guest's default. + overlay_mode: Option, + /// Whether the scratch may have a journal, unless the guest's default. + scratch_journal: Option, } /// The options of a command. @@ -185,6 +194,10 @@ struct Options { upper: Option>, /// `--runtime-dir`. runtime: Option, + /// `--overlay-mode`. + overlay_mode: Option, + /// `--scratch-journal`. + scratch_journal: Option, } fn main() -> ExitCode { @@ -273,6 +286,9 @@ fn parse(arguments: Vec) -> Result { Ok(match name.as_str() { "launch" => { let layered = match (options.distro, options.upper) { + (Some(_), Some(None)) if options.scratch_journal.is_some() => { + return Err("--scratch-journal needs --upper scratch:PATH".to_owned()); + } (Some((format, distro)), Some(scratch)) => Some(Layered { distro, image: RootImage { @@ -281,16 +297,25 @@ fn parse(arguments: Vec) -> Result { }, scratch, runtime: options.runtime, + overlay_mode: options.overlay_mode, + scratch_journal: options.scratch_journal, }), (Some(_), None) => return Err("--distro needs --upper".to_owned()), (None, _) if options.identity.is_some() => { return Err("--distro-identity needs --distro".to_owned()); } (None, Some(_)) => return Err("--upper needs --distro".to_owned()), - (None, None) if options.runtime.is_some() => { - return Err("--runtime-dir needs --distro".to_owned()); + (None, None) => { + let layered = [ + ("--runtime-dir", options.runtime.is_some()), + ("--overlay-mode", options.overlay_mode.is_some()), + ("--scratch-journal", options.scratch_journal.is_some()), + ]; + if let Some((option, _)) = layered.iter().find(|(_, given)| *given) { + return Err(format!("{option} needs --distro")); + } + None } - (None, None) => None, }; Command::Launch { store, @@ -365,6 +390,20 @@ fn parse_option( ("launch", "--runtime-dir") if options.runtime.is_none() => { options.runtime = Some(text(value)?); } + ("launch", "--overlay-mode") if options.overlay_mode.is_none() => { + let value = text(value)?; + let mode = OverlayMode::from_name(&value).ok_or_else(|| { + format!("--overlay-mode takes trusted-lowers or untrusted-lowers, not {value:?}") + })?; + options.overlay_mode = Some(mode); + } + ("launch", "--scratch-journal") if options.scratch_journal.is_none() => { + let value = text(value)?; + let policy = JournalPolicy::from_name(&value).ok_or_else(|| { + format!("--scratch-journal takes allowed or forbidden, not {value:?}") + })?; + options.scratch_journal = Some(policy); + } (_, "--guest-build") if options.guest_build.is_none() => { options.guest_build = Some(text(value)?); } @@ -393,13 +432,16 @@ fn launch( .probe() .map_err(|error| describe(&error))?; let layered = layered - .map(|layered| { - LayeredLaunch::new( + .map(|layered| -> nvx::Result { + let mut launch = LayeredLaunch::new( &layered.distro, layered.image, layered.scratch.as_deref(), layered.runtime.as_deref(), - ) + )?; + launch.config.overlay_mode = layered.overlay_mode; + launch.config.scratch_journal = layered.scratch_journal; + Ok(launch) }) .transpose() .map_err(|error| describe(&error))?; diff --git a/tests/runtime-adapter/src/root.rs b/tests/runtime-adapter/src/root.rs index b40662d80..72bdbe861 100644 --- a/tests/runtime-adapter/src/root.rs +++ b/tests/runtime-adapter/src/root.rs @@ -5,7 +5,8 @@ //! format and the identity that the guest expects of it, and the upper. A [`RootConfig`] carries //! the rest that the root needs, which only the adapter defines, after the host attaches: the //! runtime directory, the cgroups, their resource values, the sizes that the block devices must -//! have, and how long the guest waits for them. +//! have, how long the guest waits for them, and how the guest assembles the overlay over the +//! upper. //! //! A root configuration is UTF-8 text: one `NAME=VALUE` line per field, each ended by a newline, //! in any order, and each at most once. Numbers are decimal, without a sign or leading zeros: @@ -22,6 +23,8 @@ //! | `workload-pids-max` | The workload cgroup's `pids.max` | no | //! | `distro-sectors` | The size of the image's block device, in 512-byte sectors | no | //! | `scratch-sectors` | The size of the scratch block device, in 512-byte sectors | no | +//! | `overlay-mode` | The overlay's [mode](OverlayMode): `trusted-lowers`, the guest's default, or `untrusted-lowers` | no | +//! | `scratch-journal` | Whether an ext4 scratch may have a journal, a [`JournalPolicy`]: `allowed`, the guest's default, or `forbidden` | no | //! //! The paths are absolute and canonical guest paths: not `/`, without an empty, `.`, or `..` //! component, or a trailing `/`. The runtime directory holds no `,`, `:`, `\`, or `=`, which the @@ -83,8 +86,12 @@ const WORKLOAD_PIDS_MAX: &str = "workload-pids-max"; const DISTRO_SECTORS: &str = "distro-sectors"; /// See [`VERSION`]. const SCRATCH_SECTORS: &str = "scratch-sectors"; +/// See [`VERSION`]. +const OVERLAY_MODE: &str = "overlay-mode"; +/// See [`VERSION`]. +const SCRATCH_JOURNAL: &str = "scratch-journal"; /// Every field, in the order in which [`RootConfig::encode`] writes them. -const FIELDS: [&str; 10] = [ +const FIELDS: [&str; 12] = [ VERSION, RUNTIME, AGENT_CGROUP, @@ -95,8 +102,68 @@ const FIELDS: [&str; 10] = [ WORKLOAD_PIDS_MAX, DISTRO_SECTORS, SCRATCH_SECTORS, + OVERLAY_MODE, + SCRATCH_JOURNAL, ]; +/// The overlay mode that a configuration names, which the guest's assembly applies: the +/// `nvx::guest::root::OverlayMode` of the same name, which selects the overlay's one option set. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum OverlayMode { + /// `trusted-lowers`: `metacopy=on,xino=on`, for lowers whose overlay xattrs the guest trusts. + TrustedLowers, + /// `untrusted-lowers`: `metacopy=off,redirect_dir=nofollow`, for lowers that may carry + /// untrusted overlay xattrs. + UntrustedLowers, +} + +impl OverlayMode { + /// Every mode. + pub const ALL: [Self; 2] = [Self::TrustedLowers, Self::UntrustedLowers]; + + /// Returns the mode's name in a configuration. + pub fn as_str(self) -> &'static str { + match self { + Self::TrustedLowers => "trusted-lowers", + Self::UntrustedLowers => "untrusted-lowers", + } + } + + /// Returns the mode named `name`, if any. + pub fn from_name(name: &str) -> Option { + Self::ALL.into_iter().find(|mode| mode.as_str() == name) + } +} + +/// Whether the guest's assembly allows an ext4 scratch a journal, which a configuration names: +/// the `nvx::guest::root::Journal` of the same name, which the assembly checks in the scratch's +/// superblock. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum JournalPolicy { + /// `allowed`: the scratch may have a journal. + Allowed, + /// `forbidden`: the assembly refuses a scratch with a journal. + Forbidden, +} + +impl JournalPolicy { + /// Every policy. + pub const ALL: [Self; 2] = [Self::Allowed, Self::Forbidden]; + + /// Returns the policy's name in a configuration. + pub fn as_str(self) -> &'static str { + match self { + Self::Allowed => "allowed", + Self::Forbidden => "forbidden", + } + } + + /// Returns the policy named `name`, if any. + pub fn from_name(name: &str) -> Option { + Self::ALL.into_iter().find(|policy| policy.as_str() == name) + } +} + /// The host's configuration of a guest's layered root: everything that the root needs besides its /// boot tokens, as the [module documentation](self) lists it. /// @@ -121,12 +188,17 @@ pub struct RootConfig { pub distro_sectors: Option, /// The size that the scratch block device must have, in 512-byte sectors, if any. pub scratch_sectors: Option, + /// The overlay's mode, unless the guest's default. + pub overlay_mode: Option, + /// Whether an ext4 scratch may have a journal, unless the guest's default. + pub scratch_journal: Option, } impl Default for RootConfig { /// Returns the configuration of the defaults: [`DEFAULT_RUNTIME`], [`DEFAULT_AGENT_CGROUP`], /// [`DEFAULT_WORKLOAD_CGROUP`], [`DEFAULT_DEVICE_TIMEOUT`], [`DEFAULT_AGENT_MEMORY_LOW`], - /// [`DEFAULT_WORKLOAD_MEMORY_MAX`], and [`DEFAULT_WORKLOAD_PIDS_MAX`], with no sizes. + /// [`DEFAULT_WORKLOAD_MEMORY_MAX`], and [`DEFAULT_WORKLOAD_PIDS_MAX`], with no sizes, and the + /// guest's default overlay mode and journal policy. fn default() -> Self { Self { runtime: DEFAULT_RUNTIME.to_owned(), @@ -138,6 +210,8 @@ impl Default for RootConfig { workload_pids_max: Some(DEFAULT_WORKLOAD_PIDS_MAX), distro_sectors: None, scratch_sectors: None, + overlay_mode: None, + scratch_journal: None, } } } @@ -165,6 +239,18 @@ impl RootConfig { .into_iter() .filter_map(|(name, value)| Some((name, value?.to_string()))), ); + let named = [ + (OVERLAY_MODE, self.overlay_mode.map(OverlayMode::as_str)), + ( + SCRATCH_JOURNAL, + self.scratch_journal.map(JournalPolicy::as_str), + ), + ]; + fields.extend( + named + .into_iter() + .filter_map(|(name, value)| Some((name, value?.to_owned()))), + ); fields } @@ -262,6 +348,18 @@ impl RootConfig { MAX_DEVICE_TIMEOUT.as_millis() ))); } + let unknown = + |name: &str, text: &str| refused(format!("names the unknown {name} {text:?}")); + let overlay_mode = values + .get(OVERLAY_MODE) + .map(|text| OverlayMode::from_name(text).ok_or_else(|| unknown(OVERLAY_MODE, text))) + .transpose()?; + let scratch_journal = values + .get(SCRATCH_JOURNAL) + .map(|text| { + JournalPolicy::from_name(text).ok_or_else(|| unknown(SCRATCH_JOURNAL, text)) + }) + .transpose()?; Ok(Self { runtime, agent_cgroup, @@ -272,6 +370,8 @@ impl RootConfig { workload_pids_max: number(WORKLOAD_PIDS_MAX)?, distro_sectors: number(DISTRO_SECTORS)?, scratch_sectors: number(SCRATCH_SECTORS)?, + overlay_mode, + scratch_journal, }) } } @@ -477,6 +577,8 @@ mod tests { RootConfig { distro_sectors: Some(116_960), scratch_sectors: Some(131_072), + overlay_mode: Some(OverlayMode::UntrustedLowers), + scratch_journal: Some(JournalPolicy::Forbidden), ..RootConfig::default() } } @@ -494,8 +596,17 @@ mod tests { workload-memory-max=134217728\n\ workload-pids-max=128\n\ distro-sectors=116960\n\ - scratch-sectors=131072\n" + scratch-sectors=131072\n\ + overlay-mode=untrusted-lowers\n\ + scratch-journal=forbidden\n" ); + // Each mode and policy has its own name. + for mode in OverlayMode::ALL { + assert_eq!(OverlayMode::from_name(mode.as_str()), Some(mode)); + } + for policy in JournalPolicy::ALL { + assert_eq!(JournalPolicy::from_name(policy.as_str()), Some(policy)); + } let minimal = RootConfig { agent_memory_low: None, workload_memory_max: None, @@ -659,6 +770,14 @@ mod tests { format!("{base}scratch-sectors= 1\n").into_bytes(), "not a positive decimal number", ), + ( + format!("{base}overlay-mode=TrustedLowers\n").into_bytes(), + "names the unknown overlay-mode \"TrustedLowers\"", + ), + ( + format!("{base}scratch-journal=\n").into_bytes(), + "names the unknown scratch-journal \"\"", + ), ]; for (body, problem) in cases { let error = RootConfig::decode(&body).unwrap_err(); diff --git a/tests/runtime-adapter/src/worker.rs b/tests/runtime-adapter/src/worker.rs index a86ca8f25..922655632 100644 --- a/tests/runtime-adapter/src/worker.rs +++ b/tests/runtime-adapter/src/worker.rs @@ -28,8 +28,9 @@ //! confirms the stop with `Stopped`, and ends. //! //! Before the root configuration, it accepts only the configuration, or a `Stop`, which ends it -//! without a root. It assembles an EROFS image over an ext4 scratch, and refuses the other layouts -//! that the boot tokens can name at the layout stage. +//! without a root. It assembles an EROFS image, over an ext4 scratch or over RAM, in the overlay +//! mode that the configuration names, and refuses a GPT image at the layout stage, as it refuses a +//! configuration that configures a scratch for a root over RAM. //! //! The worker, and with it the VM, ends with 0 once the host stopped it, after a teardown that //! succeeded, with the stage's [exit status](RootStage::exit_status) once a stage failed, from 11 @@ -50,7 +51,9 @@ use nix::sys::signal::{self, Signal}; use nix::sys::wait::{self, WaitPidFlag, WaitStatus}; use nix::unistd::Pid; use nvx::guest::cgroup::Layout; -use nvx::guest::root::{self, Assembly, Bootstrap, Device, Resources, Root}; +use nvx::guest::root::{ + self, Assembly, Bootstrap, Device, Image, Journal, Resources, Root, Upper, UpperKind, +}; use nvx::protocol::root::{ DISTRO_MMIO_BASE, ImageFormat, RootLayout, RootUpper, SCRATCH_MMIO_BASE, }; @@ -58,7 +61,7 @@ use nvx::protocol::root::{ use crate::frame::{MAX_BODY, MAX_TEXT_BODY, Message, truncated}; use crate::guest::{self, Ended, ServeError, Step}; use crate::probe::{Facts, Probe}; -use crate::root::{RootConfig, RootFailure, RootStage}; +use crate::root::{JournalPolicy, OverlayMode, RootConfig, RootFailure, RootStage}; /// The status with which the worker ends once its session failed or the broker closed it, and /// with which PID 1 ends the VM if a child cannot start or the worker's status cannot be collected. @@ -184,8 +187,8 @@ fn serve_root( fn assemble(operations: &mut dyn RootOperations, body: &[u8]) -> Result<(), RootFailure> { let config = RootConfig::decode(body) .map_err(|error| RootFailure::new(RootStage::Configuration, error.problem()))?; - let layout = - layout(operations).map_err(|problem| RootFailure::new(RootStage::Layout, problem))?; + let layout = layout(operations, &config) + .map_err(|problem| RootFailure::new(RootStage::Layout, problem))?; operations .bootstrap(&config) .map_err(|problem| RootFailure::new(RootStage::Bootstrap, problem))?; @@ -195,18 +198,27 @@ fn assemble(operations: &mut dyn RootOperations, body: &[u8]) -> Result<(), Root } /// Reads the boot layout of the root from the kernel command line, through `operations`, and -/// checks that the worker assembles it: an EROFS image over an ext4 scratch. -fn layout(operations: &mut dyn RootOperations) -> Result { +/// checks that the worker assembles it as `config` configures it: an EROFS image, over an ext4 +/// scratch, or over RAM if `config` configures no scratch. +fn layout(operations: &mut dyn RootOperations, config: &RootConfig) -> Result { let command_line = operations.command_line()?; let layout = RootLayout::parse(&command_line) .map_err(|error| format!("the kernel command line has invalid boot tokens: {error}"))? .ok_or("the kernel command line names no layered root")?; - if (layout.image.format, layout.upper) != (ImageFormat::Erofs, RootUpper::Scratch) { + if layout.image.format != ImageFormat::Erofs { return Err(format!( - "the worker assembles only an EROFS image over an ext4 scratch, not {} over {}", - layout.image.format, layout.upper + "the worker assembles only an EROFS image, not a {} image", + layout.image.format )); } + if layout.upper == RootUpper::Ram + && (config.scratch_sectors.is_some() || config.scratch_journal.is_some()) + { + return Err( + "the root configuration configures a scratch, but the boot tokens keep the upper in RAM" + .to_owned(), + ); + } Ok(layout) } @@ -281,21 +293,42 @@ impl RootOperations for LibraryRoot { Ok(()) } - /// Runs `nvx::guest::root::assemble` on the block devices at the MMIO bases of their roles, - /// with the identity that the boot tokens expect of the image, and the configuration's - /// runtime directory, device sizes, and timeout. + /// Runs `nvx::guest::root::assemble` on the block devices at the MMIO bases of their roles: + /// the EROFS image, with the identity that the boot tokens expect of it, over the ext4 + /// scratch, with the configuration's journal policy, or over a tmpfs, in the configuration's + /// overlay mode, with the configuration's runtime directory, device sizes, and timeout. A + /// policy or a mode that the configuration does not name is the guest's default: a scratch + /// may have a journal, and the overlay trusts its lowers. fn assemble(&mut self, config: &RootConfig, layout: &RootLayout) -> Result<(), String> { + let upper = match layout.upper { + RootUpper::Scratch => Upper::Scratch { + device: Device { + mmio: SCRATCH_MMIO_BASE, + sectors: config.scratch_sectors, + }, + journal: match config.scratch_journal { + None | Some(JournalPolicy::Allowed) => Journal::Allowed, + Some(JournalPolicy::Forbidden) => Journal::Forbidden, + }, + }, + RootUpper::Ram => Upper::Tmpfs, + }; + let overlay = match config.overlay_mode { + None => root::OverlayMode::default(), + Some(OverlayMode::TrustedLowers) => root::OverlayMode::TrustedLowers, + Some(OverlayMode::UntrustedLowers) => root::OverlayMode::UntrustedLowers, + }; let assembly = Assembly { runtime: PathBuf::from(&config.runtime), distro: Device { mmio: DISTRO_MMIO_BASE, sectors: config.distro_sectors, }, - distro_uuid: layout.image.identity, - scratch: Device { - mmio: SCRATCH_MMIO_BASE, - sectors: config.scratch_sectors, + image: Image::Erofs { + uuid: layout.image.identity, }, + upper, + overlay, timeout: config.device_timeout, }; let root = root::assemble(&assembly).map_err(|error| error.to_string())?; @@ -308,10 +341,24 @@ impl RootOperations for LibraryRoot { let (Some(bootstrap), Some(root)) = (&self.bootstrap, &mut self.root) else { return Err("the worker holds no root to probe".to_owned()); }; + // The upper's mount point, below the runtime directory, as `nvx::guest::root` names it. + let upper = match root.upper() { + UpperKind::Scratch => "scratch", + UpperKind::Tmpfs => "tmpfs", + kind => return Err(format!("the worker probes no root over a {kind:?} upper")), + }; let (runtime, path) = (root.runtime().to_owned(), root.path().to_owned()); + let upper = runtime.join(upper); + let lowers = root + .lowers() + .iter() + .map(|lower| lower.path().to_owned()) + .collect::>(); let context = probes::Context { runtime: &runtime, root: &path, + lowers: &lowers, + upper: &upper, agent: bootstrap.layout().agent(), workload: bootstrap.layout().workload(), sibling: self.sibling, diff --git a/tests/runtime-adapter/src/worker/mapping.rs b/tests/runtime-adapter/src/worker/mapping.rs index afcbb8da6..13f718a36 100644 --- a/tests/runtime-adapter/src/worker/mapping.rs +++ b/tests/runtime-adapter/src/worker/mapping.rs @@ -103,7 +103,7 @@ fn map(context: &Context<'_>, root: &mut Root) -> io::Result { facts.push("file.write", open_for_writing(&target("srv/maps/file.txt"))); facts.push("out.write", create(&target("srv/maps/out/new"))); facts.push("out.source", probes::read(&sources.join("out/new"))); - let upper = context.runtime.join("scratch/upper/srv/maps"); + let upper = context.upper.join("upper/srv/maps"); facts.push("upper.directory", yes(upper.is_dir())); let file = fs::symlink_metadata(upper.join("file.txt")); facts.push( diff --git a/tests/runtime-adapter/src/worker/probes.rs b/tests/runtime-adapter/src/worker/probes.rs index 48d3503a2..6a4245409 100644 --- a/tests/runtime-adapter/src/worker/probes.rs +++ b/tests/runtime-adapter/src/worker/probes.rs @@ -3,24 +3,30 @@ //! //! | Probe | Facts | //! | --- | --- | -//! | `mounts` | For the image's mount, `distro`, the scratch's, `scratch`, and the overlay's, `root`: the number of mounts at its mount point, `NAME.count`, and of the topmost, its `NAME.type`, `NAME.source`, `NAME.options`, and `NAME.super`, the file system's own options | -//! | `contents` | The root's `os-release.id` and `root.mode`. It writes [`PROBE_FILE`] in the root, and reports what the root, `write.root`, the upper, `write.upper`, and the image, `write.lower`, hold at that name, and whether the image refuses a new file, `lower.write` | +//! | `mounts` | For the mount of each layer, by the name of its mount point, `distro` for an EROFS image, the scratch's, `scratch`, the tmpfs's, `tmpfs`, and the overlay's, `root`: the number of mounts at its mount point, `NAME.count`, and of the topmost, its `NAME.type`, `NAME.source`, `NAME.options`, and `NAME.super`, the file system's own options | +//! | `contents` | The root's `os-release.id` and `root.mode`. It writes [`PROBE_FILE`] in the root, and reports what the root, `write.root`, the upper, `write.upper`, and the top layer, `write.lower`, hold at that name, and whether the top layer refuses a new file, `lower.write` | +//! | `copy-up FILE` | It sets the mode of FILE, a regular file of the layers, to the mode that it has, which copies the file up, and reports its size in the topmost layer that holds it, `lower.size`, and in the upper, `upper.size`; whether the upper's copy holds the file's data, `upper.data`: `copied`, `none` once the overlay copied up the file's metadata alone, or `different`; and whether the root still shows the layer's data, `root.data`: `intact` or `different` | +//! | `rename DIRECTORY` | It renames DIRECTORY, a directory of the layers, to `.runtime-adapter-renamed` at the top of the root, and back if it can: `outcome` is `allowed`, `refused` once the overlay refused it with `EXDEV`, or `failed: PROBLEM` | +//! | `fill` | It writes a file in the root until the upper is full, then removes it, and writes a file there again: the upper's size, `total`, and the bytes available in it before, `available`, in bytes; the bytes that it wrote, `written`; the error that ended the writes, `error`: `ENOSPC` once the upper was full, or the problem; what the second file holds, `after`; and the guest's memory, `memory`, in bytes | +//! | `devices` | The guest's disks, the block devices that a device backs: `disks`, each as `NAME:SECTORS:ACCESS`, its size in 512-byte sectors and its access mode, `ro` or `rw`, in the order of their names, separated by spaces | //! | `cgroups` | The file system at the cgroups' parent, `parent.type`, and its `parent.subtree-control`; whether each cgroup exists, `agent.exists` and `workload.exists`; `agent.memory.low`, `workload.memory.max`, and `workload.pids.max`; the number of processes in each, `agent.processes` and `workload.processes`; and `user-namespaces`, the guest's limit of user namespaces, or `unavailable` without them | //! | `processes` | For PID 1, `pid1`, the worker, `worker`, and PID 1's unrelated child, `sibling`: `NAME.pid`, and while the process exists, `NAME.state`, `NAME.ppid`, `NAME.root`, `NAME.cgroup`, `NAME.command`, and `NAME.namespaces`: `same` if it shares PID 1's, or the ones that differ | //! | `hold` | It opens the root, and keeps it open until the worker ends, so that the teardown finds the root busy: `held`, the path that it holds | //! -//! None takes arguments. A value that a probe cannot read is `absent` if its file does not exist, -//! and otherwise `unreadable: PROBLEM`. The probes that map beneath the root are those of the +//! FILE and DIRECTORY are relative paths in the root, of normal components, and the other probes take +//! no arguments. A value that a probe cannot read is `absent` if its file does not exist, and +//! otherwise `unreadable: PROBLEM`. The probes that map beneath the root are those of the //! [`mapping`](super::mapping) module. use std::ffi::OsString; use std::fs::{self, File}; -use std::io; +use std::io::{self, Write}; use std::os::unix::ffi::OsStringExt; use std::os::unix::fs::PermissionsExt; -use std::path::{Path, PathBuf}; +use std::path::{Component, Path, PathBuf}; use nix::errno::Errno; +use nix::sys::statvfs; use crate::probe::{Facts, Probe}; @@ -28,8 +34,21 @@ use crate::probe::{Facts, Probe}; pub const PROBE_FILE: &str = ".runtime-adapter-probe"; /// What the `contents` probe writes in [`PROBE_FILE`]. pub const PROBE_CONTENTS: &str = "runtime-adapter"; -/// The file that the `contents` probe tries to create at the top of the image. +/// The file that the `contents` probe tries to create at the top of the top layer. const LOWER_FILE: &str = ".runtime-adapter-lower"; +/// The name to which the `rename` probe renames its directory, at the top of the root. +const RENAMED_TO: &str = ".runtime-adapter-renamed"; +/// The file that the `fill` probe writes until the upper is full. +const FILL_FILE: &str = ".runtime-adapter-fill"; +/// How many bytes the `fill` probe writes at a time. +const FILL_CHUNK: usize = 1 << 20; +/// How many bytes the `fill` probe writes beyond those available in the upper before it stops, +/// which a file system that keeps its size never takes. +const FILL_SLACK: u64 = 16 << 20; +/// The guest's memory information, whose `MemTotal` the `fill` probe reports. +const MEMORY_INFORMATION: &str = "/proc/meminfo"; +/// The guest's block devices. +const BLOCK_DEVICES: &str = "/sys/block"; /// The mount table of the worker's mount namespace. pub(super) const MOUNTINFO: &str = "/proc/self/mountinfo"; /// The guest's limit of user namespaces, which only a kernel with user namespaces has. @@ -41,12 +60,17 @@ const NAMESPACES: [&str; 7] = ["cgroup", "ipc", "mnt", "net", "pid", "user", "ut /// What the probes check: the worker's root and cgroups, and PID 1's unrelated child. #[derive(Clone, Copy, Debug)] pub struct Context<'a> { - /// The runtime directory, below which `nvx::guest::root` mounts the root: the image at - /// `layers/distro`, the scratch, with the overlay's `upper` directory, at `scratch`, and the - /// overlay, the root, at `rootfs`. + /// The runtime directory, below which `nvx::guest::root` mounts the root: the layers below + /// `layers`, the upper, an ext4 scratch at `scratch` or a tmpfs at `tmpfs`, and the overlay, + /// the root, at `rootfs`. pub runtime: &'a Path, /// The root, the overlay's mount point. pub root: &'a Path, + /// The mount points of the root's layers, the base first: `layers/distro`, the one of an EROFS + /// image. + pub lowers: &'a [PathBuf], + /// The upper's mount point, which holds the overlay's `upper` directory. + pub upper: &'a Path, /// The agent's cgroup. pub agent: &'a Path, /// The workload cgroup. @@ -56,15 +80,28 @@ pub struct Context<'a> { } impl Context<'_> { - /// Returns the image's mount point. - fn distro(&self) -> PathBuf { - self.runtime.join("layers/distro") + /// Returns the top layer's mount point, which an assembled root has. + fn top(&self) -> io::Result<&Path> { + self.lowers + .last() + .map(PathBuf::as_path) + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "the root has no layer")) } /// Returns the scratch's mount point. fn scratch(&self) -> PathBuf { self.runtime.join("scratch") } + + /// Returns the tmpfs's mount point. + fn tmpfs(&self) -> PathBuf { + self.runtime.join("tmpfs") + } + + /// Returns the overlay's `upper` directory. + fn upper_directory(&self) -> PathBuf { + self.upper.join("upper") + } } /// Runs `probe` with `context`, and returns its facts. `held` keeps the descriptors that a probe @@ -74,31 +111,71 @@ impl Context<'_> { /// with arguments, and one that fails. pub fn run(probe: &Probe, context: &Context<'_>, held: &mut Vec) -> Result { let name = probe.name(); - if !["mounts", "contents", "cgroups", "processes", "hold"].contains(&name) { + let probes = [ + "mounts", + "contents", + "copy-up", + "rename", + "fill", + "devices", + "cgroups", + "processes", + "hold", + ]; + if !probes.contains(&name) { return Err(format!("the worker runs no probe named {name}")); } - if !probe.arguments().is_empty() { - return Err(format!("the {name} probe takes no arguments")); - } - let facts = match name { - "mounts" => mounts(context), - "contents" => contents(context), - "cgroups" => cgroups(context), - "processes" => processes(context), + let path = match (name, probe.arguments()) { + ("copy-up" | "rename", [argument]) => Some(path_argument(name, argument)?), + ("copy-up" | "rename", _) => return Err(format!("the {name} probe takes one argument")), + (_, []) => None, + _ => return Err(format!("the {name} probe takes no arguments")), + }; + let facts = match (name, path) { + ("mounts", _) => mounts(context), + ("contents", _) => contents(context), + ("copy-up", Some(file)) => copy_up(context, file), + ("rename", Some(directory)) => rename(context, directory), + ("fill", _) => fill(context), + ("devices", _) => devices(), + ("cgroups", _) => cgroups(context), + ("processes", _) => processes(context), _ => hold(context, held), }; facts.map_err(|error| format!("the {name} probe failed: {error}")) } +/// Returns the path that the argument `argument` of the probe `name` names in the root: a relative +/// path of normal components, or the problem. +fn path_argument<'a>(name: &str, argument: &'a str) -> Result<&'a Path, String> { + let path = Path::new(argument); + let mut components = path.components().peekable(); + let normal = components.peek().is_some() + && components.all(|component| matches!(component, Component::Normal(_))); + if normal { + Ok(path) + } else { + Err(format!( + "the {name} probe takes a relative path in the root, not {argument:?}" + )) + } +} + /// Runs the `mounts` probe. fn mounts(context: &Context<'_>) -> io::Result { let table = mount_table(&fs::read_to_string(MOUNTINFO)?)?; let mut facts = Facts::new(); - for (name, point) in [ - ("distro", context.distro()), + let lowers = context.lowers.iter().map(|lower| { + let name = lower.file_name().unwrap_or_default().to_string_lossy(); + (name.into_owned(), lower.clone()) + }); + let others = [ ("scratch", context.scratch()), + ("tmpfs", context.tmpfs()), ("root", context.root.to_owned()), - ] { + ]; + let points = lowers.chain(others.map(|(name, point)| (name.to_owned(), point))); + for (name, point) in points { let mounts = table .iter() .filter(|mount| mount.point == point) @@ -129,10 +206,13 @@ fn contents(context: &Context<'_>) -> io::Result { facts.push("root.mode", format!("{mode:o}")); fs::write(root.join(PROBE_FILE), PROBE_CONTENTS)?; facts.push("write.root", read(&root.join(PROBE_FILE))); - let upper = context.scratch().join("upper").join(PROBE_FILE); - facts.push("write.upper", read(&upper)); - facts.push("write.lower", read(&context.distro().join(PROBE_FILE))); - let lower = match File::create_new(context.distro().join(LOWER_FILE)) { + facts.push( + "write.upper", + read(&context.upper_directory().join(PROBE_FILE)), + ); + let top = context.top()?; + facts.push("write.lower", read(&top.join(PROBE_FILE))); + let lower = match File::create_new(top.join(LOWER_FILE)) { Ok(_) => "allowed".to_owned(), Err(error) if error.raw_os_error() == Some(Errno::EROFS as i32) => "refused".to_owned(), Err(error) => format!("failed: {error}"), @@ -141,6 +221,132 @@ fn contents(context: &Context<'_>) -> io::Result { Ok(facts) } +/// Runs the `copy-up` probe on `file`. +fn copy_up(context: &Context<'_>, file: &Path) -> io::Result { + let merged = context.root.join(file); + let lower = context + .lowers + .iter() + .rev() + .map(|layer| layer.join(file)) + .find(|path| path.exists()) + .ok_or_else(|| { + io::Error::new( + io::ErrorKind::NotFound, + format!("no layer holds {}", file.display()), + ) + })?; + let lower = fs::read(lower)?; + // A change of the file's metadata alone, which copies the file up. + let permissions = fs::metadata(&merged)?.permissions(); + fs::set_permissions(&merged, permissions)?; + let copy = context.upper_directory().join(file); + let mut facts = Facts::new(); + facts.push("lower.size", lower.len().to_string()); + facts.push("upper.size", fs::metadata(©)?.len().to_string()); + let copied = fs::read(©)?; + let data = if copied == lower { + "copied" + } else if copied.iter().all(|byte| *byte == 0) { + "none" + } else { + "different" + }; + facts.push("upper.data", data); + let shown = if fs::read(&merged)? == lower { + "intact" + } else { + "different" + }; + facts.push("root.data", shown); + Ok(facts) +} + +/// Runs the `rename` probe on `directory`. +fn rename(context: &Context<'_>, directory: &Path) -> io::Result { + let (from, to) = (context.root.join(directory), context.root.join(RENAMED_TO)); + let outcome = match fs::rename(&from, &to) { + Ok(()) => { + fs::rename(&to, &from)?; + "allowed".to_owned() + } + Err(error) if error.raw_os_error() == Some(Errno::EXDEV as i32) => "refused".to_owned(), + Err(error) => format!("failed: {error}"), + }; + let mut facts = Facts::new(); + facts.push("outcome", outcome); + Ok(facts) +} + +/// Runs the `fill` probe. +fn fill(context: &Context<'_>) -> io::Result { + let space = statvfs::statvfs(context.upper)?; + let total = space.blocks() * space.fragment_size(); + let available = space.blocks_available() * space.fragment_size(); + let path = context.root.join(FILL_FILE); + let mut file = File::create_new(&path)?; + let chunk = vec![0xa5; FILL_CHUNK]; + let mut written = 0; + let error = loop { + if written > available + FILL_SLACK { + break format!("the upper took {written} bytes, beyond the {available} available"); + } + match file.write(&chunk) { + Ok(0) => break "a write wrote nothing".to_owned(), + Ok(count) => written += count as u64, + Err(error) if error.kind() == io::ErrorKind::Interrupted => {} + Err(error) if error.raw_os_error() == Some(Errno::ENOSPC as i32) => { + break "ENOSPC".to_owned(); + } + Err(error) => break error.to_string(), + } + }; + drop(file); + fs::remove_file(&path)?; + fs::write(&path, PROBE_CONTENTS)?; + let after = read(&path); + fs::remove_file(&path)?; + let memory = fs::read_to_string(MEMORY_INFORMATION)?; + let memory = memory_total(&memory).ok_or_else(|| { + io::Error::new( + io::ErrorKind::InvalidData, + format!("{MEMORY_INFORMATION} has no MemTotal in kB"), + ) + })?; + let mut facts = Facts::new(); + facts.push("total", total.to_string()); + facts.push("available", available.to_string()); + facts.push("written", written.to_string()); + facts.push("error", error); + facts.push("after", after); + facts.push("memory", memory.to_string()); + Ok(facts) +} + +/// Runs the `devices` probe. +fn devices() -> io::Result { + let mut disks = Vec::new(); + for entry in fs::read_dir(BLOCK_DEVICES)? { + let directory = entry?.path(); + // A disk that a device backs, rather than a loop or RAM device. + if !directory.join("device").exists() { + continue; + } + let name = directory.file_name().unwrap_or_default().to_string_lossy(); + let access = match read(&directory.join("ro")).as_str() { + "0" => "rw".to_owned(), + "1" => "ro".to_owned(), + other => other.to_owned(), + }; + let sectors = read(&directory.join("size")); + disks.push(format!("{name}:{sectors}:{access}")); + } + disks.sort(); + let mut facts = Facts::new(); + facts.push("disks", disks.join(" ")); + Ok(facts) +} + /// Runs the `cgroups` probe. fn cgroups(context: &Context<'_>) -> io::Result { let parent = context.agent.parent().unwrap_or(Path::new("/")); @@ -372,6 +578,20 @@ fn command_text(command: &[u8]) -> String { .join(" ") } +/// Returns the `MemTotal` of `/proc/meminfo`, in bytes. +fn memory_total(text: &str) -> Option { + let line = text + .lines() + .find_map(|line| line.strip_prefix("MemTotal:"))?; + let kibibytes = line + .trim() + .strip_suffix(" kB")? + .trim() + .parse::() + .ok()?; + kibibytes.checked_mul(1024) +} + /// Returns the `ID` of an `os-release` file, without its quotes. fn os_release_id(text: &str) -> Option<&str> { text.lines() @@ -442,6 +662,15 @@ rw,lowerdir=/run/a\\040b/layers/distro,metacopy=on,xino=on assert_eq!(command_text(b""), ""); } + #[test] + fn the_memory_total_is_read_in_bytes() { + let text = "MemTotal: 241540 kB\nMemFree: 1 kB\n"; + assert_eq!(memory_total(text), Some(241_540 * 1024)); + assert_eq!(memory_total("MemFree: 1 kB\n"), None); + assert_eq!(memory_total("MemTotal: 1 MB\n"), None); + assert_eq!(memory_total("MemTotal: x kB\n"), None); + } + #[test] fn the_os_release_id_loses_its_quotes() { let text = "NAME=\"Ubuntu\"\nID=ubuntu\nID_LIKE=debian\n"; @@ -453,9 +682,12 @@ rw,lowerdir=/run/a\\040b/layers/distro,metacopy=on,xino=on #[test] fn unknown_probes_and_arguments_are_refused() { + let lowers = [PathBuf::from("/run/runtime-adapter/layers/distro")]; let context = Context { runtime: Path::new("/run/runtime-adapter"), root: Path::new("/run/runtime-adapter/rootfs"), + lowers: &lowers, + upper: Path::new("/run/runtime-adapter/scratch"), agent: Path::new("/sys/fs/cgroup/agent"), workload: Path::new("/sys/fs/cgroup/workload"), sibling: None, @@ -471,6 +703,37 @@ rw,lowerdir=/run/a\\040b/layers/distro,metacopy=on,xino=on run(&with_argument, &context, &mut held), Err("the hold probe takes no arguments".to_owned()) ); + for (text, problem) in [ + ("copy-up", "the copy-up probe takes one argument"), + ("rename a b", "the rename probe takes one argument"), + ( + "copy-up /etc/os-release", + "the copy-up probe takes a relative path in the root, not \"/etc/os-release\"", + ), + ( + "rename ../mnt", + "the rename probe takes a relative path in the root, not \"../mnt\"", + ), + ( + "rename ./mnt", + "the rename probe takes a relative path in the root, not \"./mnt\"", + ), + ( + "copy-up usr/../etc", + "the copy-up probe takes a relative path in the root, not \"usr/../etc\"", + ), + ] { + let probe = Probe::parse(text).unwrap(); + assert_eq!(run(&probe, &context, &mut held), Err(problem.to_owned())); + } + assert_eq!( + path_argument("rename", "lost+found"), + Ok(Path::new("lost+found")) + ); + assert_eq!( + path_argument("copy-up", "usr/lib/os-release"), + Ok(Path::new("usr/lib/os-release")) + ); assert!(held.is_empty()); } } diff --git a/tests/runtime-adapter/tests/guest_session.rs b/tests/runtime-adapter/tests/guest_session.rs index 0428f3332..c597e3d78 100644 --- a/tests/runtime-adapter/tests/guest_session.rs +++ b/tests/runtime-adapter/tests/guest_session.rs @@ -44,7 +44,7 @@ use nvx::uuid::Uuid; use runtime_adapter::frame::{Decoder, FrameError, HEADER_LEN, MAX_BODY, Message}; use runtime_adapter::guest::{self, NO_ROOT, RECEIVE_WINDOW, ServeError, Served}; use runtime_adapter::probe::{Facts, Probe}; -use runtime_adapter::root::{RootConfig, RootFailure, RootStage}; +use runtime_adapter::root::{JournalPolicy, OverlayMode, RootConfig, RootFailure, RootStage}; use runtime_adapter::worker::{self, RootOperations, Worked}; /// Identity of the broker's first epoch. @@ -1072,7 +1072,7 @@ fn found(probe: &Probe) -> Facts { facts } -/// Returns the layout that the worker assembles: an EROFS image, with the identity that the guest +/// Returns a layout that the worker assembles: an EROFS image, with the identity that the guest /// expects of it, over an ext4 scratch. fn layout() -> RootLayout { RootLayout { @@ -1162,6 +1162,32 @@ fn the_worker_assembles_the_root_serves_it_and_tears_it_down_on_stop() { assert_eq!(root.calls, expected); } +#[test] +fn the_worker_assembles_an_image_over_ram_in_the_overlay_mode_of_its_configuration() { + let ram = RootLayout { + upper: RootUpper::Ram, + ..layout() + }; + let config = RootConfig { + overlay_mode: Some(OverlayMode::UntrustedLowers), + ..RootConfig::default() + }; + let (mut broker, working) = start_worker(FakeRoot::booting(&ram)); + broker.attach(FIRST); + let messages = [Message::RootConfig(config.encode()), Message::Stop]; + let replies = broker.exchange(&messages, OUTER_MAX_PAYLOAD, 2); + assert_eq!(replies, [Message::RootReady, Message::Stopped]); + let (root, worked) = joined(working); + assert_eq!(worked.unwrap(), Worked::Stopped); + let expected = [ + Call::CommandLine, + Call::Bootstrap(config.clone()), + Call::Assemble(config, ram), + Call::Teardown, + ]; + assert_eq!(root.calls, expected); +} + #[test] fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { let config = RootConfig::default(); @@ -1225,15 +1251,31 @@ fn the_worker_reports_the_stage_that_failed_and_ends_with_its_status() { FakeRoot::booting(&gpt), config.encode(), RootStage::Layout, - "assembles only an EROFS image over an ext4 scratch, not gpt over scratch", + "the worker assembles only an EROFS image, not a gpt image", read(), ), ( - "a RAM upper", + "the size of a scratch, for a RAM upper", FakeRoot::booting(&ram), - config.encode(), + RootConfig { + scratch_sectors: Some(131_072), + ..RootConfig::default() + } + .encode(), + RootStage::Layout, + "configures a scratch, but the boot tokens keep the upper in RAM", + read(), + ), + ( + "the journal policy of a scratch, for a RAM upper", + FakeRoot::booting(&ram), + RootConfig { + scratch_journal: Some(JournalPolicy::Allowed), + ..RootConfig::default() + } + .encode(), RootStage::Layout, - "not erofs over ram", + "configures a scratch, but the boot tokens keep the upper in RAM", read(), ), ( diff --git a/tests/runtime-adapter/tests/layered_launch.rs b/tests/runtime-adapter/tests/layered_launch.rs index 284f00249..f6b83520d 100644 --- a/tests/runtime-adapter/tests/layered_launch.rs +++ b/tests/runtime-adapter/tests/layered_launch.rs @@ -199,7 +199,7 @@ fn a_launch_records_what_it_launched() { fn the_host_program_refuses_options_that_do_not_make_a_command() { let launch = ["launch", "store", "--guest-agent", "agent"]; let uuid = "0a490905-41be-ed5f-984e-63eb7e5e20b1"; - let cases: [(&[&str], &str); 14] = [ + let cases: [(&[&str], &str); 19] = [ (&["--distro", "erofs:image"], "--distro needs --upper"), (&["--upper", "ram"], "--upper needs --distro"), ( @@ -207,6 +207,33 @@ fn the_host_program_refuses_options_that_do_not_make_a_command() { "--distro-identity needs --distro", ), (&["--runtime-dir", "/run/a"], "--runtime-dir needs --distro"), + ( + &["--overlay-mode", "untrusted-lowers"], + "--overlay-mode needs --distro", + ), + ( + &["--scratch-journal", "forbidden"], + "--scratch-journal needs --distro", + ), + ( + &[ + "--distro", + "erofs:image", + "--upper", + "ram", + "--scratch-journal", + "allowed", + ], + "--scratch-journal needs --upper scratch:PATH", + ), + ( + &["--overlay-mode", "trusted"], + "--overlay-mode takes trusted-lowers or untrusted-lowers", + ), + ( + &["--scratch-journal", "none"], + "--scratch-journal takes allowed or forbidden", + ), ( &["--distro", "squashfs:image"], "--distro takes FORMAT:PATH", diff --git a/tests/runtime-adapter/tests/vm_layered.rs b/tests/runtime-adapter/tests/vm_layered.rs index ccd45367c..95b851c09 100644 --- a/tests/runtime-adapter/tests/vm_layered.rs +++ b/tests/runtime-adapter/tests/vm_layered.rs @@ -30,8 +30,8 @@ use std::path::{Path, PathBuf}; use nvx::host::artifacts::{ArtifactDir, Artifacts, Guest}; use runtime_adapter::layered::RECORD_NAME; -use runtime_adapter::root::{DEFAULT_RUNTIME, RootConfig, RootStage}; -use serde_json::Value; +use runtime_adapter::root::{DEFAULT_RUNTIME, JournalPolicy, OverlayMode, RootConfig, RootStage}; +use serde_json::{Value, json}; use common::{Printed, Sandbox}; @@ -40,6 +40,11 @@ const SCRATCH_TEMPLATE: &str = "ubuntu-smoke-scratch.ext4"; /// The exit status of the host program's command whose guest reported that a stage of the root /// failed. const ROOT_FAILED: i32 = 4; +/// The size of the guest's pages, in bytes. +const PAGE: u64 = 4096; +/// How many of the bytes available in an upper a file that fills it may leave, which the file's +/// own blocks of metadata take. +const FULL_SLACK: u64 = 1 << 20; /// The fixtures of the scenarios, among NVX's artifacts. struct Fixtures { @@ -101,10 +106,12 @@ enum Identity { } /// The upper of a scenario's root. -#[derive(Clone, Copy, Debug)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] enum Upper { /// A copy of the scratch template. Scratch, + /// RAM, without a scratch. + Ram, } /// The layered root of a scenario's launch. @@ -118,15 +125,21 @@ struct Root { upper: Upper, /// The guest's runtime directory, as the launch passes it, unless the default. runtime: Option<&'static str>, + /// The overlay's mode, as the launch names it, unless the guest's default. + overlay: Option, + /// The scratch's journal policy, as the launch names it, unless the guest's default. + journal: Option, } /// The root of the scenarios unless they say otherwise: NVX's EROFS image, with its own identity, -/// over a copy of the scratch template. +/// over a copy of the scratch template, with the guest's defaults. const ROOT: Root = Root { image: Image::Erofs, identity: Identity::Own, upper: Upper::Scratch, runtime: None, + overlay: None, + journal: None, }; impl Root { @@ -138,15 +151,21 @@ impl Root { } } - /// Returns the launch's options of the root, with the `fixtures` and the `scratch` image. + /// Returns the launch's options of the root, with the `fixtures` and the `scratch` image, for a + /// root over a scratch. fn options(&self, fixtures: &Fixtures, scratch: &Path) -> Vec { let Image::Erofs = self.image; let mut distro = OsString::from("erofs:"); distro.push(&fixtures.distro); let identity = self.identity(fixtures); - let Upper::Scratch = self.upper; - let mut upper = OsString::from("scratch:"); - upper.push(scratch); + let upper = match self.upper { + Upper::Scratch => { + let mut upper = OsString::from("scratch:"); + upper.push(scratch); + upper + } + Upper::Ram => "ram".into(), + }; let mut options = [ "--distro".into(), distro, @@ -159,8 +178,19 @@ impl Root { if let Some(runtime) = self.runtime { options.extend(["--runtime-dir".into(), runtime.into()]); } + if let Some(mode) = self.overlay { + options.extend(["--overlay-mode".into(), mode.as_str().into()]); + } + if let Some(policy) = self.journal { + options.extend(["--scratch-journal".into(), policy.as_str().into()]); + } options } + + /// Returns the overlay's mode, the guest's default unless the launch names another. + fn overlay_mode(&self) -> OverlayMode { + self.overlay.unwrap_or(OverlayMode::TrustedLowers) + } } /// Returns a UUID other than `identity`, which differs in its last digit. @@ -172,13 +202,13 @@ fn other_identity(identity: &str) -> String { /// What a scenario expects of its VM. enum Expect { /// The guest assembles the root, the check probes the VM, and the stop tears the root down, - /// which leaves the scratch unmounted. + /// which leaves a scratch unmounted. Ready(fn(&mut Vm<'_>)), /// The guest assembles the root, the check keeps it busy, and the stop reports the teardown - /// that failed with this problem, which leaves the scratch mounted when the VM ends. + /// that failed with this problem, which leaves a scratch mounted when the VM ends. TeardownFails(fn(&mut Vm<'_>), &'static str), /// The guest reports that the stage failed with this problem, and the VM ends with its - /// status, before the guest mounted the scratch. + /// status, before the guest mounted a scratch. Fails(RootStage, &'static str), } @@ -198,6 +228,10 @@ struct Vm<'a> { launched: Printed, /// The guest's runtime directory, below which it assembled the root. runtime: &'static str, + /// The scenario's root. + root: Root, + /// The root configuration that the launch sent. + config: RootConfig, } impl Vm<'_> { @@ -228,12 +262,55 @@ macro_rules! scenarios { scenarios! { /// An EROFS image over an ext4 scratch, of the identity that the launch expects: the merged - /// root's contents and mounts, the agent's and the workload's cgroups, PID 1's unrelated - /// child, and a clean teardown. + /// root's contents and mounts, a copy-up of metadata alone, the guest's disks, the agent's and + /// the workload's cgroups, PID 1's unrelated child, and a clean teardown. erofs_over_scratch: Scenario { root: ROOT, expect: Expect::Ready(check_erofs_over_scratch), }; + /// The image over the scratch, in the untrusted-lowers mode, with a journal that the launch + /// allows: a copy-up of the data too, and a rename of a lower directory that fails. + erofs_over_scratch_with_untrusted_lowers: Scenario { + root: Root { + overlay: Some(OverlayMode::UntrustedLowers), + journal: Some(JournalPolicy::Allowed), + ..ROOT + }, + expect: Expect::Ready(check_root), + }; + /// The image over a tmpfs in RAM, of the kernel's default size, without a scratch device. + erofs_over_ram: Scenario { + root: Root { upper: Upper::Ram, ..ROOT }, + expect: Expect::Ready(check_root), + }; + /// The image over a tmpfs in RAM, in the untrusted-lowers mode. + erofs_over_ram_with_untrusted_lowers: Scenario { + root: Root { + upper: Upper::Ram, + overlay: Some(OverlayMode::UntrustedLowers), + ..ROOT + }, + expect: Expect::Ready(check_root), + }; + /// A scratch that a file fills, which then refuses writes until the file is gone. + full_scratch_refuses_writes: Scenario { + root: ROOT, + expect: Expect::Ready(check_full_upper), + }; + /// A tmpfs that a file fills, at half of the guest's memory, which then refuses writes until + /// the file is gone. + full_ram_upper_refuses_writes: Scenario { + root: Root { upper: Upper::Ram, ..ROOT }, + expect: Expect::Ready(check_full_upper), + }; + /// A scratch with a journal, which the launch forbids, and the assembly refuses. + forbidden_journal_fails_the_assembly: Scenario { + root: Root { journal: Some(JournalPolicy::Forbidden), ..ROOT }, + expect: Expect::Fails( + RootStage::Assembly, + "has an ext4 journal, which the assembly forbids", + ), + }; /// An image of another identity than the launch expects, which the assembly refuses. other_identity_fails_the_assembly: Scenario { root: Root { identity: Identity::Other, ..ROOT }, @@ -272,9 +349,12 @@ scenarios! { fn run(name: &str, scenario: Scenario) { let fixtures = Fixtures::find(); let mut sandbox = Sandbox::new(name); - let scratch = sandbox.root.join("scratch.ext4"); - fs::copy(&fixtures.scratch, &scratch).unwrap(); let root = scenario.root; + let scratch = sandbox.root.join("scratch.ext4"); + let over_scratch = root.upper == Upper::Scratch; + if over_scratch { + fs::copy(&fixtures.scratch, &scratch).unwrap(); + } let options = root.options(&fixtures, &scratch); let options = options.iter().map(OsString::as_os_str).collect::>(); let (check, failure) = match scenario.expect { @@ -287,7 +367,10 @@ fn run(name: &str, scenario: Scenario) { check_record(&sandbox, &fixtures, &root, &scratch); // The launch ended: no runtime record names the VM. assert_eq!(sandbox.record(), None); - assert!(!needs_recovery(&scratch), "the guest mounted the scratch"); + assert!( + !(over_scratch && needs_recovery(&scratch)), + "the guest mounted the scratch" + ); return; } Expect::Ready(check) => (check, None), @@ -295,7 +378,7 @@ fn run(name: &str, scenario: Scenario) { }; let launched = sandbox.launch(&options, 0); assert_eq!(launched["root"], "ready"); - check_record(&sandbox, &fixtures, &root, &scratch); + let config = check_record(&sandbox, &fixtures, &root, &scratch); let vmm = sandbox .vmm() .expect("the VM runs after its launcher exited"); @@ -303,6 +386,8 @@ fn run(name: &str, scenario: Scenario) { sandbox: &mut sandbox, launched: launched.clone(), runtime: root.runtime.unwrap_or(DEFAULT_RUNTIME), + root, + config, }; check(&mut vm); @@ -323,14 +408,14 @@ fn run(name: &str, scenario: Scenario) { assert_eq!(stopped["outcome"], "success 0"); assert_eq!(stopped["teardown"], "complete"); assert!( - !needs_recovery(&scratch), + !(over_scratch && needs_recovery(&scratch)), "the stop left the scratch mounted" ); } Some(problem) => { assert_failed(&stopped, RootStage::Teardown, problem); assert!( - needs_recovery(&scratch), + !over_scratch || needs_recovery(&scratch), "the failed teardown unmounted the scratch" ); } @@ -349,9 +434,10 @@ fn assert_failed(printed: &Printed, stage: RootStage, problem: &str) { } /// Checks what the launch recorded in the sandbox's store: a VM with the layered `root` of -/// `fixtures` over `scratch`, the image of the identity that the launch expects, the boot tokens of -/// its plan, and the root configuration that the host sent, whether or not the guest accepted it. -fn check_record(sandbox: &Sandbox, fixtures: &Fixtures, root: &Root, scratch: &Path) { +/// `fixtures`, over `scratch` or RAM, the image of the identity that the launch expects, the boot +/// tokens of its plan, and the root configuration that the host sent, whether or not the guest +/// accepted it, which it returns. +fn check_record(sandbox: &Sandbox, fixtures: &Fixtures, root: &Root, scratch: &Path) -> RootConfig { let record = fs::read_to_string(sandbox.store.join(RECORD_NAME)).unwrap(); let record: Value = serde_json::from_str(&record).unwrap(); assert_eq!(record["format"], 1); @@ -362,26 +448,47 @@ fn check_record(sandbox: &Sandbox, fixtures: &Fixtures, root: &Root, scratch: &P assert_eq!(plan["distro"]["identity"], identity.as_str()); let path = |value: &Value| PathBuf::from(value.as_str().unwrap()); assert_eq!(path(&plan["distro"]["path"]), fixtures.distro); - assert_eq!(plan["upper"]["kind"], "scratch"); - assert_eq!(path(&plan["upper"]["path"]), scratch); + let upper = match root.upper { + Upper::Scratch => { + assert_eq!(plan["upper"]["kind"], "scratch"); + assert_eq!(path(&plan["upper"]["path"]), scratch); + "nvx_scratch=0xd0006000,ext4" + } + Upper::Ram => { + assert_eq!(plan["upper"], json!({"kind": "ram"})); + "nvx_overlay_upper=ramfs" + } + }; let tokens = record["kernelCommandLine"] .as_str() .unwrap() .split(' ') .collect::>(); let layer = format!("nvx_layer=distro,0xd0003000,erofs:{identity}"); - for token in ["nvx_sandbox=1", &layer, "nvx_scratch=0xd0006000,ext4"] { + for token in ["nvx_sandbox=1", &layer, upper] { assert!(tokens.contains(token), "{token} in {tokens:?}"); } + let uppers = tokens + .iter() + .filter(|token| { + token.starts_with("nvx_scratch=") || token.starts_with("nvx_overlay_upper=") + }) + .count(); + assert_eq!(uppers, 1, "{tokens:?}"); let sectors = |path: &Path| Some(fs::metadata(path).unwrap().len() / 512); let config = RootConfig { runtime: root.runtime.unwrap_or(DEFAULT_RUNTIME).to_owned(), distro_sectors: sectors(&fixtures.distro), - scratch_sectors: sectors(scratch), + scratch_sectors: (root.upper == Upper::Scratch) + .then(|| sectors(scratch)) + .flatten(), + overlay_mode: root.overlay, + scratch_journal: root.journal, ..RootConfig::default() }; let sent = String::from_utf8(config.encode()).unwrap(); assert_eq!(record["rootConfiguration"], sent.as_str()); + config } /// Returns whether the ext4 file system at `path` needs its journal recovered: whether a mount @@ -407,16 +514,27 @@ fn options(list: &str) -> BTreeSet<&str> { list.split(',').collect() } -/// Checks the VM of the `erofs_over_scratch` scenario. -fn check_erofs_over_scratch(vm: &mut Vm<'_>) { - let facts = vm.probe(&["mounts", "contents", "cgroups", "processes"]); +/// Checks the root of a scenario's VM, which the guest assembled, as its upper and its overlay +/// mode make it: its mounts, its contents, a copy-up, and the guest's disks. +fn check_root(vm: &mut Vm<'_>) { + let facts = vm.probe(&[ + "mounts", + "contents", + "copy-up usr/lib/os-release", + "rename mnt", + "devices", + ]); let runtime = vm.runtime; + let (upper, kind, other) = match vm.root.upper { + Upper::Scratch => ("scratch", "ext4", "tmpfs"), + Upper::Ram => ("tmpfs", "tmpfs", "scratch"), + }; - // The image, read-only, under the overlay, whose upper and work directories are on the - // scratch, all once, below the runtime directory. + // The image, read-only, under the overlay, whose upper and work directories are on the upper, + // all once, below the runtime directory, where no other upper is mounted. for (mount, kind, flags) in [ ("distro", "erofs", &["ro"][..]), - ("scratch", "ext4", &["rw", "nosuid", "nodev"]), + (upper, kind, &["rw", "nosuid", "nodev"]), ("root", "overlay", &["rw", "nosuid", "nodev"]), ] { assert_eq!(facts[&format!("mounts.{mount}.count")], "1", "{mount}"); @@ -427,18 +545,43 @@ fn check_erofs_over_scratch(vm: &mut Vm<'_>) { "{mount}: {found:?}" ); } + assert_eq!(facts[&format!("mounts.{other}.count")], "0", "{other}"); + if vm.root.upper == Upper::Ram { + // A tmpfs of the kernel's default size, whose root directory has mode 0755. + let tmpfs = options(&facts["mounts.tmpfs.super"]); + assert!(tmpfs.contains("mode=755"), "{tmpfs:?}"); + let sized = tmpfs.iter().any(|option| option.starts_with("size=")); + assert!(!sized, "{tmpfs:?}"); + } let layers = [ format!("lowerdir={runtime}/layers/distro"), - format!("upperdir={runtime}/scratch/upper"), - format!("workdir={runtime}/scratch/work"), - "metacopy=on".to_owned(), - "xino=on".to_owned(), + format!("upperdir={runtime}/{upper}/upper"), + format!("workdir={runtime}/{upper}/work"), ]; let overlay = options(&facts["mounts.root.super"]); for option in &layers { assert!(overlay.contains(option.as_str()), "{option} in {overlay:?}"); } + // The overlay's mode. Trusting its lowers, the overlay copies up a file's metadata alone, + // which leaves the data in the image, and renames a lower directory, with the options that + // the mode and metadata copy-up set. Otherwise, it copies up the data too, and refuses the + // rename, with these options off. + let trusting = ["metacopy=on", "redirect_dir=on", "xino=on"]; + let trusted = vm.root.overlay_mode() == OverlayMode::TrustedLowers; + for option in trusting { + assert_eq!(overlay.contains(option), trusted, "{option} in {overlay:?}"); + } + let (data, rename) = if trusted { + ("none", "allowed") + } else { + ("copied", "refused") + }; + assert_eq!(facts["copy-up.upper.size"], facts["copy-up.lower.size"]); + assert_eq!(facts["copy-up.upper.data"], data, "{facts:?}"); + assert_eq!(facts["copy-up.root.data"], "intact"); + assert_eq!(facts["rename.outcome"], rename, "{facts:?}"); + // The merged root: the image's files, and a file that the worker wrote, which reaches the // upper and not the image, which refuses writes. assert_eq!(facts["contents.os-release.id"], "ubuntu"); @@ -448,9 +591,52 @@ fn check_erofs_over_scratch(vm: &mut Vm<'_>) { assert_eq!(facts["contents.write.lower"], "absent"); assert_eq!(facts["contents.lower.write"], "refused"); + // The guest's disks, whatever their names: the image's, read-only, and the scratch's, + // writable, unless the upper is in RAM. + let mut disks = facts["devices.disks"] + .split(' ') + .map(|disk| disk.split_once(':').map_or(disk, |(_, size)| size)) + .collect::>(); + disks.sort_unstable(); + let distro = format!("{}:ro", vm.config.distro_sectors.unwrap()); + let mut expected = vec![distro]; + expected.extend( + vm.config + .scratch_sectors + .map(|sectors| format!("{sectors}:rw")), + ); + expected.sort_unstable(); + assert_eq!(disks, expected, "{facts:?}"); +} + +/// Fills the upper of a scenario's root through the root, and checks that the upper refuses +/// writes once full, until the file that filled it is gone. +fn check_full_upper(vm: &mut Vm<'_>) { + let facts = vm.probe(&["fill"]); + let number = |fact: &str| facts[&format!("fill.{fact}")].parse::().unwrap(); + assert_eq!(facts["fill.error"], "ENOSPC", "{facts:?}"); + let (available, written) = (number("available"), number("written")); + assert!( + written <= available && available - written < FULL_SLACK, + "{facts:?}" + ); + assert_eq!(facts["fill.after"], "runtime-adapter"); + if vm.root.upper == Upper::Ram { + // A tmpfs of the kernel's default size: half of the guest's memory, in whole pages. + let pages = number("memory") / PAGE; + assert_eq!(number("total"), pages / 2 * PAGE, "{facts:?}"); + } +} + +/// Checks the VM of the `erofs_over_scratch` scenario: its root, the agent's and the workload's +/// cgroups, and the guest's processes. +fn check_erofs_over_scratch(vm: &mut Vm<'_>) { + check_root(vm); + let facts = vm.probe(&["cgroups", "processes"]); + // The agent's and the workload's cgroups, siblings in the cgroup v2 hierarchy, with the // configuration's resource values, and no user namespaces. - let config = RootConfig::default(); + let config = &vm.config; assert_eq!(facts["cgroups.parent.type"], "cgroup2"); assert_eq!(facts["cgroups.agent.exists"], "yes"); assert_eq!(facts["cgroups.workload.exists"], "yes");