(15) n-vm ci by daniel-noland · Pull Request #1804 · githedgehog/dataplane

daniel-noland · 2026-09-08T02:22:30Z

Split out of (14) n-vm, which had grown to 115 commits and 25k insertions.

This chapter is the work of getting the VM suite, and the heavier build
profiles it competes with, to run on the lab runners: the 1 GiB hugepage
reservation and its diagnostics, the ThinLTO core budget, and keeping the
guest-driving crates out of the wasm and miri builds.

No commit here is new. Four commits that were in this range are gone, and
they netted to exactly zero: two ci(dev) probes and the two revert(ci)
commits that removed them once they had answered.

🤖 Generated with Claude Code

coderabbitai · 2026-09-08T02:22:43Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-09-08T02:45:02Z

Codecov Report

❌ Patch coverage is 77.33333% with 17 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
n-vm/src/vm.rs	71.05%	11 Missing ⚠️
n-vm-protocol/src/lib.rs	72.72%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

`vm_boots_with_host_hugepages` is the last test failing on the runners: × hugepage pool unavailable ╰─▶ the 1073741824-byte hugepage pool has 0 free page(s); this VM needs 1 It asks for a 1 GiB *host* page because that is what makes the guest's memory physically contiguous, which is the only thing DPDK-through-an-IOMMU can tell apart. Nothing else asks -- `HostPageSize`'s docs record that the default deliberately leaves this pool alone, after a version that did not made ten of eighteen tests contend for a page they had no use for. The runner cannot reserve it: measured on one, `CapEff` is zero, so writing `/sys/kernel/mm/hugepages` is not available to it. It can ask the host's daemon for a privileged container that can, and the hugepage sysfs is not namespaced, so the reservation lands on the machine and every job on it sees the result. Verified against a local daemon that a privileged container's write does reach host sysfs. Only ever raises the count. Lowering it would take pages away from whatever else is running on a shared machine, and four is a floor rather than this job's private allocation. Never fails the job. A 1 GiB reservation is a request: the kernel has to find that many physically contiguous gigabytes and on a long-lived machine it may not. If it comes up short, the test that needs a page reports it precisely, which is a better place to read it than a setup step -- so this warns and carries on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>

The reservation reached the host and the kernel declined it: "asked for 4 1 GiB hugepages, got 0 (was 0)". That is the expected answer rather than a malfunction -- a gigabyte page needs a physically contiguous gigabyte, and a machine that has been up a while rarely has one, which is why the kernel documentation reserves them at boot. Compaction is the one cheap thing worth trying first: it migrates movable pages to free contiguous runs, takes a few seconds, and disturbs nothing else on the machine. Dropping caches would probably help more and is not worth it -- it takes the page cache from every other job on a shared runner. When that still is not enough the warning now names the durable fix, a boot parameter on the runner host, rather than leaving the reader to infer it: default_hugepagesz=1G hugepagesz=1G hugepages=4 Still never fails the job. `vm_boots_with_host_hugepages` reports the missing page precisely, and that is the right place to read it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>

`vm_boots_with_host_hugepages` is the last test failing on the runners, and it is not going to pass there soon: a 1 GiB page has to come from a physically contiguous gigabyte, and these machines cannot produce one. Measured over two runs -- a privileged container does reach the host's `nr_hugepages`, and the kernel answers 0 both before and after compaction. Reserving them wants `default_hugepagesz=1G hugepagesz=1G hugepages=4` on the host's command line, and a reboot. So the 1 GiB pair is `#[ignore]`d and a 2 MiB pair added beside it. `#[ignore]` rather than n-vm's own skip, deliberately. A skip is the right answer to a genuine mismatch between a test and the machine it was handed -- "cloud-hypervisor cannot emulate aarch64" is permanent and true everywhere. This is not that: the machine could run it, given a boot parameter. An ignore says "not yet", stays visible in the run summary, and comes back with `--ignored` on a host that has the pages. What the 2 MiB version does not cover is narrow and worth naming: only DPDK driving a device through an IOMMU can tell a contiguous gigabyte from 512 contiguous megabytes. Everything between here and there -- asking for hugepage backing at all, `memfd` with `MFD_HUGE_*`, the pool accounting, both VMMs' plumbing -- is the same code, and that is what stays covered. `ci::reserve-hugepages` now reserves 2 MiB pages, 1024 of them: a 1024 MiB guest backed by 2 MiB pages needs 512, and `check` and `coverage` can be on one machine at once. Verified locally: `vm_boots_with_host_hugepages_2m` boots a guest, the 1 GiB pair reports as ignored with its reason, and `N_VM_SKIP_LOG` confirms `..._2m_on_qemu` skips under the default profile exactly as the 1 GiB one it replaces did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>

…uest honestly Two things the runners found once the tests stopped failing for other reasons. Neither had been seen before, because these crates have never been through this workspace's docs build or its coverage job. **33 rustdoc errors across `n-it` and `n-vm`.** `check-docs` builds with `-D warnings`, and the imported crates carry three kinds of broken link: - links into crates they do not depend on (`n_vm::run_in_vm` from `n-it`, which depends on `n-vm-protocol` and not on `n-vm`), - links from *public* documentation to *private* items -- `run_test_in_vm`'s summary steps through five private methods, and a reader of the public docs cannot follow any of them, - links whose target was simply missing a path: `HypervisorVerdict`, `kernel_profiles`, `KvmNotAccessible`, `ENV_PROFILE`. Fixed by kind rather than silenced: a real path where the item is reachable, a plain code span where it is not. A code span still names the thing; a link that cannot resolve is worse than no link. **A coverage guest needs more than 60 seconds.** `a_vm_boots_the_kernel_profile_it_named` overran the KVM allowance at 96s in the coverage job, and how it reported is the point: not "slow" but "no parseable test verdict from guest", because the timeout shoots the VM and the verdict dies with it. Coverage is not a small tax on a guest -- the whole suite went from 76s to 155s on the same runners. `N_VM_OVERHEAD_SCALE` multiplies the allowance, and the coverage job sets it to 3. Unset, unparseable, non-finite or non-positive all mean 1.0, so an ordinary run is untouched and a mistyped multiplier is not the reason a suite fails to run. A run-time knob rather than a `cfg`, for two reasons. The one this crate already argues for itself, at `ENV_VIRTIOFS_CACHE`: rebuilding `n-vm` to change a timeout also changes the binary under test. And the one that matters for where this crate is going -- a `cfg` would have to name dataplane's `instrumented`, which means nothing to another consumer. (`cfg(instrumented)` was tried first and `just clippy` rejected it as an unexpected condition name, which is its own argument: the flag that registers it is this workspace's, not n-vm's.) Verified with the gates that actually gate: `just clippy` and `just docs`, both through nix, and the integration suite green with the scale unset and at 3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>

`N_VM_OVERHEAD_SCALE` was set on the host and read in the container, and a container's environment is the explicit list in `build_config` and nothing else. So the coverage job showed the knob working and not working at once: its unit tests saw 180s, while the guest they launched was still shot at 60s and reported "no parseable test verdict from guest". Forwarded now, beside `N_VM_ENGINE_TIME_LIMIT`, which is in that list for exactly this reason -- the tier that consumes it never sees the invocation that chose it. The same fix to the tests, and it is the second time this shape has bitten: `vm_overhead_allowance` and `vm_test_timeout` read process-wide state, so `an_ordinary_test_gets_what_it_always_got` and `declared_work_is_added_to_the_allowance` passed here and failed in CI, where the variable is set for the whole job -- exactly what `N_VM_HOST_SHARE_DIR` did to the mount-source assertions. The scale is now a parameter (`vm_test_timeout_with`), the assertions name the scale they mean, and a new test pins that the multiplier multiplies and that 1.0 changes nothing. `the_overhead_scale_is_carried_into_the_container` covers the forwarding itself, which is the part no unit test could have caught before: it is a property of the container's environment, not of any function's return value. Verified with `N_VM_OVERHEAD_SCALE=3` set for the whole process, which is how CI runs it: 297 unit tests and the 27-test integration suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>

The 2 MiB reservation got zero on one runner -- "asked for 1024 2048kB hugepages, got 0 (was 0), even after compaction" -- and the recipe threw away the only thing that could explain it. `priv()` sent stderr to /dev/null, so a refused write and an unmet allocation looked identical, and neither is actionable from a count. Now the write's stderr is reported when it comes up short, along with `MemFree`/`MemAvailable`/`HugePages_*` and the pools the kernel actually has. Those tell the two cases apart: a write the daemon would not perform, versus memory that was not there. Also drops the explicit `-v /sys:/sys`. `--privileged` already gives a read-write sysfs, and sysfs is not namespaced, so that is the host's -- verified against a local daemon by writing the existing value back. The extra bind only added a mount whose mode depends on how the daemon is configured, which is one more thing that can differ between the runner where this worked and the one where it did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>

Raising the VM's own timeout uncovered a second deadline behind it. The coverage job's guest now survives to 98s instead of being shot at 60s, shuts down, and then all four vsock channels miss a fixed 5s drain window -- because an instrumented guest writes its coverage profile on the way out. Cutting that short is not a cosmetic loss. The verdict arrives on one of those channels, so a guest that passed is reported as "no parseable test verdict from guest", which reads as a broken harness rather than a slow one. Same knob, same reason: `DRAIN_TIMEOUT_BASE` times `N_VM_OVERHEAD_SCALE`. A test pins that the two move together, since raising only one of them is precisely the mistake this fixes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>

…n CI Reserving hugepages from CI worked mechanically and cost more than it bought. A privileged container does reach the host's `nr_hugepages` -- the daemon is on bare metal and hugepage sysfs is not namespaced -- but: - 1 GiB pages need a physically contiguous gigabyte the runners could not find. Zero allocated, before and after compaction. That wants `default_hugepagesz=1G hugepagesz=1G hugepages=4` at boot. - Pinning 2 MiB pages took memory away from everything else on a shared runner. Measured across four coverage runs: the suite went from 76s to 155-198s, and `a_vm_boots_the_kernel_profile_it_named` from **3.06s passing** to 96-139s failing, its guest producing nothing on any channel. Nothing else changed between the last good run and the first bad one. That last point also corrects the reasoning behind `N_VM_OVERHEAD_SCALE`. The 76s -> 155s slowdown was attributed to coverage instrumentation when it was in fact this reservation; run 6 did the same instrumented suite in 76s. The knob is kept -- it guards a kill deadline, where headroom is cheap -- but its comment now says what is actually true. So the tests are gated rather than deleted or skipped. They are valid tests of the path that matters, and they pass on a host that reserves pages: RUSTFLAGS='--cfg=host_hugepage_tests' cargo test -p dataplane-n-vm --test integration Not a skip, because n-vm's skips are for mismatches that are permanent and true everywhere -- "cloud-hypervisor cannot emulate aarch64". A host without a pool could have one; that is an arrangement CI has not made, not a property of the machine. `check-cfg` is declared in `n-vm/Cargo.toml` rather than the workspace's RUSTFLAGS, so it travels with the crate: n-vm opts into no workspace lints, and that is what keeps returning it to `githedgehog/testn` a code move rather than a decoupling project. Verified: ignored by default, and all four run and pass under the cfg on a host with pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>

`every_shape_leaves_the_pipeline_with_a_verdict` failed `check/debug` with `NoIp=0` after drawing 2 cases at 1.90 iterations/s. Neither the pipeline nor the guard was wrong: bolero ends a property at whichever comes first, its iteration count or its wall-clock budget, and that budget defaults to one second. The test got about a fifteenth of a core for that second. 620 tests in this workspace run on that budget, 347 of them in `dataplane-net`, and each is CPU-bound for the whole of it. Packed onto a cgroup with ten CPUs that nextest sees as 32, they starve each other. Only 24 are harmed by it: the properties in `packet_processor::fuzz` that end in `assert_covered`, where the sample size is the test rather than merely how far it searched. `dataplane` is already in the `vm` test group, so those 24 were never competing with their own package -- they were competing with everyone else's. Reproduced by pinning nextest to ten CPUs while it runs 32 threads, over `package(dataplane-net) + test(shapes::every_shape_leaves)`: without this override 3.51 iterations/s 4 cases NoIp 7 with it 18 iterations/s 20 cases NoIp 36 the test on its own 29 iterations/s 29 cases NoIp 55 Four cases is a coin toss on whether a given shape appears at all, which is how a test with nothing intermittent about it reaches CI as an intermittent failure. `threads-required` rather than a longer budget or a guard that stands down when its sample is small. A longer budget multiplies the cost of all 620 tests to help these 24, and a guard that gives up when starved stops guarding on exactly the slow and varied machines nobody here can measure. Reserving the thread budget costs the ~26 test-seconds the module already takes. Signed-off-by: Daniel Noland <daniel@githedgehog.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Cargo already spares host units most of the release profile. The emitted rustc line for a build script carries `-C embed-bitcode=no` and neither `-C opt-level` nor `-C codegen-units`, so despite `opt-level = 3`, `lto = "thin"` and `codegen-units = 1` above, they are built unoptimised, without LTO, and at the default codegen-unit count. Debuginfo is the one setting they do inherit: `-C debuginfo=2` on every build script and proc macro in the graph. That is object size and link time spent on code that exists only to emit other code and that nobody will attach a debugger to. After this the same line reads `-C strip=debuginfo`. `checked` inherits `release`, so the test profile picks this up too. Signed-off-by: Daniel Noland <daniel@githedgehog.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The `concurrency` job fails before it runs a test. `just shuttle` sets `features=shuttle` for everything it invokes, `setup-roots` forwards that to all four roots, and the build stops at error: the package 'dataplane-n-preinit' does not contain this feature: shuttle inside `setup-roots`, taking the initramfs, the kernel image and the test root down with it. The job has therefore never reported on the code it exists to check; it has only ever reported on its own scaffolding. The four roots are not one kind of thing: * `devroot` and `sysroot` are the toolchain, and they do need the feature list. `loom` and `shuttle` require `panic = "unwind"`, and `-Zbuild-std` has to build a matching std -- see `mk-needs-unwind` in default.nix. Withholding it here would quietly hand a shuttle build a `panic_abort` sysroot, which is a worse failure than this one because it would link. * `testroot` and `vmroot` are the guest's boot infrastructure: qemu, cloud-hypervisor, virtiofsd, a kernel, an initramfs, and the in-VM runner. None of it is the code under test -- the test binaries arrive at runtime under `test-bin`. Handing it the feature list only asks crates that never declared a feature to build with it. So the split is by what the root is for. This is the same trade `n-preinit-static` already makes for the sanitizer, and default.nix states the reason there: the pre-init is the scaffolding that execs the code under test, not the code under test. Verified: `just features=shuttle setup-roots` now builds all four roots. Signed-off-by: Daniel Noland <daniel@githedgehog.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

daniel-noland force-pushed the pr/daniel-noland/n-vm branch from 181eeb5 to ebf089d Compare September 8, 2026 02:49

daniel-noland force-pushed the pr/daniel-noland/n-vm-ci branch from 019c855 to acf130b Compare September 8, 2026 02:49

daniel-noland self-assigned this Sep 8, 2026

daniel-noland force-pushed the pr/daniel-noland/n-vm branch from ebf089d to a1d8933 Compare September 8, 2026 06:58

daniel-noland force-pushed the pr/daniel-noland/n-vm-ci branch from acf130b to a0dd182 Compare September 8, 2026 06:58

daniel-noland and others added 11 commits September 8, 2026 01:02

daniel-noland force-pushed the pr/daniel-noland/n-vm branch from a1d8933 to 9819709 Compare September 8, 2026 07:04

daniel-noland force-pushed the pr/daniel-noland/n-vm-ci branch from a0dd182 to 18ac1a8 Compare September 8, 2026 07:04

Navigation Menu

Uh oh!

(15) n-vm ci - #1804

(15) n-vm ci#1804
daniel-noland wants to merge 11 commits into
pr/daniel-noland/n-vmfrom
pr/daniel-noland/n-vm-ci

daniel-noland commented Sep 8, 2026

Uh oh!

coderabbitai Bot commented Sep 8, 2026

Draft PR not reviewed

Uh oh!

codecov Bot commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

daniel-noland commented Sep 8, 2026

Uh oh!

coderabbitai Bot commented Sep 8, 2026

Draft PR not reviewed

Uh oh!

codecov Bot commented Sep 8, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant