[Pedro Henrique Penna (ppenna)]

Part of #690.
Part of #430.

Summary

The KVM and MSHV legs of openvmm-vmm-tests each built the same Linux OpenVMM VMM tests before running their backend's filter, and every leg waited for the guest artifacts before it started building. This PR builds the Linux tests once, in a build-only producer that starts beside the guest-artifact build, and hands the test content to two backend consumers that only run their existing filters with OpenVMM's vmm-tests-run-target.

The WHP tests can't move yet: at the pinned OpenVMM revision, vmm-tests-run-target requires Hyper-V on Windows whatever tests it runs, which the WHP runners can't provide (see WHP below). The WHP job therefore still builds and runs its tests in one job, unchanged, under the new job ID openvmm-vmm-tests-windows.

Why

• In reference run 39585909588, the KVM and MSHV legs each spent about 4.6 and 4.9 minutes on flowey, the test-binary build, and OpenVMM before their first test, and both logs resolve the identical artifact selection (the same Resolved selections line: the x86_64-unknown-linux-gnu nextest archive, OpenVMM, openvmm_vhost, the musl pipette, guest_test_uefi, the test kernel, initrd, bzImage, and UEFI firmware) and lay out the same test content.
• In the 14 successful first-attempt full-matrix dev pushes from 2026-10-20 18:13 to 2026-10-21 13:32 UTC (runs and data on #690), the Linux legs took a median 15.6 (KVM) and 9.9 (MSHV) minutes, 25.6 Linux runner-minutes per run, and all three legs started only after the guest artifacts, which ran cold in 9 of the 14 runs for a median 5.3 minutes.
• One Linux build serves both backends: the target triple is the same, the backend is detected at run time, OpenVMM's Cargo configuration sets no host-CPU code generation, both pools run Azure Linux 3, and the GNU OpenVMM that a KVM runner builds already runs the MSHV microVM tests.

Changes

nvx.py build-openvmm-tests and test-openvmm --content (first commit)

• build-openvmm-tests --backend B [--backend B ...] --output DIR runs cargo xflowey vmm-tests-run --build-only on the union of the backends' filters ((<kvm>) | (<mshv>)), which writes the nextest archive, OpenVMM, the guest agents, the test firmware and kernels, and a flowey_hvlite that runs vmm-tests-run-target. It parses the run script that the build writes strictly (runner inside the directory, --dir ., the built filter, and only the options that the pinned flowey writes), and writes nvx-openvmm-tests.json: the format, the OpenVMM revision and whether its checkout was clean, the Rust release, each backend's filter, the build filter, the runner and its options, and every file's size and executable bit.
• test-openvmm --backend B --content DIR runs backend B's own filter from such a directory with its vmm-tests-run-target, without the OpenVMM checkout or a build. It refuses content of another OpenVMM revision than the gitlink, of another filter, or with a missing or truncated file, and it restores the executable bits that artifact downloads drop. It sets the same OPENVMM_MICROVM_TEST_KERNEL/INITRD, RUSTUP_TOOLCHAIN, and XDG_CACHE_HOME as before. Without --content, test-openvmm builds and runs the tests as before. On Windows, the build links with /Brepro, as test-openvmm does since ci: link the Windows OpenVMM test builds with /Brepro for sccache #714.

CI (second commit)

Job	Runner	Waits for	Does
build-openvmm-vmm-tests-linux (new)	general KVM, whose sccache holds the Linux test build	openvmm-changes only	build-openvmm-tests --backend kvm --backend mshv; uploads the one-day openvmm-vmm-tests-linux artifact (include-hidden-files for .config/nextest.toml, overwrite: true); restores and, on dev pushes, saves the vmm-tests input and Cargo caches
openvmm-vmm-tests-linux (KVM, MSHV legs)	general KVM / MSHV, as before	artifacts and the build	downloads the content and the guest-kernel and guest-alpine components, runs test-openvmm --content; no OpenVMM checkout, no caches, no sccache
openvmm-vmm-tests-windows	general WHP, as before	artifacts	unchanged: builds and runs the WHP tests

• Diagnostics keep the openvmm-vmm-tests-<os>-<backend>-<run-id>-<run-attempt> names; Linux now collects <runner-temp>/openvmm-vmm-tests/test_results.
• Gating: the build runs whenever run-openvmm-vmm-tests is set, which the classifier of ci: classify microVM tests, benchmarks, and artifact builds separately #712 sets only together with run-guest-artifacts, or when the change detector fails, so it runs exactly when its consumers do, and run-openvmm-vmm-tests=false skips producer and consumers together. It reads that output alone, as ci: classify microVM tests, benchmarks, and artifact builds separately #712's one-output-per-job test requires. check-required-ci expects it with them. Publish development release and Persist performance baseline, which accepted any skipped VMM-test job, now need openvmm-changes and require the three VMM jobs to succeed, or all three to be skipped because the change selected no VMM tests, so a failed or unexpectedly skipped producer or consumer blocks them. Fork, draft, and path-filter gating, cancellation, and the eight cache saves' keys and guards are unchanged.
• Reruns: a rerun of a failed consumer reuses the producer's artifact from the earlier attempt instead of rebuilding; a rerun of the producer replaces its artifact.
• Docs: doc/ci.md gains an OpenVMM VMM tests section on the producer, consumers, handoff, manifest, and WHP; doc/usage.md documents both commands.

WHP

At the pinned OpenVMM revision, 9f04d05c5 since #661, whose flowey/ is identical to that of d0bdbed8f, vmm-tests-run-target hard-codes hyperv: true for Windows targets (flowey/flowey_hvlite/src/pipelines/vmm_tests_run_target.rs), so install_vmm_tests_external_deps requires Get-VM to succeed and the AllowFirmwareLoadFromFile and EnableAdditionalComPorts values under HKLM\Software\Microsoft\Windows NT\CurrentVersion\Virtualization, or, with --install-missing-deps, runs DISM /Online, which needs elevation. The WHP runners run jobs as Network Service (validate-runner enforces S-1-5-20), which the runner setup doesn't add to Hyper-V Administrators, and the setup sets neither value. The combined vmm-tests-run computes hyperv from the selected tests, which is why the WHP job works today. Splitting WHP needs vmm-tests-run-target to take the dependencies that the build resolved, for example the --needs-hyperv and --needs-whp options of commit 79eb62981 on the copilot/vmm-tests-run-target-deps branch of nanvix/openvmm, for which I found no microsoft/openvmm pull request, and then a pin promotion. The content format and the consumer here already cover a Windows producer.

Tests

• The union build filter; both run-script formats and each rejection (filter, cd, pipeline, runner outside the directory, unknown option, missing value, other directory, missing --target).
• The build's commands, environment, and manifest; the refusal of another OS's backends, unknown backends, and used directories.
• The consumer's command, working directory, environment, and restored executable bits; the refusal of another format, revision, backend, or filter, a malformed runner or options, a file outside the directory, and a missing or truncated file.
• The CLI parsing and doc/usage.md sections.
• Workflow structure: the producer's runner, needs, and condition; the change detector's outputs that the VMM-test jobs and the two gates read; the consumers' needs and conditions; the upload's name, path, hidden files, overwrite, and retention; the download and test order; diagnostics on both OSes; the WHP job's independence of the Linux build; the release and persistence gate; the required-check policy for every event and flag combination; the cache keys and saves; and the VMM suite's inputs derived from every job that builds or runs it; and, in ci: build the guest-artifact components in parallel and download only what each job uses #711's guest component test, that the Linux test jobs download exactly the kernel and the Alpine initramfs, that the build downloads, reads, and waits for no guest component, and that test-openvmm reads only those two whether it builds the tests or runs a build's content.

Validation

• Windows: the full scripts/test_nvx_tools.py module on each commit after the rebase onto bd977e35b (504 tests on the first, 505 on the second, OK); ruff check, ruff format --check; pyright for the Linux and Windows platforms; check_markdown_links.py; git diff --check.
• WSL Ubuntu (Python 3.14): CiTests, CliTests, CiConfigurationTests, the guest-artifact download test, and the guest component selection test (134 tests, OK), which cover the Linux paths of the build and the consumer.
• actionlint v1.7.7 (release binary, checksum verified) with .github/actionlint.yaml: ci.yml reports only unexpected key "queue" for "concurrency", which ci: queue the dev push runs instead of cancelling the waiting one #734 added and GitHub accepts; actionlint 1.7.7 predates the key.
• CI on the head 07efc1873, rebased onto ci: build the guest-artifact components in parallel and download only what each job uses #711: run 39771039182 passed every job, including Required status check, apart from the expected skips. The KVM and MSHV jobs downloaded only guest-kernel and guest-alpine (1 to 2 seconds each) and the content (4 and 6 seconds), and ran 58 and 57 tests, all passing, with 49 and 50 skipped, as before, in 10.4 and 4.6 minutes. The producer's second run on its KVM runner hit sccache for all but 4 compilations, so it built the content in 5.5 minutes.
• CI on the earlier head 2f85af825, run 39691097674: every job passed, including Required status check, apart from the expected skips (the MSHV and WHP debug-kernel jobs on a pull request, the release, and the persistence). The producer built the same 214.7 MiB of content in 6.7 minutes, and the MSHV and KVM consumers ran the same 57 and 58 tests, all passing, in 4.8 and 10.3 minutes. Before the rebase, run 39625831399: the producer built the content (214.7 MiB, a 112.3 MiB artifact, uploaded in 6 seconds) in 7.0 minutes, and the MSHV and KVM consumers downloaded it in 8 and 6 seconds and ran 57 and 58 tests, all passing, with 50 and 49 skipped, exactly as before, in 4.6 and 10.5 minutes; every other job, including Required status check, passed. test-openvmm and test-openvmm-unit can't run locally.

Expected effect

From the PR runs (details on #690):

• Linux runner-minutes: 7.0 + 4.6 + 10.5 = 22.1 and 6.7 + 4.8 + 10.3 = 21.8 with the producer's first build on its runner, and 5.5 + 4.6 + 10.4 = 20.5 with its second, instead of a median 25.6 (−3.5, −3.8, and −5.1, or −14%, −15%, and −20%). The first build of the target flowey_hvlite on each KVM runner misses its sccache (197 misses, about 1.3 minutes); the second on azure-kvm-4 missed 4.
• Linux lanes: the build overlaps the guest artifacts, so with cold ones (9 of the 14 runs above) the Linux lanes finish sooner: in run 39691097674, whose guest artifacts took 5.9 minutes, the MSHV and KVM lanes finished 11.8 and 18.3 minutes after the first job started, 3.5 and 2.8 minutes before the medians of the comparable runs above. With a warm producer cache, the lanes need 5.5 + 4.6 and 5.5 + 10.4 minutes after the run starts, against 9.9 and 15.6 after the guest artifacts for the combined jobs, so they finish about as before when the guest artifacts are warm, still well before the WHP lane.
• Wall time: unchanged, because the WHP lane, the last to finish in most full runs, is unchanged until the WHP split is unblocked.

Post-merge verification

• Compare at least five comparable full-matrix runs after merge with the 14 runs above: producer and consumer queue time and duration, the content's size, upload, and download times, the Linux lanes' finish times, and the Linux runner-minutes.

Interactions with other pull requests

This branch is rebased onto bd977e35b and adapted to the pull requests that merged since its first version:

• ci: classify microVM tests, benchmarks, and artifact builds separately #712: the build job and the VMM-test clause of Publish development release and Persist performance baseline read run-openvmm-vmm-tests alone, check-required-ci expects the build whenever the VMM tests are selected (the test jobs also need the guest artifacts), and ci: classify microVM tests, benchmarks, and artifact builds separately #712's tests map the build job to that output and accept the two gates' reads of it.
• ci: key the guest images and VMM tests on the guest binaries' sources #710: its classification test checks the three VMM-test jobs instead of openvmm-vmm-tests; the VMM-test inputs derive from the actions of all three jobs and artifacts, and don't include the host side of nvx-rs.
• ci: link the Windows OpenVMM test builds with /Brepro for sccache #714: build-openvmm-tests links with /Brepro on Windows through _prepare_openvmm_test_environment, which a test now checks.
• ci: build the guest-artifact components in parallel and download only what each job uses #711: the Linux VMM-test jobs download guest-kernel and guest-alpine, as the WHP job does, and wait only for artifacts, not distro-artifacts; ci: build the guest-artifact components in parallel and download only what each job uses #711's component test lists them among the consumers, maps build-openvmm-tests to no guest file, checks that the build downloads, reads, and waits for no component, and checks both paths of test-openvmm. A negative check confirmed that the test fails when a Linux test job lacks the Alpine download or the build downloads the kernel.
• ci: run the Linux OpenVMM unit tests on a GitHub-hosted runner #717, ci: queue the dev push runs instead of cancelling the waiting one #734, openvmm: refuse kernel command lines that the guest kernel would truncate #661, ci: report nightly full-run failures #709: no change needed beyond merging the docs and tests; the flowey interfaces that this PR relies on are unchanged at OpenVMM 9f04d05c5.

Trial merges of this branch with the open siblings' heads as of 2026-10-22 11:35 UTC:

• ci: run the guest and sandbox steps beside the default microVM suite #706 (289be9cb1) and ci: reuse a pull request's guest images on its dev push #713 (7e3c899ac), both rebasing: ci: reuse a pull request's guest images on its dev push #713 already conflicts with dev in all four of the files that it shares with this branch, and ci: run the guest and sandbox steps beside the default microVM suite #706 in doc/ci.md; this branch adds a conflict in the REQUIRED_CI_* constants of ci.py for ci: run the guest and sandbox steps beside the default microVM suite #706. Keep both sides; whichever merges second resolves them.
• microvm tests: record each scenario's duration in a JSON Lines report #707 (a904264bc) merges cleanly.

[Copilot]

🟡 Changes recommended

Windows manifest paths can escape the content directory; cross-runner execution also still needs end-to-end CI validation.

1 open issue

• Windows path traversal bypasses content directory validation · New

What changed in this PR

Builds Linux OpenVMM VMM tests once for KVM and MSHV, reducing duplicate CI work while retaining the combined WHP job.

Changes:

• Adds build-only and prebuilt-content CLI commands.
• Splits Linux CI into a producer and backend consumers, updating required and publishing gates.
• Adds handoff validation tests and documentation.

File	Description
scripts/​test_nvx_tools.py	Tests CLI, manifests, handoffs, and workflow gates.
scripts/​nvx.py	Exposes build-only and content-consumer options.
scripts/​nvx_tools/​ci.py	Implements content production, validation, and execution.
doc/​usage.md	Documents new command options.
doc/​ci.md	Explains job dependencies and artifact handoffs.
.github/​workflows/​ci.yml	Shares Linux test builds and updates gating.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-runner artifact handoff needs successful producer and KVM/MSHV consumer runs before human approval.

No open issues 🎉

1 resolved since last review

• Windows path traversal bypasses content directory validation

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-runner executable handoff and release/performance gate changes warrant final human review with current-head CI results.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-runner executable-artifact handoff and release-gating changes warrant human review and completed exact-head backend CI validation.

No open issues 🎉

_{Review effort: Balanced}