[Pedro Henrique Penna (ppenna)]

Closes #695.
Part of #430.

Summary

Each backend's NVX microVM tests ran in one job: the default Alpine suite, then the guest smoke and sandbox steps, none of which needs a result of the suite. This PR runs those steps in a second job per backend, beside the suite, on the same runner pool and with the same OpenVMM and nvx-rs builds. Only the new jobs read the Ubuntu and Azure Linux artifacts, so only they download them and wait for distro-artifacts; the default suites wait only for the kernel and Alpine artifacts. The required status check, the development release, and performance persistence require both jobs. Every step still runs exactly once per backend, and the debug-kernel jobs are unchanged.

Why

• The WHP microVM job bounds every run that skips both OpenVMM suites (CI: maximize workflow parallelism and reduce turnaround time #430). In reference run 39585909588, it took 15.1 min, of which the default suite took 10.1 min and the guest and sandbox steps 4.2 min:

  Steps in run 39585909588	WHP	KVM	MSHV
  Setup: checkouts, downloads, and runner validation	46 s	28 s	46 s
  Default Alpine suite	603 s	459 s	413 s
  Guest smoke and sandbox steps	249 s	192 s	125 s
  Whole job	905 s	681 s	588 s

• Over the 41 successful first-attempt runs with microVM jobs from 2026-10-20 21:05 to 2026-10-21 13:24 UTC, the WHP job took a median of 15.1 min (p10 14.3, p90 16.1), after a median queue of 0.1 min (max 8.4), and finished a median of 20.4 min after its run started. KVM took a median of 11.3 min and MSHV 9.8 min.

Changes

• run-nvx-microvm-tests.yml: a new boolean input, guests-and-sandboxes, selects the steps that follow the default suite instead of the suite: the public managed sandbox acceptance, the Ubuntu and Azure Linux guest smoke tests, the extended Ubuntu tests (KVM only), the Ubuntu sandbox steps, the aci_edge_sandboxes lifecycle, and the runtime adapter scenarios. Each later step keeps its !inputs.debug-kernel guard. The downloads of guest-ubuntu, guest-ubuntu-distro, guest-ubuntu-scratch, and guest-azurelinux now also require the input, so the default suite downloads only guest-kernel and guest-alpine, which every job's runner validation reads. A second upload of an artifact name in a run fails, so the new job uploads its failure logs as nvx-microvm-tests-<backend>-guests and, since microvm tests: record each scenario's duration in a JSON Lines report #707, its scenario duration reports as nvx-microvm-durations-<os>-<backend>-default-guests-<run-id>-<run-attempt>, beside the default suite's ...-default-<run-id>-<run-attempt>.
• ci.yml:
  • three new jobs, nvx-microvm-guests-{kvm,mshv,whp} (NVX microVM tests / <platform> / Guests and sandboxes), with the needs, condition, run-microvm-tests selection, runner labels, OpenVMM and nvx-rs artifacts, and secret of the default job before this change;
  • the default nvx-microvm-tests-* jobs no longer wait for distro-artifacts, so a cold build of the Ubuntu or Azure Linux images, about 4 min, no longer delays them;
  • release, required-status-check, and performance-persist list the new jobs in needs, and release and persistence accept only success or skipped from them, as from the default jobs.
• check-required-ci (scripts/nvx_tools/ci.py): the new jobs join REQUIRED_CI_MICROVM_TEST_JOBS and REQUIRED_CI_RESULT_ENVIRONMENTS, so the required status check expects each to succeed when the change selects the microVM tests and to be skipped otherwise, and fails on any other result.
• doc/ci.md: describes the two jobs, the guest artifacts that each takes, their failure-log artifacts, the gates, and the general runners that they hold.

Fork, draft, path-filter, and cancellation behavior is unchanged, because the new jobs copy the conditions of the default ones. A failed part now reruns alone.

Design notes:

• Top-level jobs, rather than two jobs inside the reusable workflow, give each part its own result in the gates, so a part skipped by mistake fails the required status check instead of hiding behind its sibling's success.
• One job definition with a step selector, like debug-kernel, keeps the setup steps in one place and limits the diff to per-step conditions.
• MSHV split (recommendation: keep it). MSHV gains least: replaying the 41 baseline runs, splitting it shortens 6 of them, by a mean of 0.1 min, because its single job (9.8 min) already ends before the split WHP lane. But the split costs MSHV only about 1 runner-minute of repeated setup per run, ci: run the Linux OpenVMM unit tests on a GitHub-hosted runner #717 freed about 900 MSHV runner-minutes a week, and keeping one MSHV job would need a third mode of the reusable workflow, its gates, and its tests. I'd revisit it only if the post-merge measurements show MSHV queueing.

Tests

• New test_ci_runs_the_steps_after_the_default_microvm_suite_beside_it evaluates every step condition for each backend and checks that:
  • both jobs take the same builds and validate the runner the same way, and every other step of the former single job runs in exactly one of them;
  • the default job runs only the default suite and downloads only guest-kernel and guest-alpine, and the guest and sandbox job downloads every guest artifact that the single job did;
  • the KVM-only extended Ubuntu tests stay on KVM, and the debug-kernel jobs select the same steps as before;
  • every upload step has a distinct artifact name in the default, guest and sandbox, and debug-kernel jobs;
  • each guest and sandbox job equals its backend's default job apart from its name, the input, and its wait for distro-artifacts.
• test_every_microvm_job_uploads_its_scenario_durations (microvm tests: record each scenario's duration in a JSON Lines report #707) now derives the -guests part of each job's report name and checks that the default suite and the guest and sandbox job of a backend upload separate reports in each attempt.
• test_required_ci_job_uses_tested_policy now also checks that each job that the policy checks is in needs and reaches it as that job's result.
• Through REQUIRED_CI_MICROVM_TEST_JOBS, the existing tests cover the new jobs: ci: build the guest-artifact components in parallel and download only what each job uses #711's mapping of each step to the guest artifacts that it reads, which fails when a job lacks a component, downloads an extra one, or waits for the wrong producer; ci: classify microVM tests, benchmarks, and artifact builds separately #712's change-detector job selection and suite-input drift guard; the required-results policy; and the repository, CLI, and nvx-rs handoff tests.
• Updated: the public sandbox acceptance test, which now looks in the guest and sandbox jobs, and the OpenVMM consumer test, which expects distro-artifacts only for the guest and sandbox and platform jobs.

Validation

• Windows, Python 3.14, on 206abd182 (after ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708):
  • scripts/test_nvx_tools.py and scripts/test_microvm_tests.py pass (730 tests), including ci: build the guest-artifact components in parallel and download only what each job uses #711's step-to-component test with ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708's VMM-test jobs and this PR's jobs. On an earlier base, one test_microvm_tests.py network-script subtest errored once while pyright ran beside it, and passed alone and in a rerun of the module;
  • ruff check, ruff format --check, pyright for Linux and Windows, check_markdown_links.py, and git diff --check.
• actionlint 1.7.12 (release binary) on ci.yml and run-nvx-microvm-tests.yml: the only finding is the known false positive on concurrency.queue from ci: queue the dev push runs instead of cancelling the waiting one #734. It rejects a misspelled guests-and-sandboxes input.
• Seeded regressions: the tests fail for each of ten: a default suite that downloads the Ubuntu initramfs or waits for distro-artifacts, a guest and sandbox job that misses the Azure Linux download, distro-artifacts in its needs, or its success in its condition, one that follows run-benchmarks, a later step without the selector, a failure-log or scenario-duration artifact name without the suffix, and a guest and sandbox job missing from performance persistence.
• Left to CI: every hypervisor job, including the new ones, on KVM, MSHV, and WHP.

Expected effect

• The WHP microVM lane shortens from about 15 to about 11 min, the default suite and its setup, while the guest and sandbox job takes about 5 min beside it. KVM drops from about 11.4 to 8.5 min and MSHV from about 9.8 to 7.9 min.
• Runs that skip both OpenVMM suites should finish about 4 min sooner whenever a second general WHP runner is free.
• Cost: one more general runner per backend for about 3.5 to 5 min per run, of which 0.6 to 1.3 min repeat the setup. With the OpenVMM VMM tests, a run fills KVM's four general runners, one with ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708's Linux VMM-test build and then its KVM leg, and on dev pushes and full runs MSHV's too; WHP uses at most five of its eight. Since ci: run the Linux OpenVMM unit tests on a GitHub-hosted runner #717 moved the Linux OpenVMM unit tests off MSHV, a pull request leaves one MSHV general runner free.
• Risk: when only one general runner of a pool is free, either job of a backend can take it first, and the lane is then about as long as one job was.

Results from this PR's CI

Queue plus duration of each microVM job, and the lane of each backend, from the creation of its microVM jobs to the completion of the later one:

Run (head)	WHP default	WHP guests	WHP lane	KVM lane	MSHV lane
39617140157 (376f569)	5.3 + 10.4 min	0.1 + 5.2 min	15.8 min	12.1 min	12.7 min
39622674984 (289be9c)	0.0 + 10.7 min	0.1 + 5.2 min	10.7 min	76.3 min	10.9 min
39771927579 (0ba6caf, after #711 and #712; superseded)	0.7 + 10.8 min	5.2 + 5.6 min	11.5 min	not finished	7.9 min
39779640214 (20d0cb0, after #707)	0.2 + 10.8 min	0.2 + 6.4 min	10.9 min	12.0 min	7.8 min
Head be3a265 (after #708)	in progress

In the first run, only one general WHP runner was free, so the two WHP jobs ran one after the other. In the second, about ten other CI runs queued at once: the KVM guest and sandbox job waited 72.5 min, and the KVM debug-kernel job, which this change doesn't touch, 83.4 min. In the third, the KVM pool's four general runners served three other runs' VMM tests and microVM jobs, so the KVM OpenVMM binary / Linux GNU producer, which both KVM microVM jobs wait for, queued 17.7 min, and the KVM jobs hadn't finished when the #707 rebase superseded the run. In the fourth, each backend's two jobs started within seconds of each other, the KVM pair after waiting 3.7 to 3.9 min for runners. Compared with the single job of dev run 39768324116, the two jobs of each backend ran every one of its Run steps exactly once: KVM 1 + 12 of 13, MSHV 1 + 11 of 12, and WHP 1 + 10 of 11. The default suites downloaded only the kernel and Alpine artifacts, and the seven microVM jobs uploaded seven distinctly named duration reports. The measurements on #695 give the details and the queue time that the extra job adds.

Sibling pull requests

• microvm tests: record each scenario's duration in a JSON Lines report #707, ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708, ci: build the guest-artifact components in parallel and download only what each job uses #711, ci: classify microVM tests, benchmarks, and artifact builds separately #712, ci: run the Linux OpenVMM unit tests on a GitHub-hosted runner #717 (merged): this PR follows them. The new jobs gate on run-microvm-tests; the default suites take only the kernel and Alpine artifacts; the scenario duration reports of the two jobs of a backend have separate names; the required-results policy, the gates, and doc/ci.md keep ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708's build-openvmm-vmm-tests-linux and openvmm-vmm-tests-{linux,windows} beside the new jobs; and the documentation counts the Linux OpenVMM unit tests off MSHV and the Linux VMM-test build on KVM.
• ci: reuse a pull request's guest images on its dev push #713 (ci: reuse a pull request's guest images on its dev push #702, open): merges with this PR without conflicts, and the CI configuration tests pass on the trial merge.

Post-merge verification

• Compare the WHP microVM lane, from the jobs' creation to the later completion of the two WHP jobs, over at least five comparable successful runs after the merge with the 41-run baseline above, including queue time, and record it on ci: run the guest smoke and sandbox steps in a job beside the default microVM suite #695.

[Copilot]

🔵 Needs a closer look

No concrete defects were identified, but cross-backend execution and runner contention still need CI validation and human confirmation.

No open issues 🎉

What changed in this PR

Splits guest and sandbox tests from the default microVM suite to reduce CI turnaround while preserving coverage across KVM, MSHV, and WHP.

Changes:

• Adds parallel guest-and-sandbox jobs with distinct failure-log artifacts.
• Integrates both job groups into required-status, release, and performance-persistence gates.
• Adds regression checks and documents runner-capacity implications.

File	Description
scripts/​test_nvx_tools.py	Checks step selection, job parity, artifact names, and gate wiring.
scripts/​nvx_tools/​ci.py	Adds guest jobs to required-result policy.
doc/​ci.md	Documents parallel jobs, artifacts, gates, and runner usage.
.github/​workflows/​run-nvx-microvm-tests.yml	Selects suite or guest steps and separates artifact names.
.github/​workflows/​ci.yml	Adds three parallel jobs and downstream gate dependencies.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform validation and queue-inclusive performance confirmation remain outstanding for the new CI topology.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-platform job split and publication gates need exact-head CI confirmation and human review of runner-capacity trade-offs.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-backend topology and publication-gate changes need human confirmation of current-head CI results and runner-queue impact.

No open issues 🎉

_{Review effort: Balanced}