[Pedro Henrique Penna (ppenna)]

Closes #696.
Part of #430.

Summary

Each backend ran its default Alpine microVM suite, 36 scenarios one after another, in one job, and the WHP one bounds every run that skips both OpenVMM suites. This PR splits each backend's default suite into two shards of about equal duration, planned from the per-scenario duration reports that #707 added, and runs each shard as a leg of the backend's nvx-microvm-tests-<backend> matrix, in parallel on the backend's general runners.

• The partition is checked in as scripts/nvx_tools/microvm_shards.json, which a new nvx.py plan-microvm-shards command generates from CI's duration reports, and which tests verify.
• test-microvm --shard INDEX/COUNT runs one shard, and fails before it runs a scenario unless every default scenario is in exactly one shard of every backend and the backend has COUNT shards. A new scenario, as openvmm: refuse kernel command lines that the guest kernel would truncate #661 added command-line-limit, therefore can't silently drop out of CI.
• Every default scenario still runs exactly once per backend, guest, kernel type, and processor count. The guest and sandbox jobs and the debug-kernel jobs are unchanged, and so are their platform rules.

Why

The partition uses the per-backend medians of the default-suite reports of the 18 CI runs from October 21 and 22 that carry them, which the file lists: #707's and #706's PR runs, the dev pushes of 0e73e460f, b5af359c9, 911d5d86e, 2724c69fc, c9aa40666, and a6785cd0b, the other PR runs since #707 merged, and this PR's first two runs.

Backend	Default suite	#696's proposed groups (first plan, 13 runs)	Shards (18 runs)	Swapped by the first plan
KVM	457.7 s	247.5 / 210.1 s	228.8 / 228.9 s	smp ↔ filesystem-comma-paths
MSHV	416.0 s	203.1 / 213.3 s	208.0 / 208.0 s	time-abi-conformance ↔ directional-network-policy
WHP	601.5 s	287.2 / 309.1 s	300.4 / 301.1 s	snapshot-core ↔ host-loopback-policy

• The first plan, from the 13 runs available when I started, started from the two groups that ci: shard the default microVM suite by scenario duration #696 proposed (restore/snapshot/time ABI vs network/filesystem/lifecycle) and needed one swap per backend. Planned again from all 18 runs, as the documented regeneration does, every scenario stayed in its shard.

• The backends differ materially: console-snapshot takes 49.1 s on KVM, 1.0 on MSHV, and 3.9 on WHP; console-exit 25.2, 27.2, and 87.5 s; private-network-ingress 31.3, 31.6, and 17.4 s; managed-lifecycle 4.2, 4.3, and 17.8 s. The best partition common to all three that a local search found left the longest KVM and MSHV shards about 5% (12 s) above an even split, so each backend has its own.

• Jobs in the three runs after ci: run the guest and sandbox steps beside the default microVM suite #706 split the guest and sandbox steps off (39779640214, 39785687509, 39792079980):

  Job (min)	KVM	MSHV	WHP
  Default suite job	8.1–8.6	7.6–7.7	10.4–11.1
  of which outside the suite (setup, uploads)	0.4–1.0	0.6–0.7	0.7–0.8
  Guest and sandbox job	3.7–3.9	2.9–3.2	5.2–6.4
  Debug-kernel job	2.9–3.0	3.3–3.8	4.2–4.6
  Default suite queue	3.9, 6.8, 4.0	0.1, 1.6, 0.1	0.2, 0.2, 0.1

  The WHP microVM jobs start about 4 to 5.5 min into a run, when NVX CLI / Windows ends.

The raw per-scenario table, with sample counts and ranges, and the timings of this PR's runs are in the comment on #696.

Changes

1. microvm tests: split the default suite into duration-balanced shards
   • scripts/nvx_tools/microvm_shards.py loads and checks the partition, reads scenario duration reports, and plans the shards. A report counts only for the default suite (Alpine, default kernel, default processor counts) of a run that ended, and only for scenarios whose every execution passed. Each scenario starts in its current shard; one without a shard goes, longest first, to the shortest; a shard that is still empty takes the longest scenario of the longest shard that has more than one; then the best move or swap is applied while it shortens the longest shard by at least a second, so noise in new medians doesn't reshuffle a balanced partition.
   • scripts/nvx_tools/microvm_shards.json: per backend, the median of every default scenario, the runs whose reports gave them, and each shard's scenarios and total.
   • nvx.py plan-microvm-shards [--report PATH]... [--seconds SCENARIO=SECONDS]... [--shards BACKEND=COUNT]... [--partition PATH] regenerates it.
   • nvx.py test-microvm --shard INDEX/COUNT, with the runtime checks above; it takes no --scenario, --debug-kernel, or other --guest.
   • The classifier treats the new module and its data as inputs of the microVM tests alone.
2. ci: shard the default microVM suite by scenario duration
   • ci.yml: each nvx-microvm-tests-<backend> job is a fail-fast: false matrix, shard: [1, 2], which passes shard: ${{ matrix.shard }} and shard-count: ${{ strategy.job-total }}; jobs show as NVX microVM tests / <platform> / Shard N of 2.
   • run-nvx-microvm-tests.yml: new shard and shard-count number inputs (default 0); only the default-suite steps pass --shard. Each shard uploads its failure logs as nvx-microvm-tests-<backend>-shard-<n> and its durations as nvx-microvm-durations-<os>-<backend>-default-shard-<n>-<run-id>-<run-attempt>.
   • doc/ci.md describes the shards, how the partition is planned and regenerated, the shard jobs' results and artifacts, and the runners that they hold.

Design notes

• A matrix per backend rather than one top-level job per shard. GitHub evaluates the job's condition before it expands the matrix, so a backend's shards run together or not at all; a shard can't be skipped while its siblings run. The job keeps its ID, so Required status check, Publish development release, and Persist performance baseline keep reading needs.nvx-microvm-tests-<backend>.result, which is success only when every shard succeeded: a failed or cancelled shard fails all three closed, as a failed leg of the openvmm-unit-tests or openvmm-vmm-tests-linux matrix already does. Each leg derives COUNT from strategy.job-total, so a matrix that runs fewer or more legs than the partition has shards fails at run time, and a test checks that the matrix lists exactly shards 1..COUNT.
• Two shards per backend. Each shard then takes about as long as the guest and sandbox job beside it. A third shard (≈3.2, 3.0, 4.0 min) would end before the WHP guest and sandbox job, so it wouldn't shorten WHP, and on KVM, and on MSHV for dev pushes and full runs, the two shards, the guest and sandbox job, and the debug-kernel job already fill the four general runners, so it would mostly queue.
• Debug kernel not sharded: its jobs take 3–5 min, about as long as a shard.
• Checked-in, seeded plan rather than one computed at CI time: a run's shard set is deterministic and reviewable, and regenerations move scenarios only when that shortens the longest shard by a second or more.

Tests

• New MicrovmShardTests (scripts/test_microvm_tests.py):
  • the checked-in partition puts each default scenario in exactly one shard of each backend, with a duration, and is the planner's own plan of its durations (canonical file, no improving move or swap);
  • the default suite equals what an unsharded default run selects, and the shards of each backend together run it, each scenario once, in suite order;
  • --shard fails before any scenario runs, and writes no report, for a new, removed, doubled, or missing scenario, an empty shard, and fewer or more shards than the partition's; it rejects --scenario, --debug-kernel, another guest, and malformed values;
  • the planner: balancing, the one-second threshold, placement of new scenarios and of scenarios from dropped shards, added shards, filling every shard even when no move into it passes the threshold (scenarios of 0 and 0.3 s, and as many shards as scenarios), determinism, and too many shards;
  • which reports count (shards, failures, interrupted and killed runs, debug kernel, other guests and processor counts, duplicates, other schemas), the command's options and errors, and malformed partition files.
• New test_ci_runs_each_default_microvm_suite_in_its_shards (scripts/test_nvx_tools.py): each backend's matrix lists exactly its partition's shards with fail-fast: false; its condition doesn't use the matrix; the guest and sandbox and debug-kernel jobs pass no shard; and each leg's resolved test-microvm command, parsed with nvx.py's parser, runs every default scenario of the backend exactly once on the Alpine guest, the default kernel, and every processor count.
• Updated:
  • ci: run the guest and sandbox steps beside the default microVM suite #706's guest and sandbox test compares that job with the suite's job apart from its shards, and checks that every leg of every microVM job uploads its failure logs under a distinct name;
  • microvm tests: record each scenario's duration in a JSON Lines report #707's scenario duration test expands the matrices and checks a distinct report per shard and attempt;
  • ci: build the guest-artifact components in parallel and download only what each job uses #711's step-to-component test checks each matrix leg's downloads;
  • ci: classify microVM tests, benchmarks, and artifact builds separately #712's classification and drift-guard tests cover the new files, which select the microVM tests and not the platform jobs on pull requests.

Validation

• CI: Every check passed on the final head, 000b1a1d7, in run 39814103094, and before on bf0bfe346 in 39807968692; the skipped jobs are the MSHV and WHP debug-kernel jobs, which pull requests skip, and the release and performance persistence, which only dev pushes run. The first run, 39805377165, was superseded by the push that fixed the review finding, after all of its microVM jobs had passed. No job needed a rerun.
• Windows (Python 3.14), on a6785cd0b:
  • ruff check and ruff format --check, and pyright for the Linux and Windows platforms: clean.
  • scripts/test_nvx_tools.py (508 tests) and scripts/test_microvm_tests.py (237 tests): pass; the first commit's tree passes its own shard and workflow tests too.
  • The rest of validate-nvx's Windows sequence (compileall, every scripts/test_*.py module, test_adversarial.py, test_hosts.py, and the --help checks, plus plan-microvm-shards --help): passes, except 18 test_setup_azure failures in its Linux setup-script tests, which fail identically on an export of origin/dev on this host (its bash), so they're pre-existing and unrelated.
  • check_markdown_links.py and git diff --check: clean.
  • actionlint 1.7.12: only the two known findings (adversarial.yml's runner label, and the queue key of concurrency, which this actionlint doesn't know).
  • plan-microvm-shards without reports reproduces the checked-in file byte for byte; with --shards whp=36 it writes a valid partition.

Measured effect

This PR's three runs are full-matrix runs, since it changes ci.yml, so the OpenVMM suites (≈18–21 min) bound their wall time. Per backend, the default suite's lane runs from when its jobs became ready to when its last shard ended:

Run	Backend	Shard 1: queue, job (suite)	Shard 2: queue, job (suite)	Lane
39805377165 (e2c88625f, superseded after its microVM jobs)	KVM	8.0, 4.4 (3.8)	0.2, 4.4 (3.9)	12.4
	MSHV	0.2, 4.9 (3.5)	0.2, 4.8 (3.5)	5.1
	WHP	0.1, 5.7 (5.0)	0.1, 6.2 (5.3)	6.3
39807968692 (bf0bfe346)	KVM	0.1, 4.5 (3.8)	0.1, 4.5 (3.9)	4.7
	MSHV	0.1, 4.5 (3.5)	0.1, 4.2 (3.4)	4.7
	WHP	0.1, 6.4 (5.6)	0.1, 5.7 (4.9)	6.5
39814103094 (000b1a1d7)	KVM	2.5, 4.5 (3.8)	3.5, 4.5 (3.9)	8.0
	MSHV	0.2, 5.1 (3.5)	0.2, 4.4 (3.4)	5.3
	WHP	0.2, 5.8 (5.0)	0.2, 6.7 (5.7)	6.9

Minutes. Against the single default-suite job of the three runs after #706, measured the same way:

Backend	Lane before	Lane after	Shard queue after
KVM	12.0, 14.8, 12.6 (job 8.1–8.6 after a queue of 3.9–6.8)	12.4, 4.7, 8.0	0.1–8.0
MSHV	7.8, 9.1, 7.8	5.1, 4.7, 5.3	0.1–0.2
WHP	10.9, 11.3, 10.5	6.3, 6.5, 6.9	0.1–0.2

• WHP: the default suite's lane dropped from 10.5–11.3 to 6.3–6.9 min, and the last WHP microVM job ended 12.0–12.4 min into each run, against 14.5–18.6 before.
• KVM queues: KVM's four general runners hold the Linux VMM-test build and then its KVM leg, the two shards, the guest and sandbox job, and the debug-kernel job, and overlapping runs share them. A shard waited 8.0 min in the first run, while a dev push held two of the four, and 2.5 and 3.5 min in the third, while another run held one; the guest and sandbox job waited 3.2 min in the second. Even so, KVM's lane was shorter than any of the three before.
• Runs that skip both OpenVMM suites: without the OpenVMM suites, the second and third runs would have ended when Platform / Windows / WHP did, at 14.5 and 16.2 min, or about 15 and 16.5 with the regression gate and the required check after it. That job (8.9 min) starts after NVX CLI / Windows (3.8–5.3 min) and waits for its pool's single benchmark runner (0.1, 2.0, and 5.5 min here), so it, not the microVM tests, now bounds those runs, at about 15 min when that runner is free.

Expected effect

• The WHP default-suite lane drops from 10.5–11.3 min to 6.3–6.9 min (two shards in parallel), so WHP's microVM jobs end about 12 min into a run, together with its guest and sandbox job, instead of 14.5–18.6.
• Runs that skip both OpenVMM suites and the platform jobs (microVM-test-only changes) should end in about 12 min when no runner queues. Those that run the platform jobs are then bounded by NVX CLI / Windows (≈4–5.3 min) followed by Platform / Windows / WHP (≈8.8–8.9 min) on its single benchmark runner: about 14.5–15 min when that runner is free, which approaches CI: maximize workflow parallelism and reduce turnaround time #430's 15-min target; queueing on that benchmark runner, and on KVM's four general runners, is what can still push them past it.
• Cost: one more job setup per backend per run, about 2 runner-minutes in all. In a full run, KVM briefly needs five of its four general runners (the VMM-test build plus four microVM jobs), so one microVM job waits about three minutes for the debug-kernel job to end, as this PR's green run shows.

Post-merge verification

• Record per-shard durations, per-pool queue times, and the default-suite lane's wall time per backend from at least five comparable dev runs after merge, against five comparable runs before it, on ci: shard the default microVM suite by scenario duration #696, and say whether runs that skip both OpenVMM suites now have a median of at most 15 min.
• Re-plan the partition from the first nightly and post-merge reports if the shards of a backend drift apart by more than a few seconds.

[Copilot]

🟡 Changes recommended

The planner leaves empty shards for valid duration and shard-count inputs, causing regeneration to reject its own output.

1 open issue

• Seed empty shards before applying improvement threshold · New

What changed in this PR

This PR reduces NVX CI turnaround by splitting each backend’s default microVM suite into two duration-balanced shards, while preserving guest, sandbox, and debug-kernel coverage.

Changes:

• Adds a checked-in partition, duration-based planner, and test-microvm --shard support.
• Runs shards through backend matrices with separate diagnostic artifacts.
• Adds partition and workflow validation tests, classifier updates, and documentation.

File	Description
scripts/​test_nvx_tools.py	Checks shard matrices, coverage, artifact names, and classification.
scripts/​test_microvm_tests.py	Tests partition validation, planning, reports, and shard execution.
scripts/​nvx.py	Registers plan-microvm-shards.
scripts/​nvx_tools/​microvm_tests.py	Adds default-suite selection and shard dispatch.
scripts/​nvx_tools/​microvm_shards.py	Implements partition loading, validation, and duration-based planning.
scripts/​nvx_tools/​microvm_shards.json	Stores backend-specific durations and shard assignments.
scripts/​nvx_tools/​ci.py	Classifies shard files as microVM-only inputs.
doc/​usage.md	Documents shard execution and planner options.
doc/​ci.md	Explains shard scheduling, artifacts, capacity, and regeneration.
.github/​workflows/​run-nvx-microvm-tests.yml	Accepts shard inputs and separates shard artifacts.
.github/​workflows/​ci.yml	Runs two parallel default-suite shards per backend.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-backend matrix aggregation, rerun behavior, and runner-capacity effects need exact-head CI evidence and final human review.

No open issues 🎉

1 resolved since last review

• Seed empty shards before applying improvement threshold

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-platform matrix change needs human confirmation of execution, cancellation, rerun behavior, and fail-closed gating before approval.

No open issues 🎉

_{Review effort: Balanced}