ci: shard the default microVM suite by scenario duration #696

Activity

1. Pedro Henrique Penna (ppenna)

   self-assigned this

   on Oct 21, 2026
2. Pedro Henrique Penna (ppenna)

   added the

   Feature

   issue type on Oct 21, 2026
3. Pedro Henrique Penna (ppenna)

   added a parent issue on Oct 21, 2026

   • CI: maximize workflow parallelism and reduce turnaround time #430
4. Pedro Henrique Penna (ppenna)

   marked this as blocked on Oct 21, 2026

   • ci: record each microVM scenario's duration in a machine-readable report #694
5. Pedro Henrique Penna (ppenna)

   added 2 commits that reference this issue on Oct 21, 2026

   microvm tests: record each scenario's duration in a JSON Lines report

   ...

   dcb7f8e

   ci: upload the scenario duration reports of every microVM job

   ...

   a904264
6. ppenna commented on Oct 22, 2026

   

   Pedro Henrique Penna (ppenna)

   on Oct 22, 2026

   Last edited by ppenna

   ContributorAuthor

   More actions

   Shard plan and measurements for #763

   PR #763 implements this issue. This comment records the data behind the partition and the shard count, and the timings before and after. I'll update it with the post-merge comparison.

   Scenario durations

   Source: the nvx-microvm-durations-* artifacts (#707) of the 18 CI runs that had them by 2026-10-22 16:25 UTC, which scripts/nvx_tools/microvm_shards.json lists: the PR runs of #707 and #706, the dev pushes of 0e73e460f (39777444440), b5af359c9 (39783750996), 911d5d86e (39788615461), 2724c69fc (39792079980, the first after the #706 split), c9aa40666 (39801222132), and a6785cd0b (39804644654), the other PR runs since #707 merged, and #763's first two runs. Only the default suite's reports count (Alpine, default kernel, processors 1 2 4 8), sharded or not, of runs that ended, and a scenario only when all of its executions passed, so the two cancelled runs contribute the scenarios that they finished. The medians are per backend, to 0.1 s, as plan-microvm-shards computes them.

   Raw table: per-scenario median, sample count, and range, in seconds, and each scenario's shard

   Runs (18): 39619765285-1, 39666359892-1, 39777444440-1, 39778084391-1, 39779640214-1, 39780305955-1, 39782906524-1, 39783750996-1, 39785367867-1, 39785687509-1, 39788615461-1, 39791725058-1, 39792079980-1, 39801222132-1, 39804644654-1, 39805377165-1, 39807968692-1, 39810434285-1

   Scenario	KVM median (n, range)	shard	MSHV median (n, range)	shard	WHP median (n, range)	shard
   console-exit	25.2 (17, 24.7–25.8)	2	27.2 (18, 25.6–28.1)	2	87.5 (18, 77.1–106.8)	2
   restore-downtime	69.2 (17, 69.0–69.4)	1	71.0 (18, 69.8–71.4)	1	84.7 (17, 83.4–90.4)	1
   smp-snapshot	26.1 (17, 25.9–26.3)	1	28.5 (18, 27.5–28.8)	1	46.6 (18, 44.0–57.6)	1
   ipv6-egress-policy	29.4 (17, 29.2–29.7)	2	30.0 (18, 29.3–30.6)	2	38.6 (18, 36.5–42.0)	2
   l3-l4-egress-protocols	31.1 (17, 30.8–31.4)	2	31.3 (18, 30.7–31.7)	2	37.5 (18, 36.0–39.6)	2
   smp	21.2 (17, 21.1–21.3)	2	22.6 (18, 21.9–22.9)	1	30.9 (18, 29.5–32.0)	1
   restore-processors	21.4 (17, 21.2–21.6)	1	21.3 (18, 21.0–21.5)	1	28.8 (18, 28.2–30.9)	1
   host-loopback-policy	23.1 (17, 22.7–23.1)	2	23.5 (18, 23.0–23.8)	2	28.2 (18, 27.4–29.6)	1
   l3-l4-egress-port-ranges	22.4 (17, 22.1–22.6)	2	22.5 (18, 22.1–23.0)	2	27.4 (18, 26.4–28.9)	2
   l3-l4-egress-policy	19.3 (17, 19.3–19.4)	2	19.3 (18, 19.3–19.4)	2	19.9 (18, 19.8–20.1)	2
   managed-lifecycle	4.2 (17, 4.0–7.8)	2	4.3 (18, 3.9–8.0)	2	17.8 (18, 13.0–22.7)	2
   network-snapshot	17.0 (17, 16.9–17.1)	1	16.8 (18, 16.8–16.9)	1	17.6 (18, 17.5–18.0)	1
   private-network-ingress	31.3 (17, 31.2–31.5)	2	31.6 (18, 31.3–31.8)	2	17.4 (18, 16.9–18.2)	2
   snapshot-core	13.4 (17, 13.2–13.6)	1	12.9 (18, 12.7–13.3)	1	17.2 (17, 16.4–18.9)	2
   restore-memory	4.1 (17, 4.0–4.3)	1	4.0 (18, 3.9–4.1)	1	12.5 (17, 10.9–18.5)	1
   endpoint-policy-snapshot	10.3 (17, 10.2–10.4)	1	10.1 (18, 10.0–10.1)	1	12.4 (18, 11.9–13.6)	1
   snapshot-tiers	6.8 (17, 6.7–7.0)	1	6.3 (18, 6.2–6.6)	1	12.0 (17, 11.7–16.3)	1
   scratch-snapshot	3.4 (17, 3.2–3.5)	1	3.2 (18, 3.1–3.3)	1	8.8 (18, 8.0–11.9)	1
   filesystem-snapshot	3.2 (17, 3.1–5.2)	1	2.9 (18, 2.7–3.4)	1	8.1 (18, 6.9–10.7)	1
   directional-network-policy	5.8 (17, 5.8–5.9)	2	5.7 (18, 5.5–5.9)	1	7.0 (17, 6.7–7.2)	2
   filesystem-shares	2.0 (17, 2.0–2.1)	2	2.2 (18, 1.8–2.3)	2	5.8 (18, 5.2–7.4)	2
   filesystem-comma-paths	2.6 (17, 2.5–2.7)	1	2.9 (18, 2.6–3.0)	2	5.3 (18, 4.7–5.9)	2
   filesystem-policy	1.9 (17, 1.8–2.1)	2	1.8 (18, 1.7–2.0)	2	4.8 (18, 4.4–6.6)	2
   virtio-net	3.8 (17, 3.8–3.8)	2	3.7 (18, 3.6–3.9)	2	4.5 (17, 4.3–4.6)	2
   cpu-profile-fallback	1.5 (17, 1.3–1.7)	1	1.7 (18, 1.4–1.8)	1	4.3 (18, 3.7–5.1)	1
   console-snapshot	49.1 (17, 48.7–49.8)	1	1.0 (18, 0.9–1.0)	1	3.9 (18, 3.3–5.0)	1
   command-line-limit	1.5 (16, 1.4–1.5)	2	1.2 (17, 1.1–1.4)	2	2.3 (16, 2.1–2.5)	2
   lifecycle	1.7 (17, 1.7–1.8)	2	1.6 (18, 1.5–1.7)	2	2.1 (18, 2.0–2.2)	2
   time-abi-conformance	0.7 (17, 0.7–0.8)	1	0.8 (18, 0.7–0.9)	2	1.6 (17, 1.5–1.8)	1
   denied-filesystem-paths	0.8 (17, 0.7–0.8)	2	0.6 (18, 0.6–0.8)	2	1.3 (18, 1.2–1.4)	2
   sandbox-blocks	0.7 (17, 0.7–0.8)	2	0.6 (18, 0.5–0.7)	2	1.2 (18, 1.1–1.4)	2
   workload-identity	0.7 (17, 0.7–0.8)	2	0.6 (18, 0.5–0.6)	2	1.1 (18, 1.0–1.4)	2
   guest-boot	0.7 (17, 0.7–0.7)	2	0.5 (18, 0.4–0.6)	2	0.8 (18, 0.7–0.9)	2
   guest-identity	0.7 (17, 0.6–0.7)	2	0.5 (18, 0.4–0.6)	2	0.8 (18, 0.8–0.9)	2
   structured-outcome	0.6 (17, 0.5–0.7)	2	0.5 (18, 0.4–0.5)	2	0.8 (18, 0.8–1.0)	2
   filesystem-owner	0.8 (17, 0.8–0.9)	2	0.8 (18, 0.7–0.9)	2	0.0 (18, 0.0–0.0)	2
   Sum of medians / shards	457.7	228.8 / 228.9	416.0	208.0 / 208.0	601.5	300.4 / 301.1

   The backends differ materially: console-snapshot takes 49.1 s on KVM, 1.0 on MSHV, and 3.9 on WHP; console-exit 25.2, 27.2, and 87.5 s; private-network-ingress 31.3, 31.6, and 17.4 s; managed-lifecycle 4.2, 4.3, and 17.8 s. The best partition common to all three backends that a local search found left the longest KVM and MSHV shards about 5% (12 s) above an even split, so each backend has its own partition.

   Partition

   The first plan used the 13 runs available at 14:50 UTC. It started from the two groups proposed here: restore/snapshot/time ABI (console-snapshot, cpu-profile-fallback, endpoint-policy-snapshot, filesystem-snapshot, network-snapshot, restore-downtime, restore-memory, restore-processors, scratch-snapshot, smp, smp-snapshot, snapshot-core, snapshot-tiers, time-abi-conformance) and network/filesystem/lifecycle (the other 22). One swap per backend balanced them. Planned again from all 18 runs, as the documented regeneration does, every scenario stayed in its shard:

   Backend	Default suite	Proposed groups (13 runs)	Shards (18 runs)	Swap of the first plan
   KVM	457.7 s	247.5 / 210.1 s	228.8 / 228.9 s	smp ↔ filesystem-comma-paths
   MSHV	416.0 s	203.1 / 213.3 s	208.0 / 208.0 s	time-abi-conformance ↔ directional-network-policy
   WHP	601.5 s	287.2 / 309.1 s	300.4 / 301.1 s	snapshot-core ↔ host-loopback-policy

   doc/ci.md#microvm-test-shards describes how to regenerate the partition.

   Shard count

   Before the change, in the three runs after #706 (39779640214, 39785687509, 39792079980):

   Job (minutes)	KVM	MSHV	WHP
   Default-suite job	8.1–8.6	7.6–7.7	10.4–11.1
   of which outside the suite	0.4–1.0	0.6–0.7	0.7–0.8
   Guest and sandbox job	3.7–3.9	2.9–3.2	5.2–6.4
   Debug-kernel job	2.9–3.0	3.3–3.8	4.2–4.6
   Default-suite queue	3.9, 6.8, 4.0	0.1, 1.6, 0.1	0.2, 0.2, 0.1

   Two shards per backend make each shard job about as long as the guest and sandbox job beside it. A third shard (≈3.2, 3.0, and 4.0 min) would end before the WHP guest and sandbox job, so it wouldn't shorten WHP; and on KVM, and on MSHV for dev pushes and full runs, the two shards, the guest and sandbox job, and the debug-kernel job already fill the four general runners, so it would mostly queue. The debug-kernel jobs (3–5 min) stay unsharded.

   After: #763's CI runs

   This PR's three runs are full-matrix runs, since it changes ci.yml, so the OpenVMM suites (≈18–21 min) bound their wall time. Per backend, the default suite's lane runs from when its jobs became ready to when its last shard ended:

   Run	Backend	Shard 1: queue, job (suite)	Shard 2: queue, job (suite)	Lane
   39805377165 (e2c88625f, superseded after its microVM jobs)	KVM	8.0, 4.4 (3.8)	0.2, 4.4 (3.9)	12.4
   	MSHV	0.2, 4.9 (3.5)	0.2, 4.8 (3.5)	5.1
   	WHP	0.1, 5.7 (5.0)	0.1, 6.2 (5.3)	6.3
   39807968692 (bf0bfe346)	KVM	0.1, 4.5 (3.8)	0.1, 4.5 (3.9)	4.7
   	MSHV	0.1, 4.5 (3.5)	0.1, 4.2 (3.4)	4.7
   	WHP	0.1, 6.4 (5.6)	0.1, 5.7 (4.9)	6.5
   39814103094 (000b1a1d7)	KVM	2.5, 4.5 (3.8)	3.5, 4.5 (3.9)	8.0
   	MSHV	0.2, 5.1 (3.5)	0.2, 4.4 (3.4)	5.3
   	WHP	0.2, 5.8 (5.0)	0.2, 6.7 (5.7)	6.9

   Minutes. Against the single default-suite job of the three runs after #706, measured the same way:

   Backend	Lane before	Lane after	Shard queue after
   KVM	12.0, 14.8, 12.6 (job 8.1–8.6 after a queue of 3.9–6.8)	12.4, 4.7, 8.0	0.1–8.0
   MSHV	7.8, 9.1, 7.8	5.1, 4.7, 5.3	0.1–0.2
   WHP	10.9, 11.3, 10.5	6.3, 6.5, 6.9	0.1–0.2

   • WHP: the default suite's lane dropped from 10.5–11.3 to 6.3–6.9 min, and the last WHP microVM job ended 12.0–12.4 min into each run, against 14.5–18.6 before.
   • KVM queues: KVM's four general runners hold the Linux VMM-test build and then its KVM leg, the two shards, the guest and sandbox job, and the debug-kernel job, and overlapping runs share them. A shard waited 8.0 min in the first run, while a dev push held two of the four, and 2.5 and 3.5 min in the third, while another run held one; the guest and sandbox job waited 3.2 min in the second. Even so, KVM's lane was shorter than any of the three before.
   • Runs that skip both OpenVMM suites: without the OpenVMM suites, the second and third runs would have ended when Platform / Windows / WHP did, at 14.5 and 16.2 min, or about 15 and 16.5 with the regression gate and the required check after it. That job (8.9 min) starts after NVX CLI / Windows (3.8–5.3 min) and waits for its pool's single benchmark runner (0.1, 2.0, and 5.5 min here), so it, not the microVM tests, now bounds those runs, at about 15 min when that runner is free.

   The 15-minute target

   Runs that skip both OpenVMM suites now approach it but don't reliably meet it. The WHP microVM jobs end about 12 min into a run, so Platform / Windows / WHP bounds those runs instead. That job takes 8.9 min, starts after NVX CLI / Windows (3.8–5.3 min), and waits for its pool's single benchmark runner, so those runs take about 15 min when that runner is free, and more when it isn't. Runs that skip the platform jobs too, such as microVM-test-only changes, should end in about 12 min, unless KVM queues. KVM's four general runners, shared by overlapping runs, are the other bound: a KVM shard waited 2.5–8.0 min in two of the three runs.

   Post-merge verification

   • Five comparable dev runs after the merge against five before it: per-shard durations, per-pool queue times, each backend's default-suite lane, and the wall time of runs that skip both OpenVMM suites.
   • Re-plan the partition from the first nightly and post-merge reports if a backend's shards drift apart by more than a few seconds.
7. Pedro Henrique Penna (ppenna)

   added 2 commits that reference this issue on Oct 22, 2026

   microvm tests: split the default suite into duration-balanced shards

   ...

   a054653

   microvm tests: split the default suite into duration-balanced shards

   ...

   80c195c
8. Pedro Henrique Penna (ppenna)

   closed this as completedin #763on Oct 22, 2026