ci: investigate why the Windows OpenVMM builds miss sccache

Parent: #430, workstream 2.

Summary

The Windows OpenVMM builds hit sccache far less often than the Linux ones, and the misses don't depend on the cache size. Find which compilations miss and why, fix what can be fixed, and break down the roughly 14 minutes that the WHP VMM-test job spends before its tests start.

Data

Rust hit rates from sccache --show-stats in four full-matrix dev pushes (39592496133, 39585909588, 39472818902, and 39254314492):

Job	Rust hit rate	Cache size after the job
OpenVMM unit tests / Windows	77.7-77.8%, and 18.7% once	4-10 GiB
OpenVMM vmm-tests / Windows / WHP	82.6%	4-10 GiB
OpenVMM unit tests / Linux	99.3%	8-10 GiB
OpenVMM vmm-tests / Linux / KVM	99.7%	5-7 GiB

The jobs ran on several runners, and each runner keeps its own cache on local disk (C:\actions-runner\_work\_sccache on Windows) with a 10 GiB maximum. The setup scripts set SCCACHE_CACHE_SIZE=10G, and validate-runner checks it.
The rates repeat almost exactly across runners and cache sizes, so the same compilations probably miss every time. ci: cut OpenVMM and CLI work from the critical path and the WHP pool #436 attributed the misses to a full cache, but the same rates appear when the cache holds 4 or 7 GiB.
The 18.7% case ran on azure-windows-1, whose cache held 4 GiB, which suggests that runner hadn't built the unit tests recently. There are nine WHP runners (ci: dedicate the benchmark runners to the platform jobs #649), and since ci: run the OpenVMM suites on dev pushes only when their inputs change #660 the OpenVMM suites run less often, so any one runner's cache sees each build less often.
In the reference WHP VMM-test job: 1,779 compile requests, 1,174 hits, 247 misses, and 324 non-cacheable calls (273 because of their crate type). The average compilation took 3.4 seconds.

Tasks

List the crates that miss in consecutive runs on the same runner, for example with SCCACHE_LOG=debug, and find what changes in their arguments, environment, or inputs.
Fix the causes that NVX controls, and report those in OpenVMM's build upstream.
Break down the WHP test-openvmm step into flowey's own build, restore-packages, downloads, compilation, linking, and test execution.
Evaluate a cache shared by each pool's runners, which sccache supports through remote storage backends, against the per-runner local caches.
Revisit SCCACHE_CACHE_SIZE once the misses are understood.

Acceptance criteria

The causes of the Windows misses and the step's breakdown are recorded here.
Any fix reports hit rates and step times before and after, over at least five comparable runs.

Findings

The NVX-side fix is #714. The data below come from the 25 Windows OpenVMM jobs of dev pushes from October 19 to 21 (table at the end) and from one diagnostic run on a temporary branch, 39616159470. That run built each suite twice in one job on the same WHP runner with SCCACHE_LOG=debug, and uploaded the sccache logs and the hashes of every proc-macro DLL.

Why the Windows builds miss

Cause	Platforms	Misses per WHP VMM-test job	Per Windows unit-test job	Fix
Proc-macro DLLs differ in every build	Windows	243	286	#714: `_LINK_=/Brepro`
smoltcp 0.12 writes `config.rs` in `HashMap` order	all	4	7	OpenVMM: smoltcp 0.14 (draft below)
Build-script probes that fail by design	all	17 failed compilations	8	none needed
Runner cache wiped, as by a VMSS image upgrade	per runner	none in the sample	779 more, in 4 of 12 jobs	shared cache (below)

Proc macros, Windows only. sccache can't cache proc macros, so every build compiles and links them again, and it keys every compilation on the contents of the files that it names with --extern. MSVC's link.exe stamps each DLL with the time of the link and a random PDB signature unless it gets /Brepro, so every proc macro came out different in every build, and every crate that uses one directly missed. On Linux, the linker reproduces the shared library exactly. In the diagnostic run, without /Brepro (job, azure-windows-3):
- two consecutive builds both had 1,174 hits and 247 misses. 1,431 of the 1,438 compilations that sccache handled had identical arguments, and the other 7 differ only in the order of their --cfg flags, which sccache sorts. But all 99 proc-macro DLLs differed between the builds;
- 244 of the 247 misses named a proc-macro DLL with --extern, and none of the 1,174 hits did;
- the misses are 124 crates that use one of 30 proc macros, led by bitfield_struct (80 misses) and async_trait (70), then mesh_derive (18), paste (11), and windows_implement and windows_interface (10 each). The costliest were flowey_lib_hvlite (45 s), openvmm_core (35 s), petri (17 s), and netvsp (15 s): 791 s of compiler time per job;
- flowey builds openvmm and pipette in target directories of their own, and the light, host, and target builds compile some crates separately. The misses were spread over seven target directories (119 in target/openvmm, 54 in the artifact-discovery build, 39 in target/light, and the rest in four others), and zerocopy, thiserror, and futures-util each missed seven times per job.
With _LINK_=/Brepro (job, azure-windows-6), the 99 DLLs were byte-identical between the two builds, and the second build hit 1,417 of 1,421 compilations (99.72%). The unit tests (job, azure-windows-1) went from 293 to 7 misses (99.47%). All 29 WHP VMM tests and 4,933 unit tests passed in every build. A local reproduction with the pinned sccache 0.18.0 and Rust 1.95.0 behaves the same way: only the crate that names the proc macro misses, and /Brepro fixes it. ci: run the OpenVMM unit tests off the hypervisor runners #697's experiments corroborate it on another machine: on windows-latest, a second build with the sccache that the first had just filled still missed the same 293 of 1,318 compilations (39615224333, details).
smoltcp, all platforms. OpenVMM pins smoltcp 0.12.0, whose build script collects its configuration in a HashMap and writes OUT_DIR/config.rs in the map's iteration order, which changes in every process. smoltcp therefore misses in every build, and so do consomme, net_consomme, and openvmm_resources, whose smoltcp dependency changed: these are the 4 misses that remain with /Brepro, and probably Linux's 4 as well. smoltcp sorts the output since smoltcp-rs/smoltcp@e4b5b68, released in 0.14.0, so this needs an OpenVMM dependency update; NVX can't fix it.
Probes. The build scripts of proc-macro2, anyhow, and thiserror compile probes that fail on stable Rust by design. sccache counts them as failed compilations and never caches them: 17 per VMM-test job and 8 per unit-test job, at about 0.2–0.3 s each.
Cold runners. Each runner's cache lives in _work\_sccache on its OS disk. The four unit-test jobs with 1,072 misses (18.66%) ran on azure-windows-6, -8, -4, and -1, whose caches held 3–4 GiB after the job, although -4 and -1 had held 10 GiB the evening before. Their caches had been wiped, consistent with the VMSS image upgrades that ci: recover Windows runners after VMSS image upgrades #669 handles, which reimage the OS disk. Their 246 hits match the light builds of flowey and xtask, which the VMM-test jobs share. Such a job took a median of 1,039 s instead of 876 s.

Not causes: the cache size, since the same 247 and 293 compilations missed at 4 to 10 GiB; paths and the environment, which were identical between consecutive builds, with CARGO_HOME at the same _work\_temp\cargo-home in every job; and C or C++. cc-rs finds MSVC's cl.exe itself, without sccache, so the Windows statistics show no C or C++ requests; its objects are reproducible, and its static libraries differ only in archive member timestamps, which sccache ignores.

Breakdown of the WHP `test-openvmm` step

The reference job, the median of the 13 WHP VMM-test jobs below, and the second build with /Brepro in the diagnostic run:

Part	Reference	Median of 13	With `/Brepro`
flowey's own build (`cargo xflowey`, `light` profile)	116 s	111 s	34 s
`restore-packages`	5 s	4 s	2 s
VMM-test build for artifact discovery (`release`)	295 s	288 s	241 s
Artifact query and downloads (azcopy, `sample-vmgs.vhd`)	9 s	8 s	6 s
`openvmm` build, in `target/openvmm`	369 s	366 s	299 s
`pipette` build, in `target/pipette`	29 s	28 s	25 s
`guest_test_uefi` build	11 s	11 s	5 s
UEFI image (`xtask` build), nextest archive, and test listing	34 s	33 s	28 s
Before tests start	868 s	852 s	641 s
Test execution (29 tests)	283 s	280 s	268 s

Downloads are small: the step downloads a 4-MB VMGS image and streams the larger disk images during the tests. Before the step, the job restores the OpenVMM inputs (938 MB) and the Cargo registry (261 MB) from the Actions cache in about 53 s.
Linking dominates once the cache works. The release profile has lto = "thin" and debug = true, so the final crate of each executable does the program's codegen and its link, which sccache can't cache. In the /Brepro build, they took 188 s for the VMM tests' tests executable, 193 s for openvmm, 10 s each for pipette and xtask, and 3 s for flowey_hvlite: about 405 of the 641 s.
Compilation is the rest: 1,417 cache hits that are read in milliseconds but whose keys need a rustc --emit=dep-info pass each (564 s in total across the 8 vCPUs), 99 proc macros and 172 build scripts and executables that sccache can't cache, and the 4 smoltcp misses. Before the fix, the misses added 791 s of compiler time per job.
Beyond this issue: flowey's separate target directories repeat the proc macros, build scripts, and key computations of the shared crates, about 100 s before openvmm's final crate, and the ~405 s of final codegen and linking remain in any job structure. A test profile without ThinLTO or with line-table debug information would shorten the latter, but it would change the binaries that the tests exercise, so both are OpenVMM's decisions.

The Windows unit-test step: xtask build for the fuzz-crate query 134 → 55 s, test build 418 → 339 s, test listing 76 → 59 s, tests 45 s, doctest build 104 → 42 s, and doctests 99 s; the whole step 876 → 643 s (median of the 8 warm jobs below, then the second build with /Brepro).

First builds with `/Brepro`

Each runner's first build with /Brepro still misses the affected crates once, because the new DLLs give them new keys. In #714's run 39626778440, the WHP VMM tests on azure-windows-1, which already held the light entries from the diagnostic run, hit 86.00% (199 misses), spent 43 s instead of 111 s on flowey's build and 798 s before the tests started, and passed all 29 tests. The unit tests on azure-windows-4 hit 77.77%, as before, in their first /Brepro build there. In the next run, 39641375387, the WHP VMM tests on azure-windows-9 missed 238 compilations (83.25%), exactly like the diagnostic run's first /Brepro build, because the target directories now share the proc-macro DLLs, and the unit tests on azure-windows-7 missed 293. Later builds on a runner should hit like the diagnostic run's second builds; the five-run comparison will be added here after #714 merges.

Shared cache

	Per-runner disk (now)	Azure Blob behind the disk (`SCCACHE_MULTILEVEL_CHAIN=disk,azure`)	GitHub Actions cache (`gha`)
Runner after a reimage or a new OpenVMM pin	compiles everything that its cache lacks	reads the pool's entries and backfills its disk	reads the repository's entries
Hit latency	local	local, or one blob read for an entry that the disk lacks	one Actions cache request per entry
Cost	none	storage and operations, a few US dollars a month for 20–50 GB with lifecycle deletion	none within the budget, which the repository's 186 caches (10.85 GB) already exceed
Pull requests	write the runner's cache	read-only, if writes are limited to trusted runs	isolated: a pull request's entries are visible only to it
Credentials	none	jobs can't use the VMSS managed identity, whose instance metadata is blocked for them, so GitHub OIDC with workload identity federation, or SAS secrets	the job's runtime token
Operations	none	storage account, container, identities and federated credentials, role assignments, and a lifecycle rule, since sccache never evicts remote entries	about 1,300–1,400 cache requests per job, against a repository-wide limit of 200 uploads and 1,500 downloads a minute, and evictions of the OpenVMM and guest caches unless the 10-GB limit is raised ($0.07/GB-month)

Correctness. Cache keys contain absolute paths, which match across runners: C:\actions-runner\_work on every Windows runner and /opt/nvx-runner/_work on every Linux one, so SCCACHE_BASEDIRS isn't needed. With /Brepro, the proc-macro DLLs are identical across runners too.
Security. Every job, including same-repository pull requests, can already write its runner's cache, and sccache trusts an entry by its key alone. A pull request could plant an object under a key that a later dev build on that runner computes, including in the OpenVMM producers that feed the development release. A shared cache that pull requests can write would extend that from one runner to the pool. Writes must therefore come only from dev pushes and full runs, through an identity that pull-request jobs can't obtain: two federated credentials, repo:microsoft/nvx:ref:refs/heads/dev with Storage Blob Data Contributor and repo:microsoft/nvx:pull_request with Storage Blob Data Reader and SCCACHE_AZURE_RW_MODE=READ_ONLY. The local tier keeps today's per-runner exposure.
Benefit. It would remove the cold-runner misses, which cost about 160 s in 4 of the 12 sampled unit-test jobs and will weigh more once the warm jobs are faster, and it would compile each new OpenVMM pin once per pool instead of once on each of the eight general WHP runners. It doesn't help the ~405 s of final codegen and linking.
Recommendation. Don't adopt one yet. After ci: link the Windows OpenVMM test builds with /Brepro for sccache #714, measure how often a Windows job misses more than the 4 or 7 smoltcp compilations. If cold runners remain frequent, use disk,azure with OIDC federation and read-only pull requests, with key prefixes per operating system and a lifecycle rule of 14–30 days. The gha backend needs no Azure resources and isolates pull requests by design, but it would evict the repository's other caches and is likely to be throttled. Either is a maintainer decision about Azure resources, identities, and repository variables.

`SCCACHE_CACHE_SIZE`

Keep 10 GiB for now. The misses never depended on the size, and the working set of both Windows suites fits: a wiped runner's cache held 3–4 GiB after one unit-test job, and runners that had run both suites held 5–10 GiB. Until #714, each job also wrote 240–290 entries that no later job could hit, about 650 MiB per Windows unit-test build by #697's measurement, which filled the caches to the limit within a dozen or so jobs and made sccache evict live entries; with /Brepro, a job writes only the 4–7 smoltcp entries. During the jobs, C: had about 70 GB free (51.9–60.4 GB used of 126–127 GB). Revisit after the post-merge runs: if a runner stays at 10 GiB and warm jobs miss more than the smoltcp chain, raise the limit to 20 GiB in both setup scripts, validate-runner, and their tests together, and rerun the setup on every runner before merging, since validate-runner requires the exact value. A larger local cache doesn't survive a reimage.

Follow-ups

OpenVMM: update smoltcp to 0.14. A draft report follows.
The Windows OpenVMM producer (build-openvmm-binary.yml) and the nvx-rs builds on the Windows runners also link proc macros with link.exe and would gain the same way, but ci: link the Windows OpenVMM test builds with /Brepro for sccache #714 leaves them out: the producer builds only when its binary cache misses, and /Brepro would change the published openvmm.exe, which needs its own decision. Setting _LINK_=/Brepro in the runner environment would cover every build, but it needs the setup scripts, validate-runner, and every runner updated together.

Draft upstream report for microsoft/openvmm (not filed)

Title: Update smoltcp to 0.14 so that builds are reproducible and cacheable

smoltcp 0.12.0's build.rs collects its configuration in a std::collections::HashMap and writes OUT_DIR/config.rs by iterating it, so the order of the generated constants changes in every build process. The generated source therefore differs between two builds of the same revision, and build caches that hash sources, such as sccache, miss smoltcp every time. consomme, net_consomme, and openvmm_resources miss after it, because the smoltcp rlib that they depend on changed.

In NVX's CI, with SCCACHE_LOG=debug, two consecutive builds of the VMM tests on the same Windows runner gave these four crates different cache keys while every other key matched once link.exe got /Brepro (https://github.com/microsoft/nvx/actions/runs/39616159470). They miss 4 times per VMM-test build and 7 times per workspace unit-test build, on Linux too.

smoltcp sorts the output since smoltcp-rs/smoltcp@e4b5b68179 ("Sort generated config.rs outputs"), released in 0.14.0. Proposed fix: update the workspace dependency to smoltcp = { version = "0.14", default-features = false } and adapt consomme to the API changes between 0.12 and 0.14.

The 25 Windows OpenVMM jobs of dev pushes

Time is the VMM-test time before tests start, or the whole unit-test step, in seconds.

Start (UTC)	Run and job	Suite	Runner	Cache after	Hits	Misses	Hit rate	Time
10-19 12:43	39254314492	unit	azure-windows-4	10 GiB	1019	293	77.67 %	888
10-19 12:48	39254314492	VMM	azure-windows-3	10 GiB	1169	247	82.56 %	803
10-20 18:13	39472818902	unit	azure-windows-3	10 GiB	1025	293	77.77 %	885
10-20 18:13	39472818902	VMM	azure-windows-2	10 GiB	1174	247	82.62 %	835
10-20 20:11	39489608020	unit	azure-windows-6	4 GiB	246	1072	18.66 %	1040
10-20 20:16	39489608020	VMM	azure-windows-4	10 GiB	1174	247	82.62 %	850
10-20 20:43	39492078575	unit	azure-windows-8	3 GiB	246	1072	18.66 %	1016
10-20 20:58	39492078575	VMM	azure-windows-3	10 GiB	1174	247	82.62 %	859
10-20 22:11	39506181609	unit	azure-windows-3	10 GiB	1025	293	77.77 %	888
10-20 22:11	39506181609	VMM	azure-windows-1	10 GiB	1174	247	82.62 %	852
10-20 22:34	39506407533	unit	azure-windows-9	5 GiB	1025	293	77.77 %	866
10-20 22:39	39506407533	VMM	azure-windows-1	10 GiB	1174	247	82.62 %	850
10-20 23:51	39517713849	unit	azure-windows-2	10 GiB	1025	293	77.77 %	867
10-20 23:58	39517713849	VMM	azure-windows-4	10 GiB	1174	247	82.62 %	845
10-21 01:55	39529338356	unit	azure-windows-7	7 GiB	1025	293	77.77 %	918
10-21 01:59	39529338356	VMM	azure-windows-6	6 GiB	1174	247	82.62 %	886
10-21 10:37	39585909588	unit	azure-windows-9	9 GiB	1025	293	77.77 %	851
10-21 10:37	39585909588	VMM	azure-windows-8	7 GiB	1174	247	82.62 %	868
10-21 11:02	39588920850	unit	azure-windows-4	4 GiB	246	1072	18.66 %	1039
10-21 11:02	39588920850	VMM	azure-windows-3	4 GiB	1174	247	82.62 %	866
10-21 11:31	39592496133	unit	azure-windows-1	4 GiB	246	1072	18.66 %	1053
10-21 11:36	39592496133	VMM	azure-windows-4	4 GiB	1174	247	82.62 %	855
10-21 12:04	39596666239	unit	azure-windows-3	5 GiB	1025	293	77.77 %	855
10-21 12:04	39596666239	VMM	azure-windows-8	7 GiB	1174	247	82.62 %	875
10-21 13:01	39603556404	VMM	azure-windows-4	5 GiB	1174	247	82.62 %	842

ci: link the Windows OpenVMM test builds with /Brepro for sccache

Navigation Menu

Repository navigation

ci: investigate why the Windows OpenVMM builds miss sccache #693

Summary

Data

Tasks

Acceptance criteria

Findings

Why the Windows builds miss

Breakdown of the WHP `test-openvmm` step

First builds with `/Brepro`

Shared cache

`SCCACHE_CACHE_SIZE`

Follow-ups

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ci: investigate why the Windows OpenVMM builds miss sccache #693

Description

Summary

Data

Tasks

Acceptance criteria

Activity

ppenna commented on Oct 21, 2026

Findings

Why the Windows builds miss

Breakdown of the WHP test-openvmm step

First builds with /Brepro

Shared cache

SCCACHE_CACHE_SIZE

Follow-ups

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Breakdown of the WHP `test-openvmm` step

First builds with `/Brepro`

`SCCACHE_CACHE_SIZE`