Skip to content

ci: classify microVM tests, benchmarks, and artifact builds separately - #712

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/classify-ci-outputs-separately
Oct 21, 2026
Merged

ci: classify microVM tests, benchmarks, and artifact builds separately#712
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/classify-ci-outputs-separately

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #698.

Part of #430.

Summary

The change classifier selected the CI jobs with two coarse outputs: run-tests enabled the NVX microVM tests, and run-workloads enabled the guest artifacts, the three OpenVMM builds, and the platform jobs for every change but documentation. This PR replaces them with four outputs, beside the two OpenVMM suite outputs, so that a change runs only the jobs that its inputs affect:

Output Enables Set when
run-microvm-tests NVX microVM tests and their debug-kernel jobs An input of those jobs, or of a job that builds an artifact that they use, changed
run-benchmarks Platform jobs (benchmarks and packaging) and the PR regression gate Pull request: an input of those jobs changed. Push: any change but documentation
run-guest-artifacts artifacts and debug-kernel A job that uses the guest artifacts runs: the VMM tests, the microVM tests, or the platform jobs
run-openvmm-binaries The three build-openvmm-* producers A job that uses them runs: the microVM tests or the platform jobs
run-openvmm-unit-tests, run-openvmm-vmm-tests OpenVMM suites Unchanged

Both new suites take all of nvx-rs, whose CLI the microVM tests run and every package ships, while the VMM tests keep #710's narrower inputs, only the files that the guest binaries' build reads. It also stops documentation inside a suite's directory, such as scripts/setup/README.md, from selecting suites.

Important

Decision for the maintainer to confirm: dev pushes keep running the platform jobs, and the jobs that build their artifacts, for every change but documentation, exactly as before. Every such commit therefore still publishes a complete development release and records a complete performance-history entry. Only pull requests skip the platform jobs when none of their inputs changed. The data and the alternative are under Dev pushes.

Why

The issue's reference run is 39585909588.

Changes

  1. ci: ignore documentation files when selecting the CI suites: no suite builds, boots, or runs documentation, so the classifier selects each suite only from the changed files that are not documentation.
  2. ci: classify microVM tests, benchmarks, and artifact builds separately:
    • scripts/nvx_tools/ci.py: the six outputs. MICROVM_TEST_INPUT_PATHS and BENCHMARK_INPUT_PATHS replace CI_TEST_INPUT_PATHS, and both keep all of nvx-rs/. Every CLI module is an input of both unless none of that suite's jobs runs it, so a new module selects both until someone excludes it. Only the microVM tests run microvm_tests.py and the modules and guest scripts that only it, nvx.py sandbox, and test-aci-edge-sandboxes use. Only the platform jobs run performance.py. benchmark.py, which microvm_tests.py imports, doctor.py, which every runner validation runs, and release.py, whose verify the OpenVMM builds run, select both. The two producer outputs follow their consumers, so no job ever waits for an artifact that its run skipped. The job summary explains each output, and check-required-ci mirrors the job conditions: it expects a job only when the jobs that build its artifacts run too.
    • scripts/nvx.py: check-required-ci takes the six flags.
    • .github/workflows/ci.yml: the detector emits, checks, and falls back on the six outputs. Each job's condition reads its own output, and Required status check passes all six. The release, gate, and persistence conditions are unchanged.
    • Docs: doc/ci.md (Change classification, and a new Development releases and performance history section), doc/usage.md, doc/distribution.md, and doc/contribute.md.

The detector's fallbacks to the full matrix are unchanged: a missing base or commit, a failed diff or classifier, or a missing, duplicated, or malformed output. Nightly and dispatched runs still select everything before classification.

Dev pushes: what they package, publish, and persist

A dev push publishes its commit's development release and appends the platform results to data/. Publish development release and Persist performance baseline keep requiring all three platform jobs to succeed, plus the debug kernel and the CLI jobs. A push that skipped the platform jobs would therefore publish nothing partial or stale, but its commit would have no release. download would then install an ancestor's release, which is older still when that ancestor's release failed.

So this PR keeps the conservative behavior: a push runs the platform jobs, and their producers, for every change but documentation, as before. The debug kernel still builds with the guest artifacts, because the release and persistence require it.

Data from the 108 merges to dev between October 10 and 21, through #710, that changed more than documentation:

If you prefer the alternative, it's a one-line change in classify_ci_changes, dropping or (event_name == "push" and bool(inputs)), plus the matching test and doc updates.

Tests

  • Classifier: every path class for pull requests and pushes, including documentation, the empty diff, the OpenVMM pin, runner setup, workflows, and the shared, microVM-only, platform-only, dev-only, and GitHub-hosted-only actions. Also every module class, module data such as benchmark_scripts/, microvm_test_scripts/, host_time_probe.rs, and process_supervisor.py, the guests, crates, data/, VERSION, LICENSE, .gitattributes, and unions.
  • Producers: over every combination of ten representative paths, a host-side nvx-rs file among them, both producer outputs equal the OR of their consumers.
  • nvx-rs (ci: key the guest images and VMM tests on the guest binaries' sources #710's rows and acceptance test, ported): every file of the crate selects the microVM tests and the platform jobs, and only the guest binaries' inputs select the VMM tests and change the guest image keys, for pull requests and pushes.
  • Job summary and CLI: the per-output job summary and reasons, and the CLI output order. check-required-ci requires and validates each flag, and passes each one to its own suite: flipping any one flag fails the check.
  • Required-result policy: every output, including inconsistent flags, where a job is expected only beside its producers, plus fork PRs, pushes, nightly runs, and dispatched runs.
  • Workflow structure: each job's condition reads exactly its output with the fail-safe result != 'success' guard, only those jobs and Required status check read the outputs, and the detector lists exactly the classifier's outputs.
  • Detector (bash, POSIX): the full-matrix fallback when each output is missing or duplicated, and real git paths.
  • Drift guard: derives the microVM tests' and platform jobs' inputs from their jobs and the jobs that build their artifacts. That covers the reusable workflows and actions that these reach, the cache keys there, and the CLI modules of the nvx.py commands that they run, which the guard maps from nvx.py's parser with the AST. It also covers every module that those modules import, every file that they read beside them through Path(__file__).with_name, and the nvx crate that the CLI jobs build. The guest-artifact inputs come from the cache keys and, since ci: key the guest images and VMM tests on the guest binaries' sources #710 took nvx-rs out of the image keys' hashFiles, from the files of the guest binaries' source identity, with which every image key ends. The guard therefore follows key changes such as ci: key the guest images and VMM tests on the guest binaries' sources #710's.

Validation

All on the head rebased onto dev at 6f0facefd, which includes #710, #717, #714, #661, and #656:

  • Windows (Python 3.14):
    • The validate-nvx unittest set ran 1,100 tests, and every test_nvx_tools test passed. The 18 failures were in test_setup_azure's Linux shell-script tests, which fail the same way on origin/dev in this environment.
    • Each commit's focused classifier, policy, and nvx-rs tests passed on its own tree.
    • test_adversarial.py, test_hosts.py, compileall, every nvx.py ... --help in validate-nvx, and check_markdown_links.py passed.
  • WSL Ubuntu (Python 3.14), on an LF clone of the branch: the POSIX-only detector tests and the whole CiConfigurationTests class (67 tests) passed.
  • Linters and type checks:
    • ruff check, ruff format --check, and pyright for Linux and Windows passed, all at the pinned versions.
    • actionlint v1.7.7 (checksum-verified release binary) found nothing in ci.yml or the three reusable workflows.
  • Seeded regressions: each of 20 pattern mutations fails the drift guard, for example excluding doctor.py, release.py, or process_supervisor.py from the microVM tests, or dropping data/, VERSION, run-platform, or gate-performance from the platform jobs. After the rebase, so does each of 8 more: dropping nvx-rs/ or the Rust pin from either suite, dropping the guest binaries' source pattern from the VMM tests, or excluding guest_binaries.py from either suite.
  • Left to CI: every hypervisor job. This PR changes ci.yml, an input of every output, so its own CI runs the full matrix.

Expected effect

Replaying the new classifier over the same diffs that the old one classified:

Job group Old New (pushes) Alternative: pushes like PRs
Guest artifacts / OpenVMM builds 108 108 106
NVX microVM tests 108 104 104
Platform jobs 108 108 98
OpenVMM unit tests 44 42 42
OpenVMM VMM tests 70 68 68

108 merges to dev from October 10 to 21, through #710, that changed more than documentation, plus 24 documentation-only merges, which run nothing in every case. "Old" is dev's classifier with #710, whose narrower VMM-test inputs already skip the VMM tests for host-side nvx-rs changes.

Job group Old New
Guest artifacts / OpenVMM builds 122 117
NVX microVM tests 119 116
Platform jobs and gate 122 114

124 completed, non-draft pull-request runs of this repository's branches from October 15 to 21, across 74 PRs; each head is diffed from its base before merging.

  • In the 9 affected pull-request runs, the median wall time drops from 34.8 to 20.2 minutes, by up to 105 minutes when the WHP platform job would have queued for the benchmark runner. They free 87 KVM, 78 MSHV, and 120 WHP runner-minutes.
  • In the 9% of non-documentation changes that change no platform input (10 of the 108 merges), the run no longer waits for the WHP benchmark runner, which queued a median 3.8 and a p90 27.2 minutes in the 82 sampled runs. Pure tooling PRs, such as the CLI tests or the GitHub-hosted checks, run only the GitHub-hosted jobs.
  • The effect is modest because most recent pull requests change the nvx crate. Since ci: key the guest images and VMM tests on the guest binaries' sources #710, a change to its host side or tests skips the VMM tests, but it still runs the microVM tests, which run its CLI, and the platform jobs, whose packages ship it. A follow-up that packages the release outside the benchmark runners could let run-benchmarks drop the package inputs, so that such changes would skip the benchmarks too.

Interactions with other #430 PRs

Merged into dev since this PR opened; the rebase adapted to the first five:

Still open; whichever merges second adapts:

Post-merge verification

Copilot AI balanced review requested due to automatic review settings October 21, 2026 15:23
@ppenna Pedro Henrique Penna (ppenna) mentioned this pull request Oct 21, 2026
Closed
5 tasks
Copilot AI reviewed Oct 21, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-platform required checks and artifact dependencies warrant human review, and the development-push policy explicitly awaits maintainer confirmation.

No open issues 🎉

What changed in this PR

Refines NVX CI selection to reduce unnecessary jobs while preserving development-release and performance-history behavior.

Changes:

  • Separates microVM tests, benchmarks, guest artifacts, and OpenVMM binary selection.
  • Updates workflow gating, required-status validation, and regression tests.
  • Excludes documentation from suite selection and documents the retained dev-push policy.
File Description
scripts/​test_nvx_tools.py Tests classification, fallbacks, dependencies, and workflow wiring.
scripts/​nvx.py Accepts six required-status flags.
scripts/​nvx_tools/​ci.py Implements separate selections and result expectations.
doc/​usage.md Updates CLI flags and selection behavior.
doc/​distribution.md Clarifies when platform packaging runs.
doc/​contribute.md Updates runner-selection guidance.
doc/​ci.md Documents classification and development-push policy.
.github/​workflows/​ci.yml Wires job conditions to the new outputs.

Review effort: Balanced

This was referenced Oct 21, 2026
Open
Open
@ppenna
ci: ignore documentation files when selecting the CI suites
9ad0468 
The change classifier counted a documentation file as a suite input
whenever its path matched the suite's pattern. A change only to
scripts/setup/README.md therefore set run-tests and both OpenVMM suite
outputs, although it skipped the workloads: it built OpenVMM on a KVM,
an MSHV, and a WHP runner and ran the OpenVMM unit tests on an MSHV and
a WHP runner, none of which reads the file. Beside any other change, a
README in a test input directory also ran the NVX microVM tests and
both OpenVMM suites.

No suite builds, boots, or runs documentation, which the classifier
already defines as every path under doc/ or ending in .md, and the
GitHub-hosted jobs that check it, the Markdown link check and the CLI
tests, run for every change. Select each suite only from the changed
files that are not documentation, and cite only those in the job
summary. A documentation-only change still skips the workloads, and an
empty diff still runs them.

The classifier tests now expect no suite for documentation in the
runner setup, nvx-rs, and runtime adapter directories, alone or beside
another change, and the summary test checks that a guest README is not
cited as an input of the VMM tests. doc/ci.md and doc/usage.md state
the rule.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
ci: classify microVM tests, benchmarks, and artifact builds separately
7ccb8d5 
The change classifier reported run-tests, which enabled the NVX
microVM tests, and run-workloads, which enabled the guest artifacts,
the three OpenVMM builds, and the platform jobs for every change but
documentation. A change only to the performance tooling therefore ran
every microVM job; a change only to a microVM scenario, the
aci_edge_sandboxes crate, or the runtime adapter ran the platform jobs,
which wait for their pool's single benchmark runner; and a change only
to the CLI tests or the adversarial tooling built every artifact and
ran the benchmarks.

Replace the two outputs with four, beside the two OpenVMM suite
outputs:

- run-microvm-tests enables the microVM and debug-kernel jobs, from the
  inputs of those jobs and of every job that builds an artifact that
  they use.
- run-benchmarks enables the platform jobs, which benchmark and package
  each backend, and so a pull request's regression gate, from the
  inputs of those jobs, the gate, and the jobs that build their
  artifacts, including the history and expected hosts in data/,
  VERSION, and LICENSE.
- run-guest-artifacts and run-openvmm-binaries enable the jobs that
  build the guest artifacts and OpenVMM. Neither has a use of its own,
  so the classifier sets each exactly when a job that uses its
  artifacts runs. The guest artifacts thus follow the inputs of the
  OpenVMM VMM tests, which take only the guest binaries' inputs of
  nvx-rs.

Both new suites take all of nvx-rs, whose CLI the microVM tests run and
every package ships. Every CLI module is an input of both unless none
of their jobs runs it, so a new module selects both. Only the microVM
tests run microvm_tests.py and the modules and guest scripts that only
it, nvx.py sandbox, and test-aci-edge-sandboxes use, and only the
platform jobs run performance.py. benchmark.py, which microvm_tests.py
imports, doctor.py, which every runner validation runs, and release.py,
whose verify command the OpenVMM builds run, select both. The OpenVMM
suites keep their inputs and no longer need run-tests, which every one
of their inputs set.

A dev push publishes its commit's development release and appends to
the performance history, and both still require all three platform
jobs to succeed, so a push that skipped them would publish nothing
partial or stale but would leave its commit without a release, and
download would install an older one. Pushes therefore keep running the
platform jobs, and the jobs that build their artifacts, for every
change but documentation, as before, and pull requests, which publish
and persist nothing, run them only when one of their inputs changed.
Of the 108 merges from October 10 to 21 that changed more than
documentation, 10 changed no input of the platform jobs, whose pushes
each still spend about 19 benchmark-runner minutes on them. An empty
diff still runs the platform jobs, and the detector's fallbacks, the
nightly and dispatched full runs, and the release, gate, and
persistence conditions are unchanged.

check-required-ci takes the six outputs and, like the job conditions,
expects a job only when the jobs that build its artifacts run too.
Replayed over the 124 completed pull-request runs of this repository's
branches from October 15 to 21 that ran CI, the classifier skips the
platform jobs in 8, the microVM tests in 3, and every artifact build in
5.

Tests cover each path class for pull requests and pushes, nvx-rs's
guest and host sides included, the artifact builds following the jobs
that use them, the job summary, the CLI, the required-result policy for
every output, the job condition that reads each output, and the
detector's fallback when any output is missing or duplicated. The drift
test now derives the inputs of the microVM tests and the platform jobs
from those jobs and the jobs that build their artifacts: the reusable
workflows and actions that these reach, the cache keys there and the
files of the guest binaries' source identity, with which every image
key ends, the CLI modules of the nvx.py commands that they run, which
it maps from nvx.py's parser, with every module that those import and
every file that those read beside them, and the nvx crate that the CLI
jobs build. doc/ci.md and the other affected guides describe the
outputs and the push rule.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/classify-ci-outputs-separately branch from e4f61d2 to 7ccb8d5 Compare October 21, 2026 20:00
Copilot AI balanced review requested due to automatic review settings October 21, 2026 20:00
Copilot AI reviewed Oct 21, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-platform CI gating needs final human validation, and the development-push release/history policy explicitly awaits maintainer confirmation.

No open issues 🎉

Review effort: Balanced

@ppenna Pedro Henrique Penna (ppenna) mentioned this pull request Oct 21, 2026
Open
2 tasks
@ppenna
Pedro Henrique Penna (ppenna) merged commit 823aff0 into dev Oct 21, 2026
34 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/classify-ci-outputs-separately branch October 21, 2026 22:40
This was referenced Oct 22, 2026
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ci: classify the microVM tests, benchmarks, guest artifacts, and OpenVMM binaries separately

2 participants

@ppenna