Skip to content

ci: run the samples and integration tests in their own jobs - #5809

Draft
ric-oliv wants to merge 2 commits into
ci/speed-up-windows-jobsfrom
ci/run-samples-and-integration-tests-in-own-jobs
Draft

ric-oliv wants to merge 2 commits into
ci/speed-up-windows-jobsfrom
ci/run-samples-and-integration-tests-in-own-jobs

Conversation

@ric-oliv

@ric-oliv ric-oliv commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Part 3 of 6 for #5806, based on #5808. The build legs built, ran the unit tests and then ran the integration tests, one step after another, and the app samples made their Build step longer. This PR moves both into jobs that run next to the legs, one commit each.

Build the app samples in their own job. The app samples only check that they compile, but their Release builds trim, AOT-compile and package each app. On macOS that was about 13.7 of the Build step's 28.3 task-minutes.

  • A new Samples job builds all of them on macOS, the Android and MAUI samples on Linux, and the MAUI sample on Windows, which is the only build of its Windows target.
  • The legs' solution filters drop them, regenerated from generate-solution-filters-config.yaml.
  • So that msbuild.exe still builds an Android and a Windows app, the MSBuild job builds the MAUI sample after the solution filter. That takes about 3 minutes and the job still ends well before the slowest check.
  • The only change in coverage is that win-arm64 no longer builds the MAUI sample. win-x64 builds the same targets.

Run the integration tests in their own jobs. A new integration-test matrix job runs next to the legs:

  • On macOS it runs the MAUI cases, which build three Release apps and took 8 to 10 minutes of the macOS leg. They move unchanged from cli.Tests.ps1 to cli-maui.Tests.ps1, and the macOS leg skips them through SKIP_MAUI_TESTS.
  • On linux-x64, win-x64 and win-arm64 it runs every integration test file, which those legs now skip.
  • Each entry builds and packs only the packages its tests use. They aren't the packages the macOS leg publishes, but CI_PUBLISHING_BUILD only adds the other desktop platforms' native libraries and sentry-cli binaries, which these tests don't use.
  • linux-arm64, linux-musl-arm64 and the macOS leg keep their integration tests, because they finish well before the slowest job.

On the stack's own three runs, #5812 with all 6 parts, the slowest check finished at 23.3, 25.5 and 24.3 minutes. On #5802, with the integration tests still in the legs, it was 32.4, and the win-arm64 leg was the slowest job at 31. On #5812, runs 1, 2 and 3:

Leg Integration job
win-x64 15.0, 16.4, 15.4 min 17.5, 17.5, 19.4
linux-x64 13.2, 13.7, 13.3 14.4, 13.6, 14.3
win-arm64 15.9, 16.7, 18.1 22.4, 21.0, 22.4
macOS 21.6, 16.0, 23.6 11.7, 24.5, 12.8, the MAUI cases

The macOS MAUI job depends a lot on the Mac it gets: on run 2 its build took 5.3 minutes against 2.5, and its tests 16.3 against 7.2. The three runs of the throwaway #5805 looked the same: legs 15.0, 11.2 and 17.9, integration jobs 18.5, 12.7 and 22.1. #5802 didn't have the other parts' later fixes either, so not all of those 9 minutes come from this PR.

The linux-x64, win-x64 and win-arm64 integration jobs each passed the same 69 tests and skipped the same 35 as their legs did before, and on all three runs of #5812. The first measurements of each change are in #5783 and #5797.

#skip-changelog

@codecov

codecov Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.15%. Comparing base (d673987) to head (13de6a8).

Additional details and impacted files
@@                     Coverage Diff                      @@
##           ci/speed-up-windows-jobs    #5809      +/-   ##
============================================================
+ Coverage                     76.13%   76.15%   +0.02%     
============================================================
  Files                           528      528              
  Lines                         19579    19579              
  Branches                       3829     3829              
============================================================
+ Hits                          14906    14910       +4     
+ Misses                         3791     3788       -3     
+ Partials                        882      881       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ric-oliv and others added 2 commits October 25, 2026 17:22
The app samples only check that they compile, but their Release builds
trim, AOT-compile and package each app. In the macOS leg that's about
13.7 of the Build step's 28.3 task-minutes, and it's the whole tail of
the step once the Cocoa SDK comes from the cache. On Windows, the MAUI
sample's Android build is the last thing the Build step finishes.

The Android, iOS, Mac Catalyst, macOS and MAUI samples now build in a
separate job that starts with the .NET legs: all of them on macOS, the
Android and MAUI samples on Linux, and the MAUI sample on Windows,
which is the only build of its Windows target. The legs' solution
filters no longer include them, and generated filters list them. The
samples job's Cocoa SDK steps match the macOS leg's.

So that msbuild.exe still builds an Android and a Windows app, the
MSBuild job builds the MAUI sample after the solution filter. That took
180 to 211 seconds on three runs, and the job still ended about 10
minutes before the slowest check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build legs built, ran the unit tests, and then ran the integration
tests, one step after another. On macOS the MAUI cases in cli.Tests.ps1
took 8 to 10 of the leg's 13 to 15 integration test minutes, building
three Release apps for Android, iOS and Mac Catalyst, and the Windows
legs spent 12 to 16 minutes in the integration tests.

The MAUI cases move unchanged to cli-maui.Tests.ps1. A new
integration-test job runs next to the legs: on macOS it runs the MAUI
cases, which the macOS leg now skips, and on linux-x64, win-x64 and
win-arm64 it runs every integration test file, which those legs now
skip. Each entry builds and packs only the packages its tests use. It
packs with --no-build like the legs, because packing Sentry.Maui while
building it on macOS fails with NU5026.

These packages aren't the ones the macOS leg publishes, but
CI_PUBLISHING_BUILD only adds the other desktop platforms' native
libraries and sentry-cli binaries, which these tests don't use.

On three runs with the other CI changes, the win-x64, linux-x64 and
win-arm64 legs took 15.0, 11.2 and 17.9 minutes, and their integration
jobs 18.5, 12.7 and 22.1. Before, the win-arm64 leg took 31 and was the
slowest job. The new jobs passed the same 69 tests and skipped the same
35 as the legs did before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ric-oliv
ric-oliv force-pushed the ci/speed-up-windows-jobs branch from d4f0e61 to d673987 Compare October 25, 2026 16:22
@ric-oliv
ric-oliv force-pushed the ci/run-samples-and-integration-tests-in-own-jobs branch from 7cf192e to 13de6a8 Compare October 25, 2026 16:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant