Skip to content

CI: cut repeated work in the C++ builds - #1610

Merged
Gin890 merged 1 commit into
mainfrom
ci-efficiency
Oct 18, 2026
Merged

Gin890 merged 1 commit into
mainfrom
ci-efficiency

Conversation

@Gin890

@Gin890 Gin890 commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Why

A push to main uses about 240 runner-minutes, and the slowest job (Ubuntu clang) takes 72–89 minutes. Per-step timings from recent runs show that most of this is the same work repeated on every run:

Job Total Where the time goes
Ubuntu clang 72–89 min clang-query 42–55 min, build 23–29 min
Mac Intel 49–51 min brew 16–18 min (13 of it compiling libdpp), build 18–24 min
Ubuntu Default 41 min clang-query 26–28 min, build 10–11 min
Windows ×2 22–32 min each build 19–27 min
Mac 21–22 min build 11–13 min, brew 5–7 min (4 of it compiling libdpp)

Changes

  • clang-query runs in one job, one batch per core. It checks the source rather than the compiled code, so running it in both Ubuntu jobs was redundant. It now runs only in Ubuntu Default, the fastest Linux build. It runs nproc batches of 10 files at once, each batch writing to its own output file. Each file is now listed only once: compile_commands.json has one entry per target for files built into several targets, and clang-query already checks every entry of each file it's given, so a file listed twice had each of its entries parsed twice. If clang-query fails on a file, the step still prints the collected output and then fails.
  • libdpp is cached on macOS. Our pinned brew formula has no bottle, so every Mac job compiled DPP from source. The installed keg is now cached with actions/cache, keyed on the formula file and on the installed versions of libsodium, openssl@3 and opus, so a Homebrew update to any of them triggers a rebuild. A new pkg-config --exact-version=10.1.5 dpp check fails the job if DPP isn't found. Without it, CMake would build without Discord integration and the job would pass.
  • ccache on Linux and macOS via hendrikmuhs/ccache-action. Only pushes to main save the cache; PRs restore main's latest one, so PR runs don't use up the repo's 10 GB cache quota. evict-old-files: job keeps each saved cache to what that build used. Windows isn't covered because ccache can't cache MSVC builds that use /Zi (PDB debug info); that would need /Z7.
  • Qt is cached (cache: true on install-qt-action) instead of being downloaded in every job.
  • tabs.yml runs on pushes to main only. It ran on pushes to every branch as well as on pull_request, so it ran twice for each PR push.

Expected effect

  • Ubuntu clang should drop to about 30 minutes and Ubuntu Default to about 20, even before ccache. The slowest job would then be Mac Intel, at about 35 minutes once libdpp is cached.
  • The caches start out empty: this PR's own run populates the Qt and libdpp caches, and the first push to main after merge populates ccache. The ccache savings show up from the following runs.

Testing

  • I tested the new clang-query script with GNU xargs on Linux, using a stub clang-query and a real compile_commands.json (1,545 entries, 1,419 unique files). Every file was checked exactly once, and a path containing a space worked. A match fails with the existing "Forbidden code" error, and a clang-query error fails with the new message after printing the output.
  • All workflow files parse.
  • CI passes on this PR (libdpp and Qt caches populated)
  • After merge: the second push to main shows ccache hits in the job summary and a cache-hit restore for libdpp

🤖 Generated with Claude Code

A push to main used about 240 runner-minutes, and the Ubuntu clang job
took 72-89 minutes end to end. Most of that was work repeated on every
run:

- clang-query ran in both Ubuntu jobs on the same Linux sources, one
  100-file batch at a time: 26-55 minutes per job. It now runs only in
  Ubuntu Default (the fastest Linux build), runs one batch per core,
  and lists each file once. A file compiled into several targets has
  one compile_commands.json entry per target, and clang-query already
  checks every entry of each file it's given, so passing a file twice
  made it parse each of those entries twice.
- The macOS jobs compiled libdpp from source with our pinned brew
  formula: 4 minutes on Apple Silicon, 13 on Intel. The installed keg is
  now cached, keyed on the formula and on the versions of the libraries
  it links against. The step now also fails if pkg-config can't find
  DPP 10.1.5, because CMake would otherwise build without Discord
  integration and pass.
- Linux and macOS builds now use ccache. Only pushes to main save the
  cache; PRs restore main's latest one. Windows is left out because
  ccache can't cache MSVC builds that write debug info to a PDB (/Zi).
- install-qt-action now caches Qt instead of downloading it every job.
- tabs.yml ran on pushes to every branch as well as on pull requests,
  so it ran twice for each PR push. It now runs on pushes to main only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin890 pushed a commit that referenced this pull request Oct 17, 2026
The two Windows jobs spend 19-27 minutes each compiling, and #1610 left
them out of ccache because three things blocked it:

- ccache can't wrap the compiler under the Visual Studio generator, so
  Windows CI now configures with "Ninja Multi-Config". It keeps Visual
  Studio's output layout (bin/RelWithDebInfo/), so the upload and test
  steps are unchanged. A new step runs vcvars64.bat from the same Visual
  Studio and passes its environment to the later steps. For the clang
  job it puts that Visual Studio's own clang-cl first on PATH, the one
  the ClangCL toolset used.
- ccache can't cache /Zi, which writes debug info to a shared PDB. CI
  now builds with /Z7 (CMP0141 and CMAKE_MSVC_DEBUG_INFORMATION_FORMAT
  =Embedded); the linker still writes the PDB that gets uploaded.
- ccache can't cache a compile that writes a /FAs assembly listing. The
  listings are now behind a PA_ASSEMBLY_LISTINGS option, ON by default
  so local builds keep them, and CI turns it off.

CMakeLists.txt and CoreLib.cmake picked the clang-cl flags (-march,
-Wno-unused-function) by checking for the ClangCL toolset, which is
empty under Ninja. They now check CMAKE_CXX_COMPILER_ID, which is
"Clang" for clang-cl under either generator.

Ninja runs nproc + 2 compiles at once on Windows instead of 10, close to
the one compile per core that Visual Studio's /MP ran, so memory use
stays near what the runner handled before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gin890
Gin890 merged commit 683ae02 into main Oct 18, 2026
7 checks passed
@Gin890
Gin890 deleted the ci-efficiency branch October 18, 2026 04:45
Gin890 pushed a commit that referenced this pull request Oct 18, 2026
The two Windows jobs spend 19-27 minutes each compiling, and #1610 left
them out of ccache because three things blocked it:

- ccache can't wrap the compiler under the Visual Studio generator, so
  Windows CI now configures with "Ninja Multi-Config". It keeps Visual
  Studio's output layout (bin/RelWithDebInfo/), so the upload and test
  steps are unchanged. A new step runs vcvars64.bat from the same Visual
  Studio and passes its environment to the later steps. For the clang
  job it puts that Visual Studio's own clang-cl first on PATH, the one
  the ClangCL toolset used.
- ccache can't cache /Zi, which writes debug info to a shared PDB. CI
  now builds with /Z7 (CMP0141 and CMAKE_MSVC_DEBUG_INFORMATION_FORMAT
  =Embedded); the linker still writes the PDB that gets uploaded.
- ccache can't cache a compile that writes a /FAs assembly listing. The
  listings are now behind a PA_ASSEMBLY_LISTINGS option, ON by default
  so local builds keep them, and CI turns it off.

CMakeLists.txt and CoreLib.cmake picked the clang-cl flags (-march,
-Wno-unused-function) by checking for the ClangCL toolset, which is
empty under Ninja. They now check CMAKE_CXX_COMPILER_ID, which is
"Clang" for clang-cl under either generator.

Ninja runs nproc + 2 compiles at once on Windows instead of 10, close to
the one compile per core that Visual Studio's /MP ran, so memory use
stays near what the runner handled before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin890 pushed a commit that referenced this pull request Oct 18, 2026
The two Windows jobs spend 19-27 minutes each compiling, and #1610 left
them out of ccache because three things blocked it:

- ccache can't wrap the compiler under the Visual Studio generator, so
  Windows CI now configures with "Ninja Multi-Config". It keeps Visual
  Studio's output layout (bin/RelWithDebInfo/), so the upload and test
  steps are unchanged. A new step runs vcvars64.bat from the same Visual
  Studio and passes its environment to the later steps. For the clang
  job it puts that Visual Studio's own clang-cl first on PATH, the one
  the ClangCL toolset used.
- ccache can't cache /Zi, which writes debug info to a shared PDB. CI
  now builds with /Z7 (CMP0141 and CMAKE_MSVC_DEBUG_INFORMATION_FORMAT
  =Embedded); the linker still writes the PDB that gets uploaded.
- ccache can't cache a compile that writes a /FAs assembly listing. The
  listings are now behind a PA_ASSEMBLY_LISTINGS option, ON by default
  so local builds keep them, and CI turns it off.

CMakeLists.txt and CoreLib.cmake picked the clang-cl flags (-march,
-Wno-unused-function) by checking for the ClangCL toolset, which is
empty under Ninja. They now check CMAKE_CXX_COMPILER_ID, which is
"Clang" for clang-cl under either generator.

Ninja runs nproc + 2 compiles at once on Windows instead of 10, close to
the one compile per core that Visual Studio's /MP ran, so memory use
stays near what the runner handled before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants