Skip to content

Fix CI ccache SIGILL by building Linux for x86-64-v3 - #272

Merged
zfergus merged 11 commits into
mainfrom
fix/ci-x86-64-v3
Oct 12, 2026
Merged

zfergus merged 11 commits into
mainfrom
fix/ci-x86-64-v3

Conversation

@zfergus

@zfergus zfergus commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Description

Builds the Linux CI jobs for x86-64-v3 instead of -march=native, so one ccache serves every runner CPU. Also fixes CI comments that named MeshFEMCore as a source of -march=native.

  • Pinned baseline: continuous.yml, cuda.yml, python.yml, and coverage.yml (Linux) pass -DSIMD_CXX_FLAGS=-march=x86-64-v3 (AVX2, FMA, BMI1/2, LZCNT, F16C). The flag string ccache hashes now fully describes the generated code.
  • One cache: the continuous, cuda, python, and coverage cache keys have no CPU component on any OS, so one cache per OS and config serves every runner CPU (for example <os>-<config>-cache, with a cuda- prefix for CUDA). The CPU fingerprint steps are gone, apart from a CPU log line in the Linux guard.
  • Linux wheels: pypi.yml passes CMAKE_ARGS=-DSIMD_CXX_FLAGS=-march=x86-64-v3 through CIBW_ENVIRONMENT_LINUX. Without it, wheels take -march=native from the runner that built them. The 1.6.0 Linux wheels contain no AVX-512 but already need AVX2, FMA, BMI and LZCNT. macOS and Windows wheels are unchanged.
  • Docs: docs.yml gets its ccache back (reverting Fix docs SIGILL by dropping ccache from the docs build #270's removal) with the same x86-64-v3 baseline through CMAKE_ARGS and the same guard. It uses a new key, docs-ccache, because entries are immutable and the old docs-cache holds -march=native objects.
  • Debug without -g: the Debug configures in continuous.yml (Linux and macOS) and cuda.yml pass -DCMAKE_C_FLAGS_DEBUG=-O0 -DCMAKE_CXX_FLAGS_DEBUG=-O0, so asserts stay on (NDEBUG undefined) but debug info is gone. In the first Fix CI ccache SIGILL by keying on the resolved ISA #269 run the Debug ccache was 0.41 GB against 0.02 GB for Release on Linux, and 0.53 GB against 0.14 GB on CUDA. The CUDA job empties CMAKE_CUDA_FLAGS_DEBUG instead, because nvcc would apply -O0 to device code. Windows is unchanged, since MSVC's Debug flags differ.
  • SIMD flag validation: when SIMD_CXX_FLAGS is already set, CMakeLists.txt skips find_package(SIMD) and only checks that the compiler accepts the flags, through the new ipc_toolkit_check_cxx_flags() in ipc_toolkit_filter_flags.cmake. Rejected flags are a FATAL_ERROR instead of a silent fallback to a non-SIMD build, so a typo can't pass as a working build. Auto-detection is unchanged when the flags are not set.
  • Guard: the Linux jobs fail with a clear error if the runner CPU lacks an x86-64-v3 feature, instead of a SIGILL later.
  • Coverage: coverage.yml gets the same baseline, guard, and plain coverage-cache key. Its stray trailing backslashes are gone.
  • Comments: the comments in all five workflows now name FindSIMD and Tight-Inclusion. MeshFEMCore's CMake never runs, since meshfem_sparse.cmake uses DOWNLOAD_ONLY.
  • Python on Linux: python.yml no longer sets IPCTK_WITH_SIMD=0 on Linux (added in Add env var to disable SIMD in setup.py #231 with no stated reason). It builds with SIMD and the same x86-64-v3 pin and guard, so the Linux Python tests now exercise the SIMD code through the bindings.

Impact:

  • The first run after merge is cold on every OS, because every key changes. After that the cache is shared across CPUs. Only 16 of 40 recent Linux jobs restored a cache, because keys were split across six CPU models.

  • No Linux CI job compiles the -march=native default or the AVX-512 xsimd path (batch width 8 instead of 4) any more. Users on AVX-512 hardware run code CI no longer builds.

  • No source or API changes.

  • Windows already passes explicit /arch:AVX2, so its objects do not depend on the runner.

  • macOS still builds with -march=native. It shares one cache because 27 of 29 recent macOS jobs ran on the same Apple M1. A cache seeded by an M2 Pro runner could in principle reach M1 runners.

Why

  • SIGILL after restoring a cache on EPYC 9V74 runners: Extremum Sum Potential #259, Fix CMake options being forced in the cache #268, and the docs build (Fix docs SIGILL by dropping ccache from the docs build #270).
  • ccache hashes -march=native literally. On an Ice Lake runner it expands to -march=icelake-server -mavx512f …, and the cached objects are only safe on a runner with the same ISA.
  • Tight-Inclusion's own flags are -march=native -mavx2 -mno-avx512f …, which caps its objects at AVX2. I left them as they are. A switch there would make them strictly deterministic.
  • With -DSIMD_CXX_FLAGS=-march=x86-64-v3, all 94 ipc_toolkit translation units compile with it and none with -march=native.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (non-breaking change which improves existing functionality)

How Has This Been Tested?

  • All five workflows parse as YAML, and every non-Windows run block passes bash -n.
  • The guard was simulated against fake /proc/cpuinfo files: it passes on a v3-capable CPU and fails with a clear message when bmi2, fma, or avx2 is missing.
  • Configured locally with -DSIMD_CXX_FLAGS=-march=x86-64-v3 and checked compile_commands.json (see above). Configure only, no build.
  • Not tested: the wheel build. pypi.yml only runs on workflow_dispatch, and I haven't checked that cibuildwheel passes CMAKE_ARGS into the manylinux container.
  • Not tested yet: the Linux jobs themselves, including the nvcc host pass accepting -march=x86-64-v3 through -Xcompiler. CI on this PR is the test, since each workflow triggers on its own file. A re-run after the first shows the cross-CPU cache hits.

Test Configuration:

  • OS and Version: macOS 26.6.2
  • Compiler and Version: Apple clang 21.0.0 (configure only)

Checklist

  • I have followed the project style guide
  • My code follows the clang-format style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

zfergus and others added 2 commits October 12, 2026 16:44
- Pass -DSIMD_CXX_FLAGS=-march=x86-64-v3 in the Linux continuous and CUDA
  jobs so ccache's flag hash describes the generated code
- Key those caches on x86-64-v3 instead of the CPU, so one cache serves
  every runner
- Fail early if a runner lacks an x86-64-v3 feature
- Keep coverage on -march=native so CI still compiles the default path

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
zfergus and others added 9 commits October 12, 2026 16:54
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- Skip find_package(SIMD) when SIMD_CXX_FLAGS is already set
- Fail with an error if the compiler rejects the flags, instead of
  silently falling back to a build without SIMD
- Add ipc_toolkit_check_cxx_flags() to test a flag list as a whole

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.92%. Comparing base (101332c) to head (93b414e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #272      +/-   ##
==========================================
- Coverage   96.92%   96.92%   -0.01%     
==========================================
  Files         194      194              
  Lines       17348    17349       +1     
  Branches      944      943       -1     
==========================================
  Hits        16815    16815              
- Misses        533      534       +1     
Flag Coverage Δ
unittests 96.92% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@zfergus
zfergus merged commit 8a973ff into main Oct 12, 2026
14 checks passed
@zfergus
zfergus deleted the fix/ci-x86-64-v3 branch October 12, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant