Skip to content

Fix CI ccache SIGILL by keying on the resolved ISA - #269

Merged
zfergus merged 1 commit into
mainfrom
fix/ccache-isa-key
Oct 12, 2026
Merged

zfergus merged 1 commit into
mainfrom
fix/ccache-isa-key

Conversation

@zfergus

@zfergus zfergus commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Description

Keys the Linux CI ccaches on the flags -march=native expands to, not only on the CPU model name, in four of the five workflows that use ccache (continuous, cuda, python, coverage). docs is handled in #270.

  • ISA fingerprint: hashes c++ -march=native -E -v -'s resolved -m… flags together with the CPU model name, so the key is never weaker than before.
  • Diagnostic: logs ISA: … next to CPU: …, so a future failure shows what the runner exposed.
  • CUDA key prefix: cuda.yml and continuous.yml both used Linux-<config>-<cpu>-cache, so the CUDA and plain builds fought over one immutable entry. CUDA now uses cuda-….

Impact:

  • The first run after merge is cold, since every key changes. Old entries age out.
  • macOS and Windows fingerprints are unchanged. No failures there.
  • No source or API changes.

Why

  • Illegal instruction during catch_discover_tests on EPYC 9V74 runners that restored a large Debug cache: Fix CMake options being forced in the cache #268 (CUDA (Debug)) and Extremum Sum Potential #259 (Linux (Debug)).
  • The two jobs restored different entries under the same Linux-Debug-1130b74ab145-cache key: one from main (507 MB), one from refs/pull/259/merge (452 MB).
  • Among ~40 recent jobs that logged a CPU, only these two hit it. Cold builds on 9V74 (7 of 7) and cache hits on EPYC 7763 and Xeon (10 of 10) passed. The 9V74 Release hits also passed, but that entry is 17 MB, so almost everything was recompiled.
  • ccache hashes the literal -march=native, not what it expands to. FindSIMD (SIMD_CXX_FLAGS) and Tight-Inclusion (its own target, no off switch) add it. Turning off IPC_TOOLKIT_WITH_SIMD would drop the first but not the second.
  • The cause is not confirmed: no existing log prints the resolved ISA, and FindSIMD never probes AVX-512. The logging added here should settle it.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • The four workflows parse as YAML and each edited shell step passes bash -n.
  • The ISA pipeline runs and is stable across invocations with Apple clang and Homebrew g++-15 on arm64 macOS. Its output on x86 gcc under ubuntu-latest is untested locally. The CI on this PR exercises it, since each workflow triggers on its own file.

Test Configuration:

  • OS and Version: macOS 26.6.2
  • Compiler and Version: Apple clang 21.0.0, GCC 15

Checklist

  • I have followed the project style guide
  • My code follows the clang-format style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

- Hash the flags `c++ -march=native` expands to, alongside the CPU model
  name, in the Linux ccache keys of all four workflows
- Log the resolved ISA so a future failure shows which flags the runner had
- Prefix the CUDA cache key so it no longer shares an entry with the Build
  workflow

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.92%. Comparing base (61ff288) to head (9911399).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #269   +/-   ##
=======================================
  Coverage   96.92%   96.92%           
=======================================
  Files         194      194           
  Lines       17349    17349           
  Branches      943      943           
=======================================
  Hits        16816    16816           
  Misses        533      533           
Flag Coverage Δ
unittests 96.92% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@zfergus
zfergus merged commit 21ccd26 into main Oct 12, 2026
22 checks passed
@zfergus
zfergus deleted the fix/ccache-isa-key branch October 12, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant