[starkphi]

Description

A persistent build-cache folder is keyed on the program, the fingerprint of the generated source,
BUILD_CACHE_VERSION_ID (default: gt4py.__version__) and the build context (non-default DaCe
config). The toolchain is not part of the key: after a change of compiler, CUDA, DaCe, GridTools
or Python, a shared persistent cache silently reuses binaries built by the old toolchain.

This PR records the toolchain in the cache and checks it:

• config.py: BuildCacheToolchainCheck (ERROR, WARN, OFF) and BUILD_CACHE_TOOLCHAIN_CHECK,
  set via GT4PY_BUILD_CACHE_TOOLCHAIN_CHECK=error|warn|off (default error).
• otf/compilation/cache.py: current_toolchain(), check_build_cache_toolchain(base_path)
  (cached per process), BuildCacheToolchainError, TOOLCHAIN_FILE_NAME = "toolchain.json".
  get_cache_folder calls the check for the PERSISTENT lifetime only; session caches are unchanged.
  • Recorded: machine, python, versions of dace, gridtools-cpp, nanobind (if installed),
    cxx (first line of $CXX --version, default c++), cuda_cxx (the "release" line of
    $CUDACXX --version, default nvcc, only if found), cuda_archs ($CUDAARCHS, if set).
  • First use writes <cache>/toolchain.json under a file lock (gt4py._core.locking). Later uses
    compare only keys present in both the record and the current toolchain; keys missing from the
    record are added, so CPU and GPU builds can share one cache.
  • On a difference: error raises BuildCacheToolchainError listing old -> new values and the
    remedies (delete the cache, use another GT4PY_BUILD_CACHE_DIR, or set warn); warn warns once
    per process; off neither records nor checks.
• 8 new backend-free unit tests in test_cache.py.

Why only in get_cache_folder: every compile path goes through it (generic compiler.py, the
CMake and compiledb build systems, DaCe workflow/compilation.py), each taking a per-program lock
right after. The translation-cache path is resolved when a backend object is built, possibly at
import time, so the check is deliberately not in get_cache_base_path.

Cost (measured on a GH200 node, aarch64): 5-7 ms once per process (two --version subprocesses,
three metadata lookups, lock + JSON), ~140 ns for later calls. It never runs on calls of an already
loaded program.

Tested: locally (x86_64, CPU only): test_cache.py 17 passed (9 existing + 8 new);
nox -s "test_next-3.12(internal, cpu, nomesh)" 4881 passed, 119 skipped, 606 xfailed (+ 111
passed, 2 skipped); nox -s "test_next-3.12(dace, cpu, nomesh)" 1495 passed, 25 skipped, 64 xfailed
(+ 111 passed, 2 skipped); GPU and atlas sessions not run; pre-commit clean. Also an end-to-end application run with a fresh persistent cache
on GH200: toolchain.json written, no message, results bit-identical to the reference run. A mismatch
has so far only been exercised by the unit tests.

Open questions for reviewers

• Default error or warn? Existing caches are not broken (a cache without a record just gets
  one), but after a toolchain change users get an error instead of today's silent reuse.
• What to compare? Not covered: HIP/ROCm (HIPCXX, HIPARCHS), cmake/ninja versions, CuPy,
  compiler flags from the environment, and the device architecture when CUDAARCHS is unset (I
  avoided querying the device here). Should this reuse otf/compilation/common.py:get_device_arch()?
• ADR? Happy to write one if you think this qualifies as an architectural choice.
• Alternative considered: folding a toolchain hash into BUILD_CACHE_VERSION_ID. That avoids
  stale reuse automatically, but recompiles silently and leaves old folders behind; I preferred an
  explicit stop that can be downgraded to a warning.
• toolchain.json and its lock file sit in the cache root next to the program folders; tools that
  list the cache (e.g. users of CACHE_FOLDER_NAME_PATTERN) must ignore non-matching entries.

Requirements

• All fixes and/or new features come with corresponding tests.
• Important design decisions have been documented in the appropriate ADR inside the docs/development/ADRs/ folder. (see open question above)

If this PR contains code authored by new contributors please make sure:

• The PR contains an updated version of the AUTHORS.md file adding the names of all the new contributors.

🤖 Generated with Claude Code

[tehrengruber]

In my view the toolchain configuration is a property of the backend not a global one (for gpu the gpu compiler version matters, for cpu only the regular one). The backend instance contributes to the fingerprint of a program / field operator so adding the info there would result in a different path in the cache directory. Not a direct replacement for the proposal here as it would silently rebuild on a toolchain change, but a production run should be done with jit=False anyway. Alternatively one could store the backend fingerprint / hash per program / field operator and then provide a similar warning mechanism as done in this PR.

[havogt]

Can you explain your use-case? It's hard/impossible to get this right, but maybe a partial solution is good enough for your use-case.
Why it's hard? It's not only the compiler version that could influence compilation or code generation. E.g. you can tweak behavior by environment variables. But you can't make all environment variables part of the hash (because you would recompile way too often). Next best thing would be to try to list all environment variables that could influence behavior, but if you miss one, it's again weird. Therefore, our current strategy is: if you switch environment and want a guaranteed recompile, you need to use a different cache directory. But we are open for discussion...