Repository navigation
Conversation
…ay library Persistent translation-cache keys fingerprint the connectivity tables in the offset provider. A process-pool compile worker only receives a host (numpy) copy of each table, while serial and thread compilation key the original, so on GPU backends the two keys differed and the cache written in one mode missed in the other. Gather connectivities now get their own deconstructor: domain, codomain, skip value and a content fingerprint of the host table, without the connectivity class or array library. The table fingerprint is memoized per connectivity, so a device table is copied to the host once per process.
…ory order A host copy of a connectivity table can come back in a different memory order than the original (C vs. F), which split the persistent cache key again for F-ordered tables. Translation does not depend on table strides, so the table content is fingerprinted in C order.
Compilation receives a host snapshot of every gather connectivity: a NumPy connectivity over a C-contiguous copy of the table, memoized per source connectivity. The parent process, compile threads and process-pool workers (which rehydrate the shipped table directly from the memory map) then see the same objects, independent of the device and memory layout of the runtime tables, and compile-time domain inference no longer reads device memory. Runtime calls still receive the original offset provider.
Compilation now runs against a CompileTimeConnectivity for every gather connectivity: a read-only, C-contiguous host table memory-mapped from a file in the session cache directory named by the digest of its content. It is fingerprinted by the digest instead of the table and pickled as a reference to its file, so - the table is copied from the device and hashed once per runtime connectivity, not on every fingerprint or task, - the host copy lives in the page cache instead of process memory and is shared with the process-pool workers, which no longer re-hash it, - shipping no longer needs separate file references, and rehydration keeps the memory map instead of copying it into a newly allocated buffer. This supersedes the content fingerprint of runtime connectivities from the previous commits, which is reverted. ADR 0024 gets an amendment.
Contributor
Author
|
🤖 Written by an agent on behalf of @havogt |
…ce orchestration - make_compilation_task converts the offset provider to its compile-time view, and CompilationTask carries the compilable itself: the with_refs variant is gone, so every runner and every caller compiles the same input. - The table digest is xxh64 over dtype, shape and the C-contiguous buffer, without copies and independent of numpy's pickling internals. - CompileTimeConnectivity derives its digest from the table when it is not given, so copies made through its constructor stay valid, and writes its file on pickling if needed. A failed table write leaves no file behind. - The dace orchestration path compiles against the same view. - Stale comments and the ADR amendment are updated.
- A worker unpickles the compilable inside the task, so an unreadable connectivity table file fails that task instead of breaking the pool. - The digest of a compile-time connectivity is always derived from its table (xxh3_128); only unpickling trusts the content-addressed file name, and checks shape and dtype of the mapped table. The class is private. - A failed rename of a table file no longer leaves the temporary file. - Tests with independent witnesses: identical gtfn translation against the runtime tables, a persistent cache filled in process mode hitting in the other modes, and the table content seen by every runner. - ADR 0024 and ADR 0019 notes updated.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Compilation received the runtime offset provider, so it depended on where and how connectivity tables are stored:
GT4PY_BUILD_JOBS_MODE=process, workers get a host copy of each table, while serial and thread compilation use the device original. On GPU backends the compiled input and its persistent translation-cache key therefore differed between modes. In an icon4py warm start, thread mode on a cache filled by process mode re-translated about 90 of 110 programs._max_domain_range_sizes,_unstructured_translate_range_statically) reduced device arrays, which means implicit syncs and cupy 0-d results (see fix[next]: format domain warnings for array scalars #2924).Compilation now runs against a compile-time view of the offset provider.
make_compilation_taskreplaces each gather connectivity, once per runtime connectivity, by a compile-time connectivity: a read-only, C-contiguous host table, memory-mapped from a file in the session cache directory named by a digest of its content (dtype, shape and data, hashed in place). It is fingerprinted by the digest instead of the table and pickled as a reference to its file, which replaces the previous file references and thewith_refsvariant ofCompilationTask. Workers unpickle the compilable inside the task, so an unreadable table file fails only that task instead of breaking the pool. All runners therefore compile the same input with the same key, and the host tables live in the page cache, shared with the workers. The dace orchestration path uses the same view; compilation that bypasses both (e.g. a directBackend.compile) still sees the runtime tables. Execution always receives the runtime offset provider.ADR 0024 gets an amendment and ADR 0019 a note. The amendment also states the new contract: tables must not be modified in place after they have been used for compilation.
Translation-cache keys of programs with connectivities change on all backends, so they are re-translated once.
Supersedes #2951. #2952 is not addressed here.
Requirements