v1 fetch integration: Go (#396) + Rust (#397) fetch lanes + strict test CA (#421) - #422
Merged
Merged
Conversation
Implements go/internal/ocifetch, the Go half of the v1 OCI fetch layer against the frozen seam (lane 0A's constants_gen.go, docs/guides/fetch-v1.md): - http.go: retries with one shared table, Retry-After in both forms (and refused once it exceeds the retry schedule's own total budget), redirects with Authorization dropped cross-origin, the anonymous Bearer-token flow for mirrors, and the static CHTYPES_DOWNLOAD_TOKEN sent only to configured hosts. - oci.go: resolve an image index by tag, select the platform manifest, verify it by digest, and the tag/digest 404 policies (next-base-then- unpublished vs next-base-then-retry-on-last). - referrers.go / dsse.go: referrers-API-or-fallback-tag discovery, DSSE's pre-authentication encoding and ed25519 verification against the trusted key list, and the in-toto statement checks (abi, os, arch, version-within- request, subject-equals-layer). - unpack.go: zstd decode with the window-log cap honored before the decoder allocates for it, and tar extraction refusing anything but regular files and directories, with the unpacked-bytes cap enforced as it is written. - layout.go: the OCI image layout cache (temp-then-rename blobs, an index.json read-check-rename loop that survives a concurrent writer) and each unpacked directory's own verified.json record as the source of truth for offline reads. - lock.go: schema 3, refusing a v0 (schema 1/2) or wrong-ABI lock outright. - ensure.go: the seam's four entry points (Ensure, ResolveInstalled, ListInstalled, VerifyInstalled, FetchSigned) and --frozen/--offline/update. - errors.go: the ten shared v1 codes over the generated exit-code table. - conformance_test.go: TestConformanceV1, which skips loudly without CHTYPES_V1_CONFORMANCE; the fixtures and scripted server it depends on have not landed from the fixtures lane yet, so it is written against the documented case/report shape and will need a pass once they exist. go.mod/go.sum add github.com/klauspost/compress, pinned, for the zstd decoder (the zstd encoder used by fixture generation is a separate module). 64 unit tests exercise every piece directly, including a full fake-registry integration test of the happy path, an index.json concurrent-writer race, every tar refusal rule, and the retry/Retry-After/redirect/token behavior over real HTTP round trips — all green, including under -race. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements the Rust binding's v1 fetch layer behind a new, off-by-default `fetch-v1` feature: HTTP with retries/redirects/the anonymous Bearer-token flow, OCI resolve (image index + manifest, the version-spelling rules), referrer-based trust (DSSE + ed25519 against the pinned release key, with a duplicate-key JSON check), verify-then-decompress zstd (ruzstd, looping over frames, enforcing the window-log cap) and tar unpack (regular files and directories only), an OCI-layout cache with atomic installs, and a schema-3 lock file. The seam (`ensure`, `resolve_installed`, `list_installed`, `verify_installed`, `fetch_signed`) matches the plan's §1.3 contract. MSRV rises from 1.85 to 1.87 (ruling): ruzstd 0.9.0, the v1 zstd decoder, declares `rust-version = "1.87"`. `rust/Cargo.toml` and the v1-conformance matrix's rust leg are updated to match; noted in `rust/CHANGELOG.md`. Referrer discovery also covers an edge case measured on the staging registry: a 200 response with no matching-artifactType referrer can be edge-cache staleness rather than genuine absence, so a zero-candidate result from the referrers API always also tries the fallback tag before a trust decision is final. `rust/src/ocifetch` is wired into `lib.rs` as a single private `mod` declaration (one line), per the lane's scope: nothing else in this crate calls it yet, pending the FFI/switch lanes. `tests/ocifetch_conformance.rs` reaches it by recompiling the module tree under its own crate root (`#[path]`), the only way an integration test can reach a private `mod`; it reads `CHTYPES_V1_CONFORMANCE` and skips loudly when unset. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Every manifest GET (by tag, by digest, and the referrers fallback tag) now sends exactly Accept: image.index, image.manifest, per the delivery side's review — dropped the bundle media type, which never belongs in a manifest Accept header. - Reworded the referrers-fallback doc comment: the fallback is for mirrors that may not proxy the referrers API faithfully, not edge-cache staleness on our own host (corrected per the delivery side). - tests/ocifetch_conformance.rs: build the lane-0B server script's path from components rather than one string literal, so lint-cited-paths.sh does not read the forward reference (0B has not merged yet) as a dead repository path. - rust/src/ocifetch/constants.rs: cargo fmt expanded the struct-literal arrays rustfmt's own run had already touched; left in the rustfmt- canonical form per this lane's brief. This is the known conflict between rustfmt and scripts/fetch-v1/gen-constants.py's own emitted style (not this lane's file to fix) — see MERGE NOTES. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two corrections from the delivery side's review: - Every GET …/manifests/<ref>, by tag and by digest, now sends Accept: application/vnd.oci.image.index.v1+json, application/vnd.oci.image.manifest.v1+json. Our host ignores it; a mirror may need it for content negotiation. Covers resolveIndex, fetchManifestByDigest, the referrers fallback tag's own manifests/ GET, fetchReferrerContent, FetchSigned and the --frozen path — every one of them, not only the tag-based resolve. - referrers.go's comment no longer frames the fallback-tag fallback as working around a cache lag on our own host (that endpoint is served no-store): it is for mirrors, which may serve only the fallback tag and never the referrers API at all. The fallback logic itself is unchanged. Also removes five dead citations of scripts/fetch-v1/server.py (lane 0B's scripted server, not yet on this branch) that lint-cited-paths.sh correctly flagged once the new test files were tracked — CI caught this on the first push; fixed and re-verified clean (lint-cited-paths, lint-public, lint-spelling, policy-merge-check --check-carve-out, check-selftests-wired.py, gen-constants.py --check). 67 tests pass (two new ones assert the Accept header is sent/absent as configured); go vet, golangci-lint and gofmt all clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by re-reading ensure.go end to end: resolveOptions already parsed Options.HookBeforeIndexRename into resolvedOptions.hookBeforeIndexRename, but both real addIndexEntry call sites in ensure.go hardcoded nil instead of using it, so the option had no effect on an actual Ensure call — only layout_test.go's own direct addIndexEntry test exercised the race-survival logic itself. session now carries hookBeforeIndexRename (set from resolvedOptions in newSession) and both installManifest and the already-installed shortcut pass it through. A new integration test, TestEnsureInvokesHookBeforeIndexRename, proves the hook actually fires during a real Ensure call — the gap existing tests could not have caught, since they called addIndexEntry directly rather than through the public entry point the conformance runner will use for the index-race-reapply case. 68 tests pass; go vet, golangci-lint and gofmt all clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
VerifiedStatement now carries the digests of the referrer manifest and layer that held the verified bundle (set by referrers.rs, which is what fetched them — dsse.rs only ever sees the layer's bytes). ensure.rs no longer discards them: Digests.bundle (the lock's "bundle" entry, re-fetched and re-verified under --frozen) and the new Digests.bundle_manifest field are both populated from the real verified statement instead of being left None/empty, in every path that builds a Resolved. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scripts/support-matrix.sh reads rust/Cargo.toml's rust-version to populate docs/support.md's Rust row; this PR's 1.85 -> 1.87 bump left the committed file stale, failing the required docs check. Mechanical regeneration, no other row changed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ed predicate
A second, independent review of ensure.rs found two real bugs:
- find_installed and monotonic_warning compared ClickHouse versions as
plain strings. Components are not zero-padded, so "26.10.1.5" sorted
BELOW "26.9.3.38" ('1' < '9' lexically) despite being the newer version
-- --offline/resolve_installed could pick an older build across any
digit-width boundary (9->10, 99->100), which is routine for these
version strings. Now compared as [u64; 4] (version_key). Build strings
stay a plain string compare -- they are fixed-width per spec, so that
was already correct.
- A verified (non-null) predicate missing library_sha256/library/
library_bytes silently skipped the library hash/size check entirely
(library_sha256.as_str().unwrap_or_default() -> "" -> the `if
!is_empty()` guard never ran) instead of refusing a malformed-but-signed
artifact. library_fields() now requires all three and fails
ArtifactCorrupt if any is absent; ensure_frozen's install path, which
never checked the library at all, now does (verify_unpacked_library,
shared with ensure_online).
Also: the "already installed" short-circuit (both ensure_online and
ensure_frozen) checked only that verified.json exists, not that the
library file it names is actually on disk -- a record surviving after the
library was removed or corrupted out of band would report success with a
library_path that does not resolve, breaking ensure()'s own "the library
file exists on disk" guarantee on the cache-hit path. is_fully_installed()
now checks both.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… MSRV bump itself broke integration.rs v1-constants vs rustfmt: scripts/fetch-v1/gen-constants.py (lane 0A's file, untouched here) and rustfmt wanted different formatting of constants.rs's struct-literal arrays. Fixed in-lane instead, the standard approach for generated code: #[rustfmt::skip] on `pub mod constants;` in mod.rs, then constants.rs restored to the generator's exact output (gen-constants.py --write). rustfmt follows the mod declaration, so this also covers tests/ocifetch_conformance.rs's #[path] recompilation of the same file. Confirmed: `cargo fmt --check` and `gen-constants.py --check` both pass now. tests/integration.rs:1479, clippy::manual_is_multiple_of: this is NOT unrelated toolchain drift as I'd reported -- it's a direct consequence of this PR's own rust-version bump. Measured: the lint is MSRV-aware (checked by toggling a scratch crate's rust-version between 1.85, where it is silent, and 1.87, where it fires as a hard error under -D warnings), and `is_multiple_of` is the stable method this bump newly permits suggesting. #394 passed at the old MSRV; this PR's 1.85 -> 1.87 is what exposed it here. Fixed at the call site (`n % 2 == 0` -> `n.is_multiple_of(2)`), valid at 1.87. `cargo clippy --all-targets -- -D warnings` is clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/v1.yml
mirror-failover origin alternation, the generic-fetch convention, and requests.max/none_matching checking Merges lane 0B's fixtures/server/parity/v1-conformance-job-body PR (#401) and the FFI lane's ABI description work (#395, abi-v1/, unrelated to this lane) into this branch. ensure.rs / referrers.rs: - A pre-seeded index.json entry (a layout with no corresponding verified.json yet) is now actually installable offline, from local blobs only, per docs/guides/fetch-v1.md §1 and §10's "Cache fixtures and installed.json": find_index_candidate picks the best candidate by platform and by the index's own org.opencontainers.image.ref.name annotation, install_from_local_blobs verifies and unpacks it without any network call (find_installed and the offline+lock path in resolve_offline both use it), and referrers::find_local_referrer reconstructs the signature referrer by scanning blobs/sha256/ for a manifest whose own `subject` points back at the target digest -- a plain OCI layout's index.json lists only the artifact manifest itself, never its referrers. rust/tests/ocifetch_conformance.rs: - Fixed the lock fixture path to locks/inputs/<name>.json (it was locks/<name>.json, guessed before lane 0B's real layout existed). - Implemented the installed.json pre-seed convention: before a case with one runs, each listed digest is installed from the staged cache's own local blobs via the same install_from_local_blobs. - `{base}` occurrences in request.bases now alternate between the server's two origins for http transport (first -> primary, second -> the second origin server.py already starts for redirect cases). This is what makes mirror-failover-5xx/mirror-failover-digest-404 solvable at all: both entries are the textually identical "{base}" template, and the second origin's own http-script routes are empty for those case ids, so it falls straight through to the real tree once the first origin's retry budget is exhausted. Confirmed against retry-exhausted, which uses the IDENTICAL 503x5 script on a single base and correctly fails -- the only difference for the mirror cases is the second, distinct origin. - The generic-fetch convention (a goldens-/fixtures- case id calls fetch_signed instead of ensure(); its library_sha256 checks against the fetched content's own hash, there being no "library" file for this content). - requests.max/none_matching are now actually checked, by reading each case's request log from the server's own GET /_log/s-<id> endpoint (http transport only -- there is no request log for a file:// read). A small, specific pattern matcher covers the fixture set's actual vocabulary (a GET .*<suffix> wildcard, a bare substring, and the one negative- lookahead pattern fixtures-digest-pin-no-tag-fallback needs) rather than a general regex engine, which this crate has no dependency on. Still not implemented, reported rather than faked (see hand-back MERGE NOTES): before_index_rename_hook, the lock_after fixture comparison, and writing a lock for every platform the index offers (lock-write-all-platforms). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…arison ensure(): a request with update:true but lock_write:false never rewrote the lock at all (the write was gated on lock_write alone). Plan §6: "update re-resolves every locked request against the current index and rewrites the lock" -- update's whole point is to refresh a stale entry, so it must write regardless of lock_write. Fixed: the write now happens on lock_write || update (update-re-resolves). The conformance runner's expect.lock_after was an unconditional reported gap; it now actually compares the lock file the case just wrote against locks/expected/<name>.json (parsed as JSON before comparing, so key order never matters). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-all-platforms) Plan §6: "Writing a lock for every platform the index offers... downloads and verifies every platform's bundle but fetches the layer only for the host's own platform." ensure()'s lock_write path previously recorded only the single requested platform. lock_entries_for_all_platforms re-fetches the index (one extra request, acceptable -- lock_write is not a hot path), reuses the already-resolved host platform's data as-is (no extra fetch), and for every other platform fetches its manifest by digest and verifies its signature referrer without ever fetching its layer blob, matching lock-write-all-platforms' own assertion (zero layer GETs for the two non-host platform digests). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eep in real time v1-conformance (rust 1.87) and (rust stable) were cancelled after the full 15-minute job timeout, with no report artifact uploaded at all -- the test process was still running, not merely slow. Root cause: the runner always passed Options.clock = None, so every retry/backoff case (ensure.rs's http client has no idea it is under test) slept for REAL seconds via std::thread::sleep. Summed across the suite's several dozen retry-shaped cases (retry-5xx, retry-408, retry-429-*, retry-after-*, retry-exhausted at 4+8+16+32=60s alone, both mirror-failover cases at another 60s each, plus cold-compile time for ring/rustls/ruzstd with no cargo cache in this job), real wall-clock time plausibly exceeded 15 minutes outright. Go/Python/TS's suites have no such per-attempt real-sleep cost and passed comfortably in the same run. The seam was built for exactly this (plan §1.1.1: "every binding's fetch core takes an injected sleep(seconds) and now(), so retry cases run instantly") but the runner never actually injected one. FakeClock now does: sleep() only records the duration into a shared Vec, which the runner then checks exactly against expect.sleeps (not implemented before this commit either) instead of a real thread::sleep. Wired into both ensure() and fetch_signed() -- the latter was separately hardcoding RealClock regardless of Options.clock, now fixed to honor it the same way resources() already did. One case, stall-timeout-retried, still costs one real ~60s wait: the server genuinely holds the connection open, and detecting that is ureq's own OS-level read-timeout, not a sleep() call this Clock can intercept. Acceptable on its own; removing the other several hundred seconds of backoff sleeps should bring the suite comfortably under the job's 15-minute budget. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- conformance_test.go: wire Options.SystemDirs from the fixture's own
system-dir copies, instead of leaving it unset (which silently fell
back to the real machine's SystemCacheDirs). This was the cause of
system-dir-readonly's only failure.
- ensure.go: ensureOnline's monotonic-build check now runs BEFORE the
layer is fetched/installed, using the signed predicate's own
version/build, and keeps the existing (newer) installed entry rather
than installing the registry's older one and only warning about it
after the fact. Matches monotonic-warning's documented intent ("the
existing (newer) install is kept, with a warning").
Also carried over from the previous round's review feedback: the Accept
header on every manifest GET, the referrers-empty-then-fallback-tag
trust rule, and the referrers.go comment reword — already in this
branch before this commit.
14 (case, transport) pairs still fail; all traced to fixture/script
defects outside this lane's scope (genfixtures/*, server.py) and
reported separately rather than fixed here, per brief.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Lane 0B's round-3 fixes (merged #407) moved mirror-failover-5xx/ -digest-404 onto two genuinely independent bases ({base}, {base2}) served from server.py's two origins, each with its own response- sequence cursor. expandBase now also substitutes {base2} against the server's second port for http transport (docs/guides/fetch-v1.md's "{base2}" section). With #407's other fixes (real-version tags for existing-install-noop/ index-race-reapply, predicate-wrong-version's numeric spelling, an actually-oversize zstd window, preseed-oras dropped while its layout is absent, corrected redirect templates), the full local conformance run is now 139/139 passing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
…ssed 139/139 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
…notonic keep, 401/403 mapping, refused-connection skips retry, index.json race loop, registry transport Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
…'s config blob and reports itself unsigned Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
…he patch release Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
… the sandbox (25 passed, 0 failed) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
…t MSRV or stable), not the compiler version Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
…port from v1-network, which does not run a suite yet Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
… v1-network) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
The CA had no keyUsage, which Python's default (VERIFY_X509_STRICT) context refuses. Generate CA and server certificates with the proper extensions, and verify with openssl -x509_strict plus a Python default-context handshake. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012NdkF6p8Q3qkdxKCgbdjgb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One integration PR for three green v1 PRs, merged with
--no-ff(no rebase), so they share one CI run. Merged one at a time, each would leave the other two behind under strict protection, forcing a re-sync and a fresh CI run per merge.v1-go980bc9f), enrolled:v1-conformance139/139;v1-networkregistry 5/5 through the system CA store;v1-paritygreen with Go enrolled. Pre-cleared (binding code, its tests, marker, klauspost/compress).v1-rust98d22c9), enrolled:ureq/platform-verifier.docs/support.mdwas regenerated by the generator, not by hand, after TS v1 fetch layer, TypeScript lane: resolve, trust, bytes, cache and lock #400 merged.v1-network(v1-network-strict-cae3bad30): RFC 5280 CA and server certificate, plus fail-earlyopenssl verify -x509_strictand Python default-context handshake gates. Cleared.No merge conflicts. Gates run on this branch, each rc=0:
support-matrix.sh --checklint-actions,lint-public,lint-cited-pathscheck-selftests-wiredgen-constants --checkpolicy-merge-check --check-carve-out🤖 Generated with Claude Code