v1 fetch layer, Go lane: resolve, trust, bytes, cache, lock

[EricAndrechek]

Summary

The Go lane of the v1 fetch layer: go/internal/ocifetch, built against the frozen seam in docs/guides/fetch-v1.md and lane 0A's generated constants_gen.go. It implements resolve, trust, bytes/unpack, the cache, the lock, and the seam's four entry points, with no edits to go/chtypes/* or the generated contract files.

• http.go — one retry table (constants-driven), Retry-After in both the delta-seconds and HTTP-date forms, redirects (capped, Authorization dropped on any cross-origin hop), the anonymous Bearer-token flow for mirrors, and the static CHTYPES_DOWNLOAD_TOKEN sent only to configured hosts. An injectable Clock makes every retry/backoff path deterministic in tests.
• oci.go — spelling validation before any network call, resolving an image index by tag, selecting and verifying the platform manifest by digest, and the tag/digest 404 policies from the constants (next_base_then_unpublished vs next_base_then_retry_on_last). Base-URL construction is transport-aware: http(s) inserts /v2/ right after the authority and carries the rest of the base's path through as the repository (so a test server's routing prefix and the real repository path both pass through unchanged); file:// is used exactly as given, with no insertion, per the plan's "a file base names the repository path."
• referrers.go / dsse.go — referrers-API-or-fallback-tag discovery (filtered by artifactType, accepting the manifest once any candidate verifies), DSSE's pre-authentication encoding, ed25519 verification against the trusted key list (never the bundle's own untrusted publicKey.hint), and the in-toto statement checks (abi — never a stray abi_revision — os, arch, version-within-request, subject-equals-layer, duplicate-JSON-key refusal).
• unpack.go — zstd decode with the window-log cap honored before the decoder allocates for it, multi-frame streams accepted, and tar extraction refusing anything but regular files and directories (symlink, hardlink, device, absolute path, .., duplicate entry all refused), with the unpacked-bytes cap enforced as the stream is written, not after.
• layout.go — the OCI image layout cache: content-addressed blobs by temp-then-rename, an index.json read-check-rename loop that re-merges onto a concurrent writer's result rather than clobbering it, and each unpacked directory's own verified.json record as the durable source of truth for offline reads (never index.json).
• lock.go — schema 3, refusing a v0 (schema 1/2) or wrong-ABI-generation lock outright, naming re-lock.
• ensure.go — the seam: Ensure, ResolveInstalled, ListInstalled, VerifyInstalled, FetchSigned, plus --frozen (by-digest only, no index lookup), --offline, and update (full lock rewrite from re-resolution, never a merge of stale and fresh entries).
• errors.go — the ten shared v1 codes over the generated exit-code table; CHTYPES_ARTIFACT_INCOMPATIBLE is reserved for the FFI/loader layer and this package never constructs one.
• conformance_test.go — TestConformanceV1, the command v1.yml's conformance job runs. It skips loudly when CHTYPES_V1_CONFORMANCE is unset.

Spec commit used: the delivery side's layout-v2 design doc at commit aa1980f521c791a296d31b68d30cafb4ff6ff2bb. No disagreement found between its §0/§4.1/§7 and what this PR implements.

A mid-task correction folded in: on a staging registry, the referrers API can answer 200 with an empty (or goldens-only) list for a manifest that was just pushed, for up to five minutes, before its referrer cache catches up. findReferrers therefore always tries the fallback tag too whenever the referrers API's own match set for the artifact type being searched for comes up empty — regardless of whether that is because the API isn't served at all or because it is simply still catching up. This applies uniformly to signature-bundle discovery and to the generic FetchSigned path (goldens, fixtures).

Design points decided here (the spec is silent or ambiguous)

• The Retry-After "budget" is read as the full retry schedule's own total (the sum of every scheduled wait — 60s under today's constants), since no separate numeric threshold is given anywhere in the constants or the guide.
• The lock's bundle digest is the bundle's own content (its blob/layer digest), not the wrapping referrer manifest's digest — lock3.schema.json's pin has no bundle_manifest field, so --frozen fetches the bundle directly as a blob by that digest, never through the manifest-wrapper indirection the normal (non-frozen) discovery path uses.
• FetchSigned's repository parameter is a suffix appended to every configured base (matching how fixtures_repository_suffix is named and shaped in the constants), not a full repository-path replacement.
• "Already installed" skips everything, not just the layer GET. Once a manifest digest already has a verified.json record, Ensure returns immediately with zero further requests at all (referrers, bundle, and layer) — a superset of the documented "existing-install-noop: zero layer requests," on the reasoning that content at a given digest never changes, so nothing past that point needs re-checking.
• HTTP timeouts are combined into one per-attempt deadline (ConnectTimeoutSeconds + IdleReadTimeoutSeconds) rather than separately modeling a dial timeout and a stream-idle timeout; Options.ConnectTimeout/IdleReadTimeout let a caller (a test) override both to something short.
• Least confident area: pre-seeded (oras copy -r --to-oci-layout) local verification in ResolveInstalled. It is implemented against this lane's own best understanding of how such a copy represents a referrer in a plain OCI layout's index.json (filtering entries by artifactType), without having seen a real layouts/oras-preseed/ fixture — that is the fixtures lane's own job, and this will need a pass once it exists.

Prose elsewhere claiming the Go module is dependency-free

This PR gives go.mod its first require (pinned github.com/klauspost/compress, for the zstd decoder). A search for prose asserting the opposite found these, all outside this lane's scope and left untouched:

• .github/dependabot.yml:15 — "the module is stdlib-only (no require block)"
• CONTRIBUTING.md:75 — "this tree has zero external Go dependencies today (no require block in go/go.mod, so go/go.sum does not exist yet either)"
• docs/reference/go.md:101 — "The implementation is stdlib only"
• scripts/policy-merge-check.py:729 — a comment making the same claim
• .github/workflows/ci.yml (9 occurrences) — cache: false # the module is stdlib-only: there is no go.sum to cache

None of these are wrong about go/chtypes itself (still stdlib-only); they become stale once go.mod carries any require block at all, which happens the moment this PR lands on v1.

Test plan

• go vet ./internal/ocifetch/ — clean.
• golangci-lint run ./internal/ocifetch/... — 0 issues.
• gofmt -l internal/ocifetch/ — clean.
• go test ./internal/ocifetch/... (and again with -race) — 64 unit tests, all passing, covering every file directly: DSSE PAE against a hand-countable vector, bundle verification with this repository's own fixture test key, every tar refusal rule, the index.json concurrent-writer race, retry/Retry-After/redirect/token behavior over real HTTP round trips (httptest), and a full fake-registry integration test of the happy path (Ensure end to end, already-installed reuse, offline resolution, untrusted-key refusal, bad-spelling refusal).
• go test -count=1 -run '^TestConformanceV1$' ./internal/ocifetch/ — skips loudly (CHTYPES_V1_CONFORMANCE unset), which is the correct, expected behavior in a bare copy and on this PR today: the fixtures lane's tests/fixtures/fetch-v1/{cases.json,trees/,layouts/} and scripts/fetch-v1/server.py have not landed on v1 yet. conformance_test.go is written against the documented case/report schema shapes as precisely as they can be read today, but has not run against real fixtures — expect an iteration pass once that lane merges and this branch is updated.
• scripts/policy-merge-check.py --check-carve-out, scripts/check-selftests-wired.py, scripts/lint-public.sh, scripts/lint-spelling.sh, scripts/lint-cited-paths.sh, scripts/fetch-v1/gen-constants.py --check — all clean, run unpiped.

Related issues

No tracking issue filed yet for the v1 fetch layer as a whole.

🤖 Generated with Claude Code

[EricAndrechek]

Two follow-up commits after review:

1. Accept header on every manifests/<ref> GET (by tag and by digest): application/vnd.oci.image.index.v1+json, application/vnd.oci.image.manifest.v1+json. Our host ignores it; a mirror may need it for content negotiation. Covers resolveIndex, fetchManifestByDigest, the referrers fallback tag's own manifests/ GET, fetchReferrerContent, FetchSigned and the --frozen path. Two new tests assert it is sent when configured and absent otherwise. Also reworded referrers.go's comment: the fallback-tag path is for mirrors that may serve only the fallback tag, not a cache-lag workaround on our own host (which is no-store) — the fallback logic itself is unchanged.
2. Found by re-reading ensure.go end to end (not from review): Options.HookBeforeIndexRename was parsed into resolvedOptions but never threaded into either real addIndexEntry call site — both hardcoded nil. Existing tests didn't catch it because they exercised addIndexEntry's race-survival logic directly rather than through Ensure. Fixed (session now carries the hook), with a new integration test (TestEnsureInvokesHookBeforeIndexRename) that proves it fires during a real Ensure call — this is what the index-race-reapply conformance case will depend on.

Also removed five dead citations of the fixtures lane's not-yet-landed scripted server that lint-cited-paths.sh correctly caught on the first CI run once the new test files were tracked (not present in a pre-commit local run against the untracked files — worth remembering for the next lane).

68 unit tests pass (-race included); go vet, golangci-lint and gofmt all clean. CI on this PR: every main-required context is green (go, lint-go, abi, misspell, public, prose, python, ts, lint-ts, lint-actions, docs, divergences, api-surface, with abi-fixtures/artifacts/rust/security still running at last check); v1-conformance/v1-fixtures/v1-network/v1-refcheck/v1-parity are red as expected — lane 0B's fixtures and scripted server have not landed on v1 yet.

[EricAndrechek]

First live run of the Go conformance suite inside the no-network sandbox (run 37135773065, head 027b94d), leg v1-conformance (go go1.27.x), measured from the job log and the uploaded report:

• Sandbox selftest: proof 1a exit 0, 1b exit 0 (uid 1001 = runner uid), 1c env survives, proof 2 curl exit 6 (no resolve), proof 3 loopback round trip exit 0, sandbox.sh --selftest: OK.
• Suite exit code: 0 (ok github.com/wave-rf/chtypes/go/internal/ocifetch 36.475s).
• Report: 139 pairs, 139 pass, 0 fail (http 79/79, file 60/60). Failing ids: none. Registry transport is the v1-network job's.
• All checks on the PR were green, including v1-parity and v1-sandbox. I am enrolling Go now (spec/fetch-v1/enrolled/go) and will confirm v1-parity on that push.

[EricAndrechek]

Update on Go enrollment (measured, runs 37136299058 and 37136947820):

• Enrolling Go turned v1-parity red twice, for two different causes.
  1. Fixed in dd00d79: the Go report said toolchain: go1.27.1 (runtime.Version()) but parity.py and the matrix require go1.27.x. The runner now reports the matrix's minor-line id. Resulting report: 139/139 pass, toolchain go1.27.x.
  2. NOT mine: parity now reports go: registry-transport cases are required but no report with toolchain 'registry' (the v1-network job) was found and lists the 5 registry cases (referrers-goldens-alongside-signature, goldens-artifact-ok, goldens-predicate-type-wrong, ...) as FAIL for go. The v1-network job's final step is still the placeholder "No binding is enrolled yet, so no conformance run happens here" (v1.yml), so no binding can ever be green while enrolled. PM to route.
• I have therefore un-enrolled Go again (head below) so this PR is not red on parity. To enroll, add spec/fetch-v1/enrolled/go once v1-network runs a suite and uploads report-go-registry; Go already honors CHTYPES_V1_REGISTRY_BASE.
• v1-abi-linked and v1-abi-rust fail with "not yet provided (lane F-Go / F-Rust) — add scripts/abi-v1/check-*.sh". Not touched by this PR's diff.
• Contract note for PM: the toolchain id has no env var from the workflow; each runner must hard-map its runtime to the matrix string.