Skip to content

feat(query,storage,http): download queries and ranged storage reads - #944

Merged
sebyx07 merged 6 commits into
mainfrom
feat/download-route-ranged-storage
Oct 31, 2026
Merged

sebyx07 merged 6 commits into
mainfrom
feat/download-route-ranged-storage

Conversation

@sebyx07

@sebyx07 sebyx07 commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Why

Gaps from a production app (tesote/bank-integrations docs/upstream-issues.md §23–24):

  • Downloads: no generated surface could answer raw bytes behind a policy. The app shipped a sealed MP4 recording as base64 inside a query's JSON (+33 %, parsed as one string).
  • Ranged reads: storage drivers had no ranged read.

What

  • Download queries are an option on query(), not a ninth primitive, so they keep the query's input, policy, name, route, manifest row and typed client.
    • Mint: GET /_x/query/<kebab> without _token runs the policy, rate limit and audit exactly as a read does, and answers { url, expiresAt }.
    • Serve: with _token it serves the file without a session. Headers are content-disposition (ASCII filename plus filename*), content-length, cache-control: no-store, nosniff and content-security-policy: sandbox. One Range: window gives a 206; past the end, a 416.
    • Token: a core seal() (AES-256-GCM, purpose ultimate.query.download:<name>) over the version, name, input fingerprint, expiry and minting actor (no facts). The seal is checked before the input is parsed. A forged or re-pointed token is 403 (X_QUERY_DOWNLOAD_TOKEN_INVALID); an expired one is 410.
    • Client: the typed client exposes only .url(input) for a download query (type-pinned). On the server, q.url(input, { ctx }).
    • Refusals: sql, live, single, rows, subscribes, cache and mcp beside download are type errors and X_QUERY_DOWNLOAD_INVALID. .live(), .page(), .as() and MCP are X_QUERY_DOWNLOAD_ONLY.
    • OpenAPI: includes the download operation. diff-operations reports rows ↔ download as breaking.
  • disk.get(key, { range: [first, last] }) on memory, local and s3:
    • inclusive, like HTTP and S3; the end is clamped;
    • X_STORAGE_RANGE_UNSATISFIABLE (416) at or past the end;
    • maxGetBytes bounds the window, not the object;
    • s3 sends a HEAD then a ranged GET.
    • A sealed object is one AES-GCM envelope and can't be sliced (documented).
  • @ultimat3/http byte-range helper: parseByteRange, byteWindow, contentRangeOf. The two cli copies (/_storage, site assets) now use it; both now ignore an empty bytes=-.

Tests

  • Storage: per-driver claims (memory, local, fake-s3), plus a live s3 ranged read against a Versity gateway.
  • Range parser: byte-range parser tests.
  • Download: declaration refusals, policy at mint, every token forgery and expiry case, the actor unreadable from the token, and the full HTTP pipeline (mint 401/403/200, sealed bytes served decrypted, headers, Range 206 / suffix / 416, stream with and without size, token checked before input).
  • Pins: type pins, openapi and the operation diff.

bun run verify is green. Pins raised with measured numbers:

  • serve module-graph ceilings;
  • examples/dummy /feed 167.5 → 167.6 kB (+128 B for the client's url()).

The stale dummy/social-media-clone x_job_workers.sql dump is regenerated (draining_at). The app-gate e2e needs Chrome, which wasn't on the authoring box, so CI runs it.

🤖 Generated with Claude Code

…e URL) and ranged storage reads

`query({ input, policy, download: { expiresInMs?, handle } })`: GET /_x/query/<name> without
`_token` runs the policy and mints `{ url, expiresAt }`; with `_token` it streams the file
(content-disposition, no-store, nosniff, CSP sandbox, one Range window → 206, 416). The token is a
core seal() of the query name, the input's fingerprint, the expiry and the minting actor: forged,
re-pointed or expired tokens are 403 / 410. The typed client exposes `.url(input)` for a download.

`disk.get(key, { range: [first, last] })` on the memory, local and s3 drivers (inclusive, clamped,
X_STORAGE_RANGE_UNSATISFIABLE past the end; maxGetBytes bounds the window). The byte-range parser is
one helper in @ultimat3/http, shared by /_storage and site assets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@developerz-ai

developerz-ai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Fix failing CI on #944

Queued — waiting for a free box.

Sessions: review 1 · review 2


🤖 developerz.ai, an automated agent, not a human — developerz-ai[bot]

…ries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@developerz-ai

developerz-ai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

✅ Reviewed — nothing blocking

0 actionable comment(s) · grounded on your code

⏱ 49s wall clock · gpt-6.1-sol via router · 2 model call(s) · 219 output token(s)

🤖 developerz.ai — automated review, running on your model and your box.

@developerz-ai developerz-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary — 59 file(s), 14 finding(s).

Critical 0 · Major 2 · Minor 12 · Nit 0

completed — reviewer[bot]: Adds download queries and ranged-read contracts. Found a declaration compatibility break, inaccurate OpenAPI file responses, and misleading repair/privacy guidance. completed [bot] — Adds ranged storage reads and shared HTTP range handling for downloads.

File Findings
packages/query/README.md 1 minor
packages/query/src/download-errors.ts 1 minor
packages/query/src/download-http.test.ts 1 minor
packages/query/src/download-http.ts 1 major, 1 minor
packages/query/src/openapi.ts 2 minor
packages/query/src/query.ts 1 minor
packages/storage/src/driver-s3.ts 1 major
packages/storage/src/errors.ts 1 minor
packages/storage/src/range.test.ts 2 minor
wiki/Queries-And-Live-Queries.md 2 minor

🤖 developerz.ai review — automated, running on your model and your box. What is this? · Reviewed by router/gpt-6.1-sol on box hv-dedicated-dev-marshall — 953.1k in / 22.5k out over 19 calls; cost unknown: no price on record for router/gpt-6.1-sol.

Comment thread packages/query/README.md Resolved
Comment thread packages/query/src/download-errors.ts Resolved
Comment thread packages/query/src/download-http.ts Resolved
Comment thread packages/query/src/download-http.ts Resolved
Comment thread packages/storage/src/errors.ts Resolved
Comment thread packages/storage/src/range.test.ts Resolved
Comment thread packages/storage/src/range.test.ts Resolved
Comment thread wiki/Queries-And-Live-Queries.md Resolved
Comment thread wiki/Queries-And-Live-Queries.md Resolved
…o-referrer, s3 ranged read re-checked against a second HEAD, coded refusals and runnable fixes, OpenAPI file bodies and 401

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@developerz-ai developerz-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary — 24 file(s), 6 finding(s).

Critical 0 · Major 2 · Minor 4 · Nit 0

[bot] completed: Hardens download headers and range-error contracts and retries raced S3 reads. Provider 416 responses still bypass race recovery, and the EOF repair can exceed the read ceiling. completed [bot] — Tightens download security, OpenAPI/type contracts and ranged-read diagnostics.

File Findings
packages/query/src/download-http.test.ts 1 minor
packages/query/src/download-http.ts 1 minor
packages/storage/src/driver-s3.ts 2 major
packages/storage/src/errors-range.ts 1 minor
packages/storage/src/range.test.ts 1 minor
4 non-blocking finding(s) below major (no thread on a re-review — only blocking findings open threads)
  • packages/query/src/download-http.test.ts:223 minor · test [bot] This isolation test still passes with img-src *, which permits its hostile image to load evil.example despite the claimed restriction. Assert the complete expected handler CSP as a literal, or add a browser test proving that the external resource is blocked.
  • packages/query/src/download-http.ts:105 minor · docs [bot] img-src 'self' and media-src 'self' allow any same-origin URL, not just this file, so inline HTML can request other app paths despite the stated guarantee. Describe the same-origin allowance here and in fileResponse's comment instead of claiming only the file itself can load.
  • packages/storage/src/errors-range.ts:26 minor · bug [bot] With a 10-byte object and maxGetBytes: 6, this EOF fix retries [0,9] and immediately fails with X_STORAGE_TOO_LARGE. Pass the read ceiling to rangeUnsatisfiable and clamp the suggested end so the repair succeeds (CLAUDE.md: errors are instructions).
  • packages/storage/src/range.test.ts:190 minor · test [bot] This shrink-past-start test receives an empty slice from FakeS3Client, whereas S3 rejects bytes=6-9 on a four-byte object with 416, so the provider-failure path is never exercised. Make the fake emit S3Error InvalidRange/416 for this request and retain the X_STORAGE_RANGE_UNSATISFIABLE assert…

🤖 developerz.ai review — automated, running on your model and your box. What is this? · Reviewed by router/gpt-6.1-sol on box hv-dedicated-dev-din — 499.8k in / 9.67k out over 11 calls; cost unknown: no price on record for router/gpt-6.1-sol.

Comment thread packages/storage/src/driver-s3.ts Resolved
Comment thread packages/storage/src/driver-s3.ts Resolved
sebyx07 and others added 2 commits October 31, 2026 09:39
…ze and etag from the same response; the provider's 416 is X_STORAGE_RANGE_UNSATISFIABLE

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…anged-storage

# Conflicts:
#	CHANGELOG.md
#	framework.manifest.json
#	packages/cli/src/serve-graph.test.ts

@developerz-ai developerz-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary — 14 file(s), 4 finding(s).

Critical 0 · Major 1 · Minor 3 · Nit 0

completed: Reviewed ranged storage reads and their published contracts. Found uncoded S3 body-read failures, acceptance of incomplete ranges, and non-executable invariant fixes. Completed reviewed the supplied ranged-storage diff.

File Findings
packages/storage/src/driver-s3-range.ts 1 major, 1 minor
packages/storage/src/range.test.ts 1 minor
packages/storage/src/range.ts 1 minor
3 non-blocking finding(s) below major (no thread on a re-review — only blocking findings open threads)
  • packages/storage/src/driver-s3-range.ts:57 minor · bug A GET for [2,5] answered with Content-Range: bytes 2-3/10 and two bytes passes every check and returns an incomplete window as success. Require the served end to equal Math.min(capped, size - 1) and validate the total size before accepting the response.
  • packages/storage/src/range.test.ts:114 minor · test Moving readS3Range's arrayBuffer() before its ceiling check leaves this test green, so it does not enforce the promised refusal before buffering. Inject a response with an observable body-read seam and assert it remains unread when X_STORAGE_TOO_LARGE is raised.
  • packages/storage/src/range.ts:46 minor · bug Both malformed-range fixes append uncommented em-dash prose to a JavaScript call, so pasting either X_INVARIANT repair produces a syntax error. Put the explanation inside a JavaScript comment and supply concrete valid offsets, as CLAUDE.md's executable-fix rule requires.

🤖 developerz.ai review — automated, running on your model and your box. What is this? · Reviewed by router/gpt-6.1-sol on box hv-dedicated-5 — 505.9k in / 2.4k out over 8 calls; cost unknown: no price on record for router/gpt-6.1-sol.

Comment thread packages/storage/src/driver-s3-range.ts Resolved
…_FAILED

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@developerz-ai developerz-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary — 3 file(s), 0 finding(s).

completed: S3 ranged reads now translate response-body failures into coded storage errors, with a regression test for a broken 206 stream. No defects found in this increment.

🤖 developerz.ai review — automated, running on your model and your box. What is this? · Reviewed by router/gpt-6.1-sol on box hv-dedicated-dev-marshall — 59.5k in / 219 out over 2 calls; cost unknown: no price on record for router/gpt-6.1-sol.

@sebyx07
sebyx07 merged commit fa59505 into main Oct 31, 2026
16 checks passed
@sebyx07
sebyx07 deleted the feat/download-route-ranged-storage branch October 31, 2026 15:10
@sebyx07 sebyx07 mentioned this pull request Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant