Repository navigation
feature: support gs:// urls in load jobs - #103
Merged
Merged
Conversation
ohaibbq
requested review from
a team,
colincadams,
emilyemilyemilyemilyemilyemily,
incidentist and
klickyfan
November 2, 2026 17:56
incidentist
requested changes
Nov 3, 2026
| /// Go's --dataset: one empty dataset created at startup. | ||
| dataset: ?[]const u8 = null, | ||
| /// Where load jobs read gs:// sources: a Cloud Storage emulator's `host:port` or | ||
| /// `http://host:port`, as STORAGE_EMULATOR_HOST spells it. Null leaves gs:// loads failing. |
There was a problem hiding this comment.
It should also fail on an empty string.
Member
Author
There was a problem hiding this comment.
added some validations - left empty string as allowed. emulator gcs jobs fail at runtime when it is not set.
Both client suites can now start fake-gcs-server (1.56.1, pinned by digest, plain HTTP) beside the emulator and point STORAGE_EMULATOR_HOST at it: a published port when the emulator runs as a local binary, a per-file Docker network when it runs as a container. Tests seed buckets and objects through fake-gcs-server's JSON API, so neither suite gains a storage client library. New gs:// load tests cover a single CSV object, a wildcard over sharded CSVs with a header row each, NEWLINE_DELIMITED_JSON, a missing object and a wildcard that matches nothing. They are listed as known failures until the server reads sourceUris; the Python test that asserted the refusal is gone.
configuration.load.sourceUris was refused. A load job now reads the objects it names from a Cloud Storage emulator such as fake-gcs-server, over plain HTTP through the JSON API, at the address STORAGE_EMULATOR_HOST or the new --storage-emulator-host flag gives (host:port or http://host:port). Real Cloud Storage is never contacted and there is no authentication. - A URI may hold one *, which lists the objects under the text before it and keeps those ending in the text after it, in name order. - Each object is read on its own, so skipLeadingRows applies per file and autodetect settles the columns from the first; maxBadRecords counts across the whole job. statistics.load reports inputFiles and inputFileBytes, and the resource echoes sourceUris. - A missing object, an empty wildcard, an unreachable or silent emulator (30 s per request), more than 256 MiB in total, or no emulator configured completes the job with status.errorResult. A malformed URI is a 400. src/server/gcs.zig holds a get/list/put interface over the objects (put is for extract jobs and unimplemented over HTTP), the HTTP store on std.http.Client, and an in-memory store the unit tests use. No new C or C++ dependency.
- The deadline timer started with Io.async, which a busy Threaded pool runs in place, so every storage request slept its full 30 s before the reply was looked at. The timer now runs concurrently with the request; where the Io cannot do that, the request runs without a deadline. - Reading a load's sources has a deadline of its own, 120 s, so a slow emulator holds the server for at most that long rather than 30 s for each of a wildcard's matches. - An answer other than 200 returns before its body is read, so a 404 near the byte cap is a not-found rather than a load that is too large. - The first source with content settles the columns and how many leading CSV rows every source skips: an empty first shard or a header-only shard no longer breaks an autodetected multi-file load. - A gs:// source that cannot be read as its format completes the job with an errorResult; an upload's still answers 400. The reader's messages no longer say "upload". - jobs.insert claims a load's job id once and passes it on.
incidentist
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack created with GitHub Stacks CLI • Give Feedback 💬