Skip to content

Request CloudFetch chunks ahead of the in-memory window while the reader waits - #1812

Open
vuanhphung wants to merge 4 commits into
mainfrom
perf/lrpipe
Open

vuanhphung wants to merge 4 commits into
mainfrom
perf/lrpipe

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

On large CloudFetch reads the reader spends about 85% of the drain parked waiting for chunk downloads, on Thrift and SEA alike (JFR, large_results_2050mb). Each ~5.8 MB chunk takes ~80 ms to its response headers plus ~90 ms of body, with a p95 about twice that. Only 16 chunks may be in memory and they are read in order, so while the reader waits on one chunk, 4–6 finished chunks hold slots: only 9–11 downloads are in flight and the 16 download threads sit idle in the pool 38% of the drain. Decode CPU is not the limit: the download threads use about 2 of 8 cores. (The dashboard's 1→2 GB "cliff" is the data itself: TPC-DS seasonality gives the 2050mb query 3.29× the rows of 1025mb.)

While the reader is blocked on a download, both chunk providers now submit up to 8 chunks beyond the 16-chunk window (ChunkAdmissionWindow.MAX_REQUEST_LEAD). Those downloads send their GET at once, but the driver reads and decodes each body only once its chunk enters the window (positions in consumption order), so the storage round trip overlaps earlier downloads. Meanwhile the kernel may receive those bodies into socket buffers: a result set holds at most 8 extra compressed chunks outside the heap (~47 MB with 5.8 MB chunks; per drain, peak unread socket data was 26–28 MB median and 45 MB max, vs 6–7 MB and 37 MB on main). The decoded-memory bound, thread count and peak connection count are unchanged, and a reader that keeps up never triggers requests ahead. A response held for 10 s, because the reader paused or reads slowly after a wait, is closed, which releases its pooled connection, and its chunk is requested again once admitted: at most 8 chunks per wait or pause. The shared part is in ArrowResultChunk.downloadData; RemoteChunkProvider (Thrift) and StreamingChunkProvider (SEA) each gate their submissions. It is on by default with no server-side flag: CloudFetchRequestAhead=0 is the only off switch.

Testing

ChunkRequestAheadTest (20 tests; a mutant that requests a full window ahead fails the cap and paused-reader tests) and tests for the default and the opt-out. The full jdbc-core suite passes except the known ArrowBufferAllocatorTest.testCreateDatabricksBufferAllocator, which needs the NIO profile. Live ThriftCloudFetchTests and LargeQueryExecutionTests pass with the switch on and off. Against the warehouse, on Thrift and SEA, these match main: checksums of every value of a sorted 3.2M-row, 34-column result and of a setMaxRows read, and an order-sensitive key checksum of the sorted 21M-row 2050mb result with requests ahead active. Early close and cancel behave as on main. At -Xmx512m with Arrow's on-heap allocator, 2 GB drains complete with no more GC time than main. A reader that pauses 15 s after the first row finishes the drain within 0.2 s of main.

The lead of 8 is the smallest that kept most of the gain: in a live 2050mb sweep, leads of 2/4/8/16 kept 37/40/69/100% (Thrift) and 53/74/98/100% (SEA) of the full-window drain gain. Live drain-only (while (rs.next()) {}) on the nightly Large Results queries with the default settings: medians of warm iterations in interleaved fresh JVMs (8 per arm at 1 and 2 GB, 2 at 4 GB), total time in ms (execute + drain). The drain alone drops 18–29%; a 50 MB result is within noise.

Query Thrift main Thrift PR SEA main SEA PR
large_results_1025mb 3,964 3,306 (−17%) 3,757 3,238 (−14%)
large_results_2050mb 10,560 9,852 (−7%) 10,148 7,987 (−21%)
large_results_4100mb 19,748 17,170 (−13%) 17,832 14,366 (−19%)

Adversarial review (2 rounds, independent live re-runs): gain and checksums reproduced on both backends; its two mediums (bodies held in socket buffers, re-downloads after a pause) were fixed by capping the lead at 8; one low (slow or batch-and-flush readers re-download up to 8 chunks per wait or pause, +1–9% transfer, no latency change) is documented.

Telemetry Errors

  • Not applicable — this PR does not add or change a telemetry-visible error. A released response is requested again without a retry, log or failure telemetry. Interrupting a download that waits for admission is a cancellation, as for an interrupted download.
  • Applicable — the error uses DatabricksDriverErrorCode where appropriate, and any
    new code is uniquely numbered and tested.
  • Applicable — its driver/server/user classification is linked, or maintainer help is
    requested because the author cannot access the classification.

Additional Notes to the Reviewer

Overlaps: #1799 edits the same provider and task files and should take its per-host permit after admission, so a download waiting for admission does not hold one (its description lists this headers/body split as a follow-up). #1786 and #1794 touch the Thrift provider path. The textual conflicts are small.

This pull request and its description were written by Isaac.

vuanhphung and others added 4 commits November 16, 2026 07:45
…der waits

When the reader is blocked on a chunk download, idle download threads now send
the requests for up to a window's worth of chunks beyond the 16-chunk memory
window. Those downloads read their bodies only once their chunk enters the
window, so memory, thread and connection counts are unchanged; a response left
unread for 10 s is released and requested again. Gated by CloudFetchRequestAhead
and a server-side flag.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Drop the enableCloudFetchRequestAheadForJdbc server flag and its resolution:
request-ahead is now on for both chunk providers unless CloudFetchRequestAhead=0.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
A full window of requests ahead let a paused reader's held responses (bodies
already in kernel buffers, ~80-90 MB per result) be discarded and downloaded
again. Limit the lead to MAX_REQUEST_LEAD = 8, the smallest lead that kept most
of the live 2050mb drain gain on Thrift and SEA, and correct the changelog.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The 10 s unread limit runs from when a response requested ahead arrives, so a
reader that reads slowly after a wait, not only one that pauses, can have up to
8 chunks requested again. Say so in the changelog and comments.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung
vuanhphung marked this pull request as ready for review November 16, 2026 22:34

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good: one low concern about how the opt-out flag is parsed. I traced the concurrency in both providers: the admission window releases in consumption order, request-ahead tasks are queued in FIFO order behind the in-window tasks, so there is no thread starvation, a released response is re-requested at most once per chunk, and interrupt/close still counts as a cancellation. Nit: readerPausedPastUnreadLimit_requestsAgainAtMostMaxLead sleeps about 11.5 s per parameter (~23 s of unit-test time) because it uses the default 10 s unread limit instead of a shorter configured window.

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

On the nit about readerPausedPastUnreadLimit_requestsAgainAtMostMaxLead (~11.5 s per parameter): it is deliberate. Both providers create their ChunkAdmissionWindow internally with the production limit, so the test pins the shipped 10 s behaviour end to end on Thrift and SEA. A shorter limit would need a test-only hook threaded through both providers' construction. The unit-level path with a short limit is already covered by download_releasesResponseHeldPastLimit_thenRequestsAgain (100 ms). The cost is ~23 s on a 16–24 min unit-test job, so I've left it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant