Repository navigation
Request CloudFetch chunks ahead of the in-memory window while the reader waits - #1812
vuanhphung wants to merge 4 commits into
Conversation
…der waits When the reader is blocked on a chunk download, idle download threads now send the requests for up to a window's worth of chunks beyond the 16-chunk memory window. Those downloads read their bodies only once their chunk enters the window, so memory, thread and connection counts are unchanged; a response left unread for 10 s is released and requested again. Gated by CloudFetchRequestAhead and a server-side flag. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Drop the enableCloudFetchRequestAheadForJdbc server flag and its resolution: request-ahead is now on for both chunk providers unless CloudFetchRequestAhead=0. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
A full window of requests ahead let a paused reader's held responses (bodies already in kernel buffers, ~80-90 MB per result) be discarded and downloaded again. Limit the lead to MAX_REQUEST_LEAD = 8, the smallest lead that kept most of the live 2050mb drain gain on Thrift and SEA, and correct the changelog. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
The 10 s unread limit runs from when a response requested ahead arrives, so a reader that reads slowly after a wait, not only one that pauses, can have up to 8 chunks requested again. Say so in the changelog and comments. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good: one low concern about how the opt-out flag is parsed. I traced the concurrency in both providers: the admission window releases in consumption order, request-ahead tasks are queued in FIFO order behind the in-window tasks, so there is no thread starvation, a released response is re-requested at most once per chunk, and interrupt/close still counts as a cancellation. Nit: readerPausedPastUnreadLimit_requestsAgainAtMostMaxLead sleeps about 11.5 s per parameter (~23 s of unit-test time) because it uses the default 10 s unread limit instead of a shorter configured window.
|
On the nit about |
Description
On large CloudFetch reads the reader spends about 85% of the drain parked waiting for chunk downloads, on Thrift and SEA alike (JFR,
large_results_2050mb). Each ~5.8 MB chunk takes ~80 ms to its response headers plus ~90 ms of body, with a p95 about twice that. Only 16 chunks may be in memory and they are read in order, so while the reader waits on one chunk, 4–6 finished chunks hold slots: only 9–11 downloads are in flight and the 16 download threads sit idle in the pool 38% of the drain. Decode CPU is not the limit: the download threads use about 2 of 8 cores. (The dashboard's 1→2 GB "cliff" is the data itself: TPC-DS seasonality gives the2050mbquery 3.29× the rows of1025mb.)While the reader is blocked on a download, both chunk providers now submit up to 8 chunks beyond the 16-chunk window (
ChunkAdmissionWindow.MAX_REQUEST_LEAD). Those downloads send their GET at once, but the driver reads and decodes each body only once its chunk enters the window (positions in consumption order), so the storage round trip overlaps earlier downloads. Meanwhile the kernel may receive those bodies into socket buffers: a result set holds at most 8 extra compressed chunks outside the heap (~47 MB with 5.8 MB chunks; per drain, peak unread socket data was 26–28 MB median and 45 MB max, vs 6–7 MB and 37 MB onmain). The decoded-memory bound, thread count and peak connection count are unchanged, and a reader that keeps up never triggers requests ahead. A response held for 10 s, because the reader paused or reads slowly after a wait, is closed, which releases its pooled connection, and its chunk is requested again once admitted: at most 8 chunks per wait or pause. The shared part is inArrowResultChunk.downloadData;RemoteChunkProvider(Thrift) andStreamingChunkProvider(SEA) each gate their submissions. It is on by default with no server-side flag:CloudFetchRequestAhead=0is the only off switch.Testing
ChunkRequestAheadTest(20 tests; a mutant that requests a full window ahead fails the cap and paused-reader tests) and tests for the default and the opt-out. The fulljdbc-coresuite passes except the knownArrowBufferAllocatorTest.testCreateDatabricksBufferAllocator, which needs the NIO profile. LiveThriftCloudFetchTestsandLargeQueryExecutionTestspass with the switch on and off. Against the warehouse, on Thrift and SEA, these matchmain: checksums of every value of a sorted 3.2M-row, 34-column result and of asetMaxRowsread, and an order-sensitive key checksum of the sorted 21M-row2050mbresult with requests ahead active. Early close and cancel behave as onmain. At-Xmx512mwith Arrow's on-heap allocator, 2 GB drains complete with no more GC time thanmain. A reader that pauses 15 s after the first row finishes the drain within 0.2 s ofmain.The lead of 8 is the smallest that kept most of the gain: in a live
2050mbsweep, leads of 2/4/8/16 kept 37/40/69/100% (Thrift) and 53/74/98/100% (SEA) of the full-window drain gain. Live drain-only (while (rs.next()) {}) on the nightly Large Results queries with the default settings: medians of warm iterations in interleaved fresh JVMs (8 per arm at 1 and 2 GB, 2 at 4 GB), total time in ms (execute + drain). The drain alone drops 18–29%; a 50 MB result is within noise.mainmainlarge_results_1025mblarge_results_2050mblarge_results_4100mbAdversarial review (2 rounds, independent live re-runs): gain and checksums reproduced on both backends; its two mediums (bodies held in socket buffers, re-downloads after a pause) were fixed by capping the lead at 8; one low (slow or batch-and-flush readers re-download up to 8 chunks per wait or pause, +1–9% transfer, no latency change) is documented.
Telemetry Errors
DatabricksDriverErrorCodewhere appropriate, and anynew code is uniquely numbered and tested.
requested because the author cannot access the classification.
Additional Notes to the Reviewer
Overlaps: #1799 edits the same provider and task files and should take its per-host permit after admission, so a download waiting for admission does not hold one (its description lists this headers/body split as a follow-up). #1786 and #1794 touch the Thrift provider path. The textual conflicts are small.
This pull request and its description were written by Isaac.