Repository navigation
fix: serialize concurrent fetches on one operation across backends - #622
vuanhphung wants to merge 9 commits into
Conversation
Share result pipeline initialization and provider priming, and serialize kernel cursor consumption through the converter and slicer. Keep Thrift interleaving and the public API unchanged. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Keep provider priming unchanged behind the backend lock. Check queued readers' lifecycle and cover high-entropy kernel CloudFetch overlap, while documenting the unchanged Thrift CloudFetch gap. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Give result consumption one owner in DBSQLOperation through fetching, conversion and slicing. Remove the redundant kernel lock and initialization guard. Cover duplicate Thrift FETCH_FIRST requests and both backends' inline and CloudFetch exact unions. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Low
Looks good, no blocking concerns. The promise-chain lock in DBSQLOperation runs each fetchChunk/hasMore call to completion before the next starts, keeps waiting for readiness outside the lock, rechecks closed/cancelled once inside, and releases the lock when a read fails. Nit: the CHANGELOG entries don't reference the PR (#622), unlike the entries around them.
This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge and GPT-6.1 Terra added findings it doesn't mention — see the findings below.
🔀 1 finding considered and dismissed by the judge
- 🟡 Medium
lib/DBSQLOperation.ts:376— A queued reader cannot observe a completed close or cancel until the preceding fetch settles, so a stalled request can leave it pending for minutes after the op
Judge's evidence:await this.failIfClosed(); const defaultMaxRows = this.context.getConfig().fetchChunkDefaultMaxRows;
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
Addressed the review-body nit in 0978bcf: all three concurrency CHANGELOG entries now reference #622. Full unit suite: 1,818 passing; lint, Prettier, type-check, build and all six live overlap e2e cases pass. This comment was generated with GitHub MCP. |
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Low
Looks good, no blocking issues. The facade lock serializes fetchChunk/hasMore correctly, rejections don't poison the chain, and close/cancel are re-checked inside the lock. Nit: the description says redundant kernel guards were removed, but KernelOperationBackend is not changed in this diff.
This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge and GPT-6.1 Terra added findings it doesn't mention — see the findings below.
🔀 2 findings considered and dismissed by the judge
- 🟡 Medium
tests/e2e/concurrent_fetch.test.ts:9— The kernel cases run unconditionally even when the optional native binding is absent, causing the general e2e suite to fail on Thrift-only installations or unsu
Judge's evidence:for (const useKernel of [false, true]) { - 🟡 Medium
lib/DBSQLOperation.ts:376— A queued reader cannot observe a successful close or cancellation until the preceding fetch settles, so a stalled Thrift request can leave it pending for the 15
Judge's evidence:const result = this.resultConsumption.then(consume);
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
Addressed the second review-body nit: the description now states that both backends share one result-consumption boundary in DBSQLOperation; KernelOperationBackend and KernelResultsProvider are unchanged in the final diff. Follow-up 4bb5f46 also places the overlap spec in the kernel CI glob. Validation: 1,818 units; lint, Prettier, type-check and build; six live overlap cases on each of the published and pinned-built kernel bindings, all passing. This comment was generated with GitHub MCP. |
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Medium · 1 Low
Looks good, no blocking issues. The queue in withResultLock only wraps consuming calls, and fetchAll stays outside it, so nesting can't deadlock; close and cancel are still checked inside the lock. Nits: the Thrift cases sit under tests/e2e/kernel/, one test sets the private resultConsumption, and the 3 CHANGELOG bullets could be merged.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.
|
For the round-3 body nits: the shared dual-backend e2e spec sits under kernel/ so both the standard suite and the dedicated kernel glob exercise it, addressing the accepted B-F3 finding in the previous thread. The private resultConsumption barrier is intentional test-only synchronization: the queued-close regression deterministically reaches the queue; removing only the in-lock lifecycle recheck makes the metadata-close case fail with two metadata RPCs instead of one. The three separate CHANGELOG lines were explicitly requested in adversarial R2-1/R2-3/R2-4. Keeping these as-is avoids workflow edits and test refactors. The missing-native guard finding is accepted and being fixed separately. This comment was generated with GitHub MCP. |
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Low
Looks good, no blocking concerns. The facade promise-chain lock serializes fetchChunk/hasMore, re-checks close/cancel inside the lock and releases on rejection. Nits: the e2e file sits under tests/e2e/kernel/ but also covers Thrift; the kernel test writes private resultConsumption; the FETCH_FIRST test's finally never calls release().
This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.
|
Round-4 body nits: the dual-backend spec location and private queue barrier are intentional for the reasons in #622 (comment). In the FETCH_FIRST regression, release() runs before runAllAsync and the assertions; the gate is only an in-memory promise with no I/O, timer or external resource, so the finally's clock.restore() completes resource cleanup. No further test refactor is needed for this patch. The only inline suggestion repeats immediate waiter cancellation; answered in both threads with the explicit documentation-only R2-4 decision. All latest code changes are test-only and the production diff is unchanged. This comment was generated with GitHub MCP. |
Concurrent fetches on one operation can duplicate or lose rows on both backends. At main
bd8cb8040, the latest live tripwire returns 102,777 rows/100,000 ids on Thrift inline, 61,111 ids on Thrift CloudFetch, and 98,611 ids on kernel inline/CloudFetch. Thrift first readers can both send FETCH_FIRST; kernel first readers can construct competing pipelines over one native cursor, and shared Arrow prefetch state also races.Serialize each consuming fetchChunk/hasMore call in DBSQLOperation through response processing, conversion and slicing. Both backends share one boundary in the facade. When every read succeeds, each row is returned exactly once across the concurrent readers. They split the stream with no call-order guarantee; concurrent hasMoreRows waits for an in-flight fetch. After a failed Thrift read, other readers may continue past that read's rows, matching existing retry-after-error behavior; the kernel closes its statement on a failed fetch. Post-error hardening is a separate workstream. Readiness callbacks, status, cancel, close and retry APIs stay outside the queue, and queued readers honor existing lifecycle behavior. Related PR #432 locks a complete drain; this change owns each consuming call.
Verification: 1,818 units, lint, Prettier, type-check and build pass. Deterministic Thrift and kernel pipeline/prefetch regressions fail on main; queued-close tests cover in-flight Thrift metadata/fetch RPCs, and Thrift Arrow overlap uses the real converter/slicer. The Arrow regression gates distinct source batches independently: main loses ids 0 and 1; the shared queue preserves all six ids. Latest live overlap e2e passes 6/6 with both the published and pinned-built kernel binding, zero skips; 18 existing live batched/direct-results/CloudFetch/kernel cases also pass. Latest coverage
105af917areturns exact unions 3 consecutive times per backend and delivery mode, API overlap 2 and 3 CloudFetch downloads per operation. Its old xfails report exact SEA inline and both CloudFetch results as XPASS; Thrift inline passes normally. No coverage code changed.Ten fresh processes per build/backend/query, alternating build order, all 80 samples retained; actual delivery and all 100,000 ids verified. Medians in ms (main → fixed):
Fetch medians change -1.0% to +0.8%. No detectable paired-median latency regression; bootstrap 95% intervals include zero for every group. The shared warehouse has multi-second outliers on both builds, limiting precision.
Cross-driver state
StreamingChunkProvider.java:86,226@d5bdaa3); synchronous cursor methods do not establish simultaneous-reader support.UseThriftClient=1; DatabricksSQLException.use_kernel=False; deque/local drain (backend/kernel/result_set.py:95,157@b8259e4), PyO3 exclusive mutable borrow; competing-borrow errors map to OperationalError.WithUseKernel(false)default; unguarded pending slot (backend/kernel/rows.go:365@d9008a9). C ABI requires caller exclusion; KernelError/EOF.useKernel=false; duplicate FETCH_FIRST (RowSetProvider.ts:93,128-159@bd8cb8040) and CloudFetch prefetch races. Facade queue (DBSQLOperation.ts:161,300@b2d8ccf) covers both. Existing errors/false-on-closed hasMoreRows retained.useKernel=true; competing pipelines (KernelOperationBackend.ts:841@bd8cb8040). Same facade queue protects inline/CloudFetch; kernel backend/provider unchanged.napi/src/async_statement.rs:483@dac32e2: mutex spans next_batch; PyO3/C ABI ownership differs. Errors map from KernelError.Adversarial review: converged after two rounds; requested wording and Thrift lifecycle/Arrow tests addressed. Bot approves; immediate queued-reader wake-up is deferred per the human R2-4 documentation-only decision.
Final-head PR CI passes units/e2e on Node 22/24/26, lint and coverage. Passthrough integration passes Thrift, SEA and SEA/Reyden on
f79269ausing commit_sha (production identical to6353d6d), within the shared CI budget. Security Scan's unchanged picomatch@2.3.2 advisory reproduces on main. Local units unset CODEX_CI; the host-agent assertion also fails on main with it inherited.This pull request and its description were written by Isaac.