Skip to content

fix: serialize concurrent fetches on one operation across backends - #622

Open
vuanhphung wants to merge 9 commits into
mainfrom
vuanhphung/sea-overlap
Open

vuanhphung wants to merge 9 commits into
mainfrom
vuanhphung/sea-overlap

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Concurrent fetches on one operation can duplicate or lose rows on both backends. At main bd8cb8040, the latest live tripwire returns 102,777 rows/100,000 ids on Thrift inline, 61,111 ids on Thrift CloudFetch, and 98,611 ids on kernel inline/CloudFetch. Thrift first readers can both send FETCH_FIRST; kernel first readers can construct competing pipelines over one native cursor, and shared Arrow prefetch state also races.

Serialize each consuming fetchChunk/hasMore call in DBSQLOperation through response processing, conversion and slicing. Both backends share one boundary in the facade. When every read succeeds, each row is returned exactly once across the concurrent readers. They split the stream with no call-order guarantee; concurrent hasMoreRows waits for an in-flight fetch. After a failed Thrift read, other readers may continue past that read's rows, matching existing retry-after-error behavior; the kernel closes its statement on a failed fetch. Post-error hardening is a separate workstream. Readiness callbacks, status, cancel, close and retry APIs stay outside the queue, and queued readers honor existing lifecycle behavior. Related PR #432 locks a complete drain; this change owns each consuming call.

Verification: 1,818 units, lint, Prettier, type-check and build pass. Deterministic Thrift and kernel pipeline/prefetch regressions fail on main; queued-close tests cover in-flight Thrift metadata/fetch RPCs, and Thrift Arrow overlap uses the real converter/slicer. The Arrow regression gates distinct source batches independently: main loses ids 0 and 1; the shared queue preserves all six ids. Latest live overlap e2e passes 6/6 with both the published and pinned-built kernel binding, zero skips; 18 existing live batched/direct-results/CloudFetch/kernel cases also pass. Latest coverage 105af917a returns exact unions 3 consecutive times per backend and delivery mode, API overlap 2 and 3 CloudFetch downloads per operation. Its old xfails report exact SEA inline and both CloudFetch results as XPASS; Thrift inline passes normally. No coverage code changed.

Ten fresh processes per build/backend/query, alternating build order, all 80 samples retained; actual delivery and all 100,000 ids verified. Medians in ms (main → fixed):

Backend / delivery Fetch Execute + fetch
Thrift inline 150.0 → 151.2 498.0 → 472.2
Thrift CloudFetch 943.2 → 933.7 2152.4 → 2078.5
Kernel inline 95.3 → 96.0 440.5 → 429.1
Kernel CloudFetch 210.8 → 211.6 1313.8 → 1379.3

Fetch medians change -1.0% to +0.8%. No detectable paired-median latency regression; bootstrap 95% intervals include zero for every group. The shared warehouse has multi-second outliers on both builds, limiting precision.

Cross-driver state

Driver State Notes
JDBC Partial FIFO prefetch (StreamingChunkProvider.java:86,226 @d5bdaa3); synchronous cursor methods do not establish simultaneous-reader support. UseThriftClient=1; DatabricksSQLException.
Python Partial use_kernel=False; deque/local drain (backend/kernel/result_set.py:95,157 @b8259e4), PyO3 exclusive mutable borrow; competing-borrow errors map to OperationalError.
Go Not supported (raw concurrent kernel reads) WithUseKernel(false) default; unguarded pending slot (backend/kernel/rows.go:365 @d9008a9). C ABI requires caller exclusion; KernelError/EOF.
Node.js Thrift Not supported → Supported (this PR) useKernel=false; duplicate FETCH_FIRST (RowSetProvider.ts:93,128-159 @bd8cb8040) and CloudFetch prefetch races. Facade queue (DBSQLOperation.ts:161,300 @b2d8ccf) covers both. Existing errors/false-on-closed hasMoreRows retained.
Node.js kernel Not supported → Supported (this PR) Opt-in useKernel=true; competing pipelines (KernelOperationBackend.ts:841 @bd8cb8040). Same facade queue protects inline/CloudFetch; kernel backend/provider unchanged.
Kernel Supported (napi) napi/src/async_statement.rs:483 @dac32e2: mutex spans next_batch; PyO3/C ABI ownership differs. Errors map from KernelError.

Adversarial review: converged after two rounds; requested wording and Thrift lifecycle/Arrow tests addressed. Bot approves; immediate queued-reader wake-up is deferred per the human R2-4 documentation-only decision.

Final-head PR CI passes units/e2e on Node 22/24/26, lint and coverage. Passthrough integration passes Thrift, SEA and SEA/Reyden on f79269a using commit_sha (production identical to 6353d6d), within the shared CI budget. Security Scan's unchanged picomatch@2.3.2 advisory reproduces on main. Local units unset CODEX_CI; the host-agent assertion also fails on main with it inherited.

This pull request and its description were written by Isaac.

Share result pipeline initialization and provider priming, and serialize
kernel cursor consumption through the converter and slicer. Keep Thrift
interleaving and the public API unchanged.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Keep provider priming unchanged behind the backend lock. Check queued
readers' lifecycle and cover high-entropy kernel CloudFetch overlap,
while documenting the unchanged Thrift CloudFetch gap.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Give result consumption one owner in DBSQLOperation through fetching,
conversion and slicing. Remove the redundant kernel lock and initialization
guard. Cover duplicate Thrift FETCH_FIRST requests and both backends' inline
and CloudFetch exact unions.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung vuanhphung changed the title fix(kernel): preserve rows across overlapping result fetches fix: preserve rows across concurrent fetches on one operation Nov 18, 2026
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung vuanhphung changed the title fix: preserve rows across concurrent fetches on one operation fix: serialize concurrent fetches on one operation across backends Nov 18, 2026
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung
vuanhphung marked this pull request as ready for review November 18, 2026 10:13

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Low

Looks good, no blocking concerns. The promise-chain lock in DBSQLOperation runs each fetchChunk/hasMore call to completion before the next starts, keeps waiting for readiness outside the lock, rechecks closed/cancelled once inside, and releases the lock when a read fails. Nit: the CHANGELOG entries don't reference the PR (#622), unlike the entries around them.

This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge and GPT-6.1 Terra added findings it doesn't mention — see the findings below.

🔀 1 finding considered and dismissed by the judge
  • 🟡 Medium lib/DBSQLOperation.ts:376 — A queued reader cannot observe a completed close or cancel until the preceding fetch settles, so a stalled request can leave it pending for minutes after the op
    Judge's evidence: await this.failIfClosed(); const defaultMaxRows = this.context.getConfig().fetchChunkDefaultMaxRows;

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Addressed the review-body nit in 0978bcf: all three concurrency CHANGELOG entries now reference #622. Full unit suite: 1,818 passing; lint, Prettier, type-check, build and all six live overlap e2e cases pass.


This comment was generated with GitHub MCP.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Low

Looks good, no blocking issues. The facade lock serializes fetchChunk/hasMore correctly, rejections don't poison the chain, and close/cancel are re-checked inside the lock. Nit: the description says redundant kernel guards were removed, but KernelOperationBackend is not changed in this diff.

This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge and GPT-6.1 Terra added findings it doesn't mention — see the findings below.

🔀 2 findings considered and dismissed by the judge
  • 🟡 Medium tests/e2e/concurrent_fetch.test.ts:9 — The kernel cases run unconditionally even when the optional native binding is absent, causing the general e2e suite to fail on Thrift-only installations or unsu
    Judge's evidence: for (const useKernel of [false, true]) {
  • 🟡 Medium lib/DBSQLOperation.ts:376 — A queued reader cannot observe a successful close or cancellation until the preceding fetch settles, so a stalled Thrift request can leave it pending for the 15
    Judge's evidence: const result = this.resultConsumption.then(consume);

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Addressed the second review-body nit: the description now states that both backends share one result-consumption boundary in DBSQLOperation; KernelOperationBackend and KernelResultsProvider are unchanged in the final diff. Follow-up 4bb5f46 also places the overlap spec in the kernel CI glob. Validation: 1,818 units; lint, Prettier, type-check and build; six live overlap cases on each of the published and pinned-built kernel bindings, all passing.


This comment was generated with GitHub MCP.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Medium · 1 Low

Looks good, no blocking issues. The queue in withResultLock only wraps consuming calls, and fetchAll stays outside it, so nesting can't deadlock; close and cancel are still checked inside the lock. Nits: the Thrift cases sit under tests/e2e/kernel/, one test sets the private resultConsumption, and the 3 CHANGELOG bullets could be merged.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Comment thread lib/DBSQLOperation.ts Resolved
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

For the round-3 body nits: the shared dual-backend e2e spec sits under kernel/ so both the standard suite and the dedicated kernel glob exercise it, addressing the accepted B-F3 finding in the previous thread. The private resultConsumption barrier is intentional test-only synchronization: the queued-close regression deterministically reaches the queue; removing only the in-lock lifecycle recheck makes the metadata-close case fail with two metadata RPCs instead of one. The three separate CHANGELOG lines were explicitly requested in adversarial R2-1/R2-3/R2-4. Keeping these as-is avoids workflow edits and test refactors. The missing-native guard finding is accepted and being fixed separately.


This comment was generated with GitHub MCP.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Low

Looks good, no blocking concerns. The facade promise-chain lock serializes fetchChunk/hasMore, re-checks close/cancel inside the lock and releases on rejection. Nits: the e2e file sits under tests/e2e/kernel/ but also covers Thrift; the kernel test writes private resultConsumption; the FETCH_FIRST test's finally never calls release().

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Comment thread lib/DBSQLOperation.ts Resolved
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Round-4 body nits: the dual-backend spec location and private queue barrier are intentional for the reasons in #622 (comment). In the FETCH_FIRST regression, release() runs before runAllAsync and the assertions; the gate is only an in-memory promise with no I/O, timer or external resource, so the finally's clock.restore() completes resource cleanup. No further test refactor is needed for this patch. The only inline suggestion repeats immediate waiter cancellation; answered in both threads with the explicit documentation-only R2-4 decision. All latest code changes are test-only and the production diff is unchanged.


This comment was generated with GitHub MCP.

This branch was successfully deployed

1 active deployment
azure-prod — 6353d6d6 Deployed Nov 18, 2026 by vuanhphung via e2e-test (24.22.1, 24_22_1) #2117
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant