Skip to content

feat(cloudfetch): apply download settings to kernel sessions - #616

Merged
aakash-saravanan-db merged 5 commits into
mainfrom
aakash-saravanan_data/feat/cloudfetch-download-settings-20261114
Nov 17, 2026
Merged

aakash-saravanan-db merged 5 commits into
mainfrom
aakash-saravanan_data/feat/cloudfetch-download-settings-20261114

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Node.js applications could set cloudFetchConcurrentDownloads on Thrift connections, but kernel sessions ignored it.

What this PR addresses

Snapshot the existing setting at session open, retaining defaults of 16 on the kernel and 10 on Thrift. Preserve explicit assignments, including 10 before connecting, and raw chunk overrides; document validation, when mutations take effect and what the chunk window bounds.

Dependencies

Existing native kernel APIs; no API change. Thrift parity with runAsync: true requires the kernel CloudFetch link-prefetch fix (#466), so a Node.js release must use a native build that includes it.

Tests and validation

Build, type-check, repository lint, formatting and 269 focused tests pass; rebuilt-driver checks on Databricks SQL confirm window 16 and peak 16 downloads with the setting omitted, complete 5M-row results, reuse and close. With native kernel 16bc3c4b, equal warmups and five alternating measurements per setting give DBSQL submission-through-drain medians for 5M ordered rows: explicit window 16 versus 10 is 16.645 versus 17.088 s with runAsync: false (+2.7%, parity) and 17.762 versus 16.989 s with runAsync: true (-4.4%, parity); every result is complete and correct.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

The change looks correct, with one low finding: the session-open validation fires even when a raw chunk override makes the client setting irrelevant. The mapping, case-insensitive override detection, per-session snapshot and caller-config immutability are tested well. I couldn't check whether the native kernel actually reads cloudfetch_max_chunks_in_memory from sessionConf, because the kernel source isn't in this checkout.

Comment thread lib/kernel/KernelBackend.ts Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low concern. The kernel now rejects an invalid cloudFetchConcurrentDownloads even when an explicit raw override or disabled CloudFetch means the value is never used. I couldn't check the claims about native-kernel behaviour (clamping above 256, invalid raw overrides warning and falling back to 16) because the kernel source isn't in this checkout.

Comment thread lib/kernel/KernelBackend.ts Resolved

@vuanhphung vuanhphung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar to the Python PR, let's evaluate if 10 is the better default vs current number 16

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@aakash-saravanan-db
aakash-saravanan-db force-pushed the aakash-saravanan_data/feat/cloudfetch-download-settings-20261114 branch from 009a51e to 843637e Compare November 17, 2026 02:29

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@aakash-saravanan-db aakash-saravanan-db added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Nov 17, 2026
@aakash-saravanan-db
aakash-saravanan-db added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit b358888 Nov 17, 2026
33 checks passed

This branch was successfully deployed

1 active deployment
azure-prod — 843637ed Deployed Nov 17, 2026 by aakash-saravanan-db via e2e-test (26.12.1, 26_12_1) #2082
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants