Skip to content

Honor CloudFetch download settings on kernel connections - #1060

Open
aakash-saravanan-db wants to merge 4 commits into
mainfrom
aakash-saravanan_data/feat/cloudfetch-download-settings-20261114
Open

aakash-saravanan-db wants to merge 4 commits into
mainfrom
aakash-saravanan_data/feat/cloudfetch-download-settings-20261114

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Python applications could set max_download_threads on Thrift connections, but kernel connections ignored it.

What this PR addresses

Forward the existing setting to the kernel download window, retaining defaults of 16 on the kernel and 10 on Thrift. Honor explicit settings, including 10, preserve raw chunk overrides, and document connection-time configuration, validation, the 256 ceiling and the distinction from a byte limit.

Dependencies

Existing native kernel APIs; no native dependency bump is required.

Tests and validation

354 focused tests pass, with stub-kernel and real-kernel suites run separately; live checks of the built wheel on Databricks SQL confirm window 16 and peak 16 downloads with the setting omitted, complete 5M-row results, reuse and close. With native kernel 16bc3c4b, equal warmups and five alternating measurements per setting give DBSQL submission-through-drain medians for 5M ordered rows: explicit window 16 at 24.573 s versus 10 at 24.434 s (-0.6%, parity); every result is complete and correct.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung vuanhphung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you evaluated whether 10 is a good default value for Kernel? I'm not sure if kernel implementation is similar to Thrift to have the same default value. The current Kernel default is 16 which will change after this PR

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@aakash-saravanan-db aakash-saravanan-db added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Nov 17, 2026
@aakash-saravanan-db aakash-saravanan-db added the kernel-e2e Trigger preview run of the Kernel E2E workflow on this PR label Nov 17, 2026
@aakash-saravanan-db aakash-saravanan-db added kernel-e2e Trigger preview run of the Kernel E2E workflow on this PR and removed kernel-e2e Trigger preview run of the Kernel E2E workflow on this PR skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) labels Nov 17, 2026
@aakash-saravanan-db aakash-saravanan-db added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Nov 17, 2026

This branch was successfully deployed

1 active deployment
azure-prod — 8a1c2666 Deployed Nov 17, 2026 by aakash-saravanan-db via run-kernel-e2e #1080
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel-e2e Trigger preview run of the Kernel E2E workflow on this PR skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants