Repository navigation
feat(cloudfetch): apply download settings to kernel sessions - #616
aakash-saravanan-db merged 5 commits into
Conversation
There was a problem hiding this comment.
Verdict: 1 Low
The change looks correct, with one low finding: the session-open validation fires even when a raw chunk override makes the client setting irrelevant. The mapping, case-insensitive override detection, per-session snapshot and caller-config immutability are tested well. I couldn't check whether the native kernel actually reads cloudfetch_max_chunks_in_memory from sessionConf, because the kernel source isn't in this checkout.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with one low concern. The kernel now rejects an invalid cloudFetchConcurrentDownloads even when an explicit raw override or disabled CloudFetch means the value is never used. I couldn't check the claims about native-kernel behaviour (clamping above 256, invalid raw overrides warning and falling back to 16) because the kernel source isn't in this checkout.
vuanhphung
left a comment
There was a problem hiding this comment.
similar to the Python PR, let's evaluate if 10 is the better default vs current number 16
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
009a51e to
843637e
Compare
Feature or gap
Node.js applications could set
cloudFetchConcurrentDownloadson Thrift connections, but kernel sessions ignored it.What this PR addresses
Snapshot the existing setting at session open, retaining defaults of 16 on the kernel and 10 on Thrift. Preserve explicit assignments, including 10 before connecting, and raw chunk overrides; document validation, when mutations take effect and what the chunk window bounds.
Dependencies
Existing native kernel APIs; no API change. Thrift parity with
runAsync: truerequires the kernel CloudFetch link-prefetch fix (#466), so a Node.js release must use a native build that includes it.Tests and validation
Build, type-check, repository lint, formatting and 269 focused tests pass; rebuilt-driver checks on Databricks SQL confirm window 16 and peak 16 downloads with the setting omitted, complete 5M-row results, reuse and close. With native kernel
16bc3c4b, equal warmups and five alternating measurements per setting give DBSQL submission-through-drain medians for 5M ordered rows: explicit window 16 versus 10 is 16.645 versus 17.088 s withrunAsync: false(+2.7%, parity) and 17.762 versus 16.989 s withrunAsync: true(-4.4%, parity); every result is complete and correct.