From d939df79cb4395545c1823098ae5074a462d7302 Mon Sep 17 00:00:00 2001 From: Aakash Saravanan Date: Sat, 14 Nov 2026 03:22:28 +0000 Subject: [PATCH 1/6] Honor standard CloudFetch download settings with the kernel Signed-off-by: Aakash Saravanan --- CHANGELOG.md | 2 ++ CONNECTION_PARAMETERS.md | 13 ++++++--- README.md | 11 ++++++-- connector.go | 14 ++++++---- doc.go | 15 +++++------ internal/config/config.go | 8 +++--- kernel_config.go | 13 ++++++++- kernel_config_test.go | 56 ++++++++++++++++++++++++++++----------- telemetry/request.go | 2 +- 9 files changed, 95 insertions(+), 39 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 62ac379f..a08565c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,8 @@ - Fix the module-root package conflict when building, linting, or running Gazelle with the `tools` build tag (databricks/databricks-sql-go#162). - **Behavior change:** Report declared DECIMAL precision and scale through `sql.ColumnType.DecimalSize()` on Thrift and SEA, including zero-row results. Tools such as ORM migrators can now act on these qualifiers (databricks/databricks-sql-go#500). - **Behavior change:** Support Arrow TIME results on Thrift and SEA as UTC `time.Time` values on `0001-01-01`, preserving precision and using clock-only nested JSON. TIME metadata now reports `TIME`/`time.Time` with no length, including empty and NULL-only results; previously Thrift reported `STRING`/`string` with unbounded length, and SEA reported an empty name with `*interface{}` (databricks/databricks-sql-go#505). +- Honor `WithMaxDownloadThreads` / `maxDownloadThreads` on the kernel backend, with explicit chunk settings taking precedence. + - **Breaking change:** Upgrade Apache Arrow Go from v12 to v18.8.0 (`github.com/apache/arrow-go/v18`). The public `rows.ArrowBatchIterator` now uses v18 `arrow.RecordBatch` and `*arrow.Schema` types; callers using Arrow batches must migrate their Arrow imports and dependent code. This requires a major driver release (v2.0.0), including a `/v2` driver module path before publication. - Fix CloudFetch reads failing with `row number 0 is not contained in any arrow batch: EOF` when `WithMaxDownloadThreads` / `maxDownloadThreads` is 0 or negative. Such values now log a warning and use the default of 10 (databricks/databricks-sql-go#439). - **Behavior change:** On the Thrift backend, `ExecuteStatement` is now retried (per `WithRetries`) when the connection to the server or proxy can't be opened, for example connection refused or a connect timeout, because the statement was never sent. Failures after the connection exists are still not retried, so a statement never runs twice. With default retry settings, a refused connection now fails after about 15 seconds instead of immediately (databricks/databricks-sql-go#423). diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index b40d752e..c4184c9f 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -131,7 +131,7 @@ Thrift options where available. |---|---|:---:|:---:|---|---| | `cloudfetch_enabled` | `WithCloudFetch` | ⚠️ | ✅ | `true` | Set `false` to fetch complete results without cloud-storage downloads. Use `WithCloudFetch` for the standard option on either backend. | | `cloudfetch_link_prefetch_window` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `10` | Number of presigned Cloud Fetch links prefetched ahead of the consumer. | -| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | `16` | Maximum decompressed Cloud Fetch chunks held in memory. Values above 256 are clamped with a warning, not rejected. | +| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | standard download window | Downloading and queued chunks ahead of consumption; excludes consumer buffers. Values above 256 are clamped with a warning. | | `inline_max_chunks_in_memory` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `4` | Inline-Arrow prefetch window; `1` disables prefetching. | | `complex_types_as_json` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `false` | Render ARRAY / MAP / STRUCT / VARIANT / GEOMETRY / GEOGRAPHY as JSON text. | | `intervals_as_string` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `false` | Render INTERVAL / DURATION values as canonical Databricks text. | @@ -175,8 +175,15 @@ an internal setting) and non-Arrow values also retain legacy session-location pa | DSN parameter | Connector option | Thrift | Kernel | Default | Notes | |---|---|:---:|:---:|---|---| | `useCloudFetch` | `WithCloudFetch` | ✅ | ✅ | `true` | Set false to fetch complete results through the SQL connection without direct cloud-storage access. | -| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ⚠️ | `10` | Concurrent download goroutines (Thrift). A value <= 0 logs a warning and uses the default. Inert on the kernel path. | -| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. | +| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ✅ | `10` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. A value <= 0 warns and uses 10. | +| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | standard download window | Positive values override the standard setting and raw chunk session parameters. | + +The kernel snapshots these options when connecting. A positive +`WithKernelMaxChunksInMemory` wins over a case-insensitive +`cloudfetch_max_chunks_in_memory` session parameter, which wins over +`WithMaxDownloadThreads`. Values above 256 are clamped. Invalid raw chunk +parameters warn and retain the kernel default of 16. These settings count +chunks, not bytes, and do not include consumer or application buffers. ## TLS diff --git a/README.md b/README.md index cff17a51..c1c15119 100644 --- a/README.md +++ b/README.md @@ -328,8 +328,15 @@ parallel via cloud storage | DSN parameter | Connector option | Protocol | Default | Description | |---|---|---|---|---| | `useCloudFetch` | `WithCloudFetch` | Both | `true` | Enable Cloud Fetch. Set to false to fetch results through the SQL connection without direct cloud-storage access. | -| `maxDownloadThreads` | `WithMaxDownloadThreads` | Thrift only (inert on kernel) | `10` | Concurrent download goroutines (Thrift). A value <= 0 logs a warning and uses the default. Inert on the kernel path. | -| | `WithKernelMaxChunksInMemory(n)` | SEA only | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. | +| `maxDownloadThreads` | `WithMaxDownloadThreads` | Both | `10` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. A value <= 0 warns and uses 10. | +| | `WithKernelMaxChunksInMemory(n)` | SEA only | standard download window | Positive values override the standard setting and raw chunk session parameters. | + +The kernel snapshots these options when connecting. A positive +`WithKernelMaxChunksInMemory` wins over a case-insensitive +`cloudfetch_max_chunks_in_memory` session parameter, which wins over +`WithMaxDownloadThreads`. Values above 256 are clamped. Invalid raw chunk +parameters warn and retain the kernel default of 16. These settings count +chunks, not bytes, and do not include consumer or application buffers. On the Thrift backend: diff --git a/connector.go b/connector.go index cc4f9eb6..0d5f4046 100644 --- a/connector.go +++ b/connector.go @@ -549,6 +549,10 @@ func WithCloudFetch(useCloudFetch bool) ConnOption { // WithMaxDownloadThreads sets up maximum download threads for cloud fetch. Default is 10. // A value <= 0 logs a warning and uses the default. +// On the kernel backend it bounds downloading and queued chunks ahead of +// consumption, excluding consumer buffers; it is not a byte limit. Values above +// 256 are clamped. Configuration is fixed when connecting. An explicit chunk +// session parameter or positive WithKernelMaxChunksInMemory takes precedence. func WithMaxDownloadThreads(numThreads int) ConnOption { return func(c *config.Config) { c.MaxDownloadThreads = numThreads @@ -837,11 +841,11 @@ func WithKernelMaxConnections(n int) ConnOption { } } -// WithKernelMaxChunksInMemory bounds how many decompressed CloudFetch chunks the -// kernel holds in memory at once on the kernel backend — the knob that trades -// large-result throughput for peak RSS. Lower it (e.g. 4) to cap memory on wide, -// row-heavy result sets; raise it for more download parallelism at higher memory. -// A value <= 0 (the default) leaves the kernel's built-in default (16) in place. +// WithKernelMaxChunksInMemory bounds downloading and queued CloudFetch chunks +// ahead of consumption on the kernel backend. Consumer and application buffers +// are additional; this is not a byte limit. +// A value <= 0 (the default) leaves WithMaxDownloadThreads or an explicit +// cloudfetch_max_chunks_in_memory session parameter in effect. // // It is forwarded as the kernel's client-only "cloudfetch_max_chunks_in_memory" // session conf, which the kernel applies to its result config and strips before diff --git a/doc.go b/doc.go index aec4b5e1..57629b13 100644 --- a/doc.go +++ b/doc.go @@ -38,7 +38,7 @@ Supported optional connection parameters can be specified in param=value and inc - timeout: Adds timeout (in seconds) for the server query execution. Default is no timeout - userAgentEntry: Used to identify partners. Set as a string with format - useCloudFetch: Used to enable cloud fetch for the query execution. Default is true - - maxDownloadThreads: Sets up the max number of concurrent workers for cloud fetch. Default is 10; a value <= 0 logs a warning and uses the default + - maxDownloadThreads: Sets the CloudFetch download window on both backends. Default is 10; a value <= 0 logs a warning and uses the default - useArrowNativeDecimal: Returns DECIMAL columns as native Arrow decimal128 (via GetArrowBatches); when scanned through database/sql they are returned as lossless strings. Default is false - authType: Specifies the desired authentication type. Valid values are: Pat, OauthM2M, OauthU2M - accessToken: Personal access token. Required if authType set to Pat @@ -90,7 +90,7 @@ Supported functional options include: - WithClientQueryTimeout( Duration). Bounds client-side statement execution on the kernel backend. Omitted preserves the legacy 600s ceiling; zero or the maximum Duration is unlimited. Requires WithUseKernel(true). Optional - WithUserAgentEntry( string). Used to identify partners. Optional - WithCloudFetch (bool). Used to enable cloud fetch for the query execution. Default is true. Optional - - WithMaxDownloadThreads ( int). Sets up the max number of concurrent workers for cloud fetch. Default is 10; a value <= 0 logs a warning and uses the default. Optional + - WithMaxDownloadThreads ( int). Sets the CloudFetch download window on both backends. Default is 10; a value <= 0 logs a warning and uses the default. Optional - WithAuthenticator ( auth.Authenticator). Sets up authentication. Required if neither access token or client credentials are provided. - WithClientCredentials( string, string). Sets up Oauth M2M authentication. - WithUseKernel( bool). Routes execution through the SEA-via-kernel backend instead of Thrift. Requires a build with -tags databricks_kernel (CGO_ENABLED=1); the default build returns a clear error. Default is false. See the kernel-backend section below. Optional @@ -274,13 +274,12 @@ WithKernel* prefix marks them experimental): kernel's default budget (900s). - WithKernelMaxConnections(n) sets the maximum number of idle HTTP connections retained per host. The default is 100; n must be positive. - - WithKernelMaxChunksInMemory(n) bounds how many decompressed CloudFetch chunks the - kernel holds in memory at once — the knob that trades large-result download - throughput for peak RSS. Lower it (e.g. 4) to cap memory on wide, row-heavy result - sets; raise it for more parallelism at higher memory. It is forwarded as the + - WithKernelMaxChunksInMemory(n) bounds downloading and queued CloudFetch chunks + ahead of consumption, excluding consumer buffers. It is not a byte limit. + It is forwarded as the kernel's client-only cloudfetch_max_chunks_in_memory session conf, stripped before - the SEA wire so it never reaches the server. A value <= 0 keeps the kernel's - default (16). + the SEA wire so it never reaches the server. A value <= 0 keeps the standard download window + or an explicit cloudfetch_max_chunks_in_memory session parameter. Setting any of these without WithUseKernel fails Connect with an error wrapping the sentinel ErrRequiresKernelBackend, detectable with errors.Is. diff --git a/internal/config/config.go b/internal/config/config.go index de0a463f..7d3de115 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -117,10 +117,10 @@ type KernelExperimentalConfig struct { // host by the kernel. Nil keeps the kernel default (100). MaxConnections *int - // MaxChunksInMemory bounds how many decompressed CloudFetch chunks the kernel - // holds in memory at once (WithKernelMaxChunksInMemory) — the knob that trades - // throughput for peak RSS on large result sets. Zero = keep the kernel default - // (16). A positive value is forwarded as the client-only + // MaxChunksInMemory bounds downloading and queued CloudFetch chunks ahead of + // consumption, excluding consumer buffers. Zero keeps the standard download + // window or an explicit session parameter. A positive value is forwarded as + // the client-only // "cloudfetch_max_chunks_in_memory" session conf, which the kernel folds into // its result config and strips before the SEA wire; it is kernel-only because // the Thrift path has no such in-memory-chunk knob. diff --git a/kernel_config.go b/kernel_config.go index 76f70a33..2956a7d4 100644 --- a/kernel_config.go +++ b/kernel_config.go @@ -6,6 +6,7 @@ import ( "fmt" "net/url" "strconv" + "strings" "time" "github.com/databricks/databricks-sql-go/auth/noop" @@ -180,6 +181,16 @@ func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config { // An explicit true below opts into native geospatial values. EnableGeoSpatialSupport: false, } + hasChunkLimit := false + for key := range kc.SessionConf { + if strings.EqualFold(key, config.KernelMaxChunksInMemoryConfKey) { + hasChunkLimit = true + break + } + } + if !hasChunkLimit { + kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] = strconv.Itoa(cfg.MaxDownloadThreads) + } if !cfg.UseCloudFetch { enabled := false kc.CloudFetchEnabled = &enabled @@ -209,7 +220,7 @@ func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config { // Injected into the kernel backend's own SessionConf ONLY (not via // EffectiveSessionParams, which both backends share) as the client-only key // the kernel reads and strips before the SEA wire — so it never leaks to the - // server or the Thrift path. Zero/negative keeps the kernel default (16). + // server or the Thrift path. Zero/negative keeps the standard download window. if ke.MaxChunksInMemory > 0 { // EffectiveSessionParams returned a fresh map, so mutating it is safe. kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] = strconv.Itoa(ke.MaxChunksInMemory) diff --git a/kernel_config_test.go b/kernel_config_test.go index e7a44b9b..d4631f82 100644 --- a/kernel_config_test.go +++ b/kernel_config_test.go @@ -477,7 +477,7 @@ var kernelConfigFieldDisposition = map[string]string{ // CloudFetch internally (below the C ABI); each option is // classified individually so a new CloudFetch option can't slip the guard. "UseCloudFetch": "forwarded", - "MaxDownloadThreads": "inert", + "MaxDownloadThreads": "forwarded", "MaxFilesInMemory": "inert", "MinTimeToExpiry": "inert", "CloudFetchSpeedThresholdMbps": "inert", @@ -717,7 +717,8 @@ func TestBuildKernelConfig(t *testing.T) { } else if kc.CloudFetchEnabled == nil || *kc.CloudFetchEnabled { t.Fatal("disabled mode must forward the client option") } - if !reflect.DeepEqual(kc.SessionConf, original) || !reflect.DeepEqual(c.SessionParams, original) { + expected := map[string]string{"can_cloud_download": "true", "CLOUDFETCH_ENABLED": "true", "TIMEZONE": "UTC", config.KernelMaxChunksInMemoryConfKey: "10"} + if !reflect.DeepEqual(kc.SessionConf, expected) || !reflect.DeepEqual(c.SessionParams, original) { t.Fatal("session parameters must be preserved for the kernel to interpret") } }) @@ -732,19 +733,44 @@ func TestBuildKernelConfig(t *testing.T) { } }) - t.Run("MaxChunksInMemory unset leaves the key absent (kernel default)", func(t *testing.T) { - c := baseKernelConfig() // KernelExperimental nil - kc := buildKernelConfig(c, kernel.Auth{Mode: kernel.AuthPAT, Token: "dapi-x"}) - if _, ok := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]; ok { - t.Errorf("SessionConf should not carry %q when unset (kernel keeps its default)", config.KernelMaxChunksInMemoryConfKey) - } - // A zero/negative explicit value is also a no-op. - c.KernelExperimental = &config.KernelExperimentalConfig{MaxChunksInMemory: 0} - kc = buildKernelConfig(c, kernel.Auth{Mode: kernel.AuthPAT, Token: "dapi-x"}) - if _, ok := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]; ok { - t.Errorf("SessionConf should not carry %q for a zero value", config.KernelMaxChunksInMemoryConfKey) - } - }) + for _, tc := range []struct { + name string + threads int + rawKey string + explicit int + wantKey, want string + }{ + {"default", 10, "", 0, config.KernelMaxChunksInMemoryConfKey, "10"}, + {"single", 1, "", 0, config.KernelMaxChunksInMemoryConfKey, "1"}, + {"parallel", 4, "", 0, config.KernelMaxChunksInMemoryConfKey, "4"}, + {"above ceiling", 1000, "", 0, config.KernelMaxChunksInMemoryConfKey, "1000"}, + {"raw override", 4, config.KernelMaxChunksInMemoryConfKey, 0, config.KernelMaxChunksInMemoryConfKey, "2"}, + {"uppercase override", 4, "CLOUDFETCH_MAX_CHUNKS_IN_MEMORY", 0, "CLOUDFETCH_MAX_CHUNKS_IN_MEMORY", "2"}, + {"explicit override", 4, "CLOUDFETCH_MAX_CHUNKS_IN_MEMORY", 3, config.KernelMaxChunksInMemoryConfKey, "3"}, + } { + t.Run("download window "+tc.name, func(t *testing.T) { + c := baseKernelConfig() + c.MaxDownloadThreads = tc.threads + if tc.rawKey != "" { + c.SessionParams = map[string]string{tc.rawKey: "2"} + } + if tc.explicit > 0 { + c.KernelExperimental = &config.KernelExperimentalConfig{MaxChunksInMemory: tc.explicit} + } + kc := buildKernelConfig(c, kernel.Auth{Mode: kernel.AuthPAT, Token: "dapi-x"}) + if got := kc.SessionConf[tc.wantKey]; got != tc.want { + t.Errorf("window=%q, want %q", got, tc.want) + } + if tc.rawKey != "" && c.SessionParams[tc.rawKey] != "2" { + t.Fatal("mutated caller session parameters") + } + if tc.rawKey == "CLOUDFETCH_MAX_CHUNKS_IN_MEMORY" && tc.explicit == 0 { + if _, ok := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]; ok { + t.Fatal("default shadows uppercase override") + } + } + }) + } t.Run("MaxChunksInMemory is not leaked to the shared server params", func(t *testing.T) { // The knob is kernel-only: it must NOT appear in EffectiveSessionParams diff --git a/telemetry/request.go b/telemetry/request.go index f2d50d00..397f8bfe 100644 --- a/telemetry/request.go +++ b/telemetry/request.go @@ -96,7 +96,7 @@ type DriverConnectionParameters struct { // what a customer actually configured so a hung-connect or large-result-OOM // report can be diagnosed from telemetry, not just a debug log. RetryMaxAttempts // / RetryOverallTimeoutMs zero means the backend default is in effect; - // MaxChunksInMemory zero means the kernel default (16). + // MaxChunksInMemory zero means the value was not reported. RetryMaxAttempts int32 `json:"retry_max_attempts,omitempty"` RetryOverallTimeoutMs int64 `json:"retry_overall_timeout_ms,omitempty"` MaxChunksInMemory int32 `json:"max_chunks_in_memory,omitempty"` From 88bb6524f847ce1677c90f9145003a0c87b9542e Mon Sep 17 00:00:00 2001 From: Aakash Saravanan Date: Sat, 14 Nov 2026 03:32:34 +0000 Subject: [PATCH 2/6] Clarify download window defaults and option documentation Signed-off-by: Aakash Saravanan --- CHANGELOG.md | 3 +-- doc.go | 8 ++++---- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a08565c6..f5f3539f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,8 +5,7 @@ - Fix the module-root package conflict when building, linting, or running Gazelle with the `tools` build tag (databricks/databricks-sql-go#162). - **Behavior change:** Report declared DECIMAL precision and scale through `sql.ColumnType.DecimalSize()` on Thrift and SEA, including zero-row results. Tools such as ORM migrators can now act on these qualifiers (databricks/databricks-sql-go#500). - **Behavior change:** Support Arrow TIME results on Thrift and SEA as UTC `time.Time` values on `0001-01-01`, preserving precision and using clock-only nested JSON. TIME metadata now reports `TIME`/`time.Time` with no length, including empty and NULL-only results; previously Thrift reported `STRING`/`string` with unbounded length, and SEA reported an empty name with `*interface{}` (databricks/databricks-sql-go#505). -- Honor `WithMaxDownloadThreads` / `maxDownloadThreads` on the kernel backend, with explicit chunk settings taking precedence. - +- Honor `WithMaxDownloadThreads` / `maxDownloadThreads` on the kernel backend, with a default window of 10 (previously the kernel default of 16) and explicit chunk settings taking precedence. - **Breaking change:** Upgrade Apache Arrow Go from v12 to v18.8.0 (`github.com/apache/arrow-go/v18`). The public `rows.ArrowBatchIterator` now uses v18 `arrow.RecordBatch` and `*arrow.Schema` types; callers using Arrow batches must migrate their Arrow imports and dependent code. This requires a major driver release (v2.0.0), including a `/v2` driver module path before publication. - Fix CloudFetch reads failing with `row number 0 is not contained in any arrow batch: EOF` when `WithMaxDownloadThreads` / `maxDownloadThreads` is 0 or negative. Such values now log a warning and use the default of 10 (databricks/databricks-sql-go#439). - **Behavior change:** On the Thrift backend, `ExecuteStatement` is now retried (per `WithRetries`) when the connection to the server or proxy can't be opened, for example connection refused or a connect timeout, because the statement was never sent. Failures after the connection exists are still not retried, so a statement never runs twice. With default retry settings, a refused connection now fails after about 15 seconds instead of immediately (databricks/databricks-sql-go#423). diff --git a/doc.go b/doc.go index 57629b13..0d9e0fdc 100644 --- a/doc.go +++ b/doc.go @@ -276,10 +276,10 @@ WithKernel* prefix marks them experimental): retained per host. The default is 100; n must be positive. - WithKernelMaxChunksInMemory(n) bounds downloading and queued CloudFetch chunks ahead of consumption, excluding consumer buffers. It is not a byte limit. - It is forwarded as the - kernel's client-only cloudfetch_max_chunks_in_memory session conf, stripped before - the SEA wire so it never reaches the server. A value <= 0 keeps the standard download window - or an explicit cloudfetch_max_chunks_in_memory session parameter. + It is forwarded as the kernel's client-only cloudfetch_max_chunks_in_memory + session conf, stripped before the SEA wire so it never reaches the server. + A value <= 0 keeps the standard download window or an explicit + cloudfetch_max_chunks_in_memory session parameter. Setting any of these without WithUseKernel fails Connect with an error wrapping the sentinel ErrRequiresKernelBackend, detectable with errors.Is. From 1181a7a547d54c95dc7eb52a85a4be33c9a9a2b8 Mon Sep 17 00:00:00 2001 From: Aakash Saravanan Date: Sat, 14 Nov 2026 04:10:41 +0000 Subject: [PATCH 3/6] Keep the kernel download window default when unset Signed-off-by: Aakash Saravanan --- CHANGELOG.md | 2 +- CONNECTION_PARAMETERS.md | 2 +- README.md | 2 +- connector.go | 10 ++++++---- connector_cloudfetch_test.go | 30 ++++++++++++++++++++++++++++++ connector_test.go | 1 + doc.go | 4 ++-- internal/config/config.go | 3 +++ internal/config/config_test.go | 6 ++++++ kernel_config.go | 2 +- kernel_config_test.go | 19 +++++++++++++++---- 11 files changed, 67 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f3539f..20baf9bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ - Fix the module-root package conflict when building, linting, or running Gazelle with the `tools` build tag (databricks/databricks-sql-go#162). - **Behavior change:** Report declared DECIMAL precision and scale through `sql.ColumnType.DecimalSize()` on Thrift and SEA, including zero-row results. Tools such as ORM migrators can now act on these qualifiers (databricks/databricks-sql-go#500). - **Behavior change:** Support Arrow TIME results on Thrift and SEA as UTC `time.Time` values on `0001-01-01`, preserving precision and using clock-only nested JSON. TIME metadata now reports `TIME`/`time.Time` with no length, including empty and NULL-only results; previously Thrift reported `STRING`/`string` with unbounded length, and SEA reported an empty name with `*interface{}` (databricks/databricks-sql-go#505). -- Honor `WithMaxDownloadThreads` / `maxDownloadThreads` on the kernel backend, with a default window of 10 (previously the kernel default of 16) and explicit chunk settings taking precedence. +- Honor explicit `WithMaxDownloadThreads` / `maxDownloadThreads` settings on the kernel backend, preserving its default window of 16 when unset and explicit chunk-setting precedence. - **Breaking change:** Upgrade Apache Arrow Go from v12 to v18.8.0 (`github.com/apache/arrow-go/v18`). The public `rows.ArrowBatchIterator` now uses v18 `arrow.RecordBatch` and `*arrow.Schema` types; callers using Arrow batches must migrate their Arrow imports and dependent code. This requires a major driver release (v2.0.0), including a `/v2` driver module path before publication. - Fix CloudFetch reads failing with `row number 0 is not contained in any arrow batch: EOF` when `WithMaxDownloadThreads` / `maxDownloadThreads` is 0 or negative. Such values now log a warning and use the default of 10 (databricks/databricks-sql-go#439). - **Behavior change:** On the Thrift backend, `ExecuteStatement` is now retried (per `WithRetries`) when the connection to the server or proxy can't be opened, for example connection refused or a connect timeout, because the statement was never sent. Failures after the connection exists are still not retried, so a statement never runs twice. With default retry settings, a refused connection now fails after about 15 seconds instead of immediately (databricks/databricks-sql-go#423). diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index c4184c9f..96d6ab82 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -175,7 +175,7 @@ an internal setting) and non-Arrow values also retain legacy session-location pa | DSN parameter | Connector option | Thrift | Kernel | Default | Notes | |---|---|:---:|:---:|---|---| | `useCloudFetch` | `WithCloudFetch` | ✅ | ✅ | `true` | Set false to fetch complete results through the SQL connection without direct cloud-storage access. | -| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ✅ | `10` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. A value <= 0 warns and uses 10. | +| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ✅ | Thrift `10`; kernel `16` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. An explicit value <= 0 warns and uses 10. | | | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | standard download window | Positive values override the standard setting and raw chunk session parameters. | The kernel snapshots these options when connecting. A positive diff --git a/README.md b/README.md index c1c15119..ec5916c3 100644 --- a/README.md +++ b/README.md @@ -328,7 +328,7 @@ parallel via cloud storage | DSN parameter | Connector option | Protocol | Default | Description | |---|---|---|---|---| | `useCloudFetch` | `WithCloudFetch` | Both | `true` | Enable Cloud Fetch. Set to false to fetch results through the SQL connection without direct cloud-storage access. | -| `maxDownloadThreads` | `WithMaxDownloadThreads` | Both | `10` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. A value <= 0 warns and uses 10. | +| `maxDownloadThreads` | `WithMaxDownloadThreads` | Both | Thrift `10`; kernel `16` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. An explicit value <= 0 warns and uses 10. | | | `WithKernelMaxChunksInMemory(n)` | SEA only | standard download window | Positive values override the standard setting and raw chunk session parameters. | The kernel snapshots these options when connecting. A positive diff --git a/connector.go b/connector.go index 0d5f4046..1b72c619 100644 --- a/connector.go +++ b/connector.go @@ -547,8 +547,9 @@ func WithCloudFetch(useCloudFetch bool) ConnOption { } } -// WithMaxDownloadThreads sets up maximum download threads for cloud fetch. Default is 10. -// A value <= 0 logs a warning and uses the default. +// WithMaxDownloadThreads sets up maximum download threads for cloud fetch. +// When unset, Thrift defaults to 10 and the kernel defaults to 16. +// An explicit value <= 0 logs a warning and uses 10. // On the kernel backend it bounds downloading and queued chunks ahead of // consumption, excluding consumer buffers; it is not a byte limit. Values above // 256 are clamped. Configuration is fixed when connecting. An explicit chunk @@ -556,6 +557,7 @@ func WithCloudFetch(useCloudFetch bool) ConnOption { func WithMaxDownloadThreads(numThreads int) ConnOption { return func(c *config.Config) { c.MaxDownloadThreads = numThreads + c.MaxDownloadThreadsSet = true } } @@ -844,8 +846,8 @@ func WithKernelMaxConnections(n int) ConnOption { // WithKernelMaxChunksInMemory bounds downloading and queued CloudFetch chunks // ahead of consumption on the kernel backend. Consumer and application buffers // are additional; this is not a byte limit. -// A value <= 0 (the default) leaves WithMaxDownloadThreads or an explicit -// cloudfetch_max_chunks_in_memory session parameter in effect. +// A value <= 0 (the default) leaves WithMaxDownloadThreads, an explicit +// cloudfetch_max_chunks_in_memory session parameter or the kernel default in effect. // // It is forwarded as the kernel's client-only "cloudfetch_max_chunks_in_memory" // session conf, which the kernel applies to its result config and strips before diff --git a/connector_cloudfetch_test.go b/connector_cloudfetch_test.go index ca08e9ad..6b8f3645 100644 --- a/connector_cloudfetch_test.go +++ b/connector_cloudfetch_test.go @@ -18,6 +18,7 @@ import ( "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/ipc" "github.com/apache/arrow-go/v18/arrow/memory" + "github.com/databricks/databricks-sql-go/internal/backend/kernel" "github.com/databricks/databricks-sql-go/internal/cli_service" "github.com/databricks/databricks-sql-go/internal/config" "github.com/databricks/databricks-sql-go/logger" @@ -61,6 +62,7 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { {value: -1, want: config.DefaultMaxDownloadThreads, wantWarn: true}, {value: 0, want: config.DefaultMaxDownloadThreads, wantWarn: true}, {value: 1, want: 1}, + {value: 10, want: 10}, {value: 15, want: 15}, {value: 100000, want: 100000}, } @@ -76,6 +78,9 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { ) require.NoError(t, err) assert.Equal(t, tc.want, con.(*connector).cfg.MaxDownloadThreads) + assert.True(t, con.(*connector).cfg.MaxDownloadThreadsSet) + kc := buildKernelConfig(con.(*connector).cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"}) + assert.Equal(t, strconv.Itoa(tc.want), kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]) assertMaxDownloadThreadsWarning(t, logs.String(), tc.value, tc.wantWarn) }) @@ -85,6 +90,7 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { con, err := (&databricksDriver{}).OpenConnector(dsn) require.NoError(t, err) assert.Equal(t, tc.want, con.(*connector).cfg.MaxDownloadThreads) + assert.True(t, con.(*connector).cfg.MaxDownloadThreadsSet) assertMaxDownloadThreadsWarning(t, logs.String(), tc.value, tc.wantWarn) }) } @@ -94,10 +100,34 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { con, err := NewConnector(WithServerHostname("example.cloud.databricks.com")) require.NoError(t, err) assert.Equal(t, config.DefaultMaxDownloadThreads, con.(*connector).cfg.MaxDownloadThreads) + assert.False(t, con.(*connector).cfg.MaxDownloadThreadsSet) assert.Empty(t, maxDownloadThreadsWarnings(logs.String())) }) } +func TestMaxDownloadThreadsDSNPresence(t *testing.T) { + for _, tc := range []struct { + query string + set bool + }{ + {"", false}, + {"?maxDownloadThreads=", true}, + } { + t.Run(tc.query, func(t *testing.T) { + con, err := (&databricksDriver{}).OpenConnector("token:secret@example.cloud.databricks.com:443/sql/1.0/warehouses/abc" + tc.query) + require.NoError(t, err) + cfg := con.(*connector).cfg.DeepCopy() + assert.Equal(t, tc.set, cfg.MaxDownloadThreadsSet) + kc := buildKernelConfig(cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"}) + value, present := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] + assert.Equal(t, tc.set, present) + if tc.set { + assert.Equal(t, "10", value) + } + }) + } +} + func assertMaxDownloadThreadsWarning(t *testing.T, logs string, value int, wantWarn bool) { t.Helper() warnings := maxDownloadThreadsWarnings(logs) diff --git a/connector_test.go b/connector_test.go index 3b1382dd..684a8646 100644 --- a/connector_test.go +++ b/connector_test.go @@ -165,6 +165,7 @@ func TestNewConnector(t *testing.T) { expectedCloudFetchConfig := config.CloudFetchConfig{ UseCloudFetch: true, MaxDownloadThreads: 15, + MaxDownloadThreadsSet: true, MaxFilesInMemory: 10, MinTimeToExpiry: 0 * time.Second, CloudFetchSpeedThresholdMbps: 0.1, diff --git a/doc.go b/doc.go index 0d9e0fdc..5351772b 100644 --- a/doc.go +++ b/doc.go @@ -38,7 +38,7 @@ Supported optional connection parameters can be specified in param=value and inc - timeout: Adds timeout (in seconds) for the server query execution. Default is no timeout - userAgentEntry: Used to identify partners. Set as a string with format - useCloudFetch: Used to enable cloud fetch for the query execution. Default is true - - maxDownloadThreads: Sets the CloudFetch download window on both backends. Default is 10; a value <= 0 logs a warning and uses the default + - maxDownloadThreads: Sets the CloudFetch download window on both backends. When unset, defaults to 10 on Thrift and 16 on the kernel; an explicit value <= 0 logs a warning and uses 10 - useArrowNativeDecimal: Returns DECIMAL columns as native Arrow decimal128 (via GetArrowBatches); when scanned through database/sql they are returned as lossless strings. Default is false - authType: Specifies the desired authentication type. Valid values are: Pat, OauthM2M, OauthU2M - accessToken: Personal access token. Required if authType set to Pat @@ -90,7 +90,7 @@ Supported functional options include: - WithClientQueryTimeout( Duration). Bounds client-side statement execution on the kernel backend. Omitted preserves the legacy 600s ceiling; zero or the maximum Duration is unlimited. Requires WithUseKernel(true). Optional - WithUserAgentEntry( string). Used to identify partners. Optional - WithCloudFetch (bool). Used to enable cloud fetch for the query execution. Default is true. Optional - - WithMaxDownloadThreads ( int). Sets the CloudFetch download window on both backends. Default is 10; a value <= 0 logs a warning and uses the default. Optional + - WithMaxDownloadThreads ( int). Sets the CloudFetch download window on both backends. When unset, defaults to 10 on Thrift and 16 on the kernel; an explicit value <= 0 logs a warning and uses 10. Optional - WithAuthenticator ( auth.Authenticator). Sets up authentication. Required if neither access token or client credentials are provided. - WithClientCredentials( string, string). Sets up Oauth M2M authentication. - WithUseKernel( bool). Routes execution through the SEA-via-kernel backend instead of Thrift. Requires a build with -tags databricks_kernel (CGO_ENABLED=1); the default build returns a clear error. Default is false. See the kernel-backend section below. Optional diff --git a/internal/config/config.go b/internal/config/config.go index 7d3de115..0ccb010c 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -509,6 +509,7 @@ func ParseDSN(dsn string) (UserConfig, error) { ucfg.UseCloudFetch = useCloudFetch } + ucfg.MaxDownloadThreadsSet = params.Has("maxDownloadThreads") if numThreads, ok, err := params.extractAsInt("maxDownloadThreads"); ok { if err != nil { return UserConfig{}, err @@ -792,6 +793,7 @@ const DefaultMaxDownloadThreads = 10 type CloudFetchConfig struct { UseCloudFetch bool MaxDownloadThreads int + MaxDownloadThreadsSet bool MaxFilesInMemory int MinTimeToExpiry time.Duration CloudFetchSpeedThresholdMbps float64 // Minimum download speed in MBps before WARN logging (default: 0.1) @@ -824,6 +826,7 @@ func (cfg CloudFetchConfig) DeepCopy() CloudFetchConfig { return CloudFetchConfig{ UseCloudFetch: cfg.UseCloudFetch, MaxDownloadThreads: cfg.MaxDownloadThreads, + MaxDownloadThreadsSet: cfg.MaxDownloadThreadsSet, MaxFilesInMemory: cfg.MaxFilesInMemory, MinTimeToExpiry: cfg.MinTimeToExpiry, CloudFetchSpeedThresholdMbps: cfg.CloudFetchSpeedThresholdMbps, diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 96fd5145..518ccaf2 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -297,6 +297,7 @@ func TestParseConfig(t *testing.T) { CloudFetchConfig: CloudFetchConfig{ UseCloudFetch: true, MaxDownloadThreads: 15, + MaxDownloadThreadsSet: true, MaxFilesInMemory: 10, CloudFetchSpeedThresholdMbps: 0.1, }, @@ -326,6 +327,7 @@ func TestParseConfig(t *testing.T) { CloudFetchConfig: CloudFetchConfig{ UseCloudFetch: true, MaxDownloadThreads: 15, + MaxDownloadThreadsSet: true, MaxFilesInMemory: 10, CloudFetchSpeedThresholdMbps: 0.1, }, @@ -850,6 +852,10 @@ func TestUserConfig_DeepCopy(t *testing.T) { SessionParams: map[string]string{"a": "32", "b": "4"}, UseKernel: true, WarehouseID: "wh-abc123", + CloudFetchConfig: CloudFetchConfig{ + MaxDownloadThreads: 10, + MaxDownloadThreadsSet: true, + }, } cfg_copy := cfg.DeepCopy() diff --git a/kernel_config.go b/kernel_config.go index 2956a7d4..e64ca787 100644 --- a/kernel_config.go +++ b/kernel_config.go @@ -188,7 +188,7 @@ func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config { break } } - if !hasChunkLimit { + if cfg.MaxDownloadThreadsSet && !hasChunkLimit { kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] = strconv.Itoa(cfg.MaxDownloadThreads) } if !cfg.UseCloudFetch { diff --git a/kernel_config_test.go b/kernel_config_test.go index d4631f82..d8cd3c0c 100644 --- a/kernel_config_test.go +++ b/kernel_config_test.go @@ -478,6 +478,7 @@ var kernelConfigFieldDisposition = map[string]string{ // classified individually so a new CloudFetch option can't slip the guard. "UseCloudFetch": "forwarded", "MaxDownloadThreads": "forwarded", + "MaxDownloadThreadsSet": "forwarded", "MaxFilesInMemory": "inert", "MinTimeToExpiry": "inert", "CloudFetchSpeedThresholdMbps": "inert", @@ -717,8 +718,7 @@ func TestBuildKernelConfig(t *testing.T) { } else if kc.CloudFetchEnabled == nil || *kc.CloudFetchEnabled { t.Fatal("disabled mode must forward the client option") } - expected := map[string]string{"can_cloud_download": "true", "CLOUDFETCH_ENABLED": "true", "TIMEZONE": "UTC", config.KernelMaxChunksInMemoryConfKey: "10"} - if !reflect.DeepEqual(kc.SessionConf, expected) || !reflect.DeepEqual(c.SessionParams, original) { + if !reflect.DeepEqual(kc.SessionConf, original) || !reflect.DeepEqual(c.SessionParams, original) { t.Fatal("session parameters must be preserved for the kernel to interpret") } }) @@ -733,6 +733,17 @@ func TestBuildKernelConfig(t *testing.T) { } }) + t.Run("unset download window keeps the kernel default", func(t *testing.T) { + c := baseKernelConfig() + for _, explicit := range []int{0, -1} { + c.KernelExperimental = &config.KernelExperimentalConfig{MaxChunksInMemory: explicit} + kc := buildKernelConfig(c, kernel.Auth{Mode: kernel.AuthPAT, Token: "dapi-x"}) + if _, ok := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]; ok { + t.Fatal("unset download window must leave the native default in effect") + } + } + }) + for _, tc := range []struct { name string threads int @@ -740,7 +751,7 @@ func TestBuildKernelConfig(t *testing.T) { explicit int wantKey, want string }{ - {"default", 10, "", 0, config.KernelMaxChunksInMemoryConfKey, "10"}, + {"explicit ten", 10, "", 0, config.KernelMaxChunksInMemoryConfKey, "10"}, {"single", 1, "", 0, config.KernelMaxChunksInMemoryConfKey, "1"}, {"parallel", 4, "", 0, config.KernelMaxChunksInMemoryConfKey, "4"}, {"above ceiling", 1000, "", 0, config.KernelMaxChunksInMemoryConfKey, "1000"}, @@ -750,7 +761,7 @@ func TestBuildKernelConfig(t *testing.T) { } { t.Run("download window "+tc.name, func(t *testing.T) { c := baseKernelConfig() - c.MaxDownloadThreads = tc.threads + WithMaxDownloadThreads(tc.threads)(c) if tc.rawKey != "" { c.SessionParams = map[string]string{tc.rawKey: "2"} } From 63a85351b752ff5ea4871ec58c42f6b6cd615ae0 Mon Sep 17 00:00:00 2001 From: Aakash Saravanan Date: Sat, 14 Nov 2026 04:15:40 +0000 Subject: [PATCH 4/6] Treat empty download settings as absent Signed-off-by: Aakash Saravanan --- CONNECTION_PARAMETERS.md | 4 ++-- README.md | 2 +- connector_cloudfetch_test.go | 2 +- doc.go | 4 ++-- internal/config/config.go | 8 ++++---- kernel_config.go | 2 +- 6 files changed, 11 insertions(+), 11 deletions(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 96d6ab82..28624b37 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -131,7 +131,7 @@ Thrift options where available. |---|---|:---:|:---:|---|---| | `cloudfetch_enabled` | `WithCloudFetch` | ⚠️ | ✅ | `true` | Set `false` to fetch complete results without cloud-storage downloads. Use `WithCloudFetch` for the standard option on either backend. | | `cloudfetch_link_prefetch_window` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `10` | Number of presigned Cloud Fetch links prefetched ahead of the consumer. | -| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | standard download window | Downloading and queued chunks ahead of consumption; excludes consumer buffers. Values above 256 are clamped with a warning. | +| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | `maxDownloadThreads` when set; otherwise `16` | Downloading and queued chunks ahead of consumption; excludes consumer buffers. Values above 256 are clamped with a warning. | | `inline_max_chunks_in_memory` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `4` | Inline-Arrow prefetch window; `1` disables prefetching. | | `complex_types_as_json` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `false` | Render ARRAY / MAP / STRUCT / VARIANT / GEOMETRY / GEOGRAPHY as JSON text. | | `intervals_as_string` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `false` | Render INTERVAL / DURATION values as canonical Databricks text. | @@ -176,7 +176,7 @@ an internal setting) and non-Arrow values also retain legacy session-location pa |---|---|:---:|:---:|---|---| | `useCloudFetch` | `WithCloudFetch` | ✅ | ✅ | `true` | Set false to fetch complete results through the SQL connection without direct cloud-storage access. | | `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ✅ | Thrift `10`; kernel `16` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. An explicit value <= 0 warns and uses 10. | -| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | standard download window | Positive values override the standard setting and raw chunk session parameters. | +| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | `maxDownloadThreads` when set; otherwise `16` | Positive values override the standard setting and raw chunk session parameters. | The kernel snapshots these options when connecting. A positive `WithKernelMaxChunksInMemory` wins over a case-insensitive diff --git a/README.md b/README.md index ec5916c3..c6b98b85 100644 --- a/README.md +++ b/README.md @@ -329,7 +329,7 @@ parallel via cloud storage |---|---|---|---|---| | `useCloudFetch` | `WithCloudFetch` | Both | `true` | Enable Cloud Fetch. Set to false to fetch results through the SQL connection without direct cloud-storage access. | | `maxDownloadThreads` | `WithMaxDownloadThreads` | Both | Thrift `10`; kernel `16` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. An explicit value <= 0 warns and uses 10. | -| | `WithKernelMaxChunksInMemory(n)` | SEA only | standard download window | Positive values override the standard setting and raw chunk session parameters. | +| | `WithKernelMaxChunksInMemory(n)` | SEA only | `maxDownloadThreads` when set; otherwise `16` | Positive values override the standard setting and raw chunk session parameters. | The kernel snapshots these options when connecting. A positive `WithKernelMaxChunksInMemory` wins over a case-insensitive diff --git a/connector_cloudfetch_test.go b/connector_cloudfetch_test.go index 6b8f3645..f6fbd003 100644 --- a/connector_cloudfetch_test.go +++ b/connector_cloudfetch_test.go @@ -111,7 +111,7 @@ func TestMaxDownloadThreadsDSNPresence(t *testing.T) { set bool }{ {"", false}, - {"?maxDownloadThreads=", true}, + {"?maxDownloadThreads=", false}, } { t.Run(tc.query, func(t *testing.T) { con, err := (&databricksDriver{}).OpenConnector("token:secret@example.cloud.databricks.com:443/sql/1.0/warehouses/abc" + tc.query) diff --git a/doc.go b/doc.go index 5351772b..bc5f182a 100644 --- a/doc.go +++ b/doc.go @@ -278,8 +278,8 @@ WithKernel* prefix marks them experimental): ahead of consumption, excluding consumer buffers. It is not a byte limit. It is forwarded as the kernel's client-only cloudfetch_max_chunks_in_memory session conf, stripped before the SEA wire so it never reaches the server. - A value <= 0 keeps the standard download window or an explicit - cloudfetch_max_chunks_in_memory session parameter. + A value <= 0 keeps an explicit cloudfetch_max_chunks_in_memory session + parameter, maxDownloadThreads or the kernel default of 16 when unset. Setting any of these without WithUseKernel fails Connect with an error wrapping the sentinel ErrRequiresKernelBackend, detectable with errors.Is. diff --git a/internal/config/config.go b/internal/config/config.go index 0ccb010c..f9c5143e 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -118,9 +118,9 @@ type KernelExperimentalConfig struct { MaxConnections *int // MaxChunksInMemory bounds downloading and queued CloudFetch chunks ahead of - // consumption, excluding consumer buffers. Zero keeps the standard download - // window or an explicit session parameter. A positive value is forwarded as - // the client-only + // consumption, excluding consumer buffers. Zero keeps an explicit session + // parameter, WithMaxDownloadThreads or the kernel default of 16. + // A positive value is forwarded as the client-only // "cloudfetch_max_chunks_in_memory" session conf, which the kernel folds into // its result config and strips before the SEA wire; it is kernel-only because // the Thrift path has no such in-memory-chunk knob. @@ -509,12 +509,12 @@ func ParseDSN(dsn string) (UserConfig, error) { ucfg.UseCloudFetch = useCloudFetch } - ucfg.MaxDownloadThreadsSet = params.Has("maxDownloadThreads") if numThreads, ok, err := params.extractAsInt("maxDownloadThreads"); ok { if err != nil { return UserConfig{}, err } ucfg.MaxDownloadThreads = numThreads + ucfg.MaxDownloadThreadsSet = true } // Metric View Metadata parameter diff --git a/kernel_config.go b/kernel_config.go index e64ca787..21a9f3cb 100644 --- a/kernel_config.go +++ b/kernel_config.go @@ -220,7 +220,7 @@ func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config { // Injected into the kernel backend's own SessionConf ONLY (not via // EffectiveSessionParams, which both backends share) as the client-only key // the kernel reads and strips before the SEA wire — so it never leaks to the - // server or the Thrift path. Zero/negative keeps the standard download window. + // server or the Thrift path. Zero/negative keeps the standard setting or native default. if ke.MaxChunksInMemory > 0 { // EffectiveSessionParams returned a fresh map, so mutating it is safe. kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] = strconv.Itoa(ke.MaxChunksInMemory) From c83279d4fa82147feb05df4f3ed1d4991c98c881 Mon Sep 17 00:00:00 2001 From: Aakash Saravanan Date: Sat, 14 Nov 2026 04:38:14 +0000 Subject: [PATCH 5/6] test(cloudfetch): verify DSN forwarding to kernel configuration Signed-off-by: Aakash Saravanan --- connector_cloudfetch_test.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/connector_cloudfetch_test.go b/connector_cloudfetch_test.go index f6fbd003..1ece3bd3 100644 --- a/connector_cloudfetch_test.go +++ b/connector_cloudfetch_test.go @@ -91,6 +91,8 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { require.NoError(t, err) assert.Equal(t, tc.want, con.(*connector).cfg.MaxDownloadThreads) assert.True(t, con.(*connector).cfg.MaxDownloadThreadsSet) + kc := buildKernelConfig(con.(*connector).cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"}) + assert.Equal(t, strconv.Itoa(tc.want), kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]) assertMaxDownloadThreadsWarning(t, logs.String(), tc.value, tc.wantWarn) }) } @@ -119,11 +121,8 @@ func TestMaxDownloadThreadsDSNPresence(t *testing.T) { cfg := con.(*connector).cfg.DeepCopy() assert.Equal(t, tc.set, cfg.MaxDownloadThreadsSet) kc := buildKernelConfig(cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"}) - value, present := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] + _, present := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] assert.Equal(t, tc.set, present) - if tc.set { - assert.Equal(t, "10", value) - } }) } } From 03c875fe7c06e20908f8cf0f34eff838ca0b9ec5 Mon Sep 17 00:00:00 2001 From: Aakash Saravanan Date: Sat, 14 Nov 2026 04:52:58 +0000 Subject: [PATCH 6/6] docs(cloudfetch): disclose existing setting effects on kernel buffers Signed-off-by: Aakash Saravanan --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 20baf9bf..4d92df40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ - Fix the module-root package conflict when building, linting, or running Gazelle with the `tools` build tag (databricks/databricks-sql-go#162). - **Behavior change:** Report declared DECIMAL precision and scale through `sql.ColumnType.DecimalSize()` on Thrift and SEA, including zero-row results. Tools such as ORM migrators can now act on these qualifiers (databricks/databricks-sql-go#500). - **Behavior change:** Support Arrow TIME results on Thrift and SEA as UTC `time.Time` values on `0001-01-01`, preserving precision and using clock-only nested JSON. TIME metadata now reports `TIME`/`time.Time` with no length, including empty and NULL-only results; previously Thrift reported `STRING`/`string` with unbounded length, and SEA reported an empty name with `*interface{}` (databricks/databricks-sql-go#505). -- Honor explicit `WithMaxDownloadThreads` / `maxDownloadThreads` settings on the kernel backend, preserving its default window of 16 when unset and explicit chunk-setting precedence. +- **Behavior change:** Explicit `WithMaxDownloadThreads` / `maxDownloadThreads` settings tuned for Thrift also control kernel buffering and can change memory use and throughput. The kernel window defaults to 16 when unset; explicit chunk-setting overrides take precedence. - **Breaking change:** Upgrade Apache Arrow Go from v12 to v18.8.0 (`github.com/apache/arrow-go/v18`). The public `rows.ArrowBatchIterator` now uses v18 `arrow.RecordBatch` and `*arrow.Schema` types; callers using Arrow batches must migrate their Arrow imports and dependent code. This requires a major driver release (v2.0.0), including a `/v2` driver module path before publication. - Fix CloudFetch reads failing with `row number 0 is not contained in any arrow batch: EOF` when `WithMaxDownloadThreads` / `maxDownloadThreads` is 0 or negative. Such values now log a warning and use the default of 10 (databricks/databricks-sql-go#439). - **Behavior change:** On the Thrift backend, `ExecuteStatement` is now retried (per `WithRetries`) when the connection to the server or proxy can't be opened, for example connection refused or a connect timeout, because the statement was never sent. Failures after the connection exists are still not retried, so a statement never runs twice. With default retry settings, a refused connection now fails after about 15 seconds instead of immediately (databricks/databricks-sql-go#423).