diff --git a/CHANGELOG.md b/CHANGELOG.md index 62ac379f..4d92df40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ - Fix the module-root package conflict when building, linting, or running Gazelle with the `tools` build tag (databricks/databricks-sql-go#162). - **Behavior change:** Report declared DECIMAL precision and scale through `sql.ColumnType.DecimalSize()` on Thrift and SEA, including zero-row results. Tools such as ORM migrators can now act on these qualifiers (databricks/databricks-sql-go#500). - **Behavior change:** Support Arrow TIME results on Thrift and SEA as UTC `time.Time` values on `0001-01-01`, preserving precision and using clock-only nested JSON. TIME metadata now reports `TIME`/`time.Time` with no length, including empty and NULL-only results; previously Thrift reported `STRING`/`string` with unbounded length, and SEA reported an empty name with `*interface{}` (databricks/databricks-sql-go#505). +- **Behavior change:** Explicit `WithMaxDownloadThreads` / `maxDownloadThreads` settings tuned for Thrift also control kernel buffering and can change memory use and throughput. The kernel window defaults to 16 when unset; explicit chunk-setting overrides take precedence. - **Breaking change:** Upgrade Apache Arrow Go from v12 to v18.8.0 (`github.com/apache/arrow-go/v18`). The public `rows.ArrowBatchIterator` now uses v18 `arrow.RecordBatch` and `*arrow.Schema` types; callers using Arrow batches must migrate their Arrow imports and dependent code. This requires a major driver release (v2.0.0), including a `/v2` driver module path before publication. - Fix CloudFetch reads failing with `row number 0 is not contained in any arrow batch: EOF` when `WithMaxDownloadThreads` / `maxDownloadThreads` is 0 or negative. Such values now log a warning and use the default of 10 (databricks/databricks-sql-go#439). - **Behavior change:** On the Thrift backend, `ExecuteStatement` is now retried (per `WithRetries`) when the connection to the server or proxy can't be opened, for example connection refused or a connect timeout, because the statement was never sent. Failures after the connection exists are still not retried, so a statement never runs twice. With default retry settings, a refused connection now fails after about 15 seconds instead of immediately (databricks/databricks-sql-go#423). diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index b40d752e..28624b37 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -131,7 +131,7 @@ Thrift options where available. |---|---|:---:|:---:|---|---| | `cloudfetch_enabled` | `WithCloudFetch` | ⚠️ | ✅ | `true` | Set `false` to fetch complete results without cloud-storage downloads. Use `WithCloudFetch` for the standard option on either backend. | | `cloudfetch_link_prefetch_window` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `10` | Number of presigned Cloud Fetch links prefetched ahead of the consumer. | -| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | `16` | Maximum decompressed Cloud Fetch chunks held in memory. Values above 256 are clamped with a warning, not rejected. | +| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | `maxDownloadThreads` when set; otherwise `16` | Downloading and queued chunks ahead of consumption; excludes consumer buffers. Values above 256 are clamped with a warning. | | `inline_max_chunks_in_memory` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `4` | Inline-Arrow prefetch window; `1` disables prefetching. | | `complex_types_as_json` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `false` | Render ARRAY / MAP / STRUCT / VARIANT / GEOMETRY / GEOGRAPHY as JSON text. | | `intervals_as_string` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `false` | Render INTERVAL / DURATION values as canonical Databricks text. | @@ -175,8 +175,15 @@ an internal setting) and non-Arrow values also retain legacy session-location pa | DSN parameter | Connector option | Thrift | Kernel | Default | Notes | |---|---|:---:|:---:|---|---| | `useCloudFetch` | `WithCloudFetch` | ✅ | ✅ | `true` | Set false to fetch complete results through the SQL connection without direct cloud-storage access. | -| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ⚠️ | `10` | Concurrent download goroutines (Thrift). A value <= 0 logs a warning and uses the default. Inert on the kernel path. | -| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. | +| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ✅ | Thrift `10`; kernel `16` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. An explicit value <= 0 warns and uses 10. | +| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | `maxDownloadThreads` when set; otherwise `16` | Positive values override the standard setting and raw chunk session parameters. | + +The kernel snapshots these options when connecting. A positive +`WithKernelMaxChunksInMemory` wins over a case-insensitive +`cloudfetch_max_chunks_in_memory` session parameter, which wins over +`WithMaxDownloadThreads`. Values above 256 are clamped. Invalid raw chunk +parameters warn and retain the kernel default of 16. These settings count +chunks, not bytes, and do not include consumer or application buffers. ## TLS diff --git a/README.md b/README.md index cff17a51..c6b98b85 100644 --- a/README.md +++ b/README.md @@ -328,8 +328,15 @@ parallel via cloud storage | DSN parameter | Connector option | Protocol | Default | Description | |---|---|---|---|---| | `useCloudFetch` | `WithCloudFetch` | Both | `true` | Enable Cloud Fetch. Set to false to fetch results through the SQL connection without direct cloud-storage access. | -| `maxDownloadThreads` | `WithMaxDownloadThreads` | Thrift only (inert on kernel) | `10` | Concurrent download goroutines (Thrift). A value <= 0 logs a warning and uses the default. Inert on the kernel path. | -| | `WithKernelMaxChunksInMemory(n)` | SEA only | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. | +| `maxDownloadThreads` | `WithMaxDownloadThreads` | Both | Thrift `10`; kernel `16` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. An explicit value <= 0 warns and uses 10. | +| | `WithKernelMaxChunksInMemory(n)` | SEA only | `maxDownloadThreads` when set; otherwise `16` | Positive values override the standard setting and raw chunk session parameters. | + +The kernel snapshots these options when connecting. A positive +`WithKernelMaxChunksInMemory` wins over a case-insensitive +`cloudfetch_max_chunks_in_memory` session parameter, which wins over +`WithMaxDownloadThreads`. Values above 256 are clamped. Invalid raw chunk +parameters warn and retain the kernel default of 16. These settings count +chunks, not bytes, and do not include consumer or application buffers. On the Thrift backend: diff --git a/connector.go b/connector.go index cc4f9eb6..1b72c619 100644 --- a/connector.go +++ b/connector.go @@ -547,11 +547,17 @@ func WithCloudFetch(useCloudFetch bool) ConnOption { } } -// WithMaxDownloadThreads sets up maximum download threads for cloud fetch. Default is 10. -// A value <= 0 logs a warning and uses the default. +// WithMaxDownloadThreads sets up maximum download threads for cloud fetch. +// When unset, Thrift defaults to 10 and the kernel defaults to 16. +// An explicit value <= 0 logs a warning and uses 10. +// On the kernel backend it bounds downloading and queued chunks ahead of +// consumption, excluding consumer buffers; it is not a byte limit. Values above +// 256 are clamped. Configuration is fixed when connecting. An explicit chunk +// session parameter or positive WithKernelMaxChunksInMemory takes precedence. func WithMaxDownloadThreads(numThreads int) ConnOption { return func(c *config.Config) { c.MaxDownloadThreads = numThreads + c.MaxDownloadThreadsSet = true } } @@ -837,11 +843,11 @@ func WithKernelMaxConnections(n int) ConnOption { } } -// WithKernelMaxChunksInMemory bounds how many decompressed CloudFetch chunks the -// kernel holds in memory at once on the kernel backend — the knob that trades -// large-result throughput for peak RSS. Lower it (e.g. 4) to cap memory on wide, -// row-heavy result sets; raise it for more download parallelism at higher memory. -// A value <= 0 (the default) leaves the kernel's built-in default (16) in place. +// WithKernelMaxChunksInMemory bounds downloading and queued CloudFetch chunks +// ahead of consumption on the kernel backend. Consumer and application buffers +// are additional; this is not a byte limit. +// A value <= 0 (the default) leaves WithMaxDownloadThreads, an explicit +// cloudfetch_max_chunks_in_memory session parameter or the kernel default in effect. // // It is forwarded as the kernel's client-only "cloudfetch_max_chunks_in_memory" // session conf, which the kernel applies to its result config and strips before diff --git a/connector_cloudfetch_test.go b/connector_cloudfetch_test.go index ca08e9ad..1ece3bd3 100644 --- a/connector_cloudfetch_test.go +++ b/connector_cloudfetch_test.go @@ -18,6 +18,7 @@ import ( "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/ipc" "github.com/apache/arrow-go/v18/arrow/memory" + "github.com/databricks/databricks-sql-go/internal/backend/kernel" "github.com/databricks/databricks-sql-go/internal/cli_service" "github.com/databricks/databricks-sql-go/internal/config" "github.com/databricks/databricks-sql-go/logger" @@ -61,6 +62,7 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { {value: -1, want: config.DefaultMaxDownloadThreads, wantWarn: true}, {value: 0, want: config.DefaultMaxDownloadThreads, wantWarn: true}, {value: 1, want: 1}, + {value: 10, want: 10}, {value: 15, want: 15}, {value: 100000, want: 100000}, } @@ -76,6 +78,9 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { ) require.NoError(t, err) assert.Equal(t, tc.want, con.(*connector).cfg.MaxDownloadThreads) + assert.True(t, con.(*connector).cfg.MaxDownloadThreadsSet) + kc := buildKernelConfig(con.(*connector).cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"}) + assert.Equal(t, strconv.Itoa(tc.want), kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]) assertMaxDownloadThreadsWarning(t, logs.String(), tc.value, tc.wantWarn) }) @@ -85,6 +90,9 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { con, err := (&databricksDriver{}).OpenConnector(dsn) require.NoError(t, err) assert.Equal(t, tc.want, con.(*connector).cfg.MaxDownloadThreads) + assert.True(t, con.(*connector).cfg.MaxDownloadThreadsSet) + kc := buildKernelConfig(con.(*connector).cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"}) + assert.Equal(t, strconv.Itoa(tc.want), kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]) assertMaxDownloadThreadsWarning(t, logs.String(), tc.value, tc.wantWarn) }) } @@ -94,10 +102,31 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) { con, err := NewConnector(WithServerHostname("example.cloud.databricks.com")) require.NoError(t, err) assert.Equal(t, config.DefaultMaxDownloadThreads, con.(*connector).cfg.MaxDownloadThreads) + assert.False(t, con.(*connector).cfg.MaxDownloadThreadsSet) assert.Empty(t, maxDownloadThreadsWarnings(logs.String())) }) } +func TestMaxDownloadThreadsDSNPresence(t *testing.T) { + for _, tc := range []struct { + query string + set bool + }{ + {"", false}, + {"?maxDownloadThreads=", false}, + } { + t.Run(tc.query, func(t *testing.T) { + con, err := (&databricksDriver{}).OpenConnector("token:secret@example.cloud.databricks.com:443/sql/1.0/warehouses/abc" + tc.query) + require.NoError(t, err) + cfg := con.(*connector).cfg.DeepCopy() + assert.Equal(t, tc.set, cfg.MaxDownloadThreadsSet) + kc := buildKernelConfig(cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"}) + _, present := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] + assert.Equal(t, tc.set, present) + }) + } +} + func assertMaxDownloadThreadsWarning(t *testing.T, logs string, value int, wantWarn bool) { t.Helper() warnings := maxDownloadThreadsWarnings(logs) diff --git a/connector_test.go b/connector_test.go index 3b1382dd..684a8646 100644 --- a/connector_test.go +++ b/connector_test.go @@ -165,6 +165,7 @@ func TestNewConnector(t *testing.T) { expectedCloudFetchConfig := config.CloudFetchConfig{ UseCloudFetch: true, MaxDownloadThreads: 15, + MaxDownloadThreadsSet: true, MaxFilesInMemory: 10, MinTimeToExpiry: 0 * time.Second, CloudFetchSpeedThresholdMbps: 0.1, diff --git a/doc.go b/doc.go index aec4b5e1..bc5f182a 100644 --- a/doc.go +++ b/doc.go @@ -38,7 +38,7 @@ Supported optional connection parameters can be specified in param=value and inc - timeout: Adds timeout (in seconds) for the server query execution. Default is no timeout - userAgentEntry: Used to identify partners. Set as a string with format - useCloudFetch: Used to enable cloud fetch for the query execution. Default is true - - maxDownloadThreads: Sets up the max number of concurrent workers for cloud fetch. Default is 10; a value <= 0 logs a warning and uses the default + - maxDownloadThreads: Sets the CloudFetch download window on both backends. When unset, defaults to 10 on Thrift and 16 on the kernel; an explicit value <= 0 logs a warning and uses 10 - useArrowNativeDecimal: Returns DECIMAL columns as native Arrow decimal128 (via GetArrowBatches); when scanned through database/sql they are returned as lossless strings. Default is false - authType: Specifies the desired authentication type. Valid values are: Pat, OauthM2M, OauthU2M - accessToken: Personal access token. Required if authType set to Pat @@ -90,7 +90,7 @@ Supported functional options include: - WithClientQueryTimeout( Duration). Bounds client-side statement execution on the kernel backend. Omitted preserves the legacy 600s ceiling; zero or the maximum Duration is unlimited. Requires WithUseKernel(true). Optional - WithUserAgentEntry( string). Used to identify partners. Optional - WithCloudFetch (bool). Used to enable cloud fetch for the query execution. Default is true. Optional - - WithMaxDownloadThreads ( int). Sets up the max number of concurrent workers for cloud fetch. Default is 10; a value <= 0 logs a warning and uses the default. Optional + - WithMaxDownloadThreads ( int). Sets the CloudFetch download window on both backends. When unset, defaults to 10 on Thrift and 16 on the kernel; an explicit value <= 0 logs a warning and uses 10. Optional - WithAuthenticator ( auth.Authenticator). Sets up authentication. Required if neither access token or client credentials are provided. - WithClientCredentials( string, string). Sets up Oauth M2M authentication. - WithUseKernel( bool). Routes execution through the SEA-via-kernel backend instead of Thrift. Requires a build with -tags databricks_kernel (CGO_ENABLED=1); the default build returns a clear error. Default is false. See the kernel-backend section below. Optional @@ -274,13 +274,12 @@ WithKernel* prefix marks them experimental): kernel's default budget (900s). - WithKernelMaxConnections(n) sets the maximum number of idle HTTP connections retained per host. The default is 100; n must be positive. - - WithKernelMaxChunksInMemory(n) bounds how many decompressed CloudFetch chunks the - kernel holds in memory at once — the knob that trades large-result download - throughput for peak RSS. Lower it (e.g. 4) to cap memory on wide, row-heavy result - sets; raise it for more parallelism at higher memory. It is forwarded as the - kernel's client-only cloudfetch_max_chunks_in_memory session conf, stripped before - the SEA wire so it never reaches the server. A value <= 0 keeps the kernel's - default (16). + - WithKernelMaxChunksInMemory(n) bounds downloading and queued CloudFetch chunks + ahead of consumption, excluding consumer buffers. It is not a byte limit. + It is forwarded as the kernel's client-only cloudfetch_max_chunks_in_memory + session conf, stripped before the SEA wire so it never reaches the server. + A value <= 0 keeps an explicit cloudfetch_max_chunks_in_memory session + parameter, maxDownloadThreads or the kernel default of 16 when unset. Setting any of these without WithUseKernel fails Connect with an error wrapping the sentinel ErrRequiresKernelBackend, detectable with errors.Is. diff --git a/internal/config/config.go b/internal/config/config.go index de0a463f..f9c5143e 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -117,10 +117,10 @@ type KernelExperimentalConfig struct { // host by the kernel. Nil keeps the kernel default (100). MaxConnections *int - // MaxChunksInMemory bounds how many decompressed CloudFetch chunks the kernel - // holds in memory at once (WithKernelMaxChunksInMemory) — the knob that trades - // throughput for peak RSS on large result sets. Zero = keep the kernel default - // (16). A positive value is forwarded as the client-only + // MaxChunksInMemory bounds downloading and queued CloudFetch chunks ahead of + // consumption, excluding consumer buffers. Zero keeps an explicit session + // parameter, WithMaxDownloadThreads or the kernel default of 16. + // A positive value is forwarded as the client-only // "cloudfetch_max_chunks_in_memory" session conf, which the kernel folds into // its result config and strips before the SEA wire; it is kernel-only because // the Thrift path has no such in-memory-chunk knob. @@ -514,6 +514,7 @@ func ParseDSN(dsn string) (UserConfig, error) { return UserConfig{}, err } ucfg.MaxDownloadThreads = numThreads + ucfg.MaxDownloadThreadsSet = true } // Metric View Metadata parameter @@ -792,6 +793,7 @@ const DefaultMaxDownloadThreads = 10 type CloudFetchConfig struct { UseCloudFetch bool MaxDownloadThreads int + MaxDownloadThreadsSet bool MaxFilesInMemory int MinTimeToExpiry time.Duration CloudFetchSpeedThresholdMbps float64 // Minimum download speed in MBps before WARN logging (default: 0.1) @@ -824,6 +826,7 @@ func (cfg CloudFetchConfig) DeepCopy() CloudFetchConfig { return CloudFetchConfig{ UseCloudFetch: cfg.UseCloudFetch, MaxDownloadThreads: cfg.MaxDownloadThreads, + MaxDownloadThreadsSet: cfg.MaxDownloadThreadsSet, MaxFilesInMemory: cfg.MaxFilesInMemory, MinTimeToExpiry: cfg.MinTimeToExpiry, CloudFetchSpeedThresholdMbps: cfg.CloudFetchSpeedThresholdMbps, diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 96fd5145..518ccaf2 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -297,6 +297,7 @@ func TestParseConfig(t *testing.T) { CloudFetchConfig: CloudFetchConfig{ UseCloudFetch: true, MaxDownloadThreads: 15, + MaxDownloadThreadsSet: true, MaxFilesInMemory: 10, CloudFetchSpeedThresholdMbps: 0.1, }, @@ -326,6 +327,7 @@ func TestParseConfig(t *testing.T) { CloudFetchConfig: CloudFetchConfig{ UseCloudFetch: true, MaxDownloadThreads: 15, + MaxDownloadThreadsSet: true, MaxFilesInMemory: 10, CloudFetchSpeedThresholdMbps: 0.1, }, @@ -850,6 +852,10 @@ func TestUserConfig_DeepCopy(t *testing.T) { SessionParams: map[string]string{"a": "32", "b": "4"}, UseKernel: true, WarehouseID: "wh-abc123", + CloudFetchConfig: CloudFetchConfig{ + MaxDownloadThreads: 10, + MaxDownloadThreadsSet: true, + }, } cfg_copy := cfg.DeepCopy() diff --git a/kernel_config.go b/kernel_config.go index 76f70a33..21a9f3cb 100644 --- a/kernel_config.go +++ b/kernel_config.go @@ -6,6 +6,7 @@ import ( "fmt" "net/url" "strconv" + "strings" "time" "github.com/databricks/databricks-sql-go/auth/noop" @@ -180,6 +181,16 @@ func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config { // An explicit true below opts into native geospatial values. EnableGeoSpatialSupport: false, } + hasChunkLimit := false + for key := range kc.SessionConf { + if strings.EqualFold(key, config.KernelMaxChunksInMemoryConfKey) { + hasChunkLimit = true + break + } + } + if cfg.MaxDownloadThreadsSet && !hasChunkLimit { + kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] = strconv.Itoa(cfg.MaxDownloadThreads) + } if !cfg.UseCloudFetch { enabled := false kc.CloudFetchEnabled = &enabled @@ -209,7 +220,7 @@ func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config { // Injected into the kernel backend's own SessionConf ONLY (not via // EffectiveSessionParams, which both backends share) as the client-only key // the kernel reads and strips before the SEA wire — so it never leaks to the - // server or the Thrift path. Zero/negative keeps the kernel default (16). + // server or the Thrift path. Zero/negative keeps the standard setting or native default. if ke.MaxChunksInMemory > 0 { // EffectiveSessionParams returned a fresh map, so mutating it is safe. kc.SessionConf[config.KernelMaxChunksInMemoryConfKey] = strconv.Itoa(ke.MaxChunksInMemory) diff --git a/kernel_config_test.go b/kernel_config_test.go index e7a44b9b..d8cd3c0c 100644 --- a/kernel_config_test.go +++ b/kernel_config_test.go @@ -477,7 +477,8 @@ var kernelConfigFieldDisposition = map[string]string{ // CloudFetch internally (below the C ABI); each option is // classified individually so a new CloudFetch option can't slip the guard. "UseCloudFetch": "forwarded", - "MaxDownloadThreads": "inert", + "MaxDownloadThreads": "forwarded", + "MaxDownloadThreadsSet": "forwarded", "MaxFilesInMemory": "inert", "MinTimeToExpiry": "inert", "CloudFetchSpeedThresholdMbps": "inert", @@ -732,20 +733,56 @@ func TestBuildKernelConfig(t *testing.T) { } }) - t.Run("MaxChunksInMemory unset leaves the key absent (kernel default)", func(t *testing.T) { - c := baseKernelConfig() // KernelExperimental nil - kc := buildKernelConfig(c, kernel.Auth{Mode: kernel.AuthPAT, Token: "dapi-x"}) - if _, ok := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]; ok { - t.Errorf("SessionConf should not carry %q when unset (kernel keeps its default)", config.KernelMaxChunksInMemoryConfKey) - } - // A zero/negative explicit value is also a no-op. - c.KernelExperimental = &config.KernelExperimentalConfig{MaxChunksInMemory: 0} - kc = buildKernelConfig(c, kernel.Auth{Mode: kernel.AuthPAT, Token: "dapi-x"}) - if _, ok := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]; ok { - t.Errorf("SessionConf should not carry %q for a zero value", config.KernelMaxChunksInMemoryConfKey) + t.Run("unset download window keeps the kernel default", func(t *testing.T) { + c := baseKernelConfig() + for _, explicit := range []int{0, -1} { + c.KernelExperimental = &config.KernelExperimentalConfig{MaxChunksInMemory: explicit} + kc := buildKernelConfig(c, kernel.Auth{Mode: kernel.AuthPAT, Token: "dapi-x"}) + if _, ok := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]; ok { + t.Fatal("unset download window must leave the native default in effect") + } } }) + for _, tc := range []struct { + name string + threads int + rawKey string + explicit int + wantKey, want string + }{ + {"explicit ten", 10, "", 0, config.KernelMaxChunksInMemoryConfKey, "10"}, + {"single", 1, "", 0, config.KernelMaxChunksInMemoryConfKey, "1"}, + {"parallel", 4, "", 0, config.KernelMaxChunksInMemoryConfKey, "4"}, + {"above ceiling", 1000, "", 0, config.KernelMaxChunksInMemoryConfKey, "1000"}, + {"raw override", 4, config.KernelMaxChunksInMemoryConfKey, 0, config.KernelMaxChunksInMemoryConfKey, "2"}, + {"uppercase override", 4, "CLOUDFETCH_MAX_CHUNKS_IN_MEMORY", 0, "CLOUDFETCH_MAX_CHUNKS_IN_MEMORY", "2"}, + {"explicit override", 4, "CLOUDFETCH_MAX_CHUNKS_IN_MEMORY", 3, config.KernelMaxChunksInMemoryConfKey, "3"}, + } { + t.Run("download window "+tc.name, func(t *testing.T) { + c := baseKernelConfig() + WithMaxDownloadThreads(tc.threads)(c) + if tc.rawKey != "" { + c.SessionParams = map[string]string{tc.rawKey: "2"} + } + if tc.explicit > 0 { + c.KernelExperimental = &config.KernelExperimentalConfig{MaxChunksInMemory: tc.explicit} + } + kc := buildKernelConfig(c, kernel.Auth{Mode: kernel.AuthPAT, Token: "dapi-x"}) + if got := kc.SessionConf[tc.wantKey]; got != tc.want { + t.Errorf("window=%q, want %q", got, tc.want) + } + if tc.rawKey != "" && c.SessionParams[tc.rawKey] != "2" { + t.Fatal("mutated caller session parameters") + } + if tc.rawKey == "CLOUDFETCH_MAX_CHUNKS_IN_MEMORY" && tc.explicit == 0 { + if _, ok := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]; ok { + t.Fatal("default shadows uppercase override") + } + } + }) + } + t.Run("MaxChunksInMemory is not leaked to the shared server params", func(t *testing.T) { // The knob is kernel-only: it must NOT appear in EffectiveSessionParams // (which the Thrift path also sends to the server). diff --git a/telemetry/request.go b/telemetry/request.go index f2d50d00..397f8bfe 100644 --- a/telemetry/request.go +++ b/telemetry/request.go @@ -96,7 +96,7 @@ type DriverConnectionParameters struct { // what a customer actually configured so a hung-connect or large-result-OOM // report can be diagnosed from telemetry, not just a debug log. RetryMaxAttempts // / RetryOverallTimeoutMs zero means the backend default is in effect; - // MaxChunksInMemory zero means the kernel default (16). + // MaxChunksInMemory zero means the value was not reported. RetryMaxAttempts int32 `json:"retry_max_attempts,omitempty"` RetryOverallTimeoutMs int64 `json:"retry_overall_timeout_ms,omitempty"` MaxChunksInMemory int32 `json:"max_chunks_in_memory,omitempty"`