Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
- Fix the module-root package conflict when building, linting, or running Gazelle with the `tools` build tag (databricks/databricks-sql-go#162).
- **Behavior change:** Report declared DECIMAL precision and scale through `sql.ColumnType.DecimalSize()` on Thrift and SEA, including zero-row results. Tools such as ORM migrators can now act on these qualifiers (databricks/databricks-sql-go#500).
- **Behavior change:** Support Arrow TIME results on Thrift and SEA as UTC `time.Time` values on `0001-01-01`, preserving precision and using clock-only nested JSON. TIME metadata now reports `TIME`/`time.Time` with no length, including empty and NULL-only results; previously Thrift reported `STRING`/`string` with unbounded length, and SEA reported an empty name with `*interface{}` (databricks/databricks-sql-go#505).
- **Behavior change:** Explicit `WithMaxDownloadThreads` / `maxDownloadThreads` settings tuned for Thrift also control kernel buffering and can change memory use and throughput. The kernel window defaults to 16 when unset; explicit chunk-setting overrides take precedence.
- **Breaking change:** Upgrade Apache Arrow Go from v12 to v18.8.0 (`github.com/apache/arrow-go/v18`). The public `rows.ArrowBatchIterator` now uses v18 `arrow.RecordBatch` and `*arrow.Schema` types; callers using Arrow batches must migrate their Arrow imports and dependent code. This requires a major driver release (v2.0.0), including a `/v2` driver module path before publication.
- Fix CloudFetch reads failing with `row number 0 is not contained in any arrow batch: EOF` when `WithMaxDownloadThreads` / `maxDownloadThreads` is 0 or negative. Such values now log a warning and use the default of 10 (databricks/databricks-sql-go#439).
- **Behavior change:** On the Thrift backend, `ExecuteStatement` is now retried (per `WithRetries`) when the connection to the server or proxy can't be opened, for example connection refused or a connect timeout, because the statement was never sent. Failures after the connection exists are still not retried, so a statement never runs twice. With default retry settings, a refused connection now fails after about 15 seconds instead of immediately (databricks/databricks-sql-go#423).
Expand Down
13 changes: 10 additions & 3 deletions CONNECTION_PARAMETERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ Thrift options where available.
|---|---|:---:|:---:|---|---|
| `cloudfetch_enabled` | `WithCloudFetch` | ⚠️ | ✅ | `true` | Set `false` to fetch complete results without cloud-storage downloads. Use `WithCloudFetch` for the standard option on either backend. |
| `cloudfetch_link_prefetch_window` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `10` | Number of presigned Cloud Fetch links prefetched ahead of the consumer. |
| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | `16` | Maximum decompressed Cloud Fetch chunks held in memory. Values above 256 are clamped with a warning, not rejected. |
| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | `maxDownloadThreads` when set; otherwise `16` | Downloading and queued chunks ahead of consumption; excludes consumer buffers. Values above 256 are clamped with a warning. |
| `inline_max_chunks_in_memory` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `4` | Inline-Arrow prefetch window; `1` disables prefetching. |
| `complex_types_as_json` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `false` | Render ARRAY / MAP / STRUCT / VARIANT / GEOMETRY / GEOGRAPHY as JSON text. |
| `intervals_as_string` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `false` | Render INTERVAL / DURATION values as canonical Databricks text. |
Expand Down Expand Up @@ -175,8 +175,15 @@ an internal setting) and non-Arrow values also retain legacy session-location pa
| DSN parameter | Connector option | Thrift | Kernel | Default | Notes |
|---|---|:---:|:---:|---|---|
| `useCloudFetch` | `WithCloudFetch` | ✅ | ✅ | `true` | Set false to fetch complete results through the SQL connection without direct cloud-storage access. |
| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ⚠️ | `10` | Concurrent download goroutines (Thrift). A value <= 0 logs a warning and uses the default. Inert on the kernel path. |
| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. |
| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ✅ | Thrift `10`; kernel `16` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. An explicit value <= 0 warns and uses 10. |
| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | `maxDownloadThreads` when set; otherwise `16` | Positive values override the standard setting and raw chunk session parameters. |

The kernel snapshots these options when connecting. A positive
`WithKernelMaxChunksInMemory` wins over a case-insensitive
`cloudfetch_max_chunks_in_memory` session parameter, which wins over
`WithMaxDownloadThreads`. Values above 256 are clamped. Invalid raw chunk
parameters warn and retain the kernel default of 16. These settings count
chunks, not bytes, and do not include consumer or application buffers.

## TLS

Expand Down
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -328,8 +328,15 @@ parallel via cloud storage
| DSN parameter | Connector option | Protocol | Default | Description |
|---|---|---|---|---|
| `useCloudFetch` | `WithCloudFetch` | Both | `true` | Enable Cloud Fetch. Set to false to fetch results through the SQL connection without direct cloud-storage access. |
| `maxDownloadThreads` | `WithMaxDownloadThreads` | Thrift only (inert on kernel) | `10` | Concurrent download goroutines (Thrift). A value <= 0 logs a warning and uses the default. Inert on the kernel path. |
| | `WithKernelMaxChunksInMemory(n)` | SEA only | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. |
| `maxDownloadThreads` | `WithMaxDownloadThreads` | Both | Thrift `10`; kernel `16` | CloudFetch download window. On the kernel, bounds downloading and queued chunks ahead of consumption, excluding consumer buffers. Not a byte limit. An explicit value <= 0 warns and uses 10. |
| | `WithKernelMaxChunksInMemory(n)` | SEA only | `maxDownloadThreads` when set; otherwise `16` | Positive values override the standard setting and raw chunk session parameters. |

The kernel snapshots these options when connecting. A positive
`WithKernelMaxChunksInMemory` wins over a case-insensitive
`cloudfetch_max_chunks_in_memory` session parameter, which wins over
`WithMaxDownloadThreads`. Values above 256 are clamped. Invalid raw chunk
parameters warn and retain the kernel default of 16. These settings count
chunks, not bytes, and do not include consumer or application buffers.

On the Thrift backend:

Expand Down
20 changes: 13 additions & 7 deletions connector.go
Original file line number Diff line number Diff line change
Expand Up @@ -547,11 +547,17 @@ func WithCloudFetch(useCloudFetch bool) ConnOption {
}
}

// WithMaxDownloadThreads sets up maximum download threads for cloud fetch. Default is 10.
// A value <= 0 logs a warning and uses the default.
// WithMaxDownloadThreads sets up maximum download threads for cloud fetch.
// When unset, Thrift defaults to 10 and the kernel defaults to 16.
// An explicit value <= 0 logs a warning and uses 10.
// On the kernel backend it bounds downloading and queued chunks ahead of
// consumption, excluding consumer buffers; it is not a byte limit. Values above
// 256 are clamped. Configuration is fixed when connecting. An explicit chunk
// session parameter or positive WithKernelMaxChunksInMemory takes precedence.
func WithMaxDownloadThreads(numThreads int) ConnOption {
return func(c *config.Config) {
c.MaxDownloadThreads = numThreads
c.MaxDownloadThreadsSet = true
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
}
}

Expand Down Expand Up @@ -837,11 +843,11 @@ func WithKernelMaxConnections(n int) ConnOption {
}
}

// WithKernelMaxChunksInMemory bounds how many decompressed CloudFetch chunks the
// kernel holds in memory at once on the kernel backend — the knob that trades
// large-result throughput for peak RSS. Lower it (e.g. 4) to cap memory on wide,
// row-heavy result sets; raise it for more download parallelism at higher memory.
// A value <= 0 (the default) leaves the kernel's built-in default (16) in place.
// WithKernelMaxChunksInMemory bounds downloading and queued CloudFetch chunks
// ahead of consumption on the kernel backend. Consumer and application buffers
// are additional; this is not a byte limit.
// A value <= 0 (the default) leaves WithMaxDownloadThreads, an explicit
// cloudfetch_max_chunks_in_memory session parameter or the kernel default in effect.
//
// It is forwarded as the kernel's client-only "cloudfetch_max_chunks_in_memory"
// session conf, which the kernel applies to its result config and strips before
Expand Down
29 changes: 29 additions & 0 deletions connector_cloudfetch_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/ipc"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/databricks/databricks-sql-go/internal/backend/kernel"
"github.com/databricks/databricks-sql-go/internal/cli_service"
"github.com/databricks/databricks-sql-go/internal/config"
"github.com/databricks/databricks-sql-go/logger"
Expand Down Expand Up @@ -61,6 +62,7 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) {
{value: -1, want: config.DefaultMaxDownloadThreads, wantWarn: true},
{value: 0, want: config.DefaultMaxDownloadThreads, wantWarn: true},
{value: 1, want: 1},
{value: 10, want: 10},
{value: 15, want: 15},
{value: 100000, want: 100000},
}
Expand All @@ -76,6 +78,9 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) {
)
require.NoError(t, err)
assert.Equal(t, tc.want, con.(*connector).cfg.MaxDownloadThreads)
assert.True(t, con.(*connector).cfg.MaxDownloadThreadsSet)
kc := buildKernelConfig(con.(*connector).cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"})
assert.Equal(t, strconv.Itoa(tc.want), kc.SessionConf[config.KernelMaxChunksInMemoryConfKey])
assertMaxDownloadThreadsWarning(t, logs.String(), tc.value, tc.wantWarn)
})

Expand All @@ -85,6 +90,9 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) {
con, err := (&databricksDriver{}).OpenConnector(dsn)
require.NoError(t, err)
assert.Equal(t, tc.want, con.(*connector).cfg.MaxDownloadThreads)
assert.True(t, con.(*connector).cfg.MaxDownloadThreadsSet)
kc := buildKernelConfig(con.(*connector).cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"})
assert.Equal(t, strconv.Itoa(tc.want), kc.SessionConf[config.KernelMaxChunksInMemoryConfKey])
assertMaxDownloadThreadsWarning(t, logs.String(), tc.value, tc.wantWarn)
})
}
Expand All @@ -94,10 +102,31 @@ func TestMaxDownloadThreadsNonPositiveDegradesToDefault(t *testing.T) {
con, err := NewConnector(WithServerHostname("example.cloud.databricks.com"))
require.NoError(t, err)
assert.Equal(t, config.DefaultMaxDownloadThreads, con.(*connector).cfg.MaxDownloadThreads)
assert.False(t, con.(*connector).cfg.MaxDownloadThreadsSet)
assert.Empty(t, maxDownloadThreadsWarnings(logs.String()))
})
}

func TestMaxDownloadThreadsDSNPresence(t *testing.T) {
for _, tc := range []struct {
query string
set bool
}{
{"", false},
Comment thread
aakash-saravanan-db marked this conversation as resolved.
Resolved
{"?maxDownloadThreads=", false},
} {
t.Run(tc.query, func(t *testing.T) {
con, err := (&databricksDriver{}).OpenConnector("token:secret@example.cloud.databricks.com:443/sql/1.0/warehouses/abc" + tc.query)
require.NoError(t, err)
cfg := con.(*connector).cfg.DeepCopy()
assert.Equal(t, tc.set, cfg.MaxDownloadThreadsSet)
kc := buildKernelConfig(cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "token"})
_, present := kc.SessionConf[config.KernelMaxChunksInMemoryConfKey]
assert.Equal(t, tc.set, present)
})
}
}

func assertMaxDownloadThreadsWarning(t *testing.T, logs string, value int, wantWarn bool) {
t.Helper()
warnings := maxDownloadThreadsWarnings(logs)
Expand Down
1 change: 1 addition & 0 deletions connector_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ func TestNewConnector(t *testing.T) {
expectedCloudFetchConfig := config.CloudFetchConfig{
UseCloudFetch: true,
MaxDownloadThreads: 15,
MaxDownloadThreadsSet: true,
MaxFilesInMemory: 10,
MinTimeToExpiry: 0 * time.Second,
CloudFetchSpeedThresholdMbps: 0.1,
Expand Down
17 changes: 8 additions & 9 deletions doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Supported optional connection parameters can be specified in param=value and inc
- timeout: Adds timeout (in seconds) for the server query execution. Default is no timeout
- userAgentEntry: Used to identify partners. Set as a string with format <isv-name+product-name>
- useCloudFetch: Used to enable cloud fetch for the query execution. Default is true
- maxDownloadThreads: Sets up the max number of concurrent workers for cloud fetch. Default is 10; a value <= 0 logs a warning and uses the default
- maxDownloadThreads: Sets the CloudFetch download window on both backends. When unset, defaults to 10 on Thrift and 16 on the kernel; an explicit value <= 0 logs a warning and uses 10
- useArrowNativeDecimal: Returns DECIMAL columns as native Arrow decimal128 (via GetArrowBatches); when scanned through database/sql they are returned as lossless strings. Default is false
- authType: Specifies the desired authentication type. Valid values are: Pat, OauthM2M, OauthU2M
- accessToken: Personal access token. Required if authType set to Pat
Expand Down Expand Up @@ -90,7 +90,7 @@ Supported functional options include:
- WithClientQueryTimeout(<timeout> Duration). Bounds client-side statement execution on the kernel backend. Omitted preserves the legacy 600s ceiling; zero or the maximum Duration is unlimited. Requires WithUseKernel(true). Optional
- WithUserAgentEntry(<isv-name+product-name> string). Used to identify partners. Optional
- WithCloudFetch (bool). Used to enable cloud fetch for the query execution. Default is true. Optional
- WithMaxDownloadThreads (<num_threads> int). Sets up the max number of concurrent workers for cloud fetch. Default is 10; a value <= 0 logs a warning and uses the default. Optional
- WithMaxDownloadThreads (<num_threads> int). Sets the CloudFetch download window on both backends. When unset, defaults to 10 on Thrift and 16 on the kernel; an explicit value <= 0 logs a warning and uses 10. Optional
- WithAuthenticator (<authenticator> auth.Authenticator). Sets up authentication. Required if neither access token or client credentials are provided.
- WithClientCredentials(<clientID> string, <clientSecret> string). Sets up Oauth M2M authentication.
- WithUseKernel(<useKernel> bool). Routes execution through the SEA-via-kernel backend instead of Thrift. Requires a build with -tags databricks_kernel (CGO_ENABLED=1); the default build returns a clear error. Default is false. See the kernel-backend section below. Optional
Expand Down Expand Up @@ -274,13 +274,12 @@ WithKernel* prefix marks them experimental):
kernel's default budget (900s).
- WithKernelMaxConnections(n) sets the maximum number of idle HTTP connections
retained per host. The default is 100; n must be positive.
- WithKernelMaxChunksInMemory(n) bounds how many decompressed CloudFetch chunks the
kernel holds in memory at once — the knob that trades large-result download
throughput for peak RSS. Lower it (e.g. 4) to cap memory on wide, row-heavy result
sets; raise it for more parallelism at higher memory. It is forwarded as the
kernel's client-only cloudfetch_max_chunks_in_memory session conf, stripped before
the SEA wire so it never reaches the server. A value <= 0 keeps the kernel's
default (16).
- WithKernelMaxChunksInMemory(n) bounds downloading and queued CloudFetch chunks
ahead of consumption, excluding consumer buffers. It is not a byte limit.
It is forwarded as the kernel's client-only cloudfetch_max_chunks_in_memory
session conf, stripped before the SEA wire so it never reaches the server.
A value <= 0 keeps an explicit cloudfetch_max_chunks_in_memory session
parameter, maxDownloadThreads or the kernel default of 16 when unset.

Setting any of these without WithUseKernel fails Connect with an error wrapping the
sentinel ErrRequiresKernelBackend, detectable with errors.Is.
Expand Down
11 changes: 7 additions & 4 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -117,10 +117,10 @@ type KernelExperimentalConfig struct {
// host by the kernel. Nil keeps the kernel default (100).
MaxConnections *int

// MaxChunksInMemory bounds how many decompressed CloudFetch chunks the kernel
// holds in memory at once (WithKernelMaxChunksInMemory) — the knob that trades
// throughput for peak RSS on large result sets. Zero = keep the kernel default
// (16). A positive value is forwarded as the client-only
// MaxChunksInMemory bounds downloading and queued CloudFetch chunks ahead of
// consumption, excluding consumer buffers. Zero keeps an explicit session
// parameter, WithMaxDownloadThreads or the kernel default of 16.
// A positive value is forwarded as the client-only
// "cloudfetch_max_chunks_in_memory" session conf, which the kernel folds into
// its result config and strips before the SEA wire; it is kernel-only because
// the Thrift path has no such in-memory-chunk knob.
Expand Down Expand Up @@ -514,6 +514,7 @@ func ParseDSN(dsn string) (UserConfig, error) {
return UserConfig{}, err
}
ucfg.MaxDownloadThreads = numThreads
ucfg.MaxDownloadThreadsSet = true
}

// Metric View Metadata parameter
Expand Down Expand Up @@ -792,6 +793,7 @@ const DefaultMaxDownloadThreads = 10
type CloudFetchConfig struct {
UseCloudFetch bool
MaxDownloadThreads int
MaxDownloadThreadsSet bool
MaxFilesInMemory int
MinTimeToExpiry time.Duration
CloudFetchSpeedThresholdMbps float64 // Minimum download speed in MBps before WARN logging (default: 0.1)
Expand Down Expand Up @@ -824,6 +826,7 @@ func (cfg CloudFetchConfig) DeepCopy() CloudFetchConfig {
return CloudFetchConfig{
UseCloudFetch: cfg.UseCloudFetch,
MaxDownloadThreads: cfg.MaxDownloadThreads,
MaxDownloadThreadsSet: cfg.MaxDownloadThreadsSet,
MaxFilesInMemory: cfg.MaxFilesInMemory,
MinTimeToExpiry: cfg.MinTimeToExpiry,
CloudFetchSpeedThresholdMbps: cfg.CloudFetchSpeedThresholdMbps,
Expand Down
6 changes: 6 additions & 0 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,7 @@ func TestParseConfig(t *testing.T) {
CloudFetchConfig: CloudFetchConfig{
UseCloudFetch: true,
MaxDownloadThreads: 15,
MaxDownloadThreadsSet: true,
MaxFilesInMemory: 10,
CloudFetchSpeedThresholdMbps: 0.1,
},
Expand Down Expand Up @@ -326,6 +327,7 @@ func TestParseConfig(t *testing.T) {
CloudFetchConfig: CloudFetchConfig{
UseCloudFetch: true,
MaxDownloadThreads: 15,
MaxDownloadThreadsSet: true,
MaxFilesInMemory: 10,
CloudFetchSpeedThresholdMbps: 0.1,
},
Expand Down Expand Up @@ -850,6 +852,10 @@ func TestUserConfig_DeepCopy(t *testing.T) {
SessionParams: map[string]string{"a": "32", "b": "4"},
UseKernel: true,
WarehouseID: "wh-abc123",
CloudFetchConfig: CloudFetchConfig{
MaxDownloadThreads: 10,
MaxDownloadThreadsSet: true,
},
}

cfg_copy := cfg.DeepCopy()
Expand Down
Loading
Loading