Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/go.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,9 @@ jobs:
- name: Test-Race
run: make test-race

- name: Vet live e2e tests (compile only)
run: CGO_ENABLED=0 go vet -tags e2e .

- name: Build
run: make linux

Expand Down Expand Up @@ -256,3 +259,8 @@ jobs:
# the separate nightly-e2e workflow.
- name: Build kernel lib + run tagged tests
run: make test-kernel

- name: Vet kernel live e2e tests (compile only)
env:
CGO_LDFLAGS: -L${{ github.workspace }}/internal/backend/kernel/lib/linux_amd64 -l:libdatabricks_sql_kernel.a
run: CGO_ENABLED=1 go vet -tags databricks_kernel,e2e .
10 changes: 5 additions & 5 deletions .github/workflows/nightly-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,8 @@ jobs:
DATABRICKS_PECOTESTING_TOKEN: ${{ secrets.DATABRICKS_PECOTESTING_TOKEN }}
DATABRICKS_PECOTESTING_TOKEN_PERSONAL: ${{ secrets.DATABRICKS_PECOTESTING_TOKEN_PERSONAL }}
run: |
go test -count=1 -v -timeout 15m \
-run 'TestE2EArrowBatchesSurviveQueryContextCancellation|TestE2ECloudFetchExactRowCount|TestE2ECloudFetchNonPositiveMaxDownloadThreads|TestE2EDecimalMetadata' .
go test -tags e2e -count=1 -v -timeout 15m \
-run 'TestE2EArrowBatchesSurviveQueryContextCancellation|TestE2ECloudFetchExactRowCount|TestE2ECloudFetchNonPositiveMaxDownloadThreads|TestE2EDecimalMetadata|^TestE2ETime(Results|ArrowResults)$' .

kernel-e2e:
name: E2E (kernel backend)
Expand Down Expand Up @@ -243,7 +243,7 @@ jobs:
DATABRICKS_PECOTESTING_TOKEN: ${{ secrets.DATABRICKS_PECOTESTING_TOKEN }}
DATABRICKS_PECOTESTING_TOKEN_PERSONAL: ${{ secrets.DATABRICKS_PECOTESTING_TOKEN_PERSONAL }}
run: |
CGO_ENABLED=1 go test -tags databricks_kernel -count=1 -v -timeout 30m \
CGO_ENABLED=1 go test -tags databricks_kernel,e2e -count=1 -v -timeout 30m \
-run 'TestKernelE2E|TestKernelThriftParity|TestKernelThriftColumnTypeParity|TestKernelParamsVsThrift' \
-skip 'TestKernelE2EM2M' .

Expand All @@ -265,6 +265,6 @@ jobs:
TEST_WAREHOUSE: reyden
DATABRICKS_REYDEN_HTTP_PATH: ${{ secrets.TEST_REYDEN_WAREHOUSE_HTTP_PATH }}
run: |
CGO_ENABLED=1 go test -tags databricks_kernel -count=1 -v -timeout 30m \
CGO_ENABLED=1 go test -tags databricks_kernel,e2e -count=1 -v -timeout 30m \
-run 'TestKernelE2E' \
-skip 'TestKernelE2EM2M' .
-skip 'TestKernelE2EM2M|^TestKernelE2ETimeResults$' .
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

- Fix the module-root package conflict when building, linting, or running Gazelle with the `tools` build tag (databricks/databricks-sql-go#162).
- **Behavior change:** Report declared DECIMAL precision and scale through `sql.ColumnType.DecimalSize()` on Thrift and SEA, including zero-row results. Tools such as ORM migrators can now act on these qualifiers (databricks/databricks-sql-go#500).
- **Behavior change:** Support Arrow TIME results on Thrift and SEA as UTC `time.Time` values on `0001-01-01`, preserving precision and using clock-only nested JSON. TIME metadata now reports `TIME`/`time.Time` with no length, including empty and NULL-only results; previously Thrift reported `STRING`/`string` with unbounded length, and SEA reported an empty name with `*interface{}` (databricks/databricks-sql-go#505).
- **Breaking change:** Upgrade Apache Arrow Go from v12 to v18.8.0 (`github.com/apache/arrow-go/v18`). The public `rows.ArrowBatchIterator` now uses v18 `arrow.RecordBatch` and `*arrow.Schema` types; callers using Arrow batches must migrate their Arrow imports and dependent code. This requires a major driver release (v2.0.0), including a `/v2` driver module path before publication.
- Fix CloudFetch reads failing with `row number 0 is not contained in any arrow batch: EOF` when `WithMaxDownloadThreads` / `maxDownloadThreads` is 0 or negative. Such values now log a warning and use the default of 10 (databricks/databricks-sql-go#439).
- **Behavior change:** On the Thrift backend, `ExecuteStatement` is now retried (per `WithRetries`) when the connection to the server or proxy can't be opened, for example connection refused or a connect timeout, because the statement was never sent. Failures after the connection exists are still not retried, so a statement never runs twice. With default retry settings, a refused connection now fails after about 15 seconds instead of immediately (databricks/databricks-sql-go#423).
Expand Down
29 changes: 29 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,35 @@ STRUCT and VARIANT (as JSON) are supported. BINARY is returned as `sql.RawBytes`
Both backends return GEOMETRY / GEOGRAPHY as WKT / EWKT by default.
Set `enableGeoSpatialSupport=true` on the kernel path to opt into EWKB.

| SQL type | Go value | NULL detection |
|---|---|---|
| `TIME` | `time.Time` on `0001-01-01` UTC | `sql.NullTime.Valid` or a nil `*time.Time`, never `IsZero()`; midnight is the zero time and JSON `omitzero` omits it. |

SQL TIME values in Arrow results scan into `time.Time` (or `any`) on the reference date
`0001-01-01` in UTC, preserving the Arrow second/millisecond/microsecond/nanosecond
precision. The date and location are carriers only; session time zones do not
shift the clock. Scanning into `string` uses `database/sql`'s RFC3339Nano format
(for example, `0001-01-01T09:15:00.123456Z`). NULL scans into `any` as `nil`;
use `sql.NullTime` or `*time.Time` for nullable typed values. Non-NULL midnight equals `time.Time{}`
and has `IsZero() == true`, so detect NULL with `sql.NullTime.Valid`, not `IsZero()`;
JSON fields tagged `omitzero` omit midnight. Do not call `.In(location)` on this
date-less value: it shifts the clock and may introduce historical timezone offsets.
To bind TIME in `CAST(? AS TIME)`, pass `value.Format("15:04:05.999999")`, not the
`time.Time`: `time.Time` binds as `TIMESTAMP` and can shift the clock in non-UTC sessions.
Column metadata reports `TIME` and `time.Time` with no length, even for empty or
NULL-only results. Nested TIME values in ARRAY / MAP / STRUCT JSON use clock-only
strings with trailing fractional zeros omitted, such as `"09:15:00.5"` rather than
`"09:15:00.500000"`; their text can differ from server-rendered TIME(p) values.
On Thrift, TIME column metadata requires a server-supplied Arrow schema;
without it, the legacy Thrift column metadata is retained.
On SEA, UTF-8 fields tagged TIME describe empty manifest schemas; non-empty TIME
is assumed to arrive as Arrow time32/time64, not UTF-8.

Live TIME tests require the `e2e` build tag: `go test -tags e2e -run '^TestE2ETimeResults$' .`
or `go test -tags databricks_kernel,e2e -run '^TestKernelE2ETimeResults$' .`.
They require the `DATABRICKS_PECOTESTING_*` credentials and a workspace with the TIME
beta enabled; missing credentials or unavailable TIME support fail the tests.

On Thrift, TIMESTAMP_NTZ handling requires native timestamp fields in the
server-supplied Arrow schema. Without that schema, NTZ still reports `TIMESTAMP`
and uses session-location handling. String timestamps (`UseArrowNativeTimestamp=false`,
Expand Down
6 changes: 6 additions & 0 deletions doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -536,6 +536,12 @@ STRING --> string

DATE --> time.Time

TIME --> time.Time (0001-01-01 UTC). Detect NULL with sql.NullTime.Valid or a nil
*time.Time, never IsZero(): non-NULL midnight is the zero time and JSON omitzero
omits it.
To bind TIME, pass value.Format("15:04:05.999999") to CAST(? AS TIME), not the
time.Time: time.Time parameters bind as TIMESTAMP and can shift in non-UTC sessions.

TIMESTAMP --> time.Time

DECIMAL(p,s) --> sql.RawBytes
Expand Down
19 changes: 13 additions & 6 deletions internal/arrowscan/arrowscan.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@
// - map → {"k0":v0,"k1":v1,...} (keys stringified)
// - struct → {"field0":v0,...}
// - nested NULL → null
// - time.Time → quoted .String() (matches the Thrift marshalScalar special-case)
// - DATE/TIMESTAMP → quoted time.Time.String() (matches Thrift)
// - TIME → quoted clock-only text, with trailing fractional zeros omitted
// - nested decimal → exact scale-applied JSON number literal (never a lossy
// float64), matching Thrift's marshalScalar → ValueString
// - float32 → native float32 (not widened to float64), so JSON renders
Expand All @@ -35,7 +36,7 @@ import (

// ScanCell extracts one cell as a driver.Value. Scalars map to their Go value:
// bool, all int/uint widths, float (native float32/float64), string, binary,
// date, timestamp, and top-level decimal (as an exact fixed-point string,
// date, time, timestamp, and top-level decimal (as an exact fixed-point string,
// matching the Thrift path — a float64 would lose precision beyond ~17 digits).
// Nested types (List/Map/Struct, and VARIANT which
// arrives nested) render to a JSON string byte-identical to the Thrift path.
Expand Down Expand Up @@ -210,6 +211,10 @@ func scanCell(col arrow.Array, row int, loc *time.Location, keys *StructKeyCache
return inLocation(c.Value(row).ToTime(), loc), nil
case *array.Date64:
return inLocation(c.Value(row).ToTime(), loc), nil
case *array.Time32:
return timeOfDayValue(int64(c.Value(row)), c.DataType().(*arrow.Time32Type).Unit), nil
case *array.Time64:
return timeOfDayValue(int64(c.Value(row)), c.DataType().(*arrow.Time64Type).Unit), nil
case *array.Timestamp:
dt, ok := col.DataType().(*arrow.TimestampType)
if !ok {
Expand Down Expand Up @@ -627,10 +632,8 @@ func writeScalarJSON(b *strings.Builder, v any) error {
return nil
}

// scalarForJSON returns the Go value used for a nested leaf that is not itself a
// container — today only a map key (values are written directly by writeJSON).
// It reuses ScanCell's scalar arm, so a decimal key renders via the exact-string
// path (writeJSONKey then quotes it), never a lossy float64.
// scalarForJSON converts nested scalar values and map keys. TIME uses clock-only
// text; decimals use ScanCell's exact-string path, never a lossy float64.
func scalarForJSON(col arrow.Array, row int, loc *time.Location) (any, error) {
if col.IsNull(row) {
return nil, nil
Expand All @@ -641,6 +644,10 @@ func scalarForJSON(col arrow.Array, row int, loc *time.Location) (any, error) {
return c.Value(row), nil
case *array.LargeString:
return c.Value(row), nil
case *array.Time32:
return timeOfDay(int64(c.Value(row)), c.DataType().(*arrow.Time32Type).Unit).Format("15:04:05.999999999"), nil
case *array.Time64:
return timeOfDay(int64(c.Value(row)), c.DataType().(*arrow.Time64Type).Unit).Format("15:04:05.999999999"), nil
}
// scanCell, not ScanCell: keyLoc already came from childLocation on the key
// Field, so the field-less zone-less fallback must not override it.
Expand Down
4 changes: 2 additions & 2 deletions internal/arrowscan/arrowscan_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,9 @@ func TestScanCellScalars(t *testing.T) {
// An unhandled arrow type must error, not return a silently wrong value.
// Duration/MonthInterval are now handled (see TestScanCellInterval); use a
// type with no scan arm.
b := array.NewTime32Builder(pool, &arrow.Time32Type{Unit: arrow.Second})
b := array.NewFixedSizeBinaryBuilder(pool, &arrow.FixedSizeBinaryType{ByteWidth: 1})
defer b.Release()
b.Append(1000)
b.Append([]byte{1})
arr := b.NewArray()
defer arr.Release()
if _, err := ScanCell(arr, 0, nil); err == nil {
Expand Down
18 changes: 17 additions & 1 deletion internal/arrowscan/coltype.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ var (
scanTypeFloat64 = reflect.TypeOf(float64(0))
scanTypeString = reflect.TypeOf("")
scanTypeDateTime = reflect.TypeOf(time.Time{})
scanTypeTime = reflect.TypeOf(timeOfDayValue(0, arrow.Nanosecond))
scanTypeRawBytes = reflect.TypeOf(sql.RawBytes{})
scanTypeUnknown = reflect.TypeOf(new(any)) // *interface{}, as on the Thrift path
)
Expand Down Expand Up @@ -77,6 +78,8 @@ func ColumnTypeInfoFor(dt arrow.DataType) ColumnTypeInfo {
return varLen("BINARY", scanTypeRawBytes)
case arrow.DATE32, arrow.DATE64:
return ColumnTypeInfo{DatabaseTypeName: "DATE", ScanType: scanTypeDateTime}
case arrow.TIME32, arrow.TIME64:
return ColumnTypeInfo{DatabaseTypeName: "TIME", ScanType: scanTypeTime}
case arrow.TIMESTAMP:
// TIMESTAMP and TIMESTAMP_NTZ both arrive as Arrow timestamp[us]. NTZ is
// decided by IsTimestampNTZField: the field's logical-type metadata first
Expand Down Expand Up @@ -147,12 +150,25 @@ func IsBinaryGeospatialField(field arrow.Field) bool {
return geospatial && field.Type.ID() == arrow.STRUCT
}

// ColumnTypeInfoForField preserves logical timestamp and geospatial types that
// ColumnTypeInfoForField preserves logical time, timestamp and geospatial types that
// the kernel's physical Arrow representation alone cannot distinguish.
func ColumnTypeInfoForField(field arrow.Field) ColumnTypeInfo {
if field.Type != nil && field.Type.ID() == arrow.TIMESTAMP {
return timestampColumnTypeInfo(field)
}
if field.Type != nil && (field.Type.ID() == arrow.STRING || field.Type.ID() == arrow.LARGE_STRING) {
for _, key := range [...]string{databricksTypeNameMetadataKey, sparkSQLNameMetadataKey} {
name, ok := field.Metadata.GetValue(key)
if !ok || strings.TrimSpace(name) == "" {
continue
}
switch strings.ToUpper(strings.TrimSpace(name)) {
case "TIME", "TIME(0)", "TIME(1)", "TIME(2)", "TIME(3)", "TIME(4)", "TIME(5)", "TIME(6)":
return ColumnTypeInfoFor(arrow.FixedWidthTypes.Time64ns)
}
break
}
}
name, geospatial := LogicalGeospatialType(field)
if !geospatial {
return ColumnTypeInfoFor(field.Type)
Expand Down
29 changes: 29 additions & 0 deletions internal/arrowscan/coltype_bench_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package arrowscan

import (
"testing"

"github.com/apache/arrow-go/v18/arrow"
)

func BenchmarkColumnTypeInfoForField(benchmark *testing.B) {
for _, field := range []arrow.Field{
{Name: "string", Type: arrow.BinaryTypes.String},
{Name: "int64", Type: arrow.PrimitiveTypes.Int64},
{Name: "timestamp", Type: arrow.FixedWidthTypes.Timestamp_us},
{Name: "TIME manifest", Type: arrow.BinaryTypes.String, Metadata: arrow.MetadataFrom(map[string]string{
databricksTypeNameMetadataKey: "TIME", sparkSQLNameMetadataKey: "TIME(6)",
})},
} {
benchmark.Run(field.Name, func(benchmark *testing.B) {
benchmark.ReportAllocs()
var info ColumnTypeInfo
for iteration := 0; iteration < benchmark.N; iteration++ {
info = ColumnTypeInfoForField(field)
}
if info.ScanType == nil {
benchmark.Fatal("missing scan type")
}
})
}
}
18 changes: 18 additions & 0 deletions internal/arrowscan/time.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
package arrowscan

import (
"database/sql/driver"
"time"

"github.com/apache/arrow-go/v18/arrow"
)

func timeOfDayValue(value int64, unit arrow.TimeUnit) driver.Value {
return timeOfDay(value, unit)
}

// timeOfDay carries TIME on 0001-01-01 UTC. Midnight equals time.Time{} and
// JSON omitzero omits it; detect NULL with sql.NullTime or *time.Time, not IsZero.
func timeOfDay(value int64, unit arrow.TimeUnit) time.Time {
return time.Time{}.Add(time.Duration(value) * unit.Multiplier())
}
109 changes: 109 additions & 0 deletions internal/arrowscan/time_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
package arrowscan

import (
"fmt"
"reflect"
"testing"
"time"

"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/stretchr/testify/require"
)

func TestScanTimeUnits(t *testing.T) {
for _, test := range []struct {
dataType arrow.DataType
clock string
nanos int
}{
{arrow.FixedWidthTypes.Time32s, "12:34:56", 0},
{arrow.FixedWidthTypes.Time32ms, "12:34:56.789", 789000000},
{arrow.FixedWidthTypes.Time64us, "12:34:56.789123", 789123000},
{arrow.FixedWidthTypes.Time64ns, "12:34:56.789123456", 789123456},
} {
t.Run(test.dataType.String(), func(t *testing.T) {
builder := array.NewBuilder(memory.DefaultAllocator, test.dataType)
defer builder.Release()
require.NoError(t, builder.AppendValueFromString(test.clock))
builder.AppendNull()
values := builder.NewArray()
defer values.Release()
got, err := ScanCell(values, 0, time.FixedZone("east", 9*3600))
require.NoError(t, err)
require.Equal(t, time.Date(1, time.January, 1, 12, 34, 56, test.nanos, time.UTC), got)
got, err = ScanCell(values, 1, nil)
require.NoError(t, err)
require.Nil(t, got)
})
}
}

func TestTimeNullOnlyMetadata(t *testing.T) {
for _, dataType := range []arrow.DataType{arrow.FixedWidthTypes.Time32s, arrow.FixedWidthTypes.Time32ms, arrow.FixedWidthTypes.Time64us, arrow.FixedWidthTypes.Time64ns} {
t.Run(dataType.String(), func(t *testing.T) {
builder := array.NewBuilder(memory.DefaultAllocator, dataType)
defer builder.Release()
builder.AppendNull()
values := builder.NewArray()
defer values.Release()
got, err := ScanCell(values, 0, nil)
require.NoError(t, err)
require.Nil(t, got)
metadata := ColumnTypeInfoFor(dataType)
require.Equal(t, "TIME", metadata.DatabaseTypeName)
require.Equal(t, reflect.TypeOf(time.Time{}), metadata.ScanType)
require.Zero(t, metadata.Length)
require.False(t, metadata.HasLength)
})
}
}

func TestTimeManifestFieldMetadata(t *testing.T) {
type metadataCase struct {
name string
kernelName string
sparkName string
wantTime bool
}
cases := []metadataCase{
{"manifest", "TIME", "TIME(6)", true},
{"kernel only", "TIME", "", true},
{"case and whitespace", " time ", "", true},
{"blank kernel name", " ", "TIME", true},
{"ordinary string", "STRING", "STRING", false},
{"no metadata", "", "", false},
{"timestamp", "", "TIMESTAMP", false},
{"similar name", "", "TIMEZONE", false},
{"unsupported precision", "", "TIME(7)", false},
{"kernel name takes precedence", "STRING", "TIME(6)", false},
}
for precision := range 7 {
name := fmt.Sprintf("TIME(%d)", precision)
cases = append(cases, metadataCase{name, "", name, true})
}
for _, dataType := range []arrow.DataType{arrow.BinaryTypes.String, arrow.BinaryTypes.LargeString} {
for _, test := range cases {
t.Run(dataType.String()+"/"+test.name, func(t *testing.T) {
metadata := make(map[string]string)
if test.kernelName != "" {
metadata[databricksTypeNameMetadataKey] = test.kernelName
}
if test.sparkName != "" {
metadata[sparkSQLNameMetadataKey] = test.sparkName
}
field := arrow.Field{Name: "clock", Type: dataType, Nullable: true, Metadata: arrow.MetadataFrom(metadata)}
want := ColumnTypeInfoFor(dataType)
if test.wantTime {
want = ColumnTypeInfoFor(arrow.FixedWidthTypes.Time64ns)
}
got := ColumnTypeInfoForField(field)
require.Equal(t, want.DatabaseTypeName, got.DatabaseTypeName)
require.Equal(t, want.ScanType, got.ScanType)
require.Equal(t, want.Length, got.Length)
require.Equal(t, want.HasLength, got.HasLength)
})
}
}
}
2 changes: 2 additions & 0 deletions internal/rows/arrowbased/arrowRows.go
Original file line number Diff line number Diff line change
Expand Up @@ -657,6 +657,8 @@ func (vcm *arrowValueContainerMaker) makeFieldValueContainer(field arrow.Field,

case *arrow.Date32Type:
return &dateValueContainer{location: location}, nil
case *arrow.Time32Type, *arrow.Time64Type:
return &timeValueContainer{}, nil

case *arrow.TimestampType:
// A TIMESTAMP_NTZ carries a wall clock that must not be shifted into the
Expand Down
Loading
Loading