Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Unreleased (v2.0.0 prototype)

- Render non-finite FLOAT / DOUBLE values inside ARRAY / MAP / STRUCT results as the JSON strings `"NaN"`, `"Infinity"`, and `"-Infinity"` on both backends instead of failing row iteration. SQL NULL remains `null`; top-level floats remain numeric (databricks/databricks-sql-go#573).
- Fix the module-root package conflict when building, linting, or running Gazelle with the `tools` build tag (databricks/databricks-sql-go#162).
- **Behavior change:** Report declared DECIMAL precision and scale through `sql.ColumnType.DecimalSize()` on Thrift and SEA, including zero-row results. Tools such as ORM migrators can now act on these qualifiers (databricks/databricks-sql-go#500).
- **Behavior change:** Support Arrow TIME results on Thrift and SEA as UTC `time.Time` values on `0001-01-01`, preserving precision and using clock-only nested JSON. TIME metadata now reports `TIME`/`time.Time` with no length, including empty and NULL-only results; previously Thrift reported `STRING`/`string` with unbounded length, and SEA reported an empty name with `*interface{}` (databricks/databricks-sql-go#505).
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,14 @@ STRUCT and VARIANT (as JSON) are supported. BINARY is returned as `sql.RawBytes`
Both backends return GEOMETRY / GEOGRAPHY as WKT / EWKT by default.
Set `enableGeoSpatialSupport=true` on the kernel path to opt into EWKB.

In ARRAY / MAP / STRUCT JSON, non-finite FLOAT and DOUBLE values render as the
strings `"NaN"`, `"Infinity"`, and `"-Infinity"`, distinct from SQL NULL (`null`).
Finite values remain JSON numbers. Use `any` or a custom decoder for fields that
can contain special values; `encoding/json` does not unmarshal these strings into
plain `float32` or `float64` fields. Top-level FLOAT / DOUBLE values remain numeric.
Non-finite floating-point MAP keys retain the Go spellings `"NaN"`, `"+Inf"`, and
`"-Inf"` already used by the kernel backend.

| SQL type | Go value | NULL detection |
|---|---|---|
| `TIME` | `time.Time` on `0001-01-01` UTC | `sql.NullTime.Valid` or a nil `*time.Time`, never `IsZero()`; midnight is the zero time and JSON `omitzero` omits it. |
Expand Down
6 changes: 5 additions & 1 deletion doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -558,7 +558,11 @@ INTERVAL (year-month) --> string
INTERVAL (day-time) --> string

For ARRAY, STRUCT, and MAP types, sql.Scan can cast sql.RawBytes to JSON string, which can be unmarshalled to Golang
arrays, maps, and structs. For example:
arrays, maps, and structs. Nested non-finite FLOAT and DOUBLE values are represented as the JSON strings
"NaN", "Infinity", and "-Infinity", distinct from SQL NULL (null). Finite values remain JSON numbers.
Use any or a custom decoder for fields that can contain special values; encoding/json cannot unmarshal
these strings into plain float fields. Top-level FLOAT and DOUBLE values remain numeric. Non-finite
floating-point MAP keys retain the kernel backend's Go spellings "NaN", "+Inf", and "-Inf". For example:

type structVal struct {
StringField string `json:"string_field"`
Expand Down
4 changes: 2 additions & 2 deletions internal/arrowscan/arrowscan.go
Original file line number Diff line number Diff line change
Expand Up @@ -587,6 +587,7 @@ func writeStructJSON(b *strings.Builder, s *array.Struct, row int, loc *time.Loc
// base64 "YWJj", NOT fmt "%v" [97 98 99]), then quote-wrap the result if it is
// not already a JSON string (numbers/bools become "7"/"true" keys).
func writeJSONKey(b *strings.Builder, v any) {
v = jsonfmt.MapKey(v)
// A time.Time key renders as its quoted .String() (same special-case as
// writeScalarJSON / the Thrift marshalScalar), not RFC3339.
if t, ok := v.(time.Time); ok {
Expand All @@ -610,8 +611,7 @@ func writeJSONKey(b *strings.Builder, v any) {
}

// writeScalarJSON writes a scalar leaf, mirroring the Thrift marshalScalar: a
// time.Time becomes a quoted .String(); everything else renders as json.Marshal
// would.
// time.Time becomes a quoted .String(); everything else uses jsonfmt.Append.
func writeScalarJSON(b *strings.Builder, v any) error {
if v == nil {
b.WriteString("null")
Expand Down
60 changes: 41 additions & 19 deletions internal/jsonfmt/jsonfmt.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
// Package jsonfmt renders scalar values as JSON, byte-identical to
// encoding/json.Marshal, for the nested-type renderers shared by the Thrift
// (arrowbased) and kernel (arrowscan) result paths.
// Package jsonfmt renders scalar values as JSON for the nested-type renderers
// shared by the Thrift (arrowbased) and kernel (arrowscan) result paths.
// It matches encoding/json.Marshal except that non-finite floats become the
// JSON strings "NaN", "Infinity", and "-Infinity" rather than errors.
//
// Those renderers marshal one scalar per nested element, so per-call overhead
// dominates. Go 1.27 routes encoding/json through the json/v2 engine by default,
// which made that per-call cost markedly higher; formatting the common scalar
// types directly avoids it. Anything not handled here falls back to json.Marshal,
// so output never diverges from it.
// types directly avoids it. Anything not handled here falls back to json.Marshal.
package jsonfmt

import (
Expand All @@ -16,8 +16,8 @@ import (
"unicode/utf8"
)

// Append appends the JSON encoding of v to dst, exactly as json.Marshal(v)
// would render it, and returns the extended slice.
// Append appends the JSON encoding of v to dst and returns the extended slice.
// Unlike json.Marshal, it quotes non-finite floats instead of rejecting them.
func Append(dst []byte, v any) ([]byte, error) {
switch t := v.(type) {
case nil:
Expand All @@ -43,14 +43,8 @@ func Append(dst []byte, v any) ([]byte, error) {
case uint64:
return strconv.AppendUint(dst, t, 10), nil
case float32:
if !isFinite(float64(t)) {
break
}
return appendFloat(dst, float64(t), 32), nil
case float64:
if !isFinite(t) {
break
}
return appendFloat(dst, t, 64), nil
case string:
if needsEscape(t) {
Expand All @@ -60,8 +54,7 @@ func Append(dst []byte, v any) ([]byte, error) {
dst = append(dst, t...)
return append(dst, '"'), nil
}
// NaN/Inf (json.Marshal's error), strings needing escapes, []byte (base64),
// and any other type.
// Strings needing escapes, []byte (base64), and any other type.
b, err := json.Marshal(v)
if err != nil {
return dst, err
Expand All @@ -72,13 +65,42 @@ func Append(dst []byte, v any) ([]byte, error) {
return append(dst, b...), nil
}

func isFinite(f float64) bool {
return !math.IsInf(f, 0) && !math.IsNaN(f)
// MapKey returns the scalar to encode for a MAP key. Non-finite float keys keep
// the Go spellings used by the kernel renderer before values supported them.
func MapKey(v any) any {
var f float64
switch value := v.(type) {
case float32:
f = float64(value)
case float64:
f = value
default:
return v
}
if math.IsNaN(f) {
return "NaN"
}
if math.IsInf(f, 0) {
if f < 0 {
return "-Inf"
}
return "+Inf"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we remove MapKey and use "NaN", "Infinity", and "-Infinity" consistently for both keys and values? Currently, a map containing positive infinity as both key and value renders as {"+Inf":"Infinity"}.

JDBC’s text rendering and Node’s object keys use Infinity/-Infinity; Python preserves native numeric keys. The +Inf/-Inf spelling comes from the kernel’s old formatting fallback, while Thrift previously errored on these keys. Using the same spelling throughout would simplify the contract; we can document the kernel key spelling change explicitly in the changelog.

}
return v
}

// appendFloat mirrors encoding/json's ES6-style float formatting: plain decimal
// unless the magnitude is below 1e-6 or at least 1e21, with exponents unpadded.
// appendFloat quotes non-finite values to keep valid SQL floats distinct from
// SQL NULL in JSON. Finite values retain encoding/json's ES6-style formatting.
func appendFloat(dst []byte, f float64, bits int) []byte {
if math.IsNaN(f) {
return append(dst, `"NaN"`...)
}
if math.IsInf(f, 0) {
if f < 0 {
return append(dst, `"-Infinity"`...)
}
return append(dst, `"Infinity"`...)
}
abs := math.Abs(f)
format := byte('f')
// Compare at the value's own width so float32 cutoffs match exactly.
Expand Down
54 changes: 50 additions & 4 deletions internal/jsonfmt/jsonfmt_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ func TestMarshalMatchesEncodingJSON(t *testing.T) {
math.MaxFloat64, math.SmallestNonzeroFloat64, -math.MaxFloat64,
float32(0), float32(math.Copysign(0, -1)), float32(3.14), float32(0.1), float32(1e21), float32(1e-6), float32(1e-7),
float32(math.MaxFloat32), float32(math.SmallestNonzeroFloat32),
math.NaN(), math.Inf(1), math.Inf(-1), float32(math.NaN()), float32(math.Inf(-1)),
"", "alpha", "with space", "a\"b", `back\slash`, "<tag>", "a&b", "tab\there", "nl\n", "\x00", "\x7f",
"héllo", "日本語", "

", "\xff\xfe", "emoji 😀",
[]byte("abc"), []byte(nil),
Expand Down Expand Up @@ -89,13 +88,59 @@ func TestMarshalRandomUnicodeStrings(t *testing.T) {
func TestMarshalRandomFloats(t *testing.T) {
r := rand.New(rand.NewSource(1)) //nolint:gosec // G404: deterministic test inputs, not security
for i := 0; i < 200000; i++ {
assertParity(t, math.Float64frombits(r.Uint64()))
assertParity(t, math.Float32frombits(r.Uint32()))
f64 := math.Float64frombits(r.Uint64())
if !math.IsNaN(f64) && !math.IsInf(f64, 0) {
assertParity(t, f64)
}
f32 := math.Float32frombits(r.Uint32())
if !math.IsNaN(float64(f32)) && !math.IsInf(float64(f32), 0) {
assertParity(t, f32)
}
assertParity(t, r.NormFloat64()*math.Pow(10, float64(r.Intn(50)-25)))
assertParity(t, float32(r.NormFloat64()*math.Pow(10, float64(r.Intn(30)-15))))
}
}

func TestAppendNonFiniteFloats(t *testing.T) {
cases := []struct {
name string
value any
want string
}{
{"float32_nan", float32(math.NaN()), `"NaN"`},
{"float32_infinity", float32(math.Inf(1)), `"Infinity"`},
{"float32_negative_infinity", float32(math.Inf(-1)), `"-Infinity"`},
{"float64_nan", math.NaN(), `"NaN"`},
{"float64_infinity", math.Inf(1), `"Infinity"`},
{"float64_negative_infinity", math.Inf(-1), `"-Infinity"`},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
for _, prefix := range []string{"", "[1.5,"} {
got, err := Append([]byte(prefix), tc.value)
if err != nil || string(got) != prefix+tc.want {
t.Fatalf("Append(%q, %v) = %q, %v; want %q", prefix, tc.value, got, err, prefix+tc.want)
}
}
})
}
}

func TestMapKeyPreservesEncoding(t *testing.T) {
values := []any{nil, true, int32(7), int64(-9), "NaN", "+Inf", "key", float32(1.5),
float32(math.Copysign(0, -1)), 1.5, math.Copysign(0, -1), math.MaxFloat64}
for _, value := range values {
want, err := json.Marshal(value)
if err != nil {
t.Fatal(err)
}
got, err := Append(nil, MapKey(value))
if err != nil || string(got) != string(want) {
t.Errorf("MapKey(%v): got %q, %v; want %s", value, got, err, want)
}
}
}

func TestAppendKeepsPrefix(t *testing.T) {
got, err := Append([]byte("["), int32(7))
if err != nil || string(got) != "[7" {
Expand All @@ -112,6 +157,7 @@ func TestAppendKeepsPrefix(t *testing.T) {
func TestAppendFastPathDoesNotAllocate(t *testing.T) {
buf := make([]byte, 0, 64)
for _, v := range []any{int64(-9000), int32(42), uint64(7), 3.14, 0.30000000000000004, 1e-7, float32(0.5), true,
math.NaN(), math.Inf(1), math.Inf(-1), float32(math.NaN()), float32(math.Inf(1)), float32(math.Inf(-1)),
"alpha", "café 日本語 😀"} {
if n := testing.AllocsPerRun(100, func() { buf, _ = Append(buf[:0], v) }); n != 0 {
t.Errorf("Append(%#v) allocated %v times per run, want 0", v, n)
Expand All @@ -121,7 +167,7 @@ func TestAppendFastPathDoesNotAllocate(t *testing.T) {

// jsonfmt appends into a reused buffer, as both renderers do.
func BenchmarkAppendScalars(b *testing.B) {
// The last three take json.Marshal's fallback.
// The escaped string and []byte take json.Marshal's fallback.
values := []any{int32(42), int64(-9000), 3.14, 0.30000000000000004, float32(0.5), true, "alpha0", "café crème",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

math.NaN() is still in BenchmarkAppendScalars. It's now a fast-path append for jsonfmt but an error path for the encoding_json baseline, so the comparison is skewed. Could we drop it, or add a comment that the baseline errors on it?

`a"b`, []byte("abc"), math.NaN()}
b.Run("jsonfmt", func(b *testing.B) {
Expand Down
2 changes: 1 addition & 1 deletion internal/rows/arrowbased/columnValues.go
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ func (mvc *mapValueContainer) Value(i int) (any, error) {
return nil, err
}

key, err := marshalScalar(keyScratch[:0], mvc.keys, int(i+s), k)
key, err := marshalScalar(keyScratch[:0], mvc.keys, int(i+s), jsonfmt.MapKey(k))
if err != nil {
return nil, err
}
Expand Down
Loading
Loading