Skip to content

Fix Arrow TIME results on Thrift and SEA - #567

Merged
vuanhphung merged 5 commits into
mainfrom
fix/issue-505-time-columns
Nov 16, 2026
Merged

vuanhphung merged 5 commits into
mainfrom
fix/issue-505-time-columns

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Both Go Arrow scanners reject TIME columns, preventing the result from being read. Handle time32/time64 on Thrift and SEA as UTC time.Time, report TIME/time.Time metadata, and render nested TIME as clock-only JSON strings with trailing fractional zeros omitted. Empty SEA results preserve logical TIME metadata even when the manifest's physical schema is UTF-8. No new option.

Logical TIME promotion assumes non-empty TIME always arrives as Arrow time32/time64; populated UTF-8 TIME parsing is deferred, since no supported live DBSQL path produces it.

Maintainer decision (2026-11-14): use UTC time.Time on 0001-01-01, matching gosnowflake converter.go:2572 @8cb15c3 and go-duckdb vector_getters.go:92 @d87a366.

Accepted trade-off: non-NULL midnight equals time.Time{} and JSON omitzero omits it; detect NULL with sql.NullTime.Valid or nil *time.Time, never IsZero() (documented in the TIME comment and README/doc.go type tables). Scalar string scans use database/sql's RFC3339Nano format, unlike nested clock-only text.

Bind TIME as value.Format("15:04:05.999999") in CAST(? AS TIME), not as time.Time, which binds as TIMESTAMP and can shift the clock in non-UTC sessions; both non-UTC backend tests cover the string round trip.

Live TIME-beta verification passes on both backends in three fresh processes each, without skips: TIME(0)..TIME(6), NULL versus midnight, non-UTC sessions, typed/dynamic/string scans, empty/NULL-only metadata and nested ARRAY/STRUCT/MAP. Each backend drains 250,000 large-result rows through database/sql and again through raw Arrow batches. Thrift reports TTypeId STRING_TYPE (7) with a nonempty time64[ns] ArrowSchema, both inline (ARROW_BASED_SET) and on actual CloudFetch links (URL_BASED_SET); SEA preserves the same time64[ns] schemas. origin/main still fails the original live reproducer on both scanners.

The e2e tag explicitly opts into TIME-beta warehouse tests that fail on missing credentials or support; offline tests remain credential-free, and CI vet compiles both tagged configurations. The DBSQL nightly runs the TIME tests; the Reyden reference leg excludes them until its TIME support can be verified.

make test: 1,550 tests, 8 existing credential-gated skips; make lint: 0 issues. Pure and kernel/e2e-tagged vet and builds pass. Published-bindings kernel tests pass 1,646 tests with 44 existing skips after excluding the pre-existing four-case TestKernelCloudFetchDisabledPreservesUnsupportedServerError failure, reproduced on main; touched-package race checks pass with the same exclusion. Three generated formatting failures also reproduce on main; changed Go files format cleanly.

Alternating fresh-process microbenchmarks provide 18 samples per case: existing int64/string/timestamp scan geomean 17.12 → 17.19 ns/op (+0.40%), allocations unchanged; kernel-scanner int64 +1.79%, timestamp -2.70%, other differences not significant. New metadata benchmarks (18 samples/case) measure 31.65 → 33.73 ns for STRING and 38.85 → 30.14 ns for manifest TIME, with zero allocations; metadata is initialized once per result column, not per row. Live warehouse timings are shared and not used for a performance claim.

Cross-driver state

Driver State Notes
JDBC Not supported No Arrow TIME logical mapping; ColumnInfoTypeName.java:32, ArrowToJavaObjectConverter.java:133 @5ad46ee.
Python Partial Arrow values become datetime.time; kernel TIME metadata falls back. result_set.py:103, backend/kernel/type_mapping.py:83 @b4cec22.
Go Not supported → Supported Both Arrow scanners; year-1 UTC time.Time, TIME metadata and clock-only nested values. No new option.
Node.js Partial TIME units pass through as raw numeric Arrow values; lib/result/ArrowResultConverter.ts:437 @8ccddd0.
Kernel Not applicable C Data export preserves Arrow arrays; Go owns conversion. src/c_abi/result.rs:426 @a488a05.

Alternative #508 remains open. This follows the maintainer-selected Go convention and completes Thrift column metadata; per-column fallback for other unsupported Arrow types remains tracked by #505, which this PR intentionally does not close.

Adversarial review converged in round 2 with no open high finding; remaining encoding assumptions and parameter-binding guidance are documented, and malformed-wire hardening is deferred.

Addresses #505

This pull request and its description were written by Isaac.

@vuanhphung
vuanhphung marked this pull request as ready for review November 14, 2026 09:46

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

vuanhphung and others added 5 commits November 16, 2026 06:29
Handle TIME values and metadata on both Arrow scanners, preserving precision, NULLs, and clock-only nested JSON. Cover database/sql destinations and explicitly opt-in live beta checks.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Document metadata and zero-time behavior, cover empty and nullable results, broaden precise TIME-unavailable tests, and compile live test configurations in PR CI. Retain the provisional TIME representation.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Document the maintainer-approved year-1 UTC convention and midnight trade-off.
Verify precision, NULLs, real Arrow schemas and large results on Thrift and SEA;
fail opt-in live tests when TIME is unavailable.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Honor logical TIME names when empty-result manifests use UTF-8 fields. Cover empty and NULL-only metadata on both live backends, and scope TIME beta nightly coverage to the verified DBSQL warehouse.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Document the supported TIME wire encoding assumption and timezone-safe parameter binding. Exercise string round trips on both live backends.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung
vuanhphung force-pushed the fix/issue-505-time-columns branch from b2d9400 to 20f84d3 Compare November 16, 2026 06:33

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung
vuanhphung enabled auto-merge November 16, 2026 06:58
@vuanhphung
vuanhphung added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 57bc51a Nov 16, 2026
15 checks passed
@vuanhphung
vuanhphung deleted the fix/issue-505-time-columns branch November 16, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant