Repository navigation
Fix Arrow TIME results on Thrift and SEA - #567
Merged
Merged
Conversation
vuanhphung
marked this pull request as ready for review
November 14, 2026 09:46
Handle TIME values and metadata on both Arrow scanners, preserving precision, NULLs, and clock-only nested JSON. Cover database/sql destinations and explicitly opt-in live beta checks. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Document metadata and zero-time behavior, cover empty and nullable results, broaden precise TIME-unavailable tests, and compile live test configurations in PR CI. Retain the provisional TIME representation. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Document the maintainer-approved year-1 UTC convention and midnight trade-off. Verify precision, NULLs, real Arrow schemas and large results on Thrift and SEA; fail opt-in live tests when TIME is unavailable. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Honor logical TIME names when empty-result manifests use UTF-8 fields. Cover empty and NULL-only metadata on both live backends, and scope TIME beta nightly coverage to the verified DBSQL warehouse. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Document the supported TIME wire encoding assumption and timezone-safe parameter binding. Exercise string round trips on both live backends. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
vuanhphung
force-pushed
the
fix/issue-505-time-columns
branch
from
November 16, 2026 06:33
b2d9400 to
20f84d3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both Go Arrow scanners reject TIME columns, preventing the result from being read. Handle time32/time64 on Thrift and SEA as UTC
time.Time, reportTIME/time.Timemetadata, and render nested TIME as clock-only JSON strings with trailing fractional zeros omitted. Empty SEA results preserve logical TIME metadata even when the manifest's physical schema is UTF-8. No new option.Logical TIME promotion assumes non-empty TIME always arrives as Arrow time32/time64; populated UTF-8 TIME parsing is deferred, since no supported live DBSQL path produces it.
Maintainer decision (2026-11-14): use UTC
time.Timeon0001-01-01, matching gosnowflakeconverter.go:2572@8cb15c3 and go-duckdbvector_getters.go:92@d87a366.Accepted trade-off: non-NULL midnight equals
time.Time{}and JSONomitzeroomits it; detect NULL withsql.NullTime.Validor nil*time.Time, neverIsZero()(documented in the TIME comment and README/doc.go type tables). Scalar string scans usedatabase/sql's RFC3339Nano format, unlike nested clock-only text.Bind TIME as
value.Format("15:04:05.999999")inCAST(? AS TIME), not astime.Time, which binds as TIMESTAMP and can shift the clock in non-UTC sessions; both non-UTC backend tests cover the string round trip.Live TIME-beta verification passes on both backends in three fresh processes each, without skips: TIME(0)..TIME(6), NULL versus midnight, non-UTC sessions, typed/dynamic/string scans, empty/NULL-only metadata and nested ARRAY/STRUCT/MAP. Each backend drains 250,000 large-result rows through
database/sqland again through raw Arrow batches. Thrift reports TTypeId STRING_TYPE (7) with a nonempty time64[ns] ArrowSchema, both inline (ARROW_BASED_SET) and on actual CloudFetch links (URL_BASED_SET); SEA preserves the same time64[ns] schemas.origin/mainstill fails the original live reproducer on both scanners.The
e2etag explicitly opts into TIME-beta warehouse tests that fail on missing credentials or support; offline tests remain credential-free, and CI vet compiles both tagged configurations. The DBSQL nightly runs the TIME tests; the Reyden reference leg excludes them until its TIME support can be verified.make test: 1,550 tests, 8 existing credential-gated skips;make lint: 0 issues. Pure and kernel/e2e-tagged vet and builds pass. Published-bindings kernel tests pass 1,646 tests with 44 existing skips after excluding the pre-existing four-caseTestKernelCloudFetchDisabledPreservesUnsupportedServerErrorfailure, reproduced on main; touched-package race checks pass with the same exclusion. Three generated formatting failures also reproduce on main; changed Go files format cleanly.Alternating fresh-process microbenchmarks provide 18 samples per case: existing int64/string/timestamp scan geomean 17.12 → 17.19 ns/op (+0.40%), allocations unchanged; kernel-scanner int64 +1.79%, timestamp -2.70%, other differences not significant. New metadata benchmarks (18 samples/case) measure 31.65 → 33.73 ns for STRING and 38.85 → 30.14 ns for manifest TIME, with zero allocations; metadata is initialized once per result column, not per row. Live warehouse timings are shared and not used for a performance claim.
Cross-driver state
ColumnInfoTypeName.java:32,ArrowToJavaObjectConverter.java:133@5ad46ee.datetime.time; kernel TIME metadata falls back.result_set.py:103,backend/kernel/type_mapping.py:83@b4cec22.time.Time, TIME metadata and clock-only nested values. No new option.lib/result/ArrowResultConverter.ts:437@8ccddd0.src/c_abi/result.rs:426@a488a05.Alternative #508 remains open. This follows the maintainer-selected Go convention and completes Thrift column metadata; per-column fallback for other unsupported Arrow types remains tracked by #505, which this PR intentionally does not close.
Adversarial review converged in round 2 with no open high finding; remaining encoding assumptions and parameter-binding guidance are documented, and malformed-wire hardening is deferred.
Addresses #505
This pull request and its description were written by Isaac.