diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 988c85f4..9dea419b 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -10,3 +10,232 @@ configured in `.bot/config.yaml`: No learnings have been recorded yet. Dated sections are appended below by the retrospective flow. +## Entries + +### 2026-08-13: learnings since 2026-08-12T17:42:15Z +- **Context:** PR #442 configured a bot's `MODEL_ENDPOINT`; using the generic `.../serving-endpoints/anthropic/invocations` form 400'd every scheduled run (`Unsupported native API path .../anthropic/invocations/v1/messages`), confirmed empirically against the sibling databricks-sql-python cron. + **Rule:** Point Databricks bot `MODEL_ENDPOINT` at the concrete `.../serving-endpoints//invocations` form, never `.../serving-endpoints/anthropic/invocations` — `translate_endpoint` early-returns on URLs already containing `/serving-endpoints/anthropic`, leaving the `/invocations` suffix so the CLI appends `/v1/messages` and hits HTTP 400. +- **Context:** PR #442's learning workflow declared a `workflow_dispatch` input as a string (`window-hours`) with a comment noting that `type: number` breaks the run. + **Rule:** In GitHub Actions, a `workflow_dispatch` input declared `type: number` fails the whole run at startup ("workflow file issue") when the workflow also has a `schedule` trigger — declare numeric dispatch inputs as `type: string` and coerce to int downstream (e.g. via argparse). +- **Context:** PR #442 initially set `retrospective.system_prompt: prompts/retrospective_system.md`, a file that did not exist; the maintainer confirmed the engine treats a set-but-missing prompt path as a hard error, which would have failed the daily cron every run. + **Rule:** For engine config keys that reference a file (e.g. `system_prompt`): an UNSET key falls back to the engine's built-in default, but a SET key pointing at a missing file is a hard error — omit the key entirely rather than point it at a nonexistent path. +- **Context:** PR #442 needed the retrospective flow to commit `.claude/knowledge/learning-log.md`, but the repo `.gitignore` ignored all of `.claude`; the fix scoped the ignore (`.claude/*` + `!.claude/knowledge/` + `.claude/knowledge/*` + `!.claude/knowledge/learning-log.md`) and seeded the file so the author read path never hits a missing file. + **Rule:** When a bot/tool must commit a file under a normally-ignored directory, add scoped `.gitignore` negation for exactly that file (ignoring intermediate dirs still hides children, so re-include each level) and seed the file, so both the write (commit) and read paths resolve. + +### 2026-08-21: learnings since 2026-08-20T17:32:23Z +- **Context:** PR #446 corrected CONNECTION_PARAMETERS.md to document how session confs and proxies diverge between the Thrift and kernel (SEA) backends. + **Rule:** The kernel/SEA path is stricter than Thrift: session-conf keys are matched case-insensitively against an allowlist (non-allowlisted keys are dropped with a warning, a few hard-rejected e.g. HTTP 400 INVALID_CONF_VALUE), and only http(s) proxies are accepted (socks* URLs honored on Thrift are rejected at connect) — so a conf/proxy that works on Thrift may be silently inert or rejected on kernel; verify kernel behavior separately when adding or relying on any session parameter or proxy feature. + +### 2026-08-22: learnings since 2026-08-21T17:32:18Z +- **Context:** PR #449 fixed Azure OAuth U2M on the kernel backend — `resolveKernelAuth` had been forwarding the cloud-inferred client id + scopes (`a.U2MClientID()` / `oauth.GetScopes`) that the Thrift path infers from the host, which routed the kernel's browser to a broken AAD authorize URL on Azure. + **Rule:** The kernel/SEA backend runs ONE cloud-blind in-house workspace-federated U2M flow (OIDC discovery against `{host}/oidc`, no Azure branching), so its auth mapping must forward the fixed in-house `databricks-sql-connector` client + `offline_access`+`sql` scopes uniformly across all clouds — do NOT forward the Azure Entra-direct app id / `user_impersonation` scope the Thrift path uses, even though both paths authorize identically on AWS/GCP. +- **Context:** PR #449 replaced `oauth.GetScopes(host, nil)` with a hardcoded scope slice and initially wrote `[]string{"sql", "offline_access"}`, producing a High-severity failure: one test asserted the reversed order and contradicted its sibling test, because `resolveKernelAuth` returns `[]string{"offline_access", "sql"}`. + **Rule:** OAuth scopes are a space-delimited unordered set per spec, but `reflect.DeepEqual` on `[]string` is order-sensitive — when hardcoding a scope slice that replaces `oauth.GetScopes` (which appends `offline_access` first, then `sql`), preserve the exact element order and keep all sibling assertions consistent, or DeepEqual comparisons/tests will fail spuriously. +- **Context:** PR #444 added `WithFederatedTokenProvider*` on the kernel path; for account-wide federation (no client id) the driver hands the kernel the raw un-exchanged external subject token via `set_auth_pat`, unlike the Thrift path which exchanges in-driver via `FederationProvider`. Reviewers repeatedly questioned whether this silently fails; a maintainer confirmed the kernel's behavior. + **Rule:** The kernel performs a mandatory server-side token exchange for tokens presented on the PAT path UNLESS the token is same-issuer or non-JWT — so handing a raw external-IdP JWT subject token to `set_auth_pat` (client id only set for SP-wide federation) correctly federates account-wide without driver-side exchange. Rely on this documented kernel guarantee rather than assuming un-exchanged tokens are treated as literal PATs. +- **Context:** In both PR #449 and PR #444, reviewers (Copilot, peco-review-bot) flagged multiple stale/contradictory doc+comment sites after a kernel-auth behavior change — the behavior is mirrored across `doc.go`, `README.md`, `CONNECTION_PARAMETERS.md`, `internal/backend/kernel/auth.go` (Auth struct + provider-interface docs), `backend.go` (setAuth comment), and `auth/oauth/u2m/authenticator.go`. + **Rule:** This repo documents kernel/Thrift auth semantics redundantly across many files; when changing behavior on one auth path, sweep ALL mirrored doc/comment sites in the same PR (not just the function you edited) or reviewers will flag stale contradictions. + +### 2026-08-24: learnings since 2026-08-23T17:29:41Z +- **Context:** PR #450 (kernel log-forwarding cgo bridge) — a reviewer flagged a helper that coerced a `cgo.Handle` (an integer token) into a small fabricated `void*` and passed it as the C callback's `user_data`. + **Rule:** Never pass a `cgo.Handle` or any fabricated/Go-managed pointer into a C pointer slot (`void* user_data`); the GC can detect the invalid pointer and abort via `runtime.throw`, which `recover` cannot catch — use NULL `user_data` with a package-global sink, or allocate a real C cell and pass its address. +- **Context:** PR #450 — the kernel invokes the log callback synchronously on the emitting native thread and forbids blocking/re-entry, but the first implementation ran zerolog plus an arbitrary user `SetLogOutput` writer inside the callback. + **Rule:** In a synchronous FFI callback that runs on a foreign thread, do only minimal bounded work: copy borrowed strings into owned memory and non-blockingly enqueue onto a bounded channel, then drain and do the real I/O on a separate goroutine — never run arbitrary user I/O on the callback thread (it can stall the native op or re-enter the ABI). +- **Context:** PR #450 — the logger held its destination-routing mutex while executing the user-owned `Write`, so a blocking writer (or one that called `SetLogOutput`/logged) could deadlock and could not be replaced. + **Rule:** Never invoke arbitrary user-supplied code (a `Write`, callback, or handler) while holding an internal lock; publish the target via an atomic pointer / short-held lock and invoke it only after releasing the lock, so a stuck consumer can never block retargeting or self-deadlock. +- **Context:** PR #450 — `installKernelLogCallback` returned early inside `logCallbackOnce.Do` when the level was OFF; a reviewer noted `sync.Once` still marks itself complete, latching forwarding off permanently for the process. + **Rule:** A conditional early `return` inside `sync.Once.Do` still records the Once as done, so a later attempt becomes a permanent no-op; if a step must be retryable when a precondition later changes, gate the precondition outside `Once.Do` rather than short-circuiting within it. +- **Context:** PR #450 — building the root zerolog instance over a stable proxy that implemented only `io.Writer` permanently downgraded any later `zerolog.LevelWriter` (e.g. `MultiLevelWriter`, syslog) to plain `Write`, losing severity-aware routing. + **Rule:** A long-lived proxy/wrapper placed in front of a swappable destination must implement the richest interface its underlying targets may support (e.g. `WriteLevel` for `zerolog.LevelWriter`) and delegate to it when present; wrapping only the base interface silently strips capabilities of any future richer target. +- **Context:** PR #450 — dropped kernel log records under channel backpressure were counted but the count was only read by tests, so a burst past capacity silently lost lines with no operator-visible signal. + **Rule:** When dropping data under backpressure (bounded queue overflow, sampling, truncation), surface it — at minimum a one-shot/periodic warning through the normal output path — so consumers can distinguish 'nothing was produced' from 'output was silently dropped'. + +### 2026-08-25: learnings since 2026-08-24T17:32:28Z +- **Context:** PR #456 added `WithKernelClientCertificate`; a paired credential where an empty cert/key is invalid. It introduced a dedicated `TLSClientCertConfigured` bool alongside the PEM buffers and validated at connect time (`ErrInvalidKernelConfig`). + **Rule:** For optional paired-credential/config options where empty is invalid, carry an explicit "configured" sentinel flag (not just checking for non-empty values) so an explicit call with empty input is rejected rather than silently treated as unset — prevents failing open. +- **Context:** PR #455 corrected the kernel connection docs: the connect-context deadline was described as unhonored only during U2M browser login, but it is actually unhonored mid-connect for ALL auth modes (PAT/M2M included). + **Rule:** On the kernel/SEA path the connect-context deadline is checked only at entry to session-open; the kernel's blocking C-ABI session-open then runs uninterruptibly, so a slow cold-start or network partition can block past the deadline regardless of auth mechanism — not a U2M-specific limitation. +- **Context:** PR #455's new kernel session-conf allowlist table transcribed state living in vendored kernel source (`build/kernel-src/src/config.rs`) not present in the repo checkout; a review-bot finding flagged it as un-guarded and drift-prone, and the fix pinned the source and called out the missing CI guard. + **Rule:** When documenting behavior sourced from vendored/external code that isn't in the repo checkout (so no test or CI guard anchors it), pin the exact source ref/path, name which entries have repo-side anchors, and explicitly warn the rest may lag — treat the external source as authoritative. + +### 2026-08-26: learnings since 2026-08-25T17:32:57Z +- **Context:** In PR #458 a reviewer caught an unused local variable (`k declared and not used`) in `internal/backend/kernel/backend.go`, a file behind `//go:build cgo && databricks_kernel`. It is a hard Go compile error, but the default `go test` excludes that tag so ordinary CI would not catch it — only the tagged kernel test environment fails. + **Rule:** Code behind build tags excluded from the default test run (e.g. `cgo && databricks_kernel`) is not compiled or vetted by ordinary `go test`/CI; compile and run the tagged build separately before relying on it, since even trivial compile errors (unused vars) there pass default CI. +- **Context:** In PR #458 the test seam `trySetTokenCacheConfig` originally re-implemented the `set_u2m_token_cache_config` cgo call standalone instead of routing through the production `setAuth`. A reviewer noted that a future edit dropping/mis-wiring the setter inside `setAuth`'s U2M branch would leave every test green; the fix rebuilt a real `KernelBackend` and called `k.setAuth(cfg)`. + **Rule:** Test seams should drive the real production code path (call the actual method under test), not a parallel re-implementation of the same underlying call — a standalone copy asserts the C signature but not that the production path still invokes it, so regressions in the real wiring go uncaught. +- **Context:** In PR #458 the `WithTokenCache(bool)` option calls `kernelExperimental(c)`, which allocates `KernelExperimental` unconditionally — so even `WithTokenCache(false)` makes `KernelExperimental != nil` and is rejected on the Thrift backend with `ErrRequiresKernelBackend`. The DSN carrier `tokenCache=false` deliberately does NOT forward, so only it is a true no-op. Godoc initially (wrongly) claimed disabling was "a no-op on any backend." + **Rule:** `WithKernel*` options allocate the kernel-only `KernelExperimental` struct unconditionally, so passing even the disabling/`false` value opts the connection into the kernel backend and fails on Thrift; only the DSN carrier for such a flag can be a genuine no-op. Scope any "no-op" doc claim to the DSN path, and mind that the option and DSN entry points for the same setting diverge (option always opts in; DSN `false` does not). + +### 2026-08-27: learnings since 2026-08-26T18:05:10Z +- **Context:** PR #457 forwarded the driver's ClientTimeout to the kernel C ABI (`kernel_session_config_set_request_timeout`). Reviewers noted that passing `0` does not mean unlimited or immediate — the kernel substitutes its own 120s default. + **Rule:** When forwarding a timeout/limit to the kernel C ABI, treat `0` as the "use kernel default" sentinel (120s for request timeout), not as unlimited or zero-wait; document this at every knob and account for it when reporting effective values. +- **Context:** In PR #457, telemetry's `SocketTimeout` was initially populated with a millisecond value, but reviewers flagged that the receiver proto tags millisecond durations with an explicit `_ms`/`_millis` suffix (`retry_overall_timeout_ms`, `result_set_ready_latency_millis`), whereas the bare `socket_timeout` field is interpreted in seconds — the fix converted to seconds. + **Rule:** Before populating a telemetry duration field, match its unit to the receiver schema's field-name convention: bare names (e.g. `socket_timeout`) are seconds, only `_ms`/`_millis`-suffixed names are milliseconds — mismatching over/under-reports by 1000×. +- **Context:** PR #457 had two paired duration converters: the kernel setter rounded positive sub-millisecond values *up* to 1 (so a real timeout never collapses into the `0 = default` sentinel), while the telemetry helper rounded positive sub-second values *down* to 0 — meaning a small nonzero configured timeout was forwarded as a real deadline yet reported as "unset" (the field was `omitempty`). + **Rule:** When paired converters share a `0 = default/unset` sentinel, round positive sub-unit values consistently (floor them to 1, never down to 0) so a genuinely-configured value is never misreported as absent. + +### 2026-08-28: learnings since 2026-08-27T20:43:32Z +- **Context:** PR #440 wired per-platform kernel archives as nested Go modules using an in-tree `replace` pointing at a placeholder `require ... v0.0.0`/`v0.0.1`; reviewers flagged it builds in-tree but breaks external consumers. + **Rule:** Go `replace` directives are honored ONLY in the main module — a downstream `go get` of your module ignores them, so never rely on an in-tree `replace` to shadow a placeholder `require` version; publish and pin the dependency at a real tagged version before advertising a `go get`-installable build. +- **Context:** PR #440 review: an unresolvable nested-module `require` would break even pure-Thrift (`CGO_ENABLED=0`, no build tag) consumer builds that never compile a kernel file. + **Rule:** Go module-graph resolution (MVS) is build-tag-independent — every `require`d module's `go.mod` must resolve for ALL consumers regardless of build tags, so an unpublished/placeholder version in `go.mod` fails builds that never touch that code, not just the tag-gated ones. +- **Context:** PR #440 added a `!databricks_kernel_dynlib` term to only the darwin/arm64 cgo link shim while the other shims and the `cgo_unsupported.go` compile guard stayed dynlib-unaware, so building with that tag compiles no link shim and fails with opaque `undefined reference` link errors. + **Rule:** With build-tag-gated cgo link shims plus a companion `unsupported`/compile-time guard, any new build-tag term must be applied consistently across every shim AND the guard's exclusion list — an asymmetric term silently drops all shims and defeats the guard whose job is to turn link failures into a clear compile error. +- **Context:** PR #440 CI sync step used `go mod verify || true`, which reviewers noted swallows genuine module-cache corruption/tampering failures while the surrounding comment framed the step as verifying the module graph. + **Rule:** Do not mask verification/gating commands (`go mod verify`, checksum/lint checks) with `|| true` — it defeats the check's purpose; let the command fail the job, or log an explicit justification if a non-fatal run is truly intended. +- **Context:** PR #451's `applyTelemetry` always forwarded hardcoded Go-side constants (batch size, flush interval, etc.) across the kernel C ABI even when unset, while `CONNECTION_PARAMETERS.md` promised the kernel's own default applied "when unset." + **Rule:** When docs state the downstream/native layer owns a default "when unset," do not hardcode a duplicated copy of that default on the wrapper side and always forward it — forward a sentinel that lets the native layer keep its own default, or change the docs to state the driver owns the value; duplicated defaults silently drift when the native side changes and no test catches it. +- **Context:** PRs #463/#440 documented a `make kernel-lib` → `make build-kernel` → `make test-kernel` contributor flow, but reviewers noted `build-kernel`/`test-kernel` link the published bindings modules, not the local `.a` that `kernel-lib` produces; linking the local build needs a still-unwired manual `go.work` step. + **Rule:** In this repo the kernel dev-loop does not link a locally built archive by default — `make build-kernel`/`test-kernel` link the published `databricks-sql-kernel-bindings` modules; to test a local source build you must manually point a `go.work`/replace at the built `lib/` archive, so verify which archive is actually linked before trusting a source-build change. + +### 2026-09-04: learnings since 2026-09-03T17:27:27Z +- **Context:** In PR #469 the kernel-rev sync also rewrote user-facing docs (CONNECTION_PARAMETERS.md/README) to claim the kernel now forwards `WithSessionParams` unchanged with no allowlist, but the review flagged that released-driver consumers link the *published* `databricks-sql-kernel-bindings` module in `go.mod`, whose archive contents live outside this repo checkout — the no-allowlist behavior was only verified in the source-built pinned `KERNEL_REV`. + **Rule:** When a PR documents new kernel behavior, gate the doc change on the shipped/published bindings version that released-driver users actually link — don't describe source-built `KERNEL_REV` behavior as current unless you can confirm the pinned bindings module in `go.mod` contains it (bindings archive contents are not in this checkout, so treat them as unverifiable from the repo alone). + +### 2026-09-05: learnings since 2026-09-04T17:25:10Z +- **Context:** PR #470 added per-statement query tags on the kernel backend; a reviewer caught that the serialized tags string reaching the length-less `kernel_statement_set_query_tags` C ABI setter had no interior-NUL guard, unlike SQL text (`checkQueryText`/`errQueryNUL`) and bound params (`checkParamValue`/`errParamNUL`). + **Rule:** Any user-supplied string passed to a NUL-terminated (length-less) kernel C ABI setter must be guarded for interior NUL bytes before cgo marshaling — a NUL silently truncates on the kernel path while the length-prefixed Thrift `confOverlay` carries the value whole, breaking Thrift/SEA parity; fail loudly, and place the guard in an untagged pure-Go file so it is unit-tested under CGO_ENABLED=0. +- **Context:** PR #468 bumped the vendored `databricks_kernel.h`, flipping the documented `kernel_session_close` contract from async/best-effort to awaited, but the Go-side `CloseSession` doc comment still described the old behavior and explicitly cited that header. + **Rule:** When a vendored kernel C ABI header's documented contract changes (or KERNEL_REV is bumped), audit and update the Go doc comments that cite or paraphrase that header in lockstep — they go stale silently and can end up contradicting the very ABI they reference. +- **Context:** PR #471 documented numeric defaults in CONNECTION_PARAMETERS.md (e.g. `cloudfetch_link_prefetch_window=10`, `inline_max_chunks_in_memory=4`) that live only in the Rust kernel, not in this repo; a reviewer noted they can drift out of sync silently. + **Rule:** Don't assert exact numeric defaults owned by the Rust kernel in repo docs unless the value is pinned by a cited KERNEL_REV (as the mTLS row does); otherwise soften to "kernel default" to avoid silent doc/kernel drift. + +### 2026-09-11: learnings since 2026-09-10T17:25:51Z +- **Context:** PR #479 added Reyden→kernel auto-recovery that hinges on `errors.Is(err, ErrReydenThriftUnsupported)` traversing production's wrapping chain (`thrift.Backend.OpenSession` wraps the marker via `NewRequestError` → `WithMessage`/`WithStack`), but the tests injected fakes returning the *bare* marker, so a regression that breaks the unwrap chain would silently disable recovery while every test stayed green. + **Rule:** When a feature depends on `errors.Is`/`errors.As` matching through a multi-layer wrapping chain, add a test that asserts the match against the production-shaped *wrapped* error (or drives the real backend) — fakes returning the bare sentinel do not pin the production error path. +- **Context:** PR #479's `TestReydenDefaultBuildKernelNotCompiled` asserted default-build stub behavior (`newKernelBackend` returning `ErrKernelNotCompiled`) but carried no build constraint, so it also compiled under `-tags databricks_kernel` + `CGO_ENABLED=1` where `newKernelBackend` is the real implementation, breaking `make test` for the kernel build. + **Rule:** A test asserting build-tag-gated stub behavior must carry the same `//go:build` constraint as the stub it exercises (e.g. `!cgo || !databricks_kernel`); an untagged test runs under every build and fails where the real implementation is linked. +- **Context:** PR #479 review flagged that the known-Reyden cache pre-check opened the kernel backend before the `KernelExperimental != nil` guardrail, so identical misconfigured config (a `WithKernel*` option without `WithUseKernel`) produced a hard error on a cold cache but silently succeeded on a warm cache populated by a sibling connection. + **Rule:** Run config-validation guardrails before any cache-based short-circuit/pre-check so behavior for a given config is deterministic and does not depend on process-global cache state. + +### 2026-09-17: learnings since 2026-09-16T17:26:26Z +- **Context:** PR #485 rewrote `parseHostName` in connector.go — the old code used `strings.HasPrefix(host, "https"/"http")`, which is case-sensitive and also matched non-scheme hostnames like `httpbin.example.com`. The fix uses `strings.Cut` on `:` plus `strings.EqualFold`, and this scheme detection feeds downstream backend validation (e.g. mTLS-over-HTTP must be rejected before the SEA/kernel backend regardless of scheme casing). + **Rule:** When detecting a URL/URI scheme by string matching, compare case-insensitively (URI schemes are case-insensitive per RFC 3986 — use `EqualFold`, not `HasPrefix`) and require the `:` delimiter so ordinary hostnames that merely start with `http` aren't misparsed as a scheme; verify casing variants flow correctly into downstream backend/protocol validation. + +### 2026-09-23: learnings since 2026-09-22T17:27:28Z +- **Context:** PR #484 added a detached per-statement cancel watcher on the kernel client-query-timeout path; reviewers repeatedly flagged that the watcher can call `kernel_statement_canceller_cancel`/`_free` concurrently with or after `kernel_statement_close`, and safety hinged on a kernel C ABI guarantee (canceller owns cloned Arcs + its own session ref, no back-pointer into the statement) that is not verifiable from this repo. Resolution came only from checking the pinned kernel revision and its ABI test `canceller_outlives_statement_and_session_close`. + **Rule:** When Go memory safety depends on a cgo/kernel object staying valid independently of another handle's lifetime (use/free after the related statement or session is closed), that independence is NOT verifiable from this repo — confirm it against the pinned KERNEL_REV and its ABI test, and record the load-bearing assumption in a code comment so a future kernel refactor trips a reviewer instead of silently introducing a use-after-free. +- **Context:** PR #484 introduced `MaxClientQueryTimeoutMilliseconds` as a hand-copied mirror of the C macro `DATABRICKS_KERNEL_MAX_CLIENT_QUERY_TIMEOUT_MS`, guarded only by a build-tagged runtime test; a reviewer contrasted this with the `KernelStatusCode` constants, which use a compile-time drift assertion (`_ = uint(a-b) | uint(b-a)`) so a header renumber fails the build. + **Rule:** Any Go constant hand-mirrored from a C ABI macro must be pinned with a compile-time drift assertion (the established `uint(a-b)|uint(b-a)` pattern), not a build-tagged runtime test — a test that must be remembered to run lets a KERNEL_REV bump silently validate against a stale bound. +- **Context:** In PR #484 `WithClientQueryTimeout` stores a pointer by copying into a fresh local (`configured := timeout; c.ClientQueryTimeout = &configured`) inside the option closure, and a dedicated test asserts reusing one option value across two connectors does not alias their configs. + **Rule:** A functional option that stores a pointer must allocate a fresh copy per application (copy into a local inside the closure, not `¶m`); otherwise reusing the same option value across multiple constructions aliases shared mutable state. Add a test that applies one option to two objects and mutates one. +- **Context:** PR #484 modeled `ClientQueryTimeout` as `*time.Duration` so that an omitted option (nil → legacy `kernel_statement_execute` with the 600s ceiling) is behaviorally distinct from an explicit zero (non-nil → timeout-aware entry point, unlimited), with a test named "omitted remains distinguishable from explicit zero." + **Rule:** When an omitted option must select different behavior than an explicit zero/default value, model it as a pointer (nil = omitted) rather than a bare value, and test both the omitted and explicit-zero cases — a plain zero value collapses the two and silently picks one path. + +### 2026-10-06: learnings since 2026-10-05T17:30:24Z +- **Context:** PR #493 fixed M2M OAuth so a least-privilege service-principal secret (e.g. scoped to `sql`) can authenticate. The old `GetScopes` unconditionally appended `all-apis` to any caller-supplied scope list, broadening access beyond what the caller asked for. + **Rule:** When defaulting an OAuth/credential scope set, only inject a broad default (e.g. `all-apis`) when the caller passed NONE — never append it to a non-empty list, or you silently over-grant and break least-privilege scoped secrets. +- **Context:** PR #493 reversed an earlier decision to reject custom M2M scopes on the SEA/kernel path. The rejection assumed scopes were unforwardable because `set_auth_m2m` has no scopes argument; in fact a separate `set_oauth_scopes` C-ABI call carries them, so the whole `M2MScopesSupported` rejection path was deleted. + **Rule:** Before rejecting a config option as unsupported on a backend because its primary setter can't carry it, check for a complementary API (e.g. a separate `set_oauth_scopes` call) that can — a missing argument on one setter doesn't mean the capability is absent. + +### 2026-10-10: learnings since 2026-10-09T17:29:31Z +- **Context:** PR #497 changed kernel `TelemetryConfig.Enabled` from `bool` to `*bool` so an unset `enableTelemetry` DSN value stays nil (kernel feature flag decides) while explicit `true`/`false` override it; the old `bool` conflated "unset" with "explicit false". + **Rule:** For a client knob that overrides a server/feature flag, model three states (unset / explicit-true / explicit-false) with a pointer or optional — a plain bool silently forces the "off" default and defeats server-side flag control. +- **Context:** In PR #497 the cgo shim began unconditionally calling the new `kernel_session_config_set_telemetry_config_v2` symbol, but `go.mod` still pinned the prebuilt bindings at v1.1.0; reviewers flagged that `-tags databricks_kernel` builds link the committed per-platform archives (not the `make kernel-lib` source build at `KERNEL_REV`), so they fail with an undefined-reference link error. + **Rule:** When a cgo shim starts calling a new kernel ABI symbol, bump the published bindings pin in go.mod/go.sum (to a version actually exporting that symbol) in the same change — bumping KERNEL_REV alone only fixes source builds; default kernel-tagged consumer builds link the committed archives and break at link time. +- **Context:** PR #497 hand-added the `KernelTelemetryConfig` struct and `_v2` prototype to the vendored `databricks_kernel.h` ahead of the upstream kernel landing them, and the cgo shim fills the struct with designated initializers; a reviewer noted any field reorder/padding/type divergence from the kernel's canonical layout is a silent ABI mismatch at the C boundary, not a compile error. + **Rule:** When hand-vendoring a C struct/prototype into a kernel header ahead of upstream, the field order/types/size must match the published header byte-for-byte and be re-verified once the real header lands — ABI divergence corrupts data silently rather than failing to compile. + +### 2026-10-14: learnings since 2026-10-13T17:31:21Z +- **Context:** In PR #496 (generalize driver-side feature-flag cache), reviewers flagged two unrelated changes bundled into the scoped PR — a `KERNEL_REV` bump and a large async-statement addition to the vendored `databricks_kernel.h` C ABI header; the author confirmed and removed the async surface. + **Rule:** Keep vendored-kernel changes (KERNEL_REV bumps, `databricks_kernel.h` C ABI edits) out of feature-scoped PRs — they muddy the changelog, break bisection, and risk Go bindings drifting from the declared ABI; split them into their own focused PR. +- **Context:** In PR #496 the feature-flag cache collapses concurrent cold readers through `singleflight.DoChan`, but the single executing `load()` captured the first caller's `ctx`; the cold path detached via `context.WithoutCancel` while the stale-refresh branch still ran `load(ctx)` under the winning caller's cancellable deadline, so one caller's cancellation/short deadline could abort the shared fetch and fail or perpetually re-stale other waiters. + **Rule:** When collapsing concurrent fetches behind singleflight (or any shared cache refresh), run the shared fetch on a detached, timeout-bounded context (e.g. `context.WithoutCancel` + `WithTimeout`) and let each waiter honor its own ctx via `select` — never let one caller's deadline abort work other goroutines depend on. +- **Context:** In PR #496 the generalized `GetString`/`GetInt32`/etc. feature-flag getters decode the stored flag value with `json.Unmarshal`, so a string flag only resolves when the server's `value` is a JSON literal (e.g. `"hello"`); an unquoted plain string errors instead of returning the raw value. + **Rule:** Typed getters that `json.Unmarshal` a raw string value require the stored value to be a JSON literal — document this and have consumers/producers JSON-encode string and list flag values, or non-bool getters will error on plain-string inputs. + +### 2026-10-20: learnings since 2026-10-19T17:29:11Z +- **Context:** PR #502 bumped the SQL kernel bindings to v1.2.0, syncing `databricks_kernel.h`; a reviewer found the new header dropped the "600s polling ceiling" wording from `kernel_statement_execute`, leaving `config.go` and `operation.go` Go doc comments describing behavior the shipped header no longer guarantees. + **Rule:** When bumping a vendored native/kernel header or bindings, grep the driver's own Go doc comments for behavior they mirror from that header (timeouts, ceilings, defaults) and update any that the new header contradicts — stale doc comments mislead the next reader even when code behavior is unchanged. +- **Context:** In PR #492's geospatial WKB path, a reviewer flagged that `rewriteGeometry` used `panic` in its `default` arm — reached per-row during a `database/sql` scan on server-provided bytes — while every other malformed-input case in the file returns a wrapped error that `kernelRows.next` surfaces cleanly. + **Rule:** In per-row scan/decode paths over server-provided data, return a wrapped error for malformed or unexpected input instead of panicking, so one bad row fails just the query rather than crashing the consuming application. +- **Context:** In PR #492, `ColumnTypeInfoForField`'s fallback branch reported `DatabaseTypeName = GEOMETRY/GEOGRAPHY` while the row scanner (gated on `arrow.STRUCT`) would not produce EWKB `[]byte` for a non-struct physical type, so the advertised column type and the scanned value could silently diverge. + **Rule:** Keep the reported `sql.ColumnType` (DatabaseTypeName/ScanType) and the value the scanner actually produces in lockstep; defensive fallback branches for unexpected physical representations should drop to the plain Arrow-derived mapping rather than advertise a logical type the scan path won't yield. +- **Context:** In PR #492 the `enableGeoSpatialSupport` DSN carrier allocated `KernelExperimental` whenever the value was non-nil (both true and false), unlike the adjacent `tokenCache` carrier that only allocates for the behavior-changing case — so a plain Thrift DSN carrying the option (even the no-op `=true`) failed connect with `ErrRequiresKernelBackend`, and the DSN-carrier-on-Thrift path had no test. + **Rule:** Kernel-only DSN option carriers should follow the `tokenCache` precedent — only allocate `KernelExperimental` when the value actually changes behavior — and pin the intended Thrift-path rejection with an explicit DSN-carrier test, not just an option test. + +### 2026-10-21: learnings since 2026-10-20T17:29:43Z +- **Context:** Engineer-bot author run #39539859873 (diagnosing the Thrift Arrow-path TIME/time64 bug) burned many turns re-issuing identical read-only inspection commands: `go doc github.com/apache/arrow/go/v12/arrow.Time64` ran at turns 26, 57, 63 and again in the next phase, `go env GOMODCACHE` ran twice, and repeated globs for the Arrow `datatype_fixedwidth.go` source returned nothing — the agent never cached a result or pivoted after `go doc` proved unhelpful. + **Rule:** Do not re-run an identical read-only inspection command (`go doc `, `go env GOMODCACHE`, same glob/grep) whose output you already have; when `go doc` on a third-party type yields nothing useful, resolve the module-cache path once and read the type's source directly instead of retrying the same command across turns. + +### 2026-10-22: learnings since 2026-10-21T17:29:21Z +- **Context:** Author run #39665678594 fixed issue #507, where `arrowRecordIterator` turned a failed mid-pagination `FetchResults` into `io.EOF`: `resultPageIterator.HasNext()` stored the real error in `rpf.err`, closed, and returned `false`, and the caller's `checkFinished()` ANDed `!HasNext()` into its done-decision with no error check, so a `for it.HasNext()` loop exited cleanly and swallowed the failure. + **Rule:** For iterator-style APIs (`HasNext()`/`Next()`), a `false`/termination signal must not conflate "done" with "errored" — expose and check a stored error (an `Err()` accessor) before treating end-of-iteration as success, and never overwrite a real error with an EOF/sentinel (set EOF only when `err == nil`). +- **Context:** Author run #39665654097 diagnosed that the driver reports `TIMESTAMP` for both TIMESTAMP and TIMESTAMP_NTZ because `ColumnTypeDatabaseTypeName`/`GetDBTypeName` derive the name from the Thrift `TColumnDesc`, where both types share `TTypeId_TIMESTAMP_TYPE` (id 8); the NTZ distinction exists only in the Arrow schema's `TimestampType.TimeZone` (empty for NTZ, populated for zoned). + **Rule:** The Thrift type id cannot distinguish TIMESTAMP from TIMESTAMP_NTZ — any code that must tell them apart (type-name metadata, value timezone handling) has to consult the Arrow schema's `TimestampType.TimeZone` field, not the Thrift `TColumnDesc`. +- **Context:** In author run #39665654097 the engineer-bot wrote live-warehouse e2e tests, then spent ~15 turns (go test, git status, probing existing e2e tests) rediscovering that the sandboxed bash tool scrubs credentials, so all e2e tests `Skip` and can never be observed red/green locally. + **Rule:** The bash sandbox strips warehouse/auth credentials, so live e2e tests always skip and cannot be driven red→green locally — don't burn turns probing env/creds; author unit tests at a lower in-process layer (e.g. Arrow-schema level) that run in the default build, keeping the e2e test only for the grading harness. + +### 2026-10-24: learnings since 2026-10-23T17:29:25Z +- **Context:** In PR #516 (Go 1.27.3 bump), a newer vet/lint check fired on `fmt.Errorf("%w", inner)` in `internal/errors/category_test.go`. The first fix dereferenced `inner`, so the test wrapped a `requestError` value instead of the `*requestError` pointer that production code returns. The reviewer flagged that this silently changed what the test covers. The merged fix keeps the pointer as `var inner error = ...` and adds `assert.Same`/`assert.ErrorIs` checks. + **Rule:** When a toolchain or linter upgrade flags a test, keep the type under test the same as in production (for example, a pointer error type). Satisfy the check with an interface-typed variable or a scoped `//nolint: // reason`, or add a separate case. Never swap pointer for value just to silence a check. +- **Context:** In PR #516, the Security Scan workflow briefly switched osv-scanner to `go install`. Fork PRs skip the `Setup JFrog` step, so that install would depend on GOPROXY access that the hardened protected runners may block. The merged version downloads the prebuilt binary from GitHub Releases instead. + **Rule:** CI jobs that run on fork PRs, where JFrog/GOPROXY is unavailable, must install tools from prebuilt release binaries rather than `go install`. Keep workflow comments about fork-PR behavior in sync whenever the install method changes. + +### 2026-10-27: learnings since 2026-10-26T17:30:35Z +- **Context:** PR #509 (ArrowBatchIterator masking a failed FetchResults as io.EOF). A refactor of `arrowRecordIterator.checkFinished()` split a short-circuited `&&` chain into separate statements. That made `resultPageIterator.HasNext()`, which sends a real FetchResults RPC, run on every call. Dropping the `isFinished ||` guard also meant a closed iterator re-fetched against a closed operation and reported a spurious error. + **Rule:** `resultPageIterator.HasNext()` has side effects (it sends a FetchResults RPC), so it must only run after local batches are drained AND the iterator is not finished. When refactoring boolean chains that contain it, keep the short-circuit order, and add a test that calls Close() early and asserts no FetchResults calls are made. +- **Context:** PR #509 fixed page-fetch errors being reported as io.EOF in `arrowRecordIterator`. The reviewer found the same bug in the sibling `arrowIPCStreamIterator`. Several more review rounds were needed on the semantics: report the error once, and make an explicit Close() discard a pending error while internal end-of-data keeps it (a `release()` vs `Close()` split). + **Rule:** When an iterator must tell a fetch failure apart from a normal end of results, check `Err()` when HasNext turns false. Hold the error as pending (HasNext=true until it is consumed) and report it exactly once from Next/Schema through one shared take-pending helper. An explicit Close() clears the pending error, but internal end-of-data release must not. Apply the same fix to every sibling iterator (record and IPC stream), and test each one for: row/stream count delivered before the error, fetch call count, and HasNext/Next after the error and after Close. +- **Context:** PR #510 (TIMESTAMP_NTZ reported as TIMESTAMP). The Thrift-path type-name override was read from whatever `r.RowScanner` held at the moment of the call. That scanner is nil before the first Next() (no direct results) and after the results are exhausted, and the column-based pages use a different scanner, so `ColumnTypes()` returned different names depending on page state. + **Rule:** Derive column metadata (DatabaseTypeName, ScanType) from the result-set metadata, i.e. `resultSetMetadata.ArrowSchema` (parse once and cache it). Never derive it from the current page's row scanner, whose lifecycle depends on page state. On Thrift, TIMESTAMP vs TIMESTAMP_NTZ can only be told apart by the Arrow `TimestampType.TimeZone` / field metadata, because TTypeId has no NTZ variant. String-timestamp and column-based paths cannot tell them apart. +- **Context:** PR #510 changed how TIMESTAMP_NTZ values are shifted. The unit suite passed, but `TestKernelE2ETimestampNTZ` in `kernel_e2e_test.go` still asserted the old behavior. That test is build-tagged (`databricks_kernel`) and needs credentials, so the local unit run never executed it. + **Rule:** Before claiming a behavior change is verified, grep the build-tagged and credential-gated e2e tests (`*_e2e_test.go`, `databricks_kernel` tag) for assertions on the old behavior and update them too. A passing unit suite does not cover them. Also update README.md, CONNECTION_PARAMETERS.md and doc.go together, and add a CHANGELOG "Behavior change" entry for user-visible value changes. + +### 2026-10-30: learnings since 2026-10-29T17:24:43Z +- **Context:** In engineer-bot author run #40976127852 (issue #475), the issue reported that `internal/rows/arrowbased/arrowRows.go` passes `context.Background()` to `NewCloudBatchIterator`/`NewLocalBatchIterator` and drops the caller's context. The bot found this is on purpose. `NewRows` detaches `rows.ctx` from the caller with `context.WithoutCancel` (ES-1934053, #295, #371), and `TestE2EArrowBatchesSurviveQueryContextCancellation` guards that behaviour. The bot left it unchanged. + **Rule:** Result fetching (CloudFetch/Arrow batch download) is deliberately detached from the caller's QueryContext cancellation via `context.WithoutCancel` in `NewRows`, so do not "fix" a `context.Background()` or detached ctx in the rows/arrowbased download path. Bound downloads with an explicit timeout or HTTP client option instead. +- **Context:** In engineer-bot author run #40976127852, the new e2e tests in `driver_e2e_test.go` (`TestE2ECloudFetchMinTimeToExpiryIsHonored`, `TestE2ECloudFetchHTTPClientOption`) were reported as red, but only because they did not compile. They then skipped in the fix phase because the author tool shell scrubs the `DATABRICKS_PECOTESTING_*` credentials. Their assertions never ran, and only the unit tests got a real red→green check. + **Rule:** E2E tests that use pecoTestingCreds skip in the engineer-bot author environment, and "red because it doesn't compile" is not proof of red. For each fix, also add a unit test that can actually run (a mock server via httptest, or config plumbing assertions) and that fails on its assertions before the fix, and flag any e2e-only verification as out of scope. + +### 2026-10-31: learnings since 2026-10-30T17:23:39Z +- **Context:** PR #525 added fast paths in `internal/arrowscan` `writeJSON` for Boolean and Int8–Int64 leaves, so they skip `scalarForJSON` + `json.Marshal` and allocate less. The only test was a Thrift/kernel render-parity case. A reviewer pointed out that a parity test checks that the output matches, not how much was allocated. A later edit could send these leaves back through the slow path and every test would still pass. + **Rule:** A PR that exists to cut allocations or speed things up must ship a `testing.AllocsPerRun` assertion or a `Benchmark*` for the hot path. Output-parity tests alone cannot catch a performance regression. + +### 2026-11-02: learnings since 2026-11-01T17:24:02Z +- **Context:** PR #526 upgraded go-oidc, go-jose (v3→v4) and x/oauth2 for security findings. The reviewer pointed out that the PR had no `## Unreleased` CHANGELOG line, even though earlier security bumps (klauspost/compress, thrift, x/crypto, x/mod, go-jose) all had one. The PR also added a `golang.org/x/crypto` requirement to go.mod that nothing imports. It is there as a deliberate minimum version for security scans, and a comment notes that `go mod tidy` removes it. The `osv-scanner.toml` suppression reasons depend on `go list -deps -test ./...` results, checked both with and without `-tags=databricks_kernel`. + **Rule:** For a security dependency bump, add a CHANGELOG `## Unreleased` entry that names the new versions and the advisory being fixed. After running `go mod tidy`, put back the deliberate x/crypto minimum it removes from go.mod. Re-check any osv-scanner.toml suppression reason with `go list -deps -test ./...`, both with and without `-tags=databricks_kernel`. + +### 2026-11-03: learnings since 2026-11-02T17:23:44Z +- **Context:** PR #527 added `internal/jsonfmt`, a hand-written fast path that renders nested scalars byte-for-byte the same as `json.Marshal` on both the Thrift (arrowbased) and kernel (arrowscan) renderers. Anything it doesn't handle falls back to `json.Marshal`. Parity was checked by exhaustive sweeps (every byte, every rune) and by randomized float and string inputs. Parity tests alone would still pass if the fast path were silently disabled, so the PR also pinned it with an `AllocsPerRun == 0` test. Review then noted that the ~2M-call sweeps ignored `testing.Short()` and slow down `-race` CI. + **Rule:** When you reimplement a stdlib or library function as a fast path with a fallback, assert byte-for-byte parity, including errors, against the original over exhaustive and randomized inputs. Also add a test that proves the fast path is actually taken, such as zero allocations or a fallback counter, because the fallback hides a disabled fast path. Gate the large sweeps behind `testing.Short()`. + +### 2026-11-09: learnings since 2026-11-08T17:23:32Z +- **Context:** PR #534 (OSV remediation that bumped brotli/compress) ran a bare `go mod tidy`. Tidy silently removed the deliberate `golang.org/x/crypto v0.55.0 // indirect` security-scan minimum pin (added in #464) along with its go.sum hashes. That left old, CVE-affected x/crypto pseudo-versions as the resolved version, and it broke the justification for the suppressions in `osv-scanner.toml`. The tooling also merged the separate kernel-bindings `require` block into the main block, so the lockstep-pin comment ended up above unrelated dependencies. The reviewer caught both problems, and both needed a human fix. + **Rule:** When bumping dependencies in go.mod, do not run a bare `go mod tidy`. Use `go get @`, then check the diff to confirm that deliberate `// indirect` security pins (e.g. golang.org/x/crypto) and their go.sum hashes are still there, and that the separate commented `require` blocks were not merged or reordered. The diff should show only the intended version bumps. + +### 2026-11-12: learnings since 2026-11-11T17:23:34Z +- **Context:** Three author runs (#43087748527, #43090396242, #43095026884) took on the same SEA/kernel issue (#517, duplicate column names such as `SELECT 1 AS a, 2 AS a`). All three traced it to the pinned `github.com/apache/arrow/go/v12`: `cdata.ImportCArrowSchema` and `cdata.ImportCRecordBatch` build the top-level `+s` struct with `arrow.StructOf`, which panics with `arrow: duplicate field with name`. The Thrift path decodes through `ipc.NewReader` / `arrow.NewSchema`, which accept duplicate names, so it is not affected. + **Rule:** In arrow-go v12, `arrow.StructOf` (and so the cdata importers in `internal/backend/kernel`) panics on duplicate field names, while `arrow.NewSchema` and `array.NewRecord` accept them. Check both the kernel/cdata path and the Thrift/IPC path when a bug involves column names, and don't assume one backend's behavior holds for the other. +- **Context:** On SEA/kernel issue #517, the three author runs reached different verdicts (two `blocked`, one `feasible`). The feasible run (#43090396242) edited cgo/C code in `internal/backend/kernel/rows.go` (`//go:build cgo && databricks_kernel`) even though the job builds only the pure-Go path (CGO_ENABLED=0, no tag, no native lib), so `go build ./...` never compiled that change. Its live e2e test also silently skipped because the bash tool strips the warehouse credentials. It spent several turns finding that out, then still reported `red_confirmed` based on pure-Go helper unit tests alone. + **Rule:** A bug whose fix lives only in `cgo && databricks_kernel`-tagged files cannot be compiled or reproduced in the author job. Report `blocked` early instead of editing uncompiled cgo code or treating proxy unit tests as the reproduction. When a `pecoTestingCreds` e2e test skips, treat the skip as an environment limit and report it in one step rather than spending turns investigating env vars. + +### 2026-11-13: learnings since 2026-11-12T17:23:28Z +- **Context:** PR #536 made `WithCloudFetch(false)` / `useCloudFetch=false` take effect on the kernel backend. The reviewer suggested forwarding the documented client-only key `cloudfetch_enabled=false`. The author showed that in kernel 1.4.0 this key selects INLINE+ARROW_STREAM, which has a size cap and which the server rejects, so it cannot return large results in full. Fetching complete results with no storage downloads needed the server session capability `can_cloud_download=false` (paged inline), which is the same capability JDBC uses. The repo docs that called it "not a valid SEA conf" were stale. Servers without paged-inline support reject it with `CONFIG_NOT_AVAILABLE` / SQLSTATE 42K0I, either at CreateSession or at the first statement. + **Rule:** Kernel no-CloudFetch means sending `can_cloud_download=false` (paged inline), not `cloudfetch_enabled=false` (size-capped INLINE). When you force a session key, first remove case-insensitive copies of it and of any conflicting keys from the caller's `WithSessionParams`. Do not trust repo docs about which SEA confs are valid without a live check. +- **Context:** PR #536 changed the CloudFetch option from inert to honored inside `buildKernelConfig`. The reviewer pointed out that `openSessionWithReydenFallback` (automatic Thrift→kernel recovery on KP001) builds the kernel session from the same `c.cfg`. So a Thrift-only user who never set `WithUseKernel` also gets the new kernel behavior. + **Rule:** Any change to `buildKernelConfig` also changes the Reyden auto-recovery path for Thrift users. When you make a previously-inert option active on the kernel path, add a connector test that combines the option with `openSessionWithReydenFallback`, and record the change as a behavior change in the CHANGELOG, because DSNs are often shared between Thrift and kernel. +- **Context:** PR #536 needed to show that `WithCloudFetch(false)` really avoids cloud-storage downloads on the kernel path. Unit tests that only check config-map entries were not enough. The e2e test routes the kernel through `WithKernelProxy` to an httptest CONNECT proxy. The proxy tunnels only to the workspace host:443, rejects and counts every other destination, and the test asserts zero blocked connections plus a full ordered row and hash check. + **Rule:** To prove a 'no direct storage/network egress' guarantee, run the driver through a restricted CONNECT proxy that allows only the workspace host, then assert that blocked attempts == 0 and that the result is complete. Do not rely only on asserting which conf keys were set. + +### 2026-11-14: learnings since 2026-11-13T17:23:17Z +- **Context:** PR #555 made Thrift `ExecuteStatement` retryable, but only when the connection to the server or proxy was never opened. `isDialError` matches `net.OpError{Op:"dial"}`, including a dial wrapped in `proxyconnect`. It excludes context cancellation, DNS not-found, and TLS, read and write failures. An attempt that followed a redirect is treated as possibly sent, because an earlier hop already received the request. + **Rule:** Never retry non-idempotent RPCs such as ExecuteStatement on generic transport errors. Retry them only on dial errors, which prove no request bytes were sent, and treat any redirected attempt as possibly already delivered. +- **Context:** PRs #559 and #558 fixed flaky and racy tests. Unjoined `go drainKernelLogs(...)` goroutines and CloudFetch downloads kept running after their test returned, then wrote into a later test's log buffer or raced under `-race`. The fixes join each goroutine in `t.Cleanup` with a bounded `select` timeout. They also pass a cancellable ctx with `defer cancel()` to iterators, and replace `time.Sleep` with waiting on a done channel. + **Rule:** Join every goroutine a test starts, waiting in t.Cleanup with a timeout, and give background workers a ctx that is cancelled when the test returns. Synchronize on done channels, not time.Sleep, so leaked goroutines can't corrupt later tests. +- **Context:** In a PR #555 review, a custom `http.Transport.DialContext` closure called a helper that used `require.NoError`. The transport runs `DialContext` on its own goroutine, so a failure there calls `t.FailNow` off the test goroutine. That exits the dial goroutine instead of the test, which hangs the test or fails it confusingly. + **Rule:** Do not call require.*/t.FailNow/t.Fatal inside callbacks that run on goroutines the test doesn't own, such as http handlers, DialContext and drain loops. Compute fixtures before the closure, and report failures through t.Error or a channel. +- **Context:** In a PR #550 review, a reviewer flagged as critical that a new test file imported `github.com/apache/arrow/go/v12/...`. PR #535 had already moved the repo to `github.com/apache/arrow-go/v18`, and v12 is no longer in go.mod, so the import breaks build and lint for the whole package. v18 also renames APIs: `arrow.Record` became `arrow.RecordBatch`, `ipc.Reader.Record()` became `RecordBatch()`, and `NewRecord` is deprecated in favor of `NewRecordBatch`. + **Rule:** Import Arrow only from github.com/apache/arrow-go/v18 and use the RecordBatch APIs. Never copy arrow/go/v12 imports from older code, branches or snippets into new files. +- **Context:** In PR #539, `NewResultPageIterator` took two adjacent positional bools, `isFinished` and `closedOnServer`. A swap would still compile, and passing `closedOnServer=true` by mistake skips CloseOperation and leaks the server operation. The fix takes `hasMoreRows *bool` plus `closedOnServer bool` and derives completion in one place. + **Rule:** Avoid adjacent same-typed bool parameters whose swap causes resource leaks. Use distinct types (for example *bool vs bool) or an options struct, and derive combined state in exactly one place. +- **Context:** PR #556 found that the DSN `timeout`, given in seconds, was multiplied straight into a duration with `time.Duration(n) * time.Second`. Values beyond ±9223372036 seconds silently wrapped to a different and often negative timeout. The fix range-checks the value against `math.MaxInt64/time.Second` and `math.MinInt64/time.Second` before converting, and returns an error that names the param. + **Rule:** When converting user-supplied integer units to time.Duration, check the value against MaxInt64/unit and MinInt64/unit before multiplying. Reject out-of-range values with an error naming the param instead of letting them overflow. + +### 2026-11-16: learnings since 2026-11-15T17:23:28Z +- **Context:** PR #567 put the live warehouse tests behind an `e2e` build tag. The nightly workflows had to switch to `-tags e2e` and `-tags databricks_kernel,e2e`. Plain `go test`/`make test` never compiles these tests, so CI added `go vet -tags e2e .` and `CGO_ENABLED=1 go vet -tags databricks_kernel,e2e .` steps that only check they compile. + **Rule:** Live e2e tests need the `e2e` build tag (kernel ones need `databricks_kernel,e2e`). When you add or change one, run `go vet` with those tags and add its name to the `-run` regex in nightly-e2e.yml, or it is never compiled or run. +- **Context:** PRs #563 and #567 added DECIMAL qualifiers and TIME metadata. On SEA/kernel, an empty-result manifest schema can describe a typed column as a UTF-8 Arrow field, with the real type only in `databricks.type_name` / Spark SQL-name field metadata. Thrift TIME metadata depends on the server-supplied Arrow schema. Both PRs had to test zero-row and NULL-only results separately from populated ones. + **Rule:** When adding support for a result type, get column metadata from the field (`ColumnTypeInfoForField`), not only the physical Arrow type. SEA empty results can carry the logical type only in field metadata on a string field. Test zero-row, NULL-only and populated results on both Thrift and SEA. +- **Context:** PR #561 removed a `//go:build tools` `package tools` file from the module root. Under `-tags tools` that file clashed with the root `dbsql` package and broke build, vet and Gazelle. The tool dependency moved to a go.mod `tool` directive, and a test now checks that every root `*.go` file is `dbsql` or `dbsql_test`. + **Rule:** Every `.go` file in the module root must be `package dbsql` or `dbsql_test`, including files behind build tags. Declare dev tools with the go.mod `tool` directive, not a tagged `tools.go` file.