From 77bfd522cc046918ab6a6be1a444f73aa0e2fd04 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Thu, 13 Aug 2026 17:43:12 +0000 Subject: [PATCH 01/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 988c85f4..6ff53d3c 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -10,3 +10,14 @@ configured in `.bot/config.yaml`: No learnings have been recorded yet. Dated sections are appended below by the retrospective flow. +## Entries + +### 2026-08-13: learnings since 2026-08-12T17:42:15Z +- **Context:** PR #442 configured a bot's `MODEL_ENDPOINT`; using the generic `.../serving-endpoints/anthropic/invocations` form 400'd every scheduled run (`Unsupported native API path .../anthropic/invocations/v1/messages`), confirmed empirically against the sibling databricks-sql-python cron. + **Rule:** Point Databricks bot `MODEL_ENDPOINT` at the concrete `.../serving-endpoints//invocations` form, never `.../serving-endpoints/anthropic/invocations` — `translate_endpoint` early-returns on URLs already containing `/serving-endpoints/anthropic`, leaving the `/invocations` suffix so the CLI appends `/v1/messages` and hits HTTP 400. +- **Context:** PR #442's learning workflow declared a `workflow_dispatch` input as a string (`window-hours`) with a comment noting that `type: number` breaks the run. + **Rule:** In GitHub Actions, a `workflow_dispatch` input declared `type: number` fails the whole run at startup ("workflow file issue") when the workflow also has a `schedule` trigger — declare numeric dispatch inputs as `type: string` and coerce to int downstream (e.g. via argparse). +- **Context:** PR #442 initially set `retrospective.system_prompt: prompts/retrospective_system.md`, a file that did not exist; the maintainer confirmed the engine treats a set-but-missing prompt path as a hard error, which would have failed the daily cron every run. + **Rule:** For engine config keys that reference a file (e.g. `system_prompt`): an UNSET key falls back to the engine's built-in default, but a SET key pointing at a missing file is a hard error — omit the key entirely rather than point it at a nonexistent path. +- **Context:** PR #442 needed the retrospective flow to commit `.claude/knowledge/learning-log.md`, but the repo `.gitignore` ignored all of `.claude`; the fix scoped the ignore (`.claude/*` + `!.claude/knowledge/` + `.claude/knowledge/*` + `!.claude/knowledge/learning-log.md`) and seeded the file so the author read path never hits a missing file. + **Rule:** When a bot/tool must commit a file under a normally-ignored directory, add scoped `.gitignore` negation for exactly that file (ignoring intermediate dirs still hides children, so re-include each level) and seed the file, so both the write (commit) and read paths resolve. From 7c711e70e9e8ba90f0ec1b52236e79ea332f1445 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Fri, 21 Aug 2026 17:33:48 +0000 Subject: [PATCH 02/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 6ff53d3c..8cf434d5 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -21,3 +21,7 @@ retrospective flow. **Rule:** For engine config keys that reference a file (e.g. `system_prompt`): an UNSET key falls back to the engine's built-in default, but a SET key pointing at a missing file is a hard error — omit the key entirely rather than point it at a nonexistent path. - **Context:** PR #442 needed the retrospective flow to commit `.claude/knowledge/learning-log.md`, but the repo `.gitignore` ignored all of `.claude`; the fix scoped the ignore (`.claude/*` + `!.claude/knowledge/` + `.claude/knowledge/*` + `!.claude/knowledge/learning-log.md`) and seeded the file so the author read path never hits a missing file. **Rule:** When a bot/tool must commit a file under a normally-ignored directory, add scoped `.gitignore` negation for exactly that file (ignoring intermediate dirs still hides children, so re-include each level) and seed the file, so both the write (commit) and read paths resolve. + +### 2026-08-21: learnings since 2026-08-20T17:32:23Z +- **Context:** PR #446 corrected CONNECTION_PARAMETERS.md to document how session confs and proxies diverge between the Thrift and kernel (SEA) backends. + **Rule:** The kernel/SEA path is stricter than Thrift: session-conf keys are matched case-insensitively against an allowlist (non-allowlisted keys are dropped with a warning, a few hard-rejected e.g. HTTP 400 INVALID_CONF_VALUE), and only http(s) proxies are accepted (socks* URLs honored on Thrift are rejected at connect) — so a conf/proxy that works on Thrift may be silently inert or rejected on kernel; verify kernel behavior separately when adding or relying on any session parameter or proxy feature. From e4852e957386a70423c1342313fed46c7d3d611e Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Sat, 22 Aug 2026 17:32:32 +0000 Subject: [PATCH 03/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 8cf434d5..faed0237 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -25,3 +25,13 @@ retrospective flow. ### 2026-08-21: learnings since 2026-08-20T17:32:23Z - **Context:** PR #446 corrected CONNECTION_PARAMETERS.md to document how session confs and proxies diverge between the Thrift and kernel (SEA) backends. **Rule:** The kernel/SEA path is stricter than Thrift: session-conf keys are matched case-insensitively against an allowlist (non-allowlisted keys are dropped with a warning, a few hard-rejected e.g. HTTP 400 INVALID_CONF_VALUE), and only http(s) proxies are accepted (socks* URLs honored on Thrift are rejected at connect) — so a conf/proxy that works on Thrift may be silently inert or rejected on kernel; verify kernel behavior separately when adding or relying on any session parameter or proxy feature. + +### 2026-08-22: learnings since 2026-08-21T17:32:18Z +- **Context:** PR #449 fixed Azure OAuth U2M on the kernel backend — `resolveKernelAuth` had been forwarding the cloud-inferred client id + scopes (`a.U2MClientID()` / `oauth.GetScopes`) that the Thrift path infers from the host, which routed the kernel's browser to a broken AAD authorize URL on Azure. + **Rule:** The kernel/SEA backend runs ONE cloud-blind in-house workspace-federated U2M flow (OIDC discovery against `{host}/oidc`, no Azure branching), so its auth mapping must forward the fixed in-house `databricks-sql-connector` client + `offline_access`+`sql` scopes uniformly across all clouds — do NOT forward the Azure Entra-direct app id / `user_impersonation` scope the Thrift path uses, even though both paths authorize identically on AWS/GCP. +- **Context:** PR #449 replaced `oauth.GetScopes(host, nil)` with a hardcoded scope slice and initially wrote `[]string{"sql", "offline_access"}`, producing a High-severity failure: one test asserted the reversed order and contradicted its sibling test, because `resolveKernelAuth` returns `[]string{"offline_access", "sql"}`. + **Rule:** OAuth scopes are a space-delimited unordered set per spec, but `reflect.DeepEqual` on `[]string` is order-sensitive — when hardcoding a scope slice that replaces `oauth.GetScopes` (which appends `offline_access` first, then `sql`), preserve the exact element order and keep all sibling assertions consistent, or DeepEqual comparisons/tests will fail spuriously. +- **Context:** PR #444 added `WithFederatedTokenProvider*` on the kernel path; for account-wide federation (no client id) the driver hands the kernel the raw un-exchanged external subject token via `set_auth_pat`, unlike the Thrift path which exchanges in-driver via `FederationProvider`. Reviewers repeatedly questioned whether this silently fails; a maintainer confirmed the kernel's behavior. + **Rule:** The kernel performs a mandatory server-side token exchange for tokens presented on the PAT path UNLESS the token is same-issuer or non-JWT — so handing a raw external-IdP JWT subject token to `set_auth_pat` (client id only set for SP-wide federation) correctly federates account-wide without driver-side exchange. Rely on this documented kernel guarantee rather than assuming un-exchanged tokens are treated as literal PATs. +- **Context:** In both PR #449 and PR #444, reviewers (Copilot, peco-review-bot) flagged multiple stale/contradictory doc+comment sites after a kernel-auth behavior change — the behavior is mirrored across `doc.go`, `README.md`, `CONNECTION_PARAMETERS.md`, `internal/backend/kernel/auth.go` (Auth struct + provider-interface docs), `backend.go` (setAuth comment), and `auth/oauth/u2m/authenticator.go`. + **Rule:** This repo documents kernel/Thrift auth semantics redundantly across many files; when changing behavior on one auth path, sweep ALL mirrored doc/comment sites in the same PR (not just the function you edited) or reviewers will flag stale contradictions. From 95737b8f302ece8662d808e838d1f12a05553924 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Mon, 24 Aug 2026 17:34:34 +0000 Subject: [PATCH 04/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index faed0237..36658b39 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -35,3 +35,17 @@ retrospective flow. **Rule:** The kernel performs a mandatory server-side token exchange for tokens presented on the PAT path UNLESS the token is same-issuer or non-JWT — so handing a raw external-IdP JWT subject token to `set_auth_pat` (client id only set for SP-wide federation) correctly federates account-wide without driver-side exchange. Rely on this documented kernel guarantee rather than assuming un-exchanged tokens are treated as literal PATs. - **Context:** In both PR #449 and PR #444, reviewers (Copilot, peco-review-bot) flagged multiple stale/contradictory doc+comment sites after a kernel-auth behavior change — the behavior is mirrored across `doc.go`, `README.md`, `CONNECTION_PARAMETERS.md`, `internal/backend/kernel/auth.go` (Auth struct + provider-interface docs), `backend.go` (setAuth comment), and `auth/oauth/u2m/authenticator.go`. **Rule:** This repo documents kernel/Thrift auth semantics redundantly across many files; when changing behavior on one auth path, sweep ALL mirrored doc/comment sites in the same PR (not just the function you edited) or reviewers will flag stale contradictions. + +### 2026-08-24: learnings since 2026-08-23T17:29:41Z +- **Context:** PR #450 (kernel log-forwarding cgo bridge) — a reviewer flagged a helper that coerced a `cgo.Handle` (an integer token) into a small fabricated `void*` and passed it as the C callback's `user_data`. + **Rule:** Never pass a `cgo.Handle` or any fabricated/Go-managed pointer into a C pointer slot (`void* user_data`); the GC can detect the invalid pointer and abort via `runtime.throw`, which `recover` cannot catch — use NULL `user_data` with a package-global sink, or allocate a real C cell and pass its address. +- **Context:** PR #450 — the kernel invokes the log callback synchronously on the emitting native thread and forbids blocking/re-entry, but the first implementation ran zerolog plus an arbitrary user `SetLogOutput` writer inside the callback. + **Rule:** In a synchronous FFI callback that runs on a foreign thread, do only minimal bounded work: copy borrowed strings into owned memory and non-blockingly enqueue onto a bounded channel, then drain and do the real I/O on a separate goroutine — never run arbitrary user I/O on the callback thread (it can stall the native op or re-enter the ABI). +- **Context:** PR #450 — the logger held its destination-routing mutex while executing the user-owned `Write`, so a blocking writer (or one that called `SetLogOutput`/logged) could deadlock and could not be replaced. + **Rule:** Never invoke arbitrary user-supplied code (a `Write`, callback, or handler) while holding an internal lock; publish the target via an atomic pointer / short-held lock and invoke it only after releasing the lock, so a stuck consumer can never block retargeting or self-deadlock. +- **Context:** PR #450 — `installKernelLogCallback` returned early inside `logCallbackOnce.Do` when the level was OFF; a reviewer noted `sync.Once` still marks itself complete, latching forwarding off permanently for the process. + **Rule:** A conditional early `return` inside `sync.Once.Do` still records the Once as done, so a later attempt becomes a permanent no-op; if a step must be retryable when a precondition later changes, gate the precondition outside `Once.Do` rather than short-circuiting within it. +- **Context:** PR #450 — building the root zerolog instance over a stable proxy that implemented only `io.Writer` permanently downgraded any later `zerolog.LevelWriter` (e.g. `MultiLevelWriter`, syslog) to plain `Write`, losing severity-aware routing. + **Rule:** A long-lived proxy/wrapper placed in front of a swappable destination must implement the richest interface its underlying targets may support (e.g. `WriteLevel` for `zerolog.LevelWriter`) and delegate to it when present; wrapping only the base interface silently strips capabilities of any future richer target. +- **Context:** PR #450 — dropped kernel log records under channel backpressure were counted but the count was only read by tests, so a burst past capacity silently lost lines with no operator-visible signal. + **Rule:** When dropping data under backpressure (bounded queue overflow, sampling, truncation), surface it — at minimum a one-shot/periodic warning through the normal output path — so consumers can distinguish 'nothing was produced' from 'output was silently dropped'. From 3e0ea54cb1e74dd8cfee4cd3dab95bd09315358e Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Tue, 25 Aug 2026 17:35:08 +0000 Subject: [PATCH 05/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 36658b39..9cf79e66 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -49,3 +49,11 @@ retrospective flow. **Rule:** A long-lived proxy/wrapper placed in front of a swappable destination must implement the richest interface its underlying targets may support (e.g. `WriteLevel` for `zerolog.LevelWriter`) and delegate to it when present; wrapping only the base interface silently strips capabilities of any future richer target. - **Context:** PR #450 — dropped kernel log records under channel backpressure were counted but the count was only read by tests, so a burst past capacity silently lost lines with no operator-visible signal. **Rule:** When dropping data under backpressure (bounded queue overflow, sampling, truncation), surface it — at minimum a one-shot/periodic warning through the normal output path — so consumers can distinguish 'nothing was produced' from 'output was silently dropped'. + +### 2026-08-25: learnings since 2026-08-24T17:32:28Z +- **Context:** PR #456 added `WithKernelClientCertificate`; a paired credential where an empty cert/key is invalid. It introduced a dedicated `TLSClientCertConfigured` bool alongside the PEM buffers and validated at connect time (`ErrInvalidKernelConfig`). + **Rule:** For optional paired-credential/config options where empty is invalid, carry an explicit "configured" sentinel flag (not just checking for non-empty values) so an explicit call with empty input is rejected rather than silently treated as unset — prevents failing open. +- **Context:** PR #455 corrected the kernel connection docs: the connect-context deadline was described as unhonored only during U2M browser login, but it is actually unhonored mid-connect for ALL auth modes (PAT/M2M included). + **Rule:** On the kernel/SEA path the connect-context deadline is checked only at entry to session-open; the kernel's blocking C-ABI session-open then runs uninterruptibly, so a slow cold-start or network partition can block past the deadline regardless of auth mechanism — not a U2M-specific limitation. +- **Context:** PR #455's new kernel session-conf allowlist table transcribed state living in vendored kernel source (`build/kernel-src/src/config.rs`) not present in the repo checkout; a review-bot finding flagged it as un-guarded and drift-prone, and the fix pinned the source and called out the missing CI guard. + **Rule:** When documenting behavior sourced from vendored/external code that isn't in the repo checkout (so no test or CI guard anchors it), pin the exact source ref/path, name which entries have repo-side anchors, and explicitly warn the rest may lag — treat the external source as authoritative. From 8f0e74f1b5085b8bcc4cdf790b9ccd71380bfabc Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Wed, 26 Aug 2026 18:07:08 +0000 Subject: [PATCH 06/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 9cf79e66..abc15e9f 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -57,3 +57,11 @@ retrospective flow. **Rule:** On the kernel/SEA path the connect-context deadline is checked only at entry to session-open; the kernel's blocking C-ABI session-open then runs uninterruptibly, so a slow cold-start or network partition can block past the deadline regardless of auth mechanism — not a U2M-specific limitation. - **Context:** PR #455's new kernel session-conf allowlist table transcribed state living in vendored kernel source (`build/kernel-src/src/config.rs`) not present in the repo checkout; a review-bot finding flagged it as un-guarded and drift-prone, and the fix pinned the source and called out the missing CI guard. **Rule:** When documenting behavior sourced from vendored/external code that isn't in the repo checkout (so no test or CI guard anchors it), pin the exact source ref/path, name which entries have repo-side anchors, and explicitly warn the rest may lag — treat the external source as authoritative. + +### 2026-08-26: learnings since 2026-08-25T17:32:57Z +- **Context:** In PR #458 a reviewer caught an unused local variable (`k declared and not used`) in `internal/backend/kernel/backend.go`, a file behind `//go:build cgo && databricks_kernel`. It is a hard Go compile error, but the default `go test` excludes that tag so ordinary CI would not catch it — only the tagged kernel test environment fails. + **Rule:** Code behind build tags excluded from the default test run (e.g. `cgo && databricks_kernel`) is not compiled or vetted by ordinary `go test`/CI; compile and run the tagged build separately before relying on it, since even trivial compile errors (unused vars) there pass default CI. +- **Context:** In PR #458 the test seam `trySetTokenCacheConfig` originally re-implemented the `set_u2m_token_cache_config` cgo call standalone instead of routing through the production `setAuth`. A reviewer noted that a future edit dropping/mis-wiring the setter inside `setAuth`'s U2M branch would leave every test green; the fix rebuilt a real `KernelBackend` and called `k.setAuth(cfg)`. + **Rule:** Test seams should drive the real production code path (call the actual method under test), not a parallel re-implementation of the same underlying call — a standalone copy asserts the C signature but not that the production path still invokes it, so regressions in the real wiring go uncaught. +- **Context:** In PR #458 the `WithTokenCache(bool)` option calls `kernelExperimental(c)`, which allocates `KernelExperimental` unconditionally — so even `WithTokenCache(false)` makes `KernelExperimental != nil` and is rejected on the Thrift backend with `ErrRequiresKernelBackend`. The DSN carrier `tokenCache=false` deliberately does NOT forward, so only it is a true no-op. Godoc initially (wrongly) claimed disabling was "a no-op on any backend." + **Rule:** `WithKernel*` options allocate the kernel-only `KernelExperimental` struct unconditionally, so passing even the disabling/`false` value opts the connection into the kernel backend and fails on Thrift; only the DSN carrier for such a flag can be a genuine no-op. Scope any "no-op" doc claim to the DSN path, and mind that the option and DSN entry points for the same setting diverge (option always opts in; DSN `false` does not). From 11ed01cf8d47a3248de49c1708977e676c1eef45 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Thu, 27 Aug 2026 20:45:14 +0000 Subject: [PATCH 07/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index abc15e9f..27ae8612 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -65,3 +65,11 @@ retrospective flow. **Rule:** Test seams should drive the real production code path (call the actual method under test), not a parallel re-implementation of the same underlying call — a standalone copy asserts the C signature but not that the production path still invokes it, so regressions in the real wiring go uncaught. - **Context:** In PR #458 the `WithTokenCache(bool)` option calls `kernelExperimental(c)`, which allocates `KernelExperimental` unconditionally — so even `WithTokenCache(false)` makes `KernelExperimental != nil` and is rejected on the Thrift backend with `ErrRequiresKernelBackend`. The DSN carrier `tokenCache=false` deliberately does NOT forward, so only it is a true no-op. Godoc initially (wrongly) claimed disabling was "a no-op on any backend." **Rule:** `WithKernel*` options allocate the kernel-only `KernelExperimental` struct unconditionally, so passing even the disabling/`false` value opts the connection into the kernel backend and fails on Thrift; only the DSN carrier for such a flag can be a genuine no-op. Scope any "no-op" doc claim to the DSN path, and mind that the option and DSN entry points for the same setting diverge (option always opts in; DSN `false` does not). + +### 2026-08-27: learnings since 2026-08-26T18:05:10Z +- **Context:** PR #457 forwarded the driver's ClientTimeout to the kernel C ABI (`kernel_session_config_set_request_timeout`). Reviewers noted that passing `0` does not mean unlimited or immediate — the kernel substitutes its own 120s default. + **Rule:** When forwarding a timeout/limit to the kernel C ABI, treat `0` as the "use kernel default" sentinel (120s for request timeout), not as unlimited or zero-wait; document this at every knob and account for it when reporting effective values. +- **Context:** In PR #457, telemetry's `SocketTimeout` was initially populated with a millisecond value, but reviewers flagged that the receiver proto tags millisecond durations with an explicit `_ms`/`_millis` suffix (`retry_overall_timeout_ms`, `result_set_ready_latency_millis`), whereas the bare `socket_timeout` field is interpreted in seconds — the fix converted to seconds. + **Rule:** Before populating a telemetry duration field, match its unit to the receiver schema's field-name convention: bare names (e.g. `socket_timeout`) are seconds, only `_ms`/`_millis`-suffixed names are milliseconds — mismatching over/under-reports by 1000×. +- **Context:** PR #457 had two paired duration converters: the kernel setter rounded positive sub-millisecond values *up* to 1 (so a real timeout never collapses into the `0 = default` sentinel), while the telemetry helper rounded positive sub-second values *down* to 0 — meaning a small nonzero configured timeout was forwarded as a real deadline yet reported as "unset" (the field was `omitempty`). + **Rule:** When paired converters share a `0 = default/unset` sentinel, round positive sub-unit values consistently (floor them to 1, never down to 0) so a genuinely-configured value is never misreported as absent. From b2e1d5173d5c7d7bef6b18fb302c21e2f41c126a Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Fri, 28 Aug 2026 21:05:49 +0000 Subject: [PATCH 08/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 27ae8612..66cad344 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -73,3 +73,17 @@ retrospective flow. **Rule:** Before populating a telemetry duration field, match its unit to the receiver schema's field-name convention: bare names (e.g. `socket_timeout`) are seconds, only `_ms`/`_millis`-suffixed names are milliseconds — mismatching over/under-reports by 1000×. - **Context:** PR #457 had two paired duration converters: the kernel setter rounded positive sub-millisecond values *up* to 1 (so a real timeout never collapses into the `0 = default` sentinel), while the telemetry helper rounded positive sub-second values *down* to 0 — meaning a small nonzero configured timeout was forwarded as a real deadline yet reported as "unset" (the field was `omitempty`). **Rule:** When paired converters share a `0 = default/unset` sentinel, round positive sub-unit values consistently (floor them to 1, never down to 0) so a genuinely-configured value is never misreported as absent. + +### 2026-08-28: learnings since 2026-08-27T20:43:32Z +- **Context:** PR #440 wired per-platform kernel archives as nested Go modules using an in-tree `replace` pointing at a placeholder `require ... v0.0.0`/`v0.0.1`; reviewers flagged it builds in-tree but breaks external consumers. + **Rule:** Go `replace` directives are honored ONLY in the main module — a downstream `go get` of your module ignores them, so never rely on an in-tree `replace` to shadow a placeholder `require` version; publish and pin the dependency at a real tagged version before advertising a `go get`-installable build. +- **Context:** PR #440 review: an unresolvable nested-module `require` would break even pure-Thrift (`CGO_ENABLED=0`, no build tag) consumer builds that never compile a kernel file. + **Rule:** Go module-graph resolution (MVS) is build-tag-independent — every `require`d module's `go.mod` must resolve for ALL consumers regardless of build tags, so an unpublished/placeholder version in `go.mod` fails builds that never touch that code, not just the tag-gated ones. +- **Context:** PR #440 added a `!databricks_kernel_dynlib` term to only the darwin/arm64 cgo link shim while the other shims and the `cgo_unsupported.go` compile guard stayed dynlib-unaware, so building with that tag compiles no link shim and fails with opaque `undefined reference` link errors. + **Rule:** With build-tag-gated cgo link shims plus a companion `unsupported`/compile-time guard, any new build-tag term must be applied consistently across every shim AND the guard's exclusion list — an asymmetric term silently drops all shims and defeats the guard whose job is to turn link failures into a clear compile error. +- **Context:** PR #440 CI sync step used `go mod verify || true`, which reviewers noted swallows genuine module-cache corruption/tampering failures while the surrounding comment framed the step as verifying the module graph. + **Rule:** Do not mask verification/gating commands (`go mod verify`, checksum/lint checks) with `|| true` — it defeats the check's purpose; let the command fail the job, or log an explicit justification if a non-fatal run is truly intended. +- **Context:** PR #451's `applyTelemetry` always forwarded hardcoded Go-side constants (batch size, flush interval, etc.) across the kernel C ABI even when unset, while `CONNECTION_PARAMETERS.md` promised the kernel's own default applied "when unset." + **Rule:** When docs state the downstream/native layer owns a default "when unset," do not hardcode a duplicated copy of that default on the wrapper side and always forward it — forward a sentinel that lets the native layer keep its own default, or change the docs to state the driver owns the value; duplicated defaults silently drift when the native side changes and no test catches it. +- **Context:** PRs #463/#440 documented a `make kernel-lib` → `make build-kernel` → `make test-kernel` contributor flow, but reviewers noted `build-kernel`/`test-kernel` link the published bindings modules, not the local `.a` that `kernel-lib` produces; linking the local build needs a still-unwired manual `go.work` step. + **Rule:** In this repo the kernel dev-loop does not link a locally built archive by default — `make build-kernel`/`test-kernel` link the published `databricks-sql-kernel-bindings` modules; to test a local source build you must manually point a `go.work`/replace at the built `lib/` archive, so verify which archive is actually linked before trusting a source-build change. From fd921453d0a9d5fe210efb7504f10e36fc9f4718 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Fri, 4 Sep 2026 17:26:36 +0000 Subject: [PATCH 09/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 66cad344..d2856d84 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -87,3 +87,7 @@ retrospective flow. **Rule:** When docs state the downstream/native layer owns a default "when unset," do not hardcode a duplicated copy of that default on the wrapper side and always forward it — forward a sentinel that lets the native layer keep its own default, or change the docs to state the driver owns the value; duplicated defaults silently drift when the native side changes and no test catches it. - **Context:** PRs #463/#440 documented a `make kernel-lib` → `make build-kernel` → `make test-kernel` contributor flow, but reviewers noted `build-kernel`/`test-kernel` link the published bindings modules, not the local `.a` that `kernel-lib` produces; linking the local build needs a still-unwired manual `go.work` step. **Rule:** In this repo the kernel dev-loop does not link a locally built archive by default — `make build-kernel`/`test-kernel` link the published `databricks-sql-kernel-bindings` modules; to test a local source build you must manually point a `go.work`/replace at the built `lib/` archive, so verify which archive is actually linked before trusting a source-build change. + +### 2026-09-04: learnings since 2026-09-03T17:27:27Z +- **Context:** In PR #469 the kernel-rev sync also rewrote user-facing docs (CONNECTION_PARAMETERS.md/README) to claim the kernel now forwards `WithSessionParams` unchanged with no allowlist, but the review flagged that released-driver consumers link the *published* `databricks-sql-kernel-bindings` module in `go.mod`, whose archive contents live outside this repo checkout — the no-allowlist behavior was only verified in the source-built pinned `KERNEL_REV`. + **Rule:** When a PR documents new kernel behavior, gate the doc change on the shipped/published bindings version that released-driver users actually link — don't describe source-built `KERNEL_REV` behavior as current unless you can confirm the pinned bindings module in `go.mod` contains it (bindings archive contents are not in this checkout, so treat them as unverifiable from the repo alone). From 1dd3d56dac9213482d4da59da303733eb323621c Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Sat, 5 Sep 2026 17:25:41 +0000 Subject: [PATCH 10/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index d2856d84..3a5e813e 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -91,3 +91,11 @@ retrospective flow. ### 2026-09-04: learnings since 2026-09-03T17:27:27Z - **Context:** In PR #469 the kernel-rev sync also rewrote user-facing docs (CONNECTION_PARAMETERS.md/README) to claim the kernel now forwards `WithSessionParams` unchanged with no allowlist, but the review flagged that released-driver consumers link the *published* `databricks-sql-kernel-bindings` module in `go.mod`, whose archive contents live outside this repo checkout — the no-allowlist behavior was only verified in the source-built pinned `KERNEL_REV`. **Rule:** When a PR documents new kernel behavior, gate the doc change on the shipped/published bindings version that released-driver users actually link — don't describe source-built `KERNEL_REV` behavior as current unless you can confirm the pinned bindings module in `go.mod` contains it (bindings archive contents are not in this checkout, so treat them as unverifiable from the repo alone). + +### 2026-09-05: learnings since 2026-09-04T17:25:10Z +- **Context:** PR #470 added per-statement query tags on the kernel backend; a reviewer caught that the serialized tags string reaching the length-less `kernel_statement_set_query_tags` C ABI setter had no interior-NUL guard, unlike SQL text (`checkQueryText`/`errQueryNUL`) and bound params (`checkParamValue`/`errParamNUL`). + **Rule:** Any user-supplied string passed to a NUL-terminated (length-less) kernel C ABI setter must be guarded for interior NUL bytes before cgo marshaling — a NUL silently truncates on the kernel path while the length-prefixed Thrift `confOverlay` carries the value whole, breaking Thrift/SEA parity; fail loudly, and place the guard in an untagged pure-Go file so it is unit-tested under CGO_ENABLED=0. +- **Context:** PR #468 bumped the vendored `databricks_kernel.h`, flipping the documented `kernel_session_close` contract from async/best-effort to awaited, but the Go-side `CloseSession` doc comment still described the old behavior and explicitly cited that header. + **Rule:** When a vendored kernel C ABI header's documented contract changes (or KERNEL_REV is bumped), audit and update the Go doc comments that cite or paraphrase that header in lockstep — they go stale silently and can end up contradicting the very ABI they reference. +- **Context:** PR #471 documented numeric defaults in CONNECTION_PARAMETERS.md (e.g. `cloudfetch_link_prefetch_window=10`, `inline_max_chunks_in_memory=4`) that live only in the Rust kernel, not in this repo; a reviewer noted they can drift out of sync silently. + **Rule:** Don't assert exact numeric defaults owned by the Rust kernel in repo docs unless the value is pinned by a cited KERNEL_REV (as the mTLS row does); otherwise soften to "kernel default" to avoid silent doc/kernel drift. From ac862f22c7d932d244ad550cc3a3a3f314b1962d Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Fri, 11 Sep 2026 17:28:19 +0000 Subject: [PATCH 11/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 3a5e813e..a9463a45 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -99,3 +99,11 @@ retrospective flow. **Rule:** When a vendored kernel C ABI header's documented contract changes (or KERNEL_REV is bumped), audit and update the Go doc comments that cite or paraphrase that header in lockstep — they go stale silently and can end up contradicting the very ABI they reference. - **Context:** PR #471 documented numeric defaults in CONNECTION_PARAMETERS.md (e.g. `cloudfetch_link_prefetch_window=10`, `inline_max_chunks_in_memory=4`) that live only in the Rust kernel, not in this repo; a reviewer noted they can drift out of sync silently. **Rule:** Don't assert exact numeric defaults owned by the Rust kernel in repo docs unless the value is pinned by a cited KERNEL_REV (as the mTLS row does); otherwise soften to "kernel default" to avoid silent doc/kernel drift. + +### 2026-09-11: learnings since 2026-09-10T17:25:51Z +- **Context:** PR #479 added Reyden→kernel auto-recovery that hinges on `errors.Is(err, ErrReydenThriftUnsupported)` traversing production's wrapping chain (`thrift.Backend.OpenSession` wraps the marker via `NewRequestError` → `WithMessage`/`WithStack`), but the tests injected fakes returning the *bare* marker, so a regression that breaks the unwrap chain would silently disable recovery while every test stayed green. + **Rule:** When a feature depends on `errors.Is`/`errors.As` matching through a multi-layer wrapping chain, add a test that asserts the match against the production-shaped *wrapped* error (or drives the real backend) — fakes returning the bare sentinel do not pin the production error path. +- **Context:** PR #479's `TestReydenDefaultBuildKernelNotCompiled` asserted default-build stub behavior (`newKernelBackend` returning `ErrKernelNotCompiled`) but carried no build constraint, so it also compiled under `-tags databricks_kernel` + `CGO_ENABLED=1` where `newKernelBackend` is the real implementation, breaking `make test` for the kernel build. + **Rule:** A test asserting build-tag-gated stub behavior must carry the same `//go:build` constraint as the stub it exercises (e.g. `!cgo || !databricks_kernel`); an untagged test runs under every build and fails where the real implementation is linked. +- **Context:** PR #479 review flagged that the known-Reyden cache pre-check opened the kernel backend before the `KernelExperimental != nil` guardrail, so identical misconfigured config (a `WithKernel*` option without `WithUseKernel`) produced a hard error on a cold cache but silently succeeded on a warm cache populated by a sibling connection. + **Rule:** Run config-validation guardrails before any cache-based short-circuit/pre-check so behavior for a given config is deterministic and does not depend on process-global cache state. From fe3b0bda0bc17fb3283a118708d1608cfb0f86df Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Thu, 17 Sep 2026 17:27:37 +0000 Subject: [PATCH 12/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index a9463a45..0962fbae 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -107,3 +107,7 @@ retrospective flow. **Rule:** A test asserting build-tag-gated stub behavior must carry the same `//go:build` constraint as the stub it exercises (e.g. `!cgo || !databricks_kernel`); an untagged test runs under every build and fails where the real implementation is linked. - **Context:** PR #479 review flagged that the known-Reyden cache pre-check opened the kernel backend before the `KernelExperimental != nil` guardrail, so identical misconfigured config (a `WithKernel*` option without `WithUseKernel`) produced a hard error on a cold cache but silently succeeded on a warm cache populated by a sibling connection. **Rule:** Run config-validation guardrails before any cache-based short-circuit/pre-check so behavior for a given config is deterministic and does not depend on process-global cache state. + +### 2026-09-17: learnings since 2026-09-16T17:26:26Z +- **Context:** PR #485 rewrote `parseHostName` in connector.go — the old code used `strings.HasPrefix(host, "https"/"http")`, which is case-sensitive and also matched non-scheme hostnames like `httpbin.example.com`. The fix uses `strings.Cut` on `:` plus `strings.EqualFold`, and this scheme detection feeds downstream backend validation (e.g. mTLS-over-HTTP must be rejected before the SEA/kernel backend regardless of scheme casing). + **Rule:** When detecting a URL/URI scheme by string matching, compare case-insensitively (URI schemes are case-insensitive per RFC 3986 — use `EqualFold`, not `HasPrefix`) and require the `:` delimiter so ordinary hostnames that merely start with `http` aren't misparsed as a scheme; verify casing variants flow correctly into downstream backend/protocol validation. From aaad986b6dbb2e45d979aa4b77803e026dba87d1 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Wed, 23 Sep 2026 17:31:41 +0000 Subject: [PATCH 13/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 0962fbae..ed9dfdad 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -111,3 +111,13 @@ retrospective flow. ### 2026-09-17: learnings since 2026-09-16T17:26:26Z - **Context:** PR #485 rewrote `parseHostName` in connector.go — the old code used `strings.HasPrefix(host, "https"/"http")`, which is case-sensitive and also matched non-scheme hostnames like `httpbin.example.com`. The fix uses `strings.Cut` on `:` plus `strings.EqualFold`, and this scheme detection feeds downstream backend validation (e.g. mTLS-over-HTTP must be rejected before the SEA/kernel backend regardless of scheme casing). **Rule:** When detecting a URL/URI scheme by string matching, compare case-insensitively (URI schemes are case-insensitive per RFC 3986 — use `EqualFold`, not `HasPrefix`) and require the `:` delimiter so ordinary hostnames that merely start with `http` aren't misparsed as a scheme; verify casing variants flow correctly into downstream backend/protocol validation. + +### 2026-09-23: learnings since 2026-09-22T17:27:28Z +- **Context:** PR #484 added a detached per-statement cancel watcher on the kernel client-query-timeout path; reviewers repeatedly flagged that the watcher can call `kernel_statement_canceller_cancel`/`_free` concurrently with or after `kernel_statement_close`, and safety hinged on a kernel C ABI guarantee (canceller owns cloned Arcs + its own session ref, no back-pointer into the statement) that is not verifiable from this repo. Resolution came only from checking the pinned kernel revision and its ABI test `canceller_outlives_statement_and_session_close`. + **Rule:** When Go memory safety depends on a cgo/kernel object staying valid independently of another handle's lifetime (use/free after the related statement or session is closed), that independence is NOT verifiable from this repo — confirm it against the pinned KERNEL_REV and its ABI test, and record the load-bearing assumption in a code comment so a future kernel refactor trips a reviewer instead of silently introducing a use-after-free. +- **Context:** PR #484 introduced `MaxClientQueryTimeoutMilliseconds` as a hand-copied mirror of the C macro `DATABRICKS_KERNEL_MAX_CLIENT_QUERY_TIMEOUT_MS`, guarded only by a build-tagged runtime test; a reviewer contrasted this with the `KernelStatusCode` constants, which use a compile-time drift assertion (`_ = uint(a-b) | uint(b-a)`) so a header renumber fails the build. + **Rule:** Any Go constant hand-mirrored from a C ABI macro must be pinned with a compile-time drift assertion (the established `uint(a-b)|uint(b-a)` pattern), not a build-tagged runtime test — a test that must be remembered to run lets a KERNEL_REV bump silently validate against a stale bound. +- **Context:** In PR #484 `WithClientQueryTimeout` stores a pointer by copying into a fresh local (`configured := timeout; c.ClientQueryTimeout = &configured`) inside the option closure, and a dedicated test asserts reusing one option value across two connectors does not alias their configs. + **Rule:** A functional option that stores a pointer must allocate a fresh copy per application (copy into a local inside the closure, not `¶m`); otherwise reusing the same option value across multiple constructions aliases shared mutable state. Add a test that applies one option to two objects and mutates one. +- **Context:** PR #484 modeled `ClientQueryTimeout` as `*time.Duration` so that an omitted option (nil → legacy `kernel_statement_execute` with the 600s ceiling) is behaviorally distinct from an explicit zero (non-nil → timeout-aware entry point, unlimited), with a test named "omitted remains distinguishable from explicit zero." + **Rule:** When an omitted option must select different behavior than an explicit zero/default value, model it as a pointer (nil = omitted) rather than a bare value, and test both the omitted and explicit-zero cases — a plain zero value collapses the two and silently picks one path. From 7459758339cfe6494f334151b96e1850693cdea3 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Tue, 6 Oct 2026 17:30:21 +0000 Subject: [PATCH 14/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index ed9dfdad..cdeaa61a 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -121,3 +121,9 @@ retrospective flow. **Rule:** A functional option that stores a pointer must allocate a fresh copy per application (copy into a local inside the closure, not `¶m`); otherwise reusing the same option value across multiple constructions aliases shared mutable state. Add a test that applies one option to two objects and mutates one. - **Context:** PR #484 modeled `ClientQueryTimeout` as `*time.Duration` so that an omitted option (nil → legacy `kernel_statement_execute` with the 600s ceiling) is behaviorally distinct from an explicit zero (non-nil → timeout-aware entry point, unlimited), with a test named "omitted remains distinguishable from explicit zero." **Rule:** When an omitted option must select different behavior than an explicit zero/default value, model it as a pointer (nil = omitted) rather than a bare value, and test both the omitted and explicit-zero cases — a plain zero value collapses the two and silently picks one path. + +### 2026-10-06: learnings since 2026-10-05T17:30:24Z +- **Context:** PR #493 fixed M2M OAuth so a least-privilege service-principal secret (e.g. scoped to `sql`) can authenticate. The old `GetScopes` unconditionally appended `all-apis` to any caller-supplied scope list, broadening access beyond what the caller asked for. + **Rule:** When defaulting an OAuth/credential scope set, only inject a broad default (e.g. `all-apis`) when the caller passed NONE — never append it to a non-empty list, or you silently over-grant and break least-privilege scoped secrets. +- **Context:** PR #493 reversed an earlier decision to reject custom M2M scopes on the SEA/kernel path. The rejection assumed scopes were unforwardable because `set_auth_m2m` has no scopes argument; in fact a separate `set_oauth_scopes` C-ABI call carries them, so the whole `M2MScopesSupported` rejection path was deleted. + **Rule:** Before rejecting a config option as unsupported on a backend because its primary setter can't carry it, check for a complementary API (e.g. a separate `set_oauth_scopes` call) that can — a missing argument on one setter doesn't mean the capability is absent. From 7fe1b71364c8b009cb07ad47048e875ef48630dc Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Sat, 10 Oct 2026 17:30:06 +0000 Subject: [PATCH 15/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index cdeaa61a..422f82cb 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -127,3 +127,11 @@ retrospective flow. **Rule:** When defaulting an OAuth/credential scope set, only inject a broad default (e.g. `all-apis`) when the caller passed NONE — never append it to a non-empty list, or you silently over-grant and break least-privilege scoped secrets. - **Context:** PR #493 reversed an earlier decision to reject custom M2M scopes on the SEA/kernel path. The rejection assumed scopes were unforwardable because `set_auth_m2m` has no scopes argument; in fact a separate `set_oauth_scopes` C-ABI call carries them, so the whole `M2MScopesSupported` rejection path was deleted. **Rule:** Before rejecting a config option as unsupported on a backend because its primary setter can't carry it, check for a complementary API (e.g. a separate `set_oauth_scopes` call) that can — a missing argument on one setter doesn't mean the capability is absent. + +### 2026-10-10: learnings since 2026-10-09T17:29:31Z +- **Context:** PR #497 changed kernel `TelemetryConfig.Enabled` from `bool` to `*bool` so an unset `enableTelemetry` DSN value stays nil (kernel feature flag decides) while explicit `true`/`false` override it; the old `bool` conflated "unset" with "explicit false". + **Rule:** For a client knob that overrides a server/feature flag, model three states (unset / explicit-true / explicit-false) with a pointer or optional — a plain bool silently forces the "off" default and defeats server-side flag control. +- **Context:** In PR #497 the cgo shim began unconditionally calling the new `kernel_session_config_set_telemetry_config_v2` symbol, but `go.mod` still pinned the prebuilt bindings at v1.1.0; reviewers flagged that `-tags databricks_kernel` builds link the committed per-platform archives (not the `make kernel-lib` source build at `KERNEL_REV`), so they fail with an undefined-reference link error. + **Rule:** When a cgo shim starts calling a new kernel ABI symbol, bump the published bindings pin in go.mod/go.sum (to a version actually exporting that symbol) in the same change — bumping KERNEL_REV alone only fixes source builds; default kernel-tagged consumer builds link the committed archives and break at link time. +- **Context:** PR #497 hand-added the `KernelTelemetryConfig` struct and `_v2` prototype to the vendored `databricks_kernel.h` ahead of the upstream kernel landing them, and the cgo shim fills the struct with designated initializers; a reviewer noted any field reorder/padding/type divergence from the kernel's canonical layout is a silent ABI mismatch at the C boundary, not a compile error. + **Rule:** When hand-vendoring a C struct/prototype into a kernel header ahead of upstream, the field order/types/size must match the published header byte-for-byte and be re-verified once the real header lands — ABI divergence corrupts data silently rather than failing to compile. From 246ee98270419dd65ca9a27bea06a728b5d1d63a Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Wed, 14 Oct 2026 17:32:12 +0000 Subject: [PATCH 16/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 422f82cb..8c28b67b 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -135,3 +135,11 @@ retrospective flow. **Rule:** When a cgo shim starts calling a new kernel ABI symbol, bump the published bindings pin in go.mod/go.sum (to a version actually exporting that symbol) in the same change — bumping KERNEL_REV alone only fixes source builds; default kernel-tagged consumer builds link the committed archives and break at link time. - **Context:** PR #497 hand-added the `KernelTelemetryConfig` struct and `_v2` prototype to the vendored `databricks_kernel.h` ahead of the upstream kernel landing them, and the cgo shim fills the struct with designated initializers; a reviewer noted any field reorder/padding/type divergence from the kernel's canonical layout is a silent ABI mismatch at the C boundary, not a compile error. **Rule:** When hand-vendoring a C struct/prototype into a kernel header ahead of upstream, the field order/types/size must match the published header byte-for-byte and be re-verified once the real header lands — ABI divergence corrupts data silently rather than failing to compile. + +### 2026-10-14: learnings since 2026-10-13T17:31:21Z +- **Context:** In PR #496 (generalize driver-side feature-flag cache), reviewers flagged two unrelated changes bundled into the scoped PR — a `KERNEL_REV` bump and a large async-statement addition to the vendored `databricks_kernel.h` C ABI header; the author confirmed and removed the async surface. + **Rule:** Keep vendored-kernel changes (KERNEL_REV bumps, `databricks_kernel.h` C ABI edits) out of feature-scoped PRs — they muddy the changelog, break bisection, and risk Go bindings drifting from the declared ABI; split them into their own focused PR. +- **Context:** In PR #496 the feature-flag cache collapses concurrent cold readers through `singleflight.DoChan`, but the single executing `load()` captured the first caller's `ctx`; the cold path detached via `context.WithoutCancel` while the stale-refresh branch still ran `load(ctx)` under the winning caller's cancellable deadline, so one caller's cancellation/short deadline could abort the shared fetch and fail or perpetually re-stale other waiters. + **Rule:** When collapsing concurrent fetches behind singleflight (or any shared cache refresh), run the shared fetch on a detached, timeout-bounded context (e.g. `context.WithoutCancel` + `WithTimeout`) and let each waiter honor its own ctx via `select` — never let one caller's deadline abort work other goroutines depend on. +- **Context:** In PR #496 the generalized `GetString`/`GetInt32`/etc. feature-flag getters decode the stored flag value with `json.Unmarshal`, so a string flag only resolves when the server's `value` is a JSON literal (e.g. `"hello"`); an unquoted plain string errors instead of returning the raw value. + **Rule:** Typed getters that `json.Unmarshal` a raw string value require the stored value to be a JSON literal — document this and have consumers/producers JSON-encode string and list flag values, or non-bool getters will error on plain-string inputs. From d27283bf0fc36bdd32d0fd38eaa150e2098973da Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Tue, 20 Oct 2026 17:31:28 +0000 Subject: [PATCH 17/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 8c28b67b..ab03b28b 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -143,3 +143,13 @@ retrospective flow. **Rule:** When collapsing concurrent fetches behind singleflight (or any shared cache refresh), run the shared fetch on a detached, timeout-bounded context (e.g. `context.WithoutCancel` + `WithTimeout`) and let each waiter honor its own ctx via `select` — never let one caller's deadline abort work other goroutines depend on. - **Context:** In PR #496 the generalized `GetString`/`GetInt32`/etc. feature-flag getters decode the stored flag value with `json.Unmarshal`, so a string flag only resolves when the server's `value` is a JSON literal (e.g. `"hello"`); an unquoted plain string errors instead of returning the raw value. **Rule:** Typed getters that `json.Unmarshal` a raw string value require the stored value to be a JSON literal — document this and have consumers/producers JSON-encode string and list flag values, or non-bool getters will error on plain-string inputs. + +### 2026-10-20: learnings since 2026-10-19T17:29:11Z +- **Context:** PR #502 bumped the SQL kernel bindings to v1.2.0, syncing `databricks_kernel.h`; a reviewer found the new header dropped the "600s polling ceiling" wording from `kernel_statement_execute`, leaving `config.go` and `operation.go` Go doc comments describing behavior the shipped header no longer guarantees. + **Rule:** When bumping a vendored native/kernel header or bindings, grep the driver's own Go doc comments for behavior they mirror from that header (timeouts, ceilings, defaults) and update any that the new header contradicts — stale doc comments mislead the next reader even when code behavior is unchanged. +- **Context:** In PR #492's geospatial WKB path, a reviewer flagged that `rewriteGeometry` used `panic` in its `default` arm — reached per-row during a `database/sql` scan on server-provided bytes — while every other malformed-input case in the file returns a wrapped error that `kernelRows.next` surfaces cleanly. + **Rule:** In per-row scan/decode paths over server-provided data, return a wrapped error for malformed or unexpected input instead of panicking, so one bad row fails just the query rather than crashing the consuming application. +- **Context:** In PR #492, `ColumnTypeInfoForField`'s fallback branch reported `DatabaseTypeName = GEOMETRY/GEOGRAPHY` while the row scanner (gated on `arrow.STRUCT`) would not produce EWKB `[]byte` for a non-struct physical type, so the advertised column type and the scanned value could silently diverge. + **Rule:** Keep the reported `sql.ColumnType` (DatabaseTypeName/ScanType) and the value the scanner actually produces in lockstep; defensive fallback branches for unexpected physical representations should drop to the plain Arrow-derived mapping rather than advertise a logical type the scan path won't yield. +- **Context:** In PR #492 the `enableGeoSpatialSupport` DSN carrier allocated `KernelExperimental` whenever the value was non-nil (both true and false), unlike the adjacent `tokenCache` carrier that only allocates for the behavior-changing case — so a plain Thrift DSN carrying the option (even the no-op `=true`) failed connect with `ErrRequiresKernelBackend`, and the DSN-carrier-on-Thrift path had no test. + **Rule:** Kernel-only DSN option carriers should follow the `tokenCache` precedent — only allocate `KernelExperimental` when the value actually changes behavior — and pin the intended Thrift-path rejection with an explicit DSN-carrier test, not just an option test. From 8e6f03cc999410d44da2f08eef99236accb60e7d Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Wed, 21 Oct 2026 17:31:10 +0000 Subject: [PATCH 18/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index ab03b28b..1becadf8 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -153,3 +153,7 @@ retrospective flow. **Rule:** Keep the reported `sql.ColumnType` (DatabaseTypeName/ScanType) and the value the scanner actually produces in lockstep; defensive fallback branches for unexpected physical representations should drop to the plain Arrow-derived mapping rather than advertise a logical type the scan path won't yield. - **Context:** In PR #492 the `enableGeoSpatialSupport` DSN carrier allocated `KernelExperimental` whenever the value was non-nil (both true and false), unlike the adjacent `tokenCache` carrier that only allocates for the behavior-changing case — so a plain Thrift DSN carrying the option (even the no-op `=true`) failed connect with `ErrRequiresKernelBackend`, and the DSN-carrier-on-Thrift path had no test. **Rule:** Kernel-only DSN option carriers should follow the `tokenCache` precedent — only allocate `KernelExperimental` when the value actually changes behavior — and pin the intended Thrift-path rejection with an explicit DSN-carrier test, not just an option test. + +### 2026-10-21: learnings since 2026-10-20T17:29:43Z +- **Context:** Engineer-bot author run #39539859873 (diagnosing the Thrift Arrow-path TIME/time64 bug) burned many turns re-issuing identical read-only inspection commands: `go doc github.com/apache/arrow/go/v12/arrow.Time64` ran at turns 26, 57, 63 and again in the next phase, `go env GOMODCACHE` ran twice, and repeated globs for the Arrow `datatype_fixedwidth.go` source returned nothing — the agent never cached a result or pivoted after `go doc` proved unhelpful. + **Rule:** Do not re-run an identical read-only inspection command (`go doc `, `go env GOMODCACHE`, same glob/grep) whose output you already have; when `go doc` on a third-party type yields nothing useful, resolve the module-cache path once and read the type's source directly instead of retrying the same command across turns. From bd6c2411051fe4f6508192de6f8a06957173f716 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Thu, 22 Oct 2026 17:30:55 +0000 Subject: [PATCH 19/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 1becadf8..d9048129 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -157,3 +157,11 @@ retrospective flow. ### 2026-10-21: learnings since 2026-10-20T17:29:43Z - **Context:** Engineer-bot author run #39539859873 (diagnosing the Thrift Arrow-path TIME/time64 bug) burned many turns re-issuing identical read-only inspection commands: `go doc github.com/apache/arrow/go/v12/arrow.Time64` ran at turns 26, 57, 63 and again in the next phase, `go env GOMODCACHE` ran twice, and repeated globs for the Arrow `datatype_fixedwidth.go` source returned nothing — the agent never cached a result or pivoted after `go doc` proved unhelpful. **Rule:** Do not re-run an identical read-only inspection command (`go doc `, `go env GOMODCACHE`, same glob/grep) whose output you already have; when `go doc` on a third-party type yields nothing useful, resolve the module-cache path once and read the type's source directly instead of retrying the same command across turns. + +### 2026-10-22: learnings since 2026-10-21T17:29:21Z +- **Context:** Author run #39665678594 fixed issue #507, where `arrowRecordIterator` turned a failed mid-pagination `FetchResults` into `io.EOF`: `resultPageIterator.HasNext()` stored the real error in `rpf.err`, closed, and returned `false`, and the caller's `checkFinished()` ANDed `!HasNext()` into its done-decision with no error check, so a `for it.HasNext()` loop exited cleanly and swallowed the failure. + **Rule:** For iterator-style APIs (`HasNext()`/`Next()`), a `false`/termination signal must not conflate "done" with "errored" — expose and check a stored error (an `Err()` accessor) before treating end-of-iteration as success, and never overwrite a real error with an EOF/sentinel (set EOF only when `err == nil`). +- **Context:** Author run #39665654097 diagnosed that the driver reports `TIMESTAMP` for both TIMESTAMP and TIMESTAMP_NTZ because `ColumnTypeDatabaseTypeName`/`GetDBTypeName` derive the name from the Thrift `TColumnDesc`, where both types share `TTypeId_TIMESTAMP_TYPE` (id 8); the NTZ distinction exists only in the Arrow schema's `TimestampType.TimeZone` (empty for NTZ, populated for zoned). + **Rule:** The Thrift type id cannot distinguish TIMESTAMP from TIMESTAMP_NTZ — any code that must tell them apart (type-name metadata, value timezone handling) has to consult the Arrow schema's `TimestampType.TimeZone` field, not the Thrift `TColumnDesc`. +- **Context:** In author run #39665654097 the engineer-bot wrote live-warehouse e2e tests, then spent ~15 turns (go test, git status, probing existing e2e tests) rediscovering that the sandboxed bash tool scrubs credentials, so all e2e tests `Skip` and can never be observed red/green locally. + **Rule:** The bash sandbox strips warehouse/auth credentials, so live e2e tests always skip and cannot be driven red→green locally — don't burn turns probing env/creds; author unit tests at a lower in-process layer (e.g. Arrow-schema level) that run in the default build, keeping the e2e test only for the grading harness. From 1c146c923b56d5c76e1efaf4b5464a4af1b992d0 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Sat, 24 Oct 2026 17:28:41 +0000 Subject: [PATCH 20/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index d9048129..44e03365 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -165,3 +165,9 @@ retrospective flow. **Rule:** The Thrift type id cannot distinguish TIMESTAMP from TIMESTAMP_NTZ — any code that must tell them apart (type-name metadata, value timezone handling) has to consult the Arrow schema's `TimestampType.TimeZone` field, not the Thrift `TColumnDesc`. - **Context:** In author run #39665654097 the engineer-bot wrote live-warehouse e2e tests, then spent ~15 turns (go test, git status, probing existing e2e tests) rediscovering that the sandboxed bash tool scrubs credentials, so all e2e tests `Skip` and can never be observed red/green locally. **Rule:** The bash sandbox strips warehouse/auth credentials, so live e2e tests always skip and cannot be driven red→green locally — don't burn turns probing env/creds; author unit tests at a lower in-process layer (e.g. Arrow-schema level) that run in the default build, keeping the e2e test only for the grading harness. + +### 2026-10-24: learnings since 2026-10-23T17:29:25Z +- **Context:** In PR #516 (Go 1.27.3 bump), a newer vet/lint check fired on `fmt.Errorf("%w", inner)` in `internal/errors/category_test.go`. The first fix dereferenced `inner`, so the test wrapped a `requestError` value instead of the `*requestError` pointer that production code returns. The reviewer flagged that this silently changed what the test covers. The merged fix keeps the pointer as `var inner error = ...` and adds `assert.Same`/`assert.ErrorIs` checks. + **Rule:** When a toolchain or linter upgrade flags a test, keep the type under test the same as in production (for example, a pointer error type). Satisfy the check with an interface-typed variable or a scoped `//nolint: // reason`, or add a separate case. Never swap pointer for value just to silence a check. +- **Context:** In PR #516, the Security Scan workflow briefly switched osv-scanner to `go install`. Fork PRs skip the `Setup JFrog` step, so that install would depend on GOPROXY access that the hardened protected runners may block. The merged version downloads the prebuilt binary from GitHub Releases instead. + **Rule:** CI jobs that run on fork PRs, where JFrog/GOPROXY is unavailable, must install tools from prebuilt release binaries rather than `go install`. Keep workflow comments about fork-PR behavior in sync whenever the install method changes. From 180d26fa547bd41b41763db02d67b24b67b78aeb Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Tue, 27 Oct 2026 17:26:27 +0000 Subject: [PATCH 21/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 44e03365..21ee1c82 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -171,3 +171,13 @@ retrospective flow. **Rule:** When a toolchain or linter upgrade flags a test, keep the type under test the same as in production (for example, a pointer error type). Satisfy the check with an interface-typed variable or a scoped `//nolint: // reason`, or add a separate case. Never swap pointer for value just to silence a check. - **Context:** In PR #516, the Security Scan workflow briefly switched osv-scanner to `go install`. Fork PRs skip the `Setup JFrog` step, so that install would depend on GOPROXY access that the hardened protected runners may block. The merged version downloads the prebuilt binary from GitHub Releases instead. **Rule:** CI jobs that run on fork PRs, where JFrog/GOPROXY is unavailable, must install tools from prebuilt release binaries rather than `go install`. Keep workflow comments about fork-PR behavior in sync whenever the install method changes. + +### 2026-10-27: learnings since 2026-10-26T17:30:35Z +- **Context:** PR #509 (ArrowBatchIterator masking a failed FetchResults as io.EOF). A refactor of `arrowRecordIterator.checkFinished()` split a short-circuited `&&` chain into separate statements. That made `resultPageIterator.HasNext()`, which sends a real FetchResults RPC, run on every call. Dropping the `isFinished ||` guard also meant a closed iterator re-fetched against a closed operation and reported a spurious error. + **Rule:** `resultPageIterator.HasNext()` has side effects (it sends a FetchResults RPC), so it must only run after local batches are drained AND the iterator is not finished. When refactoring boolean chains that contain it, keep the short-circuit order, and add a test that calls Close() early and asserts no FetchResults calls are made. +- **Context:** PR #509 fixed page-fetch errors being reported as io.EOF in `arrowRecordIterator`. The reviewer found the same bug in the sibling `arrowIPCStreamIterator`. Several more review rounds were needed on the semantics: report the error once, and make an explicit Close() discard a pending error while internal end-of-data keeps it (a `release()` vs `Close()` split). + **Rule:** When an iterator must tell a fetch failure apart from a normal end of results, check `Err()` when HasNext turns false. Hold the error as pending (HasNext=true until it is consumed) and report it exactly once from Next/Schema through one shared take-pending helper. An explicit Close() clears the pending error, but internal end-of-data release must not. Apply the same fix to every sibling iterator (record and IPC stream), and test each one for: row/stream count delivered before the error, fetch call count, and HasNext/Next after the error and after Close. +- **Context:** PR #510 (TIMESTAMP_NTZ reported as TIMESTAMP). The Thrift-path type-name override was read from whatever `r.RowScanner` held at the moment of the call. That scanner is nil before the first Next() (no direct results) and after the results are exhausted, and the column-based pages use a different scanner, so `ColumnTypes()` returned different names depending on page state. + **Rule:** Derive column metadata (DatabaseTypeName, ScanType) from the result-set metadata, i.e. `resultSetMetadata.ArrowSchema` (parse once and cache it). Never derive it from the current page's row scanner, whose lifecycle depends on page state. On Thrift, TIMESTAMP vs TIMESTAMP_NTZ can only be told apart by the Arrow `TimestampType.TimeZone` / field metadata, because TTypeId has no NTZ variant. String-timestamp and column-based paths cannot tell them apart. +- **Context:** PR #510 changed how TIMESTAMP_NTZ values are shifted. The unit suite passed, but `TestKernelE2ETimestampNTZ` in `kernel_e2e_test.go` still asserted the old behavior. That test is build-tagged (`databricks_kernel`) and needs credentials, so the local unit run never executed it. + **Rule:** Before claiming a behavior change is verified, grep the build-tagged and credential-gated e2e tests (`*_e2e_test.go`, `databricks_kernel` tag) for assertions on the old behavior and update them too. A passing unit suite does not cover them. Also update README.md, CONNECTION_PARAMETERS.md and doc.go together, and add a CHANGELOG "Behavior change" entry for user-visible value changes. From 317a521109763813ee9162597686b2b10ba3b00e Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Fri, 30 Oct 2026 17:25:00 +0000 Subject: [PATCH 22/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 21ee1c82..2e347bde 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -181,3 +181,9 @@ retrospective flow. **Rule:** Derive column metadata (DatabaseTypeName, ScanType) from the result-set metadata, i.e. `resultSetMetadata.ArrowSchema` (parse once and cache it). Never derive it from the current page's row scanner, whose lifecycle depends on page state. On Thrift, TIMESTAMP vs TIMESTAMP_NTZ can only be told apart by the Arrow `TimestampType.TimeZone` / field metadata, because TTypeId has no NTZ variant. String-timestamp and column-based paths cannot tell them apart. - **Context:** PR #510 changed how TIMESTAMP_NTZ values are shifted. The unit suite passed, but `TestKernelE2ETimestampNTZ` in `kernel_e2e_test.go` still asserted the old behavior. That test is build-tagged (`databricks_kernel`) and needs credentials, so the local unit run never executed it. **Rule:** Before claiming a behavior change is verified, grep the build-tagged and credential-gated e2e tests (`*_e2e_test.go`, `databricks_kernel` tag) for assertions on the old behavior and update them too. A passing unit suite does not cover them. Also update README.md, CONNECTION_PARAMETERS.md and doc.go together, and add a CHANGELOG "Behavior change" entry for user-visible value changes. + +### 2026-10-30: learnings since 2026-10-29T17:24:43Z +- **Context:** In engineer-bot author run #40976127852 (issue #475), the issue reported that `internal/rows/arrowbased/arrowRows.go` passes `context.Background()` to `NewCloudBatchIterator`/`NewLocalBatchIterator` and drops the caller's context. The bot found this is on purpose. `NewRows` detaches `rows.ctx` from the caller with `context.WithoutCancel` (ES-1934053, #295, #371), and `TestE2EArrowBatchesSurviveQueryContextCancellation` guards that behaviour. The bot left it unchanged. + **Rule:** Result fetching (CloudFetch/Arrow batch download) is deliberately detached from the caller's QueryContext cancellation via `context.WithoutCancel` in `NewRows`, so do not "fix" a `context.Background()` or detached ctx in the rows/arrowbased download path. Bound downloads with an explicit timeout or HTTP client option instead. +- **Context:** In engineer-bot author run #40976127852, the new e2e tests in `driver_e2e_test.go` (`TestE2ECloudFetchMinTimeToExpiryIsHonored`, `TestE2ECloudFetchHTTPClientOption`) were reported as red, but only because they did not compile. They then skipped in the fix phase because the author tool shell scrubs the `DATABRICKS_PECOTESTING_*` credentials. Their assertions never ran, and only the unit tests got a real red→green check. + **Rule:** E2E tests that use pecoTestingCreds skip in the engineer-bot author environment, and "red because it doesn't compile" is not proof of red. For each fix, also add a unit test that can actually run (a mock server via httptest, or config plumbing assertions) and that fails on its assertions before the fix, and flag any e2e-only verification as out of scope. From ff18246515879384cc889f7a38f408e3a5b8af8c Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Sat, 31 Oct 2026 17:25:12 +0000 Subject: [PATCH 23/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 2e347bde..81aee73a 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -187,3 +187,7 @@ retrospective flow. **Rule:** Result fetching (CloudFetch/Arrow batch download) is deliberately detached from the caller's QueryContext cancellation via `context.WithoutCancel` in `NewRows`, so do not "fix" a `context.Background()` or detached ctx in the rows/arrowbased download path. Bound downloads with an explicit timeout or HTTP client option instead. - **Context:** In engineer-bot author run #40976127852, the new e2e tests in `driver_e2e_test.go` (`TestE2ECloudFetchMinTimeToExpiryIsHonored`, `TestE2ECloudFetchHTTPClientOption`) were reported as red, but only because they did not compile. They then skipped in the fix phase because the author tool shell scrubs the `DATABRICKS_PECOTESTING_*` credentials. Their assertions never ran, and only the unit tests got a real red→green check. **Rule:** E2E tests that use pecoTestingCreds skip in the engineer-bot author environment, and "red because it doesn't compile" is not proof of red. For each fix, also add a unit test that can actually run (a mock server via httptest, or config plumbing assertions) and that fails on its assertions before the fix, and flag any e2e-only verification as out of scope. + +### 2026-10-31: learnings since 2026-10-30T17:23:39Z +- **Context:** PR #525 added fast paths in `internal/arrowscan` `writeJSON` for Boolean and Int8–Int64 leaves, so they skip `scalarForJSON` + `json.Marshal` and allocate less. The only test was a Thrift/kernel render-parity case. A reviewer pointed out that a parity test checks that the output matches, not how much was allocated. A later edit could send these leaves back through the slow path and every test would still pass. + **Rule:** A PR that exists to cut allocations or speed things up must ship a `testing.AllocsPerRun` assertion or a `Benchmark*` for the hot path. Output-parity tests alone cannot catch a performance regression. From cbf33e51f8b4de0dfac5d0a4c082e79e22c5b49c Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Mon, 2 Nov 2026 17:25:03 +0000 Subject: [PATCH 24/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 81aee73a..16bdb1d0 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -191,3 +191,7 @@ retrospective flow. ### 2026-10-31: learnings since 2026-10-30T17:23:39Z - **Context:** PR #525 added fast paths in `internal/arrowscan` `writeJSON` for Boolean and Int8–Int64 leaves, so they skip `scalarForJSON` + `json.Marshal` and allocate less. The only test was a Thrift/kernel render-parity case. A reviewer pointed out that a parity test checks that the output matches, not how much was allocated. A later edit could send these leaves back through the slow path and every test would still pass. **Rule:** A PR that exists to cut allocations or speed things up must ship a `testing.AllocsPerRun` assertion or a `Benchmark*` for the hot path. Output-parity tests alone cannot catch a performance regression. + +### 2026-11-02: learnings since 2026-11-01T17:24:02Z +- **Context:** PR #526 upgraded go-oidc, go-jose (v3→v4) and x/oauth2 for security findings. The reviewer pointed out that the PR had no `## Unreleased` CHANGELOG line, even though earlier security bumps (klauspost/compress, thrift, x/crypto, x/mod, go-jose) all had one. The PR also added a `golang.org/x/crypto` requirement to go.mod that nothing imports. It is there as a deliberate minimum version for security scans, and a comment notes that `go mod tidy` removes it. The `osv-scanner.toml` suppression reasons depend on `go list -deps -test ./...` results, checked both with and without `-tags=databricks_kernel`. + **Rule:** For a security dependency bump, add a CHANGELOG `## Unreleased` entry that names the new versions and the advisory being fixed. After running `go mod tidy`, put back the deliberate x/crypto minimum it removes from go.mod. Re-check any osv-scanner.toml suppression reason with `go list -deps -test ./...`, both with and without `-tags=databricks_kernel`. From b67ad43d71a2e7c2590ff8774d092ea707a7bafb Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Tue, 3 Nov 2026 17:25:10 +0000 Subject: [PATCH 25/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 16bdb1d0..e66c0359 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -195,3 +195,7 @@ retrospective flow. ### 2026-11-02: learnings since 2026-11-01T17:24:02Z - **Context:** PR #526 upgraded go-oidc, go-jose (v3→v4) and x/oauth2 for security findings. The reviewer pointed out that the PR had no `## Unreleased` CHANGELOG line, even though earlier security bumps (klauspost/compress, thrift, x/crypto, x/mod, go-jose) all had one. The PR also added a `golang.org/x/crypto` requirement to go.mod that nothing imports. It is there as a deliberate minimum version for security scans, and a comment notes that `go mod tidy` removes it. The `osv-scanner.toml` suppression reasons depend on `go list -deps -test ./...` results, checked both with and without `-tags=databricks_kernel`. **Rule:** For a security dependency bump, add a CHANGELOG `## Unreleased` entry that names the new versions and the advisory being fixed. After running `go mod tidy`, put back the deliberate x/crypto minimum it removes from go.mod. Re-check any osv-scanner.toml suppression reason with `go list -deps -test ./...`, both with and without `-tags=databricks_kernel`. + +### 2026-11-03: learnings since 2026-11-02T17:23:44Z +- **Context:** PR #527 added `internal/jsonfmt`, a hand-written fast path that renders nested scalars byte-for-byte the same as `json.Marshal` on both the Thrift (arrowbased) and kernel (arrowscan) renderers. Anything it doesn't handle falls back to `json.Marshal`. Parity was checked by exhaustive sweeps (every byte, every rune) and by randomized float and string inputs. Parity tests alone would still pass if the fast path were silently disabled, so the PR also pinned it with an `AllocsPerRun == 0` test. Review then noted that the ~2M-call sweeps ignored `testing.Short()` and slow down `-race` CI. + **Rule:** When you reimplement a stdlib or library function as a fast path with a fallback, assert byte-for-byte parity, including errors, against the original over exhaustive and randomized inputs. Also add a test that proves the fast path is actually taken, such as zero allocations or a fallback counter, because the fallback hides a disabled fast path. Gate the large sweeps behind `testing.Short()`. From 25984a9eed6daa924b15c1a118ca0b04af4bc9ac Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Mon, 9 Nov 2026 17:25:21 +0000 Subject: [PATCH 26/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index e66c0359..73989239 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -199,3 +199,7 @@ retrospective flow. ### 2026-11-03: learnings since 2026-11-02T17:23:44Z - **Context:** PR #527 added `internal/jsonfmt`, a hand-written fast path that renders nested scalars byte-for-byte the same as `json.Marshal` on both the Thrift (arrowbased) and kernel (arrowscan) renderers. Anything it doesn't handle falls back to `json.Marshal`. Parity was checked by exhaustive sweeps (every byte, every rune) and by randomized float and string inputs. Parity tests alone would still pass if the fast path were silently disabled, so the PR also pinned it with an `AllocsPerRun == 0` test. Review then noted that the ~2M-call sweeps ignored `testing.Short()` and slow down `-race` CI. **Rule:** When you reimplement a stdlib or library function as a fast path with a fallback, assert byte-for-byte parity, including errors, against the original over exhaustive and randomized inputs. Also add a test that proves the fast path is actually taken, such as zero allocations or a fallback counter, because the fallback hides a disabled fast path. Gate the large sweeps behind `testing.Short()`. + +### 2026-11-09: learnings since 2026-11-08T17:23:32Z +- **Context:** PR #534 (OSV remediation that bumped brotli/compress) ran a bare `go mod tidy`. Tidy silently removed the deliberate `golang.org/x/crypto v0.55.0 // indirect` security-scan minimum pin (added in #464) along with its go.sum hashes. That left old, CVE-affected x/crypto pseudo-versions as the resolved version, and it broke the justification for the suppressions in `osv-scanner.toml`. The tooling also merged the separate kernel-bindings `require` block into the main block, so the lockstep-pin comment ended up above unrelated dependencies. The reviewer caught both problems, and both needed a human fix. + **Rule:** When bumping dependencies in go.mod, do not run a bare `go mod tidy`. Use `go get @`, then check the diff to confirm that deliberate `// indirect` security pins (e.g. golang.org/x/crypto) and their go.sum hashes are still there, and that the separate commented `require` blocks were not merged or reordered. The diff should show only the intended version bumps. From 45304e29d33bdcf8d78e72a90f7b73d117c75182 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Thu, 12 Nov 2026 17:24:45 +0000 Subject: [PATCH 27/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 73989239..36aed564 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -203,3 +203,9 @@ retrospective flow. ### 2026-11-09: learnings since 2026-11-08T17:23:32Z - **Context:** PR #534 (OSV remediation that bumped brotli/compress) ran a bare `go mod tidy`. Tidy silently removed the deliberate `golang.org/x/crypto v0.55.0 // indirect` security-scan minimum pin (added in #464) along with its go.sum hashes. That left old, CVE-affected x/crypto pseudo-versions as the resolved version, and it broke the justification for the suppressions in `osv-scanner.toml`. The tooling also merged the separate kernel-bindings `require` block into the main block, so the lockstep-pin comment ended up above unrelated dependencies. The reviewer caught both problems, and both needed a human fix. **Rule:** When bumping dependencies in go.mod, do not run a bare `go mod tidy`. Use `go get @`, then check the diff to confirm that deliberate `// indirect` security pins (e.g. golang.org/x/crypto) and their go.sum hashes are still there, and that the separate commented `require` blocks were not merged or reordered. The diff should show only the intended version bumps. + +### 2026-11-12: learnings since 2026-11-11T17:23:34Z +- **Context:** Three author runs (#43087748527, #43090396242, #43095026884) took on the same SEA/kernel issue (#517, duplicate column names such as `SELECT 1 AS a, 2 AS a`). All three traced it to the pinned `github.com/apache/arrow/go/v12`: `cdata.ImportCArrowSchema` and `cdata.ImportCRecordBatch` build the top-level `+s` struct with `arrow.StructOf`, which panics with `arrow: duplicate field with name`. The Thrift path decodes through `ipc.NewReader` / `arrow.NewSchema`, which accept duplicate names, so it is not affected. + **Rule:** In arrow-go v12, `arrow.StructOf` (and so the cdata importers in `internal/backend/kernel`) panics on duplicate field names, while `arrow.NewSchema` and `array.NewRecord` accept them. Check both the kernel/cdata path and the Thrift/IPC path when a bug involves column names, and don't assume one backend's behavior holds for the other. +- **Context:** On SEA/kernel issue #517, the three author runs reached different verdicts (two `blocked`, one `feasible`). The feasible run (#43090396242) edited cgo/C code in `internal/backend/kernel/rows.go` (`//go:build cgo && databricks_kernel`) even though the job builds only the pure-Go path (CGO_ENABLED=0, no tag, no native lib), so `go build ./...` never compiled that change. Its live e2e test also silently skipped because the bash tool strips the warehouse credentials. It spent several turns finding that out, then still reported `red_confirmed` based on pure-Go helper unit tests alone. + **Rule:** A bug whose fix lives only in `cgo && databricks_kernel`-tagged files cannot be compiled or reproduced in the author job. Report `blocked` early instead of editing uncompiled cgo code or treating proxy unit tests as the reproduction. When a `pecoTestingCreds` e2e test skips, treat the skip as an environment limit and report it in one step rather than spending turns investigating env vars. From cbcf22f9d3803fade1873008b34f49182a101a51 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Fri, 13 Nov 2026 17:24:27 +0000 Subject: [PATCH 28/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index 36aed564..cdf14b79 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -209,3 +209,11 @@ retrospective flow. **Rule:** In arrow-go v12, `arrow.StructOf` (and so the cdata importers in `internal/backend/kernel`) panics on duplicate field names, while `arrow.NewSchema` and `array.NewRecord` accept them. Check both the kernel/cdata path and the Thrift/IPC path when a bug involves column names, and don't assume one backend's behavior holds for the other. - **Context:** On SEA/kernel issue #517, the three author runs reached different verdicts (two `blocked`, one `feasible`). The feasible run (#43090396242) edited cgo/C code in `internal/backend/kernel/rows.go` (`//go:build cgo && databricks_kernel`) even though the job builds only the pure-Go path (CGO_ENABLED=0, no tag, no native lib), so `go build ./...` never compiled that change. Its live e2e test also silently skipped because the bash tool strips the warehouse credentials. It spent several turns finding that out, then still reported `red_confirmed` based on pure-Go helper unit tests alone. **Rule:** A bug whose fix lives only in `cgo && databricks_kernel`-tagged files cannot be compiled or reproduced in the author job. Report `blocked` early instead of editing uncompiled cgo code or treating proxy unit tests as the reproduction. When a `pecoTestingCreds` e2e test skips, treat the skip as an environment limit and report it in one step rather than spending turns investigating env vars. + +### 2026-11-13: learnings since 2026-11-12T17:23:28Z +- **Context:** PR #536 made `WithCloudFetch(false)` / `useCloudFetch=false` take effect on the kernel backend. The reviewer suggested forwarding the documented client-only key `cloudfetch_enabled=false`. The author showed that in kernel 1.4.0 this key selects INLINE+ARROW_STREAM, which has a size cap and which the server rejects, so it cannot return large results in full. Fetching complete results with no storage downloads needed the server session capability `can_cloud_download=false` (paged inline), which is the same capability JDBC uses. The repo docs that called it "not a valid SEA conf" were stale. Servers without paged-inline support reject it with `CONFIG_NOT_AVAILABLE` / SQLSTATE 42K0I, either at CreateSession or at the first statement. + **Rule:** Kernel no-CloudFetch means sending `can_cloud_download=false` (paged inline), not `cloudfetch_enabled=false` (size-capped INLINE). When you force a session key, first remove case-insensitive copies of it and of any conflicting keys from the caller's `WithSessionParams`. Do not trust repo docs about which SEA confs are valid without a live check. +- **Context:** PR #536 changed the CloudFetch option from inert to honored inside `buildKernelConfig`. The reviewer pointed out that `openSessionWithReydenFallback` (automatic Thrift→kernel recovery on KP001) builds the kernel session from the same `c.cfg`. So a Thrift-only user who never set `WithUseKernel` also gets the new kernel behavior. + **Rule:** Any change to `buildKernelConfig` also changes the Reyden auto-recovery path for Thrift users. When you make a previously-inert option active on the kernel path, add a connector test that combines the option with `openSessionWithReydenFallback`, and record the change as a behavior change in the CHANGELOG, because DSNs are often shared between Thrift and kernel. +- **Context:** PR #536 needed to show that `WithCloudFetch(false)` really avoids cloud-storage downloads on the kernel path. Unit tests that only check config-map entries were not enough. The e2e test routes the kernel through `WithKernelProxy` to an httptest CONNECT proxy. The proxy tunnels only to the workspace host:443, rejects and counts every other destination, and the test asserts zero blocked connections plus a full ordered row and hash check. + **Rule:** To prove a 'no direct storage/network egress' guarantee, run the driver through a restricted CONNECT proxy that allows only the workspace host, then assert that blocked attempts == 0 and that the result is complete. Do not rely only on asserting which conf keys were set. From a7d23cecf2c1202c90759cd53421fd76cd2f7c62 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Sat, 14 Nov 2026 17:25:02 +0000 Subject: [PATCH 29/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index cdf14b79..df0b72de 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -217,3 +217,17 @@ retrospective flow. **Rule:** Any change to `buildKernelConfig` also changes the Reyden auto-recovery path for Thrift users. When you make a previously-inert option active on the kernel path, add a connector test that combines the option with `openSessionWithReydenFallback`, and record the change as a behavior change in the CHANGELOG, because DSNs are often shared between Thrift and kernel. - **Context:** PR #536 needed to show that `WithCloudFetch(false)` really avoids cloud-storage downloads on the kernel path. Unit tests that only check config-map entries were not enough. The e2e test routes the kernel through `WithKernelProxy` to an httptest CONNECT proxy. The proxy tunnels only to the workspace host:443, rejects and counts every other destination, and the test asserts zero blocked connections plus a full ordered row and hash check. **Rule:** To prove a 'no direct storage/network egress' guarantee, run the driver through a restricted CONNECT proxy that allows only the workspace host, then assert that blocked attempts == 0 and that the result is complete. Do not rely only on asserting which conf keys were set. + +### 2026-11-14: learnings since 2026-11-13T17:23:17Z +- **Context:** PR #555 made Thrift `ExecuteStatement` retryable, but only when the connection to the server or proxy was never opened. `isDialError` matches `net.OpError{Op:"dial"}`, including a dial wrapped in `proxyconnect`. It excludes context cancellation, DNS not-found, and TLS, read and write failures. An attempt that followed a redirect is treated as possibly sent, because an earlier hop already received the request. + **Rule:** Never retry non-idempotent RPCs such as ExecuteStatement on generic transport errors. Retry them only on dial errors, which prove no request bytes were sent, and treat any redirected attempt as possibly already delivered. +- **Context:** PRs #559 and #558 fixed flaky and racy tests. Unjoined `go drainKernelLogs(...)` goroutines and CloudFetch downloads kept running after their test returned, then wrote into a later test's log buffer or raced under `-race`. The fixes join each goroutine in `t.Cleanup` with a bounded `select` timeout. They also pass a cancellable ctx with `defer cancel()` to iterators, and replace `time.Sleep` with waiting on a done channel. + **Rule:** Join every goroutine a test starts, waiting in t.Cleanup with a timeout, and give background workers a ctx that is cancelled when the test returns. Synchronize on done channels, not time.Sleep, so leaked goroutines can't corrupt later tests. +- **Context:** In a PR #555 review, a custom `http.Transport.DialContext` closure called a helper that used `require.NoError`. The transport runs `DialContext` on its own goroutine, so a failure there calls `t.FailNow` off the test goroutine. That exits the dial goroutine instead of the test, which hangs the test or fails it confusingly. + **Rule:** Do not call require.*/t.FailNow/t.Fatal inside callbacks that run on goroutines the test doesn't own, such as http handlers, DialContext and drain loops. Compute fixtures before the closure, and report failures through t.Error or a channel. +- **Context:** In a PR #550 review, a reviewer flagged as critical that a new test file imported `github.com/apache/arrow/go/v12/...`. PR #535 had already moved the repo to `github.com/apache/arrow-go/v18`, and v12 is no longer in go.mod, so the import breaks build and lint for the whole package. v18 also renames APIs: `arrow.Record` became `arrow.RecordBatch`, `ipc.Reader.Record()` became `RecordBatch()`, and `NewRecord` is deprecated in favor of `NewRecordBatch`. + **Rule:** Import Arrow only from github.com/apache/arrow-go/v18 and use the RecordBatch APIs. Never copy arrow/go/v12 imports from older code, branches or snippets into new files. +- **Context:** In PR #539, `NewResultPageIterator` took two adjacent positional bools, `isFinished` and `closedOnServer`. A swap would still compile, and passing `closedOnServer=true` by mistake skips CloseOperation and leaks the server operation. The fix takes `hasMoreRows *bool` plus `closedOnServer bool` and derives completion in one place. + **Rule:** Avoid adjacent same-typed bool parameters whose swap causes resource leaks. Use distinct types (for example *bool vs bool) or an options struct, and derive combined state in exactly one place. +- **Context:** PR #556 found that the DSN `timeout`, given in seconds, was multiplied straight into a duration with `time.Duration(n) * time.Second`. Values beyond ±9223372036 seconds silently wrapped to a different and often negative timeout. The fix range-checks the value against `math.MaxInt64/time.Second` and `math.MinInt64/time.Second` before converting, and returns an error that names the param. + **Rule:** When converting user-supplied integer units to time.Duration, check the value against MaxInt64/unit and MinInt64/unit before multiplying. Reject out-of-range values with an error naming the param instead of letting them overflow. From 82969ffa9ad05fec1abb8bbf96b86bf2e0d0b849 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Mon, 16 Nov 2026 17:28:21 +0000 Subject: [PATCH 30/30] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/learning-log.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.claude/knowledge/learning-log.md b/.claude/knowledge/learning-log.md index df0b72de..9dea419b 100644 --- a/.claude/knowledge/learning-log.md +++ b/.claude/knowledge/learning-log.md @@ -231,3 +231,11 @@ retrospective flow. **Rule:** Avoid adjacent same-typed bool parameters whose swap causes resource leaks. Use distinct types (for example *bool vs bool) or an options struct, and derive combined state in exactly one place. - **Context:** PR #556 found that the DSN `timeout`, given in seconds, was multiplied straight into a duration with `time.Duration(n) * time.Second`. Values beyond ±9223372036 seconds silently wrapped to a different and often negative timeout. The fix range-checks the value against `math.MaxInt64/time.Second` and `math.MinInt64/time.Second` before converting, and returns an error that names the param. **Rule:** When converting user-supplied integer units to time.Duration, check the value against MaxInt64/unit and MinInt64/unit before multiplying. Reject out-of-range values with an error naming the param instead of letting them overflow. + +### 2026-11-16: learnings since 2026-11-15T17:23:28Z +- **Context:** PR #567 put the live warehouse tests behind an `e2e` build tag. The nightly workflows had to switch to `-tags e2e` and `-tags databricks_kernel,e2e`. Plain `go test`/`make test` never compiles these tests, so CI added `go vet -tags e2e .` and `CGO_ENABLED=1 go vet -tags databricks_kernel,e2e .` steps that only check they compile. + **Rule:** Live e2e tests need the `e2e` build tag (kernel ones need `databricks_kernel,e2e`). When you add or change one, run `go vet` with those tags and add its name to the `-run` regex in nightly-e2e.yml, or it is never compiled or run. +- **Context:** PRs #563 and #567 added DECIMAL qualifiers and TIME metadata. On SEA/kernel, an empty-result manifest schema can describe a typed column as a UTF-8 Arrow field, with the real type only in `databricks.type_name` / Spark SQL-name field metadata. Thrift TIME metadata depends on the server-supplied Arrow schema. Both PRs had to test zero-row and NULL-only results separately from populated ones. + **Rule:** When adding support for a result type, get column metadata from the field (`ColumnTypeInfoForField`), not only the physical Arrow type. SEA empty results can carry the logical type only in field metadata on a string field. Test zero-row, NULL-only and populated results on both Thrift and SEA. +- **Context:** PR #561 removed a `//go:build tools` `package tools` file from the module root. Under `-tags tools` that file clashed with the root `dbsql` package and broke build, vet and Gazelle. The tool dependency moved to a go.mod `tool` directive, and a test now checks that every root `*.go` file is `dbsql` or `dbsql_test`. + **Rule:** Every `.go` file in the module root must be `package dbsql` or `dbsql_test`, including files behind build tags. Declare dev tools with the go.mod `tool` directive, not a tagged `tools.go` file.