Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 10 additions & 24 deletions .bot/config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
# Engineer-bot task config for databricks-sql-go (the Databricks SQL driver for
# Go). Loaded by databricks_bot_engine.engineer_bot.bot_config.load_bot. The
# reviewer needs NO .bot/config.yaml (its prompts are engine-owned + the optional
# additive .bot/prompts/review/system.md); only the engineer-bot is config-driven.
# Engineer-bot task config and shared retrospective lesson routing.
name: databricks_sql_go_bugfix
marker_namespace: engineer-bot-go
bot_login_prefix: peco-engineer-bot
Expand Down Expand Up @@ -91,11 +88,6 @@ author:
issue_url: ISSUE_URL
context_files:
- issue_body.txt # {{issue_body}}
# Close the loop: the author phase reads this log back so fixes benefit from
# what the retrospective has learned. MUST match `retrospective.log_path` below
# — the retrospective WRITES that path; the author only READS it if the two
# agree. (Set together, they connect learning → application.)
knowledge_log: .claude/knowledge/learning-log.md

# Engine orchestration for the author phase. `bug-fix` runs the plan →
# author_tests → fix pipeline (write a failing test → fix the code → re-run to
Expand All @@ -105,20 +97,14 @@ author:
# prompts/engineer/system.md, not here.
flow: bug-fix

# Daily learning extraction (retrospective flow). Run by engineer-bot-learning.yml
# via `python -m databricks_bot_engine.engineer_bot.retrospective`: over an adaptive
# look-back window it sweeps every merged PR (diff + review comments, which the
# engine gathers ITSELF per PR) plus engineer-bot author runs, and if the model
# finds a durable, reusable learning, opens/updates a single rolling PR appending a
# dated section to log_path. Human-gated — never commits the canonical log directly.
# Omitting this block makes the retrospective a no-op.
#
# `system_prompt` is intentionally OMITTED: this repo ships no
# prompts/retrospective_system.md, so the engine's built-in base prompt is used.
# (A set-but-missing system_prompt file would be a hard error; an UNSET key uses
# the base. Add the key later only if you write a repo-specific additive prompt.)
# The learning log is created on first write — no seed file needed.
# Both bots receive matching lessons; the old log is a frozen archive.
retrospective:
log_path: .claude/knowledge/learning-log.md # MUST match author.knowledge_log above
branch_prefix: ai/learning-pr-
log_mode: distill
log_path: .claude/knowledge/learning-log.md
journal_dir: .claude/knowledge/journal
lessons_path: .claude/knowledge/lessons.md
auto_merge_learning_pr: false
author_workflow: engineer-bot.yml
# Keep the legacy append-mode PR (#443) on its original branch.
branch_prefix: ai/learning-distill-pr-
pr_label: engineer-bot-learning
245 changes: 236 additions & 9 deletions .claude/knowledge/learning-log.md

Large diffs are not rendered by default.

72 changes: 72 additions & 0 deletions .claude/knowledge/lessons.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Lessons

<!-- distilled-through: none -->

## Configuration and authentication
- L-0001 (1×, #446): Verify kernel session parameters and proxies separately from Thrift: kernel allowlists configuration keys and supports HTTP(S) proxies, while Thrift may accept additional keys/schemes.
- L-0002 (2×, #449): Kernel U2M uses workspace OIDC uniformly across clouds with its registered client/scopes, not Azure Thrift's Entra app. Preserve stable scope ordering in slice comparisons.
- L-0003 (1×, #444): External JWTs on the kernel PAT path may undergo mandatory federation exchange; verify pinned-kernel same-issuer/non-JWT exceptions before adding driver-side exchange.
- L-0004 (5×, #449 #444 #455): Update mirrored auth/ABI/default documentation together. Cite exact external kernel refs for assumptions, and avoid numeric native defaults without a pinned source.
- L-0005 (1×, #456): Track explicit configuration separately from non-empty values so deliberately supplied empty credential pairs are rejected rather than treated as absent.
- L-0006 (1×, #455): A context check before blocking native session-open does not bound the call itself; verify cancellation/deadline coverage through the pinned ABI.
- L-0007 (2×, #458 #492): WithKernel options may opt into kernel even for false values. Kernel-only DSN carriers should allocate KernelExperimental only when behavior changes; test DSN and option routing separately.
- L-0008 (3×, #457 #451): Preserve native default/unset sentinels when forwarding timeouts and limits. Round positive sub-unit durations consistently so they never become zero/unset.
- L-0009 (1×, #470): Reject interior NUL in user strings passed to length-less C setters before cgo marshaling; keep validation in pure Go so CGO_ENABLED=0 tests exercise it.
- L-0010 (1×, #479): Validate configuration before cache short-circuits so errors do not depend on global cache state.
- L-0011 (1×, #485): Match URI schemes case-insensitively with the colon delimiter, then verify the chosen backend accepts the scheme.
- L-0012 (1×, #484): Functional options storing pointers must allocate a fresh copy inside each application; test reusing one option across two independently mutable objects.
- L-0013 (2×, #484 #497): Use optional/pointer values when omitted differs from explicit zero/false, including client overrides of server rollout flags.
- L-0014 (1×, #493): Inject broad default OAuth scopes only for an empty caller scope set; never append them to explicit restricted scopes.
- L-0015 (1×, #493): Check complementary ABI setters before declaring a backend option unsupported because one primary setter lacks the argument.
- L-0016 (1×, #496): Typed JSON flag getters require stored JSON literals; producers must encode strings/lists and document this representation.
- L-0017 (1×, #536): Kernel no-CloudFetch uses can_cloud_download=false for paged inline results. Remove case-insensitive/conflicting caller keys before forcing it, and verify server acceptance live.
- L-0018 (1×, #536): buildKernelConfig also serves Thrift Reyden recovery. Test newly active options through openSessionWithReydenFallback and document changed shared-DSN behavior.
- L-0019 (1×, #539): Avoid adjacent same-typed boolean parameters controlling ownership; use distinct types/options and derive combined lifecycle state in one place.
- L-0020 (1×, #556): Check integer duration values against MinInt64/unit and MaxInt64/unit before multiplication; report overflow with the parameter name.

## Native boundaries and lifecycle
- L-0021 (1×, #450): Never place fabricated Go pointers or cgo.Handle values into C pointer slots. Use NULL with a global sink or a real C allocation.
- L-0022 (2×, #450): Foreign-thread callbacks should copy borrowed data and nonblockingly enqueue bounded work. Invoke user code and I/O outside internal locks and away from native callback threads.
- L-0023 (1×, #450): Check retryable preconditions outside sync.Once.Do; an early return inside still consumes the Once.
- L-0024 (1×, #450): Swappable writer proxies must preserve richer interfaces such as zerolog.LevelWriter and delegate when supported.
- L-0025 (1×, #450): Surface queue overflow, sampling and truncation through bounded warnings so consumers can distinguish dropped output from no output.
- L-0026 (2×, #440 #479): Keep build-tag constraints aligned across real shims, unsupported guards and stub tests; missing terms can silently select no implementation.
- L-0027 (1×, #479): Verify errors.Is/errors.As through the production-shaped wrapping chain; bare-sentinel fakes do not test real error matching.
- L-0028 (1×, #484): Handle lifetime independence across cgo objects must be confirmed against the pinned kernel ABI/tests and recorded as a load-bearing ownership assumption.
- L-0029 (1×, #484): Pin Go constants mirrored from C macros with compile-time drift assertions rather than optional tagged runtime tests.
- L-0030 (2×, #497): New ABI symbols require released bindings pins in go.mod/go.sum, not only KERNEL_REV. Hand-mirrored layouts must match the published header field order, types and size.
- L-0031 (1×, #496): Keep ABI/header/pin migrations focused enough to verify binding compatibility and bisect independently of feature behavior.
- L-0032 (1×, #496): Shared singleflight fetches need detached, bounded contexts; individual waiters honor their own cancellation without aborting shared work.
- L-0033 (1×, #295 #371): Result downloads deliberately outlive QueryContext cancellation. Preserve the detached rows-fetch contract and bound downloads with explicit timeouts/client options.
- L-0034 (1×, #555): Retry non-idempotent RPCs on proven pre-send dial failures only; generic transport failures and redirected attempts may have delivered the request.

## Arrow results and metadata
- L-0035 (2×, #492): Malformed server rows must return wrapped errors, not panic. Keep ColumnType metadata consistent with produced values and use physical-type fallback when logical conversion is unavailable.
- L-0036 (2×, #509): Preserve iterator errors separately from EOF, expose them exactly once through shared pending-error logic, and update every sibling iterator. Explicit Close clears pending errors; internal release must preserve them.
- L-0037 (2×, #510): Derive column metadata from cached result-set Arrow schema, independent of current-page scanner lifetime. Timestamp timezone metadata distinguishes Thrift TIMESTAMP from NTZ.
- L-0038 (1×, #509): HasNext may fetch remotely. Drain local batches and check finished state first; preserve boolean short-circuit order and assert early Close sends no FetchResults RPC.
- L-0039 (1×, log:2026-11-12): Duplicate-name support can differ between Arrow schemas, struct/cdata imports and IPC. Test both kernel and Thrift boundaries against the current Arrow version.
- L-0040 (1×, #550 #535): New Arrow code uses github.com/apache/arrow-go/v18 RecordBatch APIs; avoid copying legacy arrow/go/v12 imports.
- L-0041 (1×, #563 #567): Derive logical result types from field metadata with ColumnTypeInfoForField, including empty SEA string-backed schemas; test zero-row, NULL-only and populated results on both backends.

## Verification and dependencies
- L-0042 (3×, #458 #510 #567): Compile/vet relevant tagged tests explicitly, including databricks_kernel,e2e where needed, and register E2E names in nightly filters. Default unit tests omit tagged coverage.
- L-0043 (1×, #458): Test seams must call real production methods; parallel copies of native calls prove signatures without proving production wiring.
- L-0044 (1×, #457): Telemetry duration units follow receiver schema: bare names are seconds, and _ms/_millis names are milliseconds.
- L-0045 (2×, #440): Publish real dependency versions: replace directives do not apply downstream, and module graph resolution requires every go.mod dependency regardless of build tags.
- L-0046 (1×, #440): Verification, checksum and lint gates must propagate failures; do not hide them with unconditional success.
- L-0047 (2×, #463 #440 #469): Confirm which archive the build links. Default kernel builds use published bindings; local source builds need explicit go.work/replace routing, and KERNEL_REV alone does not ship native behavior.
- L-0048 (1×, #516): Linter fixes must preserve production pointer/value error types; use interface-typed test variables or narrowly justified suppressions.
- L-0049 (1×, #516): Fork CI without registry credentials must install tools from prebuilt releases rather than go install; keep fork-behavior comments accurate.
- L-0050 (1×, #525): Allocation/performance changes need AllocsPerRun assertions or benchmarks; output parity alone cannot detect a performance regression.
- L-0051 (2×, #526 #534 #464): Make targeted go get updates and preserve deliberate indirect security pins/hashes and require-block layout. Revalidate suppressions on tagged/untagged graphs and document security versions/advisories.
- L-0052 (1×, #527): Fast paths need byte/error parity against the original over exhaustive/random inputs and proof the fast path runs; keep expensive sweeps out of short tests.
- L-0053 (1×, #536): Prove no-storage-egress behavior through a workspace-only CONNECT proxy, asserting complete results and zero blocked attempts.
- L-0054 (2×, #559 #558 #555): Join test goroutines with bounded cleanup and context cancellation. Synchronize with channels and report callback failures without FailNow/Fatal from foreign goroutines.
- L-0055 (1×, #561): Root Go files must remain package dbsql/dbsql_test even behind tags; declare developer tools with the go.mod tool directive.

## Bot workflow
- L-0056 (1×, #442): Use concrete serving-endpoints/model/invocations MODEL_ENDPOINT URLs; anthropic/invocations produces an invalid translated CLI route.
- L-0057 (1×, #442): Schedule plus workflow_dispatch workflows should declare numeric knobs as strings and parse them downstream.
- L-0058 (2×, #442): Omit unused prompt-file config keys instead of referencing missing files; explicitly unignore and seed any required tracked file under ignored directories.
- L-0059 (4×, log:2026-10-21): Reuse completed inspections, report unavailable native/E2E environments promptly, and distinguish skipped coverage or compile failures from assertion-based reproduction. Add runnable unit coverage where possible.
2 changes: 1 addition & 1 deletion .github/actions/bot-prelude/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ inputs:
# value to move every bot to a new engine commit; never @main.
description: 'Engine commit SHA (full 40-char) to install.'
required: false
default: 'ac3b08eab993ef14bf997a9bf4c776fa5acb16d8'
default: '60144783c5c3e4c242df9f1f5a0f25cd71cf5f4c'
engine-repo:
description: 'owner/name of the engine repo.'
required: false
Expand Down
19 changes: 2 additions & 17 deletions .github/workflows/engineer-bot-learning.yml
Original file line number Diff line number Diff line change
@@ -1,20 +1,5 @@
# Engineer Bot — learning (retrospective) extraction — DAILY CRON.
#
# Over an adaptive look-back window the engine gathers merged PRs (diff + review
# comments) AND recent engineer-bot author-run console logs ITSELF via the GitHub
# API — no in-workflow context gathering, no per-PR trigger — and if the model
# finds durable, reusable learnings, opens ONE ROLLING PR on a stable branch
# (`ai/learning-pr`), appending a dated section per day until a human merges it.
# Human-gated by design: it NEVER commits the canonical log directly.
#
# Own job (NOT `uses: databricks/databricks-bot-engine/...`): an external repo
# can't resolve the internal engine's reusable workflows ("not found"). It shares
# the SAME prelude the other bots use — ./.github/actions/bot-prelude (tokens +
# Node + pinned engine install) — so the engine pin stays single-sourced in
# bot-prelude's `engine-ref` default (no second SHA to drift).
#
# Opt-in is purely via the `retrospective:` block in .bot/config.yaml + this
# workflow; absent that block the engine phase is a clean no-op.
# Daily journal extraction and lesson distillation in a rolling PR.
# The legacy log is frozen; learning PRs require human review.
name: Engineer Bot — Learning

on:
Expand Down
24 changes: 18 additions & 6 deletions .github/workflows/reviewer-bot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,11 @@ jobs:
labels: linux-ubuntu-latest
timeout-minutes: 30
steps:
# Checkout FIRST. Default ref = the PR merge ref on pull_request, the
# default branch on dispatch. persist-credentials:false — the reviewer
# reads this tree via read_paths/grep, so no token may sit in .git/config.
- name: Checkout
# Run trusted workflow code; inspect the resolved PR head separately.
- name: Checkout trusted workflow source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.event.pull_request.base.sha || github.sha }}
fetch-depth: 0
persist-credentials: false

Expand Down Expand Up @@ -114,16 +113,29 @@ jobs:
fi
[[ "$RAW_PR" =~ ^[0-9]+$ ]] || { echo "::error::Invalid pr_number '$RAW_PR'"; exit 1; }
case "$DRY_RUN" in true|false) ;; *) DRY_RUN="true" ;; esac # fail safe
HEAD_SHA=$(gh pr view "$RAW_PR" --repo "$REPO" --json headRefOid -q .headRefOid)
{ echo "pr_number=$RAW_PR"; echo "head_sha=$HEAD_SHA"; echo "dry_run=$DRY_RUN"; } >> "$GITHUB_OUTPUT"
PR_JSON=$(gh api "repos/$REPO/pulls/$RAW_PR")
jq -e --arg repo "$REPO" '.state == "open" and .head.repo.full_name == $repo and .head.repo.fork == false' <<< "$PR_JSON" >/dev/null \
|| { echo "::error::Reviewer requires an open PR from this repository"; exit 1; }
HEAD_SHA=$(jq -r '.head.sha' <<< "$PR_JSON")
BASE_SHA=$(jq -r '.base.sha' <<< "$PR_JSON")
{ echo "pr_number=$RAW_PR"; echo "head_sha=$HEAD_SHA"; echo "base_sha=$BASE_SHA"; echo "dry_run=$DRY_RUN"; } >> "$GITHUB_OUTPUT"

- name: Checkout reviewed PR (read-only)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ steps.inputs.outputs.head_sha }}
path: review-content
persist-credentials: false

- name: Run reviewer
env:
GH_TOKEN: ${{ steps.prelude.outputs.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
PR_NUMBER: ${{ steps.inputs.outputs.pr_number }}
HEAD_SHA: ${{ steps.inputs.outputs.head_sha }}
PR_BASE_SHA: ${{ steps.inputs.outputs.base_sha }}
DRY_RUN: ${{ steps.inputs.outputs.dry_run }}
REVIEW_CONTENT_ROOT: ${{ github.workspace }}/review-content
REVIEWER_BOT_MODEL_OVERRIDE: databricks-claude-opus-5-5
MODEL_ENDPOINT: https://${{ secrets.DATABRICKS_HOST }}/serving-endpoints/databricks-claude-opus-5-5/invocations
DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@
.claude/knowledge/*
!.claude/knowledge/learning-log.md
!.claude/knowledge/driver-pitfalls.md
!.claude/knowledge/lessons.md
!.claude/knowledge/journal/
.claude/knowledge/journal/*
!.claude/knowledge/journal/*.md

# Binaries for programs and plugins
*.exe
Expand Down
Loading