Skip to content

ci(bot): enable distillation and cross-review - #620

Merged
vuanhphung merged 2 commits into
mainfrom
vu/bot-learning-cross-review
Nov 17, 2026
Merged

vuanhphung merged 2 commits into
mainfrom
vu/bot-learning-cross-review

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Apply kernel #476's onboarding: enable distill mode with learning-PR auto-merge off, and pin the bot engine to 6014478 for two independent reviewers plus a judge. Manual reviews use trusted workflow code and a separate resolved PR checkout.

Preserve all 89 rules from #500 at 8e6b681 as a frozen archive and seed 66 lessons. New learning uses dated journal files and a fresh rolling branch.

98 focused engine tests passed. Config, archive preservation, lesson routing/source checks, seven reviewer-input cases, Bash syntax and actionlint passed. Live cross-review approved the current head; source-format nit explained.

After merge, run learning once and inspect its first journal/lessons PR. Check #500 for newer entries before retiring that legacy PR.

Lint passes. Security Scan flags picomatch@2.3.2 (GHSA-fc5q-39xc-vjpf); package manifests and lockfile are unchanged.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Nit

Looks good: 2 low provenance issues in the seeded lessons. Splitting the reviewer into a trusted-base checkout and a separate read-only head checkout, plus the same-repo open-PR gate, looks right. Follow-up, not from this PR: engineer-bot-learning.yml still reads inputs.window-hours without bracket notation.

This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge — see the findings below.

Other findings

  • ⚪ Nit — L-0051 and L-0053 cite #500, which is the legacy learning PR and not where the lesson came from. The source was author runs (#42301071194 and #42966486376). Cite the run IDs or use whatever author-run provenance format the engine accepts. This only affects provenance, not behavior. (Found by Claude Opus 5.5 · confirmed by the judge)
🔀 1 finding considered and dismissed by the judge
  • 🔵 Low .claude/knowledge/lessons.md:30 — Some seeded lesson counts don't match the PRs they cite. L-0024 says 2× but lists three PRs (#514 #506 #528), L-0033 says 2× with three, and L-0066 says 1×<br>Judge's evidence: - L-0066 (1×, #566 #562): Serialize branch-editing bot flows with one per-PR concurrency group and restore dependency-se…`

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Nit

Looks good, with one low provenance note. The reviewer workflow's split into a trusted checkout and a separate review checkout is sound: it checks that the PR is open and not from a fork before checking out the head. The distill config and the frozen archive are consistent.

Other findings

  • ⚪ Nit — L-0037 is counted 2× but cites only #565. Its other source is the 2026-11-05 author-run entry, which has no PR number. L-0047 (2×, #529) has the same gap for its 2026-11-07 hang-test entry, while L-0051 already uses a log: source. Add log:2026-11-05 to L-0037 and log:2026-11-07 to L-0047 so both counts can be traced. (Found by Claude Opus 5.5 · confirmed by the judge)

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Keeping the supported source format: the engine grammar permits up to three PR references or one log date; mixing #565 log:2026-11-05 would fail validation. The 2× counts measure archived observations. L-0037 combines the 2026-11-05 author-run observation (#41966945385/#41971447709) with PR #565. L-0047 combines PR #529 with the 2026-11-07 hang-test observation (run #42301152612). All four original Context/Rule pairs remain in the frozen archive. The compact sources remain representative references within the engine's format.

@vuanhphung
vuanhphung marked this pull request as ready for review November 17, 2026 22:50

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall, with 1 low concern: one of the 66 seeded lessons states a GitHub Actions rule that the repo's own workflow contradicts. Splitting the reviewer into a trusted base-SHA checkout plus a read-only head checkout, and the new open/non-fork PR check for manual dispatch, look sound. Follow-up, not caused by this PR: the reviewed-PR checkout is shallow, so PR_BASE_SHA is not in that tree in case the engine ever needs local history.

- L-0059 (1×, #611): Every auxiliary HTTP request, including OAuth, exchange and CloudFetch, must use the connection provider's getAgent to honor proxy/TLS settings.

## Bot workflow and CI
- L-0060 (2×, #499): Use explicit bracket notation for hyphenated dispatch input names and string-typed numeric knobs in workflows also triggered by schedule.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — L-0060 turns the #499 archive entry into a standing rule that both bots will now load: hyphenated dispatch inputs must use bracket notation. That rule is shaky. GitHub Actions expression syntax lets property names contain - under dot dereference. This repo's own learning workflow, which this PR touches, still reads the input as ${{ inputs.window-hours }} with no brackets. So the seeded lesson disagrees with the code it sits next to. Bots will likely raise findings or make edits based on it, and since lessons.md is now the only learning text injected into prompts, a wrong lesson costs more than a wrong archive note. Before the first distill run, either confirm the claim and switch the workflow to inputs['window-hours'], or reduce L-0060 to the part that is verified: use string-typed numeric inputs when the workflow also has a schedule trigger.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction proposed in #621: remove the unsupported bracket-notation requirement from L-0060, retain the string-typed numeric-input guidance, and reduce the occurrence count from 2 to 1 for the retained observation. The engine parser/renderer and actionlint pass.

@vuanhphung
vuanhphung added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit ee3c92a Nov 17, 2026
26 of 27 checks passed
@vuanhphung
vuanhphung deleted the vu/bot-learning-cross-review branch November 17, 2026 23:42

This branch was successfully deployed

1 active deployment
azure-prod — 0e522ce1 Deployed Nov 17, 2026 by vuanhphung via e2e-test (22.23.4, 22_23_4) #2095
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant