Add advanced code review metrics view by dayland · Pull Request #851 · microsoft/BC-Bench

dayland · 2026-09-03T12:47:52Z

Summary

Adds the advanced Code Review metrics view, retaining PR Review-specific diagnostics, coverage, and BCQuality lineage in category-specific result/summary/aggregate data.
Keeps the default Code Review dashboard concise and cross-runner compatible; detailed engine, CLI, BCQuality, usage, knowledge, and sub-skill evidence appears only in the advanced view.
Catches up to current main, preserving the five newly published GPT-5.6 Terra runs and regenerating the Code Review leaderboard data with optional advanced diagnostics.
Integrates the deterministic BC PR Review baseline from Validate deterministic PR review model routing #881: immutable BC-ALAgents revision, explicit root/leaf models and scheduling, strict manifest validation, complete telemetry/model attestation, and engine-owned BCQuality provenance.

Why

The default leaderboard needs comparable headline metrics, while PR Review engineering decisions need detailed evidence about the actual review stack and coverage. This keeps runner-specific provenance and optional diagnostics in the Code Review-specific models and advanced view rather than expanding the generic summary workflow or base schemas.

Validation

Focused conflict-resolution metrics/serialization/summary suite: 68 passed
Deterministic PR Review conflict-resolution suite: 71 passed
Full suite after integration: 1,108 passed, 2 skipped, 1 deselected
Changed-file pre-commit hooks, workflow/action YAML parsing, git diff --check, and leaderboard deserialization passed
Direct ty check has one pre-existing unused-ignore warning in untouched src/bcbench/redteam.py
A focused evaluation-identity PR Review smoke run is dispatched from the integrated head and will be reviewed before merge.

Design boundaries

agent_version remains generic harness identity; BCQuality and CLI provenance remain Code Review-specific.
Missing optional diagnostics stay unavailable (null); malformed artifacts fail explicitly.
The deterministic runner rejects partial, reordered, model-substituted, incomplete, or malformed review runs before scoring.
A future named PR Review definition/profile will build on this advanced view after Validate deterministic PR review model routing #881 and this PR are merged; it is intentionally not included here.

dayland · 2026-09-03T12:50:58Z

Marking this draft while the advanced data contract is completed. BC-ALAgents currently logs BCQuality filtering/consumption but does not persist article retained, pruned, cited/used, sub-skill execution, or usage-completeness diagnostics into BC-Bench result artifacts, so this page cannot yet show the requested full performance set.

Sun Haoran (haoranpb)

dayland I think lots of the changes here is not needed, if we strictly follow the versioning policy of BC-Bench.

A version of BC-Bench, e.g. v0.10.0, specifically list the version of all agent harness.

So a fixed version of BC-Bench should point to a fixed version of all agent harnesses, which should then points to fixed versions of bcquality etc

Note: the version of BC-Bench is already collected as part of the results

dayland · 2026-09-07T09:24:33Z

Sun Haoran (@haoranpb) That makes sense as the intended versioning contract. Would you be open to a thin compromise where we keep the exact pins in the persisted result identity and aggregation key, but only show them compactly as an Evaluation Stack cell on the dashboard?

The release remains the canonical description, while the abbreviated BC-Bench, BC-ALAgents, and BCQuality identities save analysts from reconstructing the transitive pins for each row. This also prevents accidental aggregation if results are produced from different commits or experimental pins under the same in-development benchmark version.

Sun Haoran (haoranpb) · 2026-09-08T07:16:13Z

Sun Haoran (Sun Haoran (@haoranpb)) That makes sense as the intended versioning contract. Would you be open to a thin compromise where we keep the exact pins in the persisted result identity and aggregation key, but only show them compactly as an Evaluation Stack cell on the dashboard?

The release remains the canonical description, while the abbreviated BC-Bench, BC-ALAgents, and BCQuality identities save analysts from reconstructing the transitive pins for each row. This also prevents accidental aggregation if results are produced from different commits or experimental pins under the same in-development benchmark version.

dayland Good point, you pointed out a very valid gap in the current design:

we can't really experiment with agent harness version.

We never had that need before PR Review Agent, after some thought, what do you think about this #860 ? Could it be an alternative to what we are trying to achieve here? Or at least partly

Copilot

🟡 Changes recommended

A few implementation details make the new “advanced diagnostics” metrics potentially inaccurate or brittle (notably knowledge-usage counting, per-task BCQuality scanning cost, and hardcoded Copilot CLI provenance output).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends the code-review (PR Review) evaluation pipeline to persist and surface richer diagnostics (usage token breakdowns, API call stats, BCQuality knowledge/sub-skill telemetry) and adds an advanced “Code Review Advanced Metrics” dashboard to display the expanded schema, while keeping the default leaderboard view focused on headline metrics.

Changes:

Persist PR Review diagnostics into result / summary / leaderboard schemas and aggregate them with “only where available” semantics.
Record evaluation stack provenance (BC-Bench commit, Copilot CLI version, BCQuality repo/commit/version) end-to-end through workflows and summaries.
Add a dedicated advanced metrics documentation page and adjust the default Code Review dashboard to emphasize production PR Review runs vs legacy direct-agent rows.

File summaries

File	Description
tests/test_review_workflows.py	Updates workflow/action pin expectations and adds docs/dashboard visibility assertions.
tests/test_pr_review_metrics.py	Expands PR Review metrics tests for new diagnostics and knowledge/sub-skill counting.
tests/test_pr_review_metrics_reporting.py	Verifies new diagnostics flow through result JSON, summaries, and aggregates.
tests/test_evaluation_summary.py	Adds coverage for provenance fields sourced from environment variables.
tests/test_agent_version_results.py	Ensures legacy `bc_alagents_commit` is restored into `agent_version`.
src/bcbench/types.py	Extends `AgentMetrics` with PR Review diagnostic fields and validation constraints.
src/bcbench/results/summary.py	Adds provenance fields to summaries and restores legacy PR Review agent identity.
src/bcbench/results/leaderboard.py	Adds provenance fields to aggregates and restores legacy PR Review agent identity.
src/bcbench/results/codereview.py	Computes/serializes new diagnostic averages and coverage rates for code-review summaries.
src/bcbench/agent/pr_review/metrics.py	Implements parsing/derivation of PR Review usage + BCQuality/filter/findings diagnostics.
src/bcbench/agent/pr_review/agent.py	Passes `bcquality_root` into metrics builder to support knowledge diagnostics.
docs/code-review.md	Reframes default dashboard into production vs legacy sections and links to advanced view.
docs/code-review-details.md	Adds the advanced metrics dashboard with expanded columns and provenance display.
.github/workflows/summarize-results.yml	Plumbs provenance inputs into summarization via environment variables.
.github/workflows/pr-review-evaluation.yml	Exports harness identity outputs and forwards them into summarization inputs.
.github/actions/install-agent-harnesses/action.yml	Adds harness identity outputs (Copilot CLI + BCQuality repo/commit/version).

Review details

Files reviewed: 16/16 changed files
Comments generated: 4
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sun Haoran (haoranpb)

With #860 merged,can we trim this to persisting and displaying the required diagnostics? benchmark_version and agent_version already identify the setup, so I’m not sure we need five extra inputs on the shared summarization workflow.

I understand the convenience of showing the BCQuality version, but the recorded BC-ALAgents SHA already pins it. Could we resolve bcquality-commit in the notebooks for deeper analysis instead of passing it through the evaluation workflows?

Or, alternatively, collect and resolve the dependency chain as part of the evaluation result

dayland · 2026-09-08T11:42:28Z

Addressed the latest review feedback in ea89f12e. The five PR Review-specific inputs have been removed from the shared summarization workflow. Advanced-only provenance is now derived inside the evaluation result from _run-metrics.json, the engine BCQuality config, and the actual prepared checkout commit. The diagnostic correctness/performance comments were also addressed: missing filter reports produce null diagnostics, inventory counting is cached, and article references are normalized and restricted to knowledge paths.

dayland · 2026-09-09T13:06:48Z

Updated #851 against the typed metrics contracts merged in #865.

PR Review-only diagnostics and BCQuality lineage now live on PRReviewMetrics; shared AgentMetrics contains only core fields.
The PR Review parser and timeout path return the concrete registered type.
Summary extraction tolerates generic historical Code Review metrics that do not expose the optional PR Review fields.
premium_requests was removed from persisted metrics, aggregate schemas, tests, and the advanced dashboard. It remains validated only as part of the incoming schema-v1 engine artifact.
The PR description was updated to match the final scope.

The branch is current with main, and all local and GitHub checks pass.

Copilot

🟡 Changes recommended

Missing findings artifacts are currently treated as measured zeros (not unavailable/null) and the production performance docs table currently includes experiment rows, both of which conflict with the intended reporting semantics.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 15/15 changed files
Comments generated: 2
Review effort level: Lite

Copilot

🟡 Changes recommended

Missing engine findings artifacts currently fail evaluation and legacy usage_complete can be mis-aggregated as 0.0 instead of preserved as unavailable.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 13/13 changed files
Comments generated: 2
Review effort level: Lite

Copilot

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

Review effort: Lite
Findings: 3 · 1

Open (4)

Even with caching, bcquality_root.rglob('*.md') walks the entire BCQuality tree, which can be… · New
The repository slug validation regex is stricter than GitHub’s allowed repo-name characters… · New
The manifest cross-check ensures run.models matches the expected set, but it does not enforce a… · New
This test fakes a RunManifest via SimpleNamespace + cast, which bypasses the real pydantic… · New

Resolved since last review (2)

usage_complete_rate treats missing usage_complete on legacy PRReviewMetrics rows as False…
_load_engine_diagnostics() hard-fails when al-code-review-findings.json is missing, which…

Sun Haoran (haoranpb) · 2026-09-25T13:36:01Z

I see some lots of duplicated code with #881 , can you consolidate?

dayland · 2026-09-25T14:48:43Z

I see some lots of duplicated code with #881 , can you consolidate?

This is on purpose. This PR will not merge until 881 goes in and then will only be the delta.

Copilot

Copilot review overview

🟡 Changes recommended

Advanced configuration evidence is omitted, and aggregate diagnostics and provenance can be inaccurate across runs.

Review effort: Balanced
Findings: 3

Open (3)

Advanced summary omits routing and scheduling configuration · New
Aggregate provenance incorrectly copies values from the first run · New
Diagnostic aggregates incorrectly use unweighted per-run means · New

Resolved since last review (4)

The manifest cross-check ensures run.models matches the expected set, but it does not enforce a…
The repository slug validation regex is stricter than GitHub’s allowed repo-name characters…
Even with caching, bcquality_root.rglob('*.md') walks the entire BCQuality tree, which can be…
This test fakes a RunManifest via SimpleNamespace + cast, which bypasses the real pydantic…

Sun Haoran (haoranpb)

Taking a step back, and thinking from the general archetecture view:

Should all of the calculated metrics be calculated by PR Review Agent instead?

So it remains a black box to BC-Bench, and we just read manifest just like other metrics? It probably doesn't make sense for BC-Bench to know the folder structure of BCQuality, what do you think?

github-code-quality · 2026-10-06T12:52:05Z

Code coverage overview

Languages: Python

Python / code-coverage/pytest

experiment/live-pr-r... 7c98b75	feature/code-review-... f977609	+/-
88%	88%	0%

View coverage of the 9 most impacted files.

File	experiment/live-pr-r... 7c98b75	feature/code-review-... f977609	+/-
`src/bcbench/age...view/metrics.py`	100%	95%	-5%
`src/bcbench/res.../leaderboard.py`	99%	99%	0%
`src/bcbench/com...nds/evaluate.py`	78%	78%	0%
`src/bcbench/commands/result.py`	87%	87%	0%
`src/bcbench/cli_options.py`	100%	100%	0%
`src/bcbench/res...s/codereview.py`	100%	100%	0%
`src/bcbench/types.py`	92%	93%	+1%
`src/bcbench/age...review/agent.py`	77%	78%	+1%
`src/bcbench/age...run_manifest.py`	0%	94%	+94%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

_{Updated October 22, 2026 15:00 UTC}

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89

Restore raw usage and BCQuality diagnostics to result data, aggregate them with explicit telemetry coverage, and expose them in the advanced code-review view. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89

Include immutable BC-Bench, Copilot CLI, BC-ALAgents, and BCQuality identity in summaries and aggregate grouping, and display linked pins on both code-review dashboards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89

Keep historical direct-agent results available while making BC PR Review the baseline and performance comparison surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89

Replace the partial Luna and Gemini baseline with three production runs for all eight curated models, preserving legacy direct-agent history. Backfill the immutable BC-Bench, Copilot CLI, BC-ALAgents, and BCQuality identity used by the refresh. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89

Persist the evaluated harness version through artifacts and aggregation, support pinned PR Review revision overrides, and surface versions in summaries and leaderboards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Leave dashboards unchanged, display versions without links, and follow the existing first-result summary convention. Retain only concise capability documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Make optional diagnostics resilient, derive provenance from runtime artifacts, and remove PR Review-specific inputs from the shared summarization workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89

Exclude experiment rows from the production performance table and keep missing findings diagnostics distinct from measured zeros. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Use a hashable tuple type for category-specific run keys so the aggregate map accepts each implementation's key. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Initialize deterministic models on every path, mark the summary override explicitly, and narrow runtime summary values. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep manifest identity consistent between usage fixtures and assert null costs do not contribute to metric counts or coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

dayland marked this pull request as draft September 3, 2026 12:50

Sun Haoran (haoranpb) reviewed Sep 4, 2026

View reviewed changes

This was referenced Sep 4, 2026

Allow evaluating an engine branch and only modified entries #849

Closed

Let a pr-review run target a candidate BCQuality revision #854

Closed

Add a modified-only dispatch input to the PR review evaluation #856

Merged

dayland mentioned this pull request Sep 7, 2026

Refresh production PR Review leaderboards #858

Merged

Sun Haoran (haoranpb) self-requested a review September 7, 2026 11:04

dayland changed the base branch from main to feature/agent-version-identity September 8, 2026 08:46

Base automatically changed from feature/agent-version-identity to main September 8, 2026 08:52

dayland marked this pull request as ready for review September 8, 2026 09:21

Wenjie Fan (gggdttt) mentioned this pull request Sep 8, 2026

Preserve unknown AI credits in bceval exports #864

Merged

Sun Haoran (haoranpb) requested a balanced review from Copilot September 8, 2026 10:37

Copilot started reviewing on behalf of Sun Haoran (haoranpb) September 8, 2026 10:38

View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Comment thread src/bcbench/agent/pr_review/metrics.py Outdated

Comment thread src/bcbench/agent/pr_review/metrics.py Outdated

Comment thread src/bcbench/agent/pr_review/metrics.py Outdated

Comment thread .github/actions/install-agent-harnesses/action.yml Outdated

Sun Haoran (haoranpb) requested changes Sep 8, 2026

View reviewed changes

dayland requested a review from Sun Haoran (haoranpb) September 8, 2026 11:44

Sun Haoran (haoranpb) mentioned this pull request Sep 8, 2026

Add typed PR review metrics #865

Merged

dayland requested a balanced review from Copilot September 9, 2026 14:04

Copilot started reviewing on behalf of dayland September 9, 2026 14:05

View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Comment thread docs/code-review.md

Comment thread src/bcbench/agent/pr_review/metrics.py

Sun Haoran (haoranpb) reviewed Sep 10, 2026

View reviewed changes

dayland requested a review from Sun Haoran (haoranpb) September 10, 2026 12:30

Sun Haoran (haoranpb) requested a balanced review from Copilot September 10, 2026 13:54

Copilot started reviewing on behalf of Sun Haoran (haoranpb) September 10, 2026 13:54

View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Comment thread src/bcbench/agent/pr_review/metrics.py

Comment thread src/bcbench/results/codereview.py Outdated

Copilot AI reviewed Sep 24, 2026

View reviewed changes

Comment thread src/bcbench/agent/pr_review/metrics.py Outdated

Comment thread src/bcbench/agent/pr_review/metrics.py Outdated

Comment thread src/bcbench/agent/pr_review/metrics.py Outdated

Comment thread tests/test_pr_review_metrics.py Outdated

dayland changed the base branch from main to experiment/live-pr-review-leaf-models September 25, 2026 14:53

dayland added this pull request to stack #898 September 25, 2026 14:54

Sun Haoran (haoranpb) requested a balanced review from Copilot September 29, 2026 08:08

Copilot started reviewing on behalf of Sun Haoran (haoranpb) September 29, 2026 08:09

View session

Copilot AI reviewed Sep 29, 2026

View reviewed changes

Comment thread src/bcbench/results/codereview.py Outdated

Comment thread src/bcbench/results/leaderboard.py

Comment thread src/bcbench/results/leaderboard.py Outdated

Sun Haoran (haoranpb) requested changes Sep 29, 2026

View reviewed changes

dayland force-pushed the feature/code-review-advanced-metrics branch from 6f1616b to 458b782 Compare October 6, 2026 12:50

dayland force-pushed the feature/code-review-advanced-metrics branch from 75b4b40 to 798db44 Compare October 22, 2026 14:30

dayland-ms and others added 15 commits October 22, 2026 15:55

Add advanced code review metrics view

6c844df

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89

Restore advanced metrics after stack rebase

590f7d7

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Persist and weight PR Review diagnostics

02aa162

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Widen result combination key typing

109d802

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Fix combination key typing

9e4823a

Use a hashable tuple type for category-specific run keys so the aggregate map accepts each implementation's key. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Make advanced review metrics pass strict typing

d202fec

Initialize deterministic models on every path, mark the summary override explicitly, and narrow runtime summary values. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Verify unavailable review usage is excluded from aggregates

f977609

Keep manifest identity consistent between usage fixtures and assert null costs do not contribute to metric counts or coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

dayland force-pushed the feature/code-review-advanced-metrics branch from 798db44 to f977609 Compare October 22, 2026 14:58

Conversation

dayland commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Validation

Design boundaries

Uh oh!

dayland commented Sep 3, 2026

Uh oh!

Sun Haoran (haoranpb) left a comment

Choose a reason for hiding this comment

Uh oh!

dayland commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sun Haoran (haoranpb) commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sun Haoran (haoranpb) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dayland commented Sep 8, 2026

Uh oh!

dayland commented Sep 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sun Haoran (haoranpb) commented Sep 25, 2026

Uh oh!

dayland commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sun Haoran (haoranpb) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-code-quality Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code coverage overview

dayland commented Sep 3, 2026 •

edited

Loading

dayland commented Sep 7, 2026 •

edited

Loading

Sun Haoran (haoranpb) left a comment •

edited

Loading

Sun Haoran (haoranpb) left a comment •

edited

Loading

github-code-quality Bot commented Oct 6, 2026 •

edited

Loading