[Wenjie Fan (gggdttt)]

Latest result: authorized API009 recovery completed; stopped

Recovery run 35736710876, attempt 1, completed successfully on 2026-09-22 13:55:16–14:34:06 UTC. It was a separately authorized, single-entry workflow_dispatch for synthetic__performance-009, not an Actions rerun or rerun-failed. The original setup-failed attempt and its evidence remain unchanged.

Execution integrity passed: 17/17 processes, exact pins/CLI/models, complete usage, valid schemas/roles and acceptable actual leaf/root/recursive outcomes. The actual matrix contained only API009; 014, drilldown and the full corpus were not rerun.

Recovery measure	Result
Bench / engine	f9cb9318e331bc2f77cc9779bf0f9cdbfe53e529 / 7fad26b703f0450ed9f95a63205128c175001e7f
Expected / generated comments	0 / 4
TP / benchmark FP / FN / ignored	0 / 4 / 0 / 0
Negative behavior	Noisy negative, not a quality improvement
Observed review AI credits	88.435023
Prompt / completion / total tokens	5,190,504 / 62,282 / 5,252,786
Review API calls / failed API calls	155 / 0
Engine wall time	1,694.605 seconds
Judge	Not invoked: no expected or ignored candidate pairs

The four generated findings concern a permission set, a temporary API/webhook, use of Name rather than SystemId as an API key, and another API's missing exposed SystemId field. They are counted against the frozen zero-gold fixture; this recovery does not adjudicate every unmatched finding as technically wrong or alter the gold. Its vacuous per-entry recall is not a recall-improvement claim.

The entry job 106776615506 ran 13:57:41–14:27:20 UTC. All three result/diagnostic/summary artifacts and the complete Actions log were preserved. Use actions.redacted.log with signed download URL query parameters removed. Evidence under postmerge-7fad26b/recovery009-once/run-35736710876 includes final-handoff.json, recovery-summary.json, and evidence-inventory.json.

Only entries changed from the original smoke inputs; source, version, engine/knowledge pins, CLI, models, severity, scorer and validator were unchanged. No new commit, push or code change was made for recovery. The user authorized exactly one additional generation after the original download failure. That authorization is now finished: no second recovery, other-entry retry or full run is scheduled. The external billing total is not claimed; judge cost is inapplicable because no judge ran.

Original post-merge smoke: not all entries completed validly

Smoke 35707243840, attempt 1, ran 2026-09-22 08:53:06–09:10:49 UTC, with Actions conclusion failure. Exactly three jobs were dispatched; only two reached generation. Its original partition remains one complete accepted entry, one rejected partial report and one setup failure. Recovery above does not rewrite that original result or turn it into a successful three-entry round.

Shared post-merge identity

Component	Value
BC-Bench commit / version	f9cb9318e331bc2f77cc9779bf0f9cdbfe53e529 / 0.14.0.dev4
Engine	7fad26b703f0450ed9f95a63205128c175001e7f, merged microsoft/BC-ALAgents#70 squash commit
Source equivalence	Same tree as reviewed head 8ac053de3a6574129e280d99fa71a117362109eb
BCQuality	b74967bc5b7a454eae19d6a1250199afd869f064, version 1.6, Microsoft layer only
CLI / root / leaves / judge	1.0.83 / gpt-5.6-sol / gpt-5.6-luna / gpt-5.3-codex
Configuration	Serial, both severity floors Medium, max leaf concurrency 4 unused

Only four pin/version lines changed from preceding Bench commit ce21d50a. Dataset, scorer, runtime validator, workflow and other parameters were unchanged. CI 35706993532 passed, including actual mock bc-eval consumption of 0.14.0.dev4.

Original smoke entry	Execution	Score	Review credits	Tokens
performance-009	Setup failed, 0 review processes	Unscored	null	null
performance-014	17 processes, root and manifest partial	Unscored	99.013474	5,151,624
Drilldown	17 processes, complete integrity	0 TP / 1 FP / 1 FN	77.663447	4,717,049
Observed total	34 started processes	No aggregate score	176.676921	9,868,673

All 34 started processes had exact identities/models/CLI, exit code 0 and complete usage. Their total includes the rejected partial entry. One normal judge invocation occurred; its cost and the total including it remain unknown. Engine wall times were 956.776 and 717.931 seconds for the two generated entries.

009: infrastructure failure before generation. Job 106679206054 failed during Setup Python with UV / uv sync, downloading rpds-py==0.28.0: Request failed after 3 retries in 12.8s, http2 error, stream error received: stream no longer needed. These were internal download retries, not evaluation reruns. The reviewer was skipped; absent manifest/score/cost fields remain null, not a clean negative or recall zero.

014: content rejection produced partial coverage. Job 106679206041 completed all 17 processes with schema-valid reports. The style leaf claimed the filename was Data Processor.Codeunit.al and should be renamed to DataProcessor.Codeunit.al, while its own location was already src/DataProcessor.Codeunit.al. Root marked that sub-result failed, with this exact reason:

al-style-review failed consumer acceptance validation because its finding alleges a space in src/DataProcessor.Codeunit.al, but the supplied source path already follows the cited naming convention; all other sub-skills completed or returned not-applicable.

The merged engine propagated the declared partial outcome to the manifest; the unchanged Bench validator rejected it before scoring. This is not a recurrence of the old partial-report/completed-manifest discrepancy. Raw evidence retains leaf-results/05-al-style-review/_review-report.json, bcquality/_review-report.json, and _run-manifest.json.

Drilldown: valid execution, scored miss. Job 106679206046 passed all integrity gates. The generated comment concerned an IsTemporary guard before DeleteAll, not restoring Group Code identity across regenerated Entry Nos. The original judge's sole pair was explicitly false. This valid 0 TP / 1 FP / 1 FN observation was not retried to obtain a preferred score.

All five actual artifacts and the whole Actions log remain under postmerge-7fad26b/smoke-35707243840, with final-handoff.json, smoke-summary.json, evidence-inventory.json and sanitized actions.redacted.log. The 009 diagnostics archive is empty; no result exists for its initial attempt. Summary/requeue skipped after failures. This original smoke had no conditional full authorization, and the later API009 recovery required separate explicit authorization.

Previous full round: finished, baseline incomplete

Full run 35608133170, attempt 1, executed all 145 declared entries once on 2026-09-21 13:49:56–14:30:57 UTC (41m01s). Independent artifact audit accepted 126, rejected 19, with none missing. Actions also reported 126 successful / 19 failed jobs, but status alone was not the acceptance rule.

That round is closed with no retry or re-judging. It is not a complete 145-entry / 233-expected-finding baseline; full measured precision, recall and F1 remain null. Later smoke/recovery outputs are not mixed into it.

• Bench ce21d50a261e93205ab9d0860e339a3cb723ee76 / 0.14.0.dev3; engine 75d67cd7c18b9ad98a1c3cbf6d173e5bdba53b9c, not later 8ac053d/7fad26b.
• Same BCQuality, CLI, models, serial scheduling and Medium floors as the post-merge smoke. Global matrix maximum 64 was preserved.
• Dataset: 145 entries, 233 expectations, 39 negatives. Blob 35bbe950213999b0cab01f8a7490425a1291d6bc; SHA-256 43bcddc685e3f81cb655fc0efe33f7c47763fe64864ecf62ab69220a231e513c. All 23 Low expectations remain in the denominator.

Measure	Accepted-subset result
Accepted / invalid / missing entries	126 / 19 / 0
Scored / unscored expected findings	182 / 51
TP / benchmark FP / scored FN	89 / 305 / 93
Micro precision / recall / F1	22.6% / 48.9% / 30.9%
Macro precision / recall / F1	30.6% / 63.9% / 34.2%
Valid output rate	86.9%

These metrics apply only to the accepted subset. Invalid executions were not scored as zero or counted as false negatives. Benchmark FP means unmatched against gold, not automatically technically incorrect.

Holding the 126 accepted outputs fixed, the 51 unscored expectations could add zero through 51 matches. The conditional mathematical recall bounds against 233 are 38.1974%–60.0858% (89–140 possible matches), not measured full recall or a confidence interval. Even crediting every unscored expectation does not make these frozen observations support 70–80%; this does not bound a fresh generation of every entry.

Gold domain	Matched / scored expected	Recall on scored expectations	Unscored expected
Style	15 / 61	24.6%	19
Performance	20 / 34	58.8%	15
Security	13 / 25	52.0%	1
Privacy	19 / 24	79.2%	0
Upgrade	9 / 17	52.9%	14
Error handling	6 / 7	85.7%	1
Breaking changes	2 / 5	40.0%	0
Data modeling	0 / 3	0.0%	0
UI	2 / 3	66.7%	1
Events	2 / 2	100.0%	0
Testing	1 / 1	100.0%	0

Style accounts for 46 of 93 scored misses. Drilldown matched in the third pilot but was missed in this fresh full round. Of 39 negatives, 4 were clean, 30 noisy and 5 unscored; the 34 accepted negatives produced 77 benchmark FPs. Of 23 Low expectations, 18 were scored (9 TP / 9 FN) and 5 unscored. Low gold may match higher-severity generated findings.

Full-round failures, cost and evidence

The 19 rejected entries comprise 12 root consumer/protocol partial outcomes, 5 leaf schema rejections followed by partial root, and 2 incomplete-usage failures. Partial runs were rejected by the unchanged validator. Usage failures were synthetic__upgrade-clean-01 and synthetic__style-018, each stopping at the security leaf after two processes, before root. Exact reasons/job URLs are in failure-evidence.json; these observations do not prove PR70 caused the model's report defects.

synthetic__security-015 was retained after recovering six original explicit false judge verdicts from evaluation.log lines 18–21, matching the normal pipeline's stdout fallback and original 0 TP / 3 FP / 2 FN. The same parser was replayed offline; no new model call or invented verdict was used. judge-stdout-evidence.json preserves the source.

2,435 review processes started. Known review subtotal: 13,497.489350 AI credits / 719,204,013 tokens. These are not complete billing totals: two failed leaves have incomplete usage and null credits, and their available tokens are partial observations. Judge-inclusive total remains unknown. Accepted entries averaged 708.28 engine seconds. The third pilot used 238.177663 review credits; that pilot-plus-full known subtotal was 13,735.667013, excluding other epochs and post-merge smoke/recovery.

All 290 artifacts and 145 entry logs were downloaded and hash-indexed. Whole Actions-log transport repeatedly failed with CANCEL/EOF; the incomplete archive is explicitly run-logs.partial.zip. Summary/requeue skipped, with no normal workflow summary or external score publication. Evidence under pr70-75d67cd/full-35608133170 includes final-handoff.json, full-report.json, full-summary.json, failure-evidence.json, evidence-inventory.json, and ledgers of 89 matched, 93 missed, 51 unscored expected findings and 305 benchmark FPs.

Earlier pilot history

Pilot	Bench / engine	Accepted TP / FP / FN: 009; 014; drilldown	Outcome
35606001580	ce21d50a dev3 / 75d67cd	0/4/0; 1/1/0; 1/2/0	3/3 complete, then the separately authorized full round
35583013319	bbd1cea9308268bb1d2a2f3e23304e2ba524633b dev2 / 560015977cbeb69a8aef85daaddf1fe91968600c	0/5/0; 1/4/0; null	Stopped on partial root despite completed manifest; no full/retry
35577242876	8297c38ef1e13968e58e1caeb0e644333b90c7c8 dev1 / 1fcaa40fa0934c22887f1a89000f33728f831a8f	0/4/0; unscored; 1/0/0	Stopped on query schema failure; no full/retry

Third pilot ran 2026-09-21 13:30:00–13:48:04 UTC: 51 processes, 238.177663 review credits / 15,672,256 tokens, seven artifacts under pr70-75d67cd/pilot-35606001580. Scores did not gate promotion; no output was reused in full.

Second pilot ran 2026-09-21 09:23:34–09:38:17 UTC. Query leaf 14 in job 106280111347 returned not-applicable with empty sub-results and skipped-sub-skills. The shared schema accepted them but the fixed DO contract prohibits them on leaves. Root returned partial while manifest said completed. Its raw 1 TP / 1 FP / 0 FN was not accepted. Credits/tokens by entry: 009 96.869397/5,365,148; 014 93.123002/5,008,258; drilldown 94.468265/5,195,990. Total 51 processes, 284.460664 credits / 15,569,396 tokens. Seven artifacts remain under pr70-5600159/pilot-35583013319, including integrity-stop-evidence.json; diagnostic artifact 10631508227 retains exact reports.

First pilot ran 2026-09-21 08:18:49–08:33:17 UTC, without PR70. 014 job 106261947854 omitted required suppressed at query leaf 14. Only 13 reports were accepted / 14 processes started there; AppSource, telemetry and root never started, and no final review, aggregate, score or judge exists for it. This was the sole schema error among 48 reports. Credits/tokens by entry: 009 88.483108/5,658,442; 014 observed process subtotal 26.171918/4,238,820; drilldown 67.465308/4,587,231. Total 182.120334 credits / 14,484,493 tokens. Six artifacts remain under pilot-35577242876. Judge-inclusive costs for these pilots are unknown.

Shared boundaries

The integration uses frozen Bench main 9b1f68928a06be31806fde6f6452237a6b9480f5 plus unmerged #881 at 39264a3bd32257afa18c8908feffc6361577c429, without its repetition-ID bypass. Later main/dependency updates were not absorbed.

Every nonempty dispatch engine-sha equals its declared immutable pin. Existing summarizer mock gating disables Braintrust/Kusto storage and leaderboard updates only; generation and normal judges are real. repeat=1 creates no ephemeral tag or automatic successor. No experiment-PR merge, score publication or release is part of this work.

Diagnostics are allowlisted reports, metrics/manifests, task context, generated review, judge evidence and logger output, not environment dumps, credentials, home directories or whole workspaces. Raw OTel is not in the frozen upload allowlist and is not claimed retained when absent. Use sanitized logs when sharing evidence.

Historical configurations or 234-expected-finding corpora are not apples-to-apples controls. The 48.9% accepted-subset recall is not a full-corpus improvement claim. No epoch isolates an individual commit's causal effect, and a matched finding does not prove its proposed repair is correct. All protocols, raw evidence and results remain separated by run and epoch.