[saloni-garg]

Summary

Fixes #1475. Code-mode reward now requires changed SQLite contents or an answer that changes the verifier outcome from others to complete. Comparison failures return judge_error; committed WAL changes are included.

Evidence

84 AWM tests pass, including arbitrary-answer bypass, corrupt DB, and WAL regressions. Changed-file lint passes. Broad suite: 3671 passed, 8 unrelated failures, 146 skipped (macOS/process permissions, Harbor dependencies, REPL subprocess). Repository lint has existing failures. Required CI remains outstanding.

Merge Danger

Reversible; limited to AWM code-mode reward classification. SQL mode is unchanged.

---

Note

Medium Risk
Changes code-mode reward classification for all AWM episodes using pure-code verifiers; behavior is intentional but may shift training/eval scores versus prior runs.

Overview
Fixes false complete rewards when code-mode verifiers pass on an unchanged SQLite database (#1475).

After a verifier returns complete, run_verifier now compares initial vs final DB contents via iterdump (including committed WAL changes). If the DB is unchanged, the reward is downgraded to others with no_db_change: true, unless the task is answer-based: with a non-empty final_answer, it reruns the same verifier with None and only keeps complete when that baseline returns others (proving the answer earned the pass). Baseline or comparison failures return judge_error (fail closed).

The verifier subprocess runner adds uses_final_answer on success and error_kind (compile, not_found, execution, invalid_return) on code-mode errors. SQL verification is unchanged. Tests cover no-op DB passes, real DB edits, answer tasks, baseline errors, corrupt DB, and WAL.

^{Reviewed by Cursor Bugbot for commit 2209de8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[burtenshaw]

The new no-change guard is bypassed whenever final_answer contains any non-whitespace text, even if the verifier ignores that argument. For the seeded-row verifier in the new regression, final_answer="done" still receives complete on the unchanged database (the existing test is changed to use precisely this bypass). This does not prevent the no-op reward exploit for agents that return an arbitrary final answer. Please distinguish answer-based tasks using an explicit task/verifier contract rather than the presence of text; the correct boundary needs task-owner input.

[burtenshaw]

Fixed answer-based bypass, fail-open comparison, and WAL handling; merged current main. 84 AWM tests pass; changed-file lint passes. Full-suite validation is still running; required CI is missing/pending.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit a5c0bf6. Configure here.}

[saloni-garg]

Thanks for reproducing, @burtenshaw. I have added the baseline contract you asked for -- the runner now reports whether a verifier actually consumes final_answer, and tags its errors by kind. The baseline rerun differs from the real run only in the answer, so when the verifier itself raises on the empty answer (float(''), json.loads('')), the pass belongs to the answer and stays. Compile errors, subprocess failures, and timeouts still fail closed as judge_error.

Regressions added: numeric and JSON answers keep their pass, the seeded-row verifier with answer text stays downgraded, and a baseline infrastructure failure stays judge_error. All AWM tests pass locally.

[burtenshaw]

Thanks for adding the numeric/JSON regressions. Those now pass, but the broad baseline execution-error exception still leaves a grading blocker at 7b06facf4467817a10f66b787ebc5106869bcc92.

Two real-subprocess regressions reproduce it: an exclusive SQLite lock during the baseline produces database is locked, and an unrelated database-value conversion raises ValueError. Both preserve complete on unchanged databases. A three-argument signature establishes that the runner passes an answer; it does not establish that the verifier uses it, or that an exception was caused by it. Narrowing this to a generic exception-type allowlist would still accept the unrelated ValueError.

The existing AWM suite passes (88 tests). A conservative proposed repair makes every baseline error return judge_error; both new regressions then pass, but the existing unguarded numeric/JSON cases fail. Guarded versions pass when they return {'result': 'others'} for an empty answer before calling float or json.loads.

Could you confirm and implement that verifier contract: an empty answer must return others, and baseline execution errors fail closed? If existing verifier producers cannot meet it, we need an explicit alternative contract for answer-dependent validation failures before treating them as evidence of completion. I have left the branch unchanged rather than silently changing that contract. CI passing alone does not clear these two reproductions.

[saloni-garg]

Thanks for the reproductions, @burtenshaw. I agree with the conservative contract: baseline execution errors should fail closed as judge_error, and an empty answer should return others before any float or json parsing. I will implement that and update the numeric and JSON regressions to the guarded form, and check what else in the AWM suite relies on the unguarded behaviour.