Tell the tester when to use the decision model - #250
Merged
Merged
Conversation
The judge tool was never called by the tester: its description was one line and no rule mentioned it, while several rules route quick page checks to see(). The description now states what the judge sees, what it is for and where it stops; the tester's system prompt gains a judge_rule when the tool is in its toolset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On today's CI regression run (2026-09-25) (29 tests, 124 Jev calls on langfuse.testomatio.work), the tester never called the
judgetool. Every call came from the built-in call sites in Navigator, Pilot and the multiple-elements fallback. The tool's description was a single line, no tester rule mentioned it, and several rules send quick "did it already happen?" checks tosee(), which uses a screenshot.What
judge-tool.ts: the description now says what Jev sees (page_aria+ recent steps, no screenshot or HTML, no test code), what it is for (in-flow judgements: did the last action take effect, which of several similar elements is meant, is this the expected page), and where it stops (final assertions go toverify/finish, visual state tosee).tester.ts: a<judge_rule>block is added to the system prompt only whenjudgeis in the tester's toolset, so prompts without the decision model are unchanged.Prompt-only change.
bun test tests/integration/passes (153 pass, 1 skip).Measuring it
The effect only shows up with real models: a regression run with the decision model on should show
judgetool calls from the tester and fewersee()calls. Not started; apply theregressionlabel if wanted.🤖 Generated with Claude Code