Repository navigation
Targets: clean history and documents for CallableTarget, system prompt and template history for HTTPAppTarget - #129
Merged
Merged
Conversation
- CallableTarget gives the function the history without the engine's markers, the scenario's documents as text through `documents`, and its files through `file_uri`, so no evaluation-only document mark reaches it (#126). - HTTPAppTarget sends the audit's system prompt: as a leading system message in "messages" mode, after which a `messages` list from request_template comes before the conversation; in other modes through the new `system_field`, with a warning when there is none (#127). - A `history` list in HTTPAppTarget's request_template comes before the conversation's earlier turns instead of replacing them (#128). - targets/base.py gains ENGINE_MARKERS, plain_history() and history_file_uri(), shared by both targets. Fixes #126 Fixes #127 Fixes #128
- system_field is honoured in "messages" mode too, so APIs that take the system prompt as a top-level field (Anthropic's native Messages API) work. A value request_template already sets at that field comes first; content-block lists get the prompt appended as a text block. - Without system_field, the audit's prompt joins the template's leading system message instead of adding a second one; many chat templates reject two system messages in a row. - The missing-system_field warning fires on every call when warnings are errors, instead of failing once and then dropping the prompt silently. - A non-list "history" in request_template is an error instead of being replaced by the conversation. - CallableTarget passes file_uri on the turn that attaches it only, and that turn keeps its file_uri in the history, so a function can attach the files where they belong. - context_marks.document_texts() replaces the copies in the callable and HTTP targets; DecisionTarget treats an explicit documents=[] as given; OpenEnvTarget uses the shared history helpers. Stale comments and the README are updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #126, #127 and #128. These are the remaining places where a target passed the system under test something other than the scenario's real input.
Changes
CallableTarget(CallableTarget passes the scenario's evaluation-only document marks to the wrapped function #126). The wrapped function now gets:documents,decision,tool_events);documents, on every turn;file_uri, on the turn that attaches them. That turn keeps itsfile_uriin the history, so the function can attach the files where they belong.No evaluation-only document mark reaches it.
HTTPAppTargetsystem prompt (HTTPAppTarget ignores the audit's system prompt #127). The audit'ssystem_promptis now sent:system_field, it goes to that request field in either body mode. That covers APIs that take the system prompt at the top level, such as Anthropic's native Messages API. A value the template already sets there comes first; content-block lists get the prompt appended as a text block.system_field, inmessagesmode, it joins the template's leading system message or becomes one. Many chat templates reject two system messages in a row. Amessageslist inrequest_templateis kept ahead of the conversation; before, it was overwritten.system_field, in other modes, it can't be sent, and a warning says so.HTTPAppTargettemplate history (A history key in HTTPAppTarget's request_template replaces the conversation history #128). Ahistorylist inrequest_templatecomes before the conversation's earlier turns instead of replacing them. Ahistoryvalue that isn't a list is now an error.Shared helpers.
targets/base.pygainsENGINE_MARKERS,plain_history()andhistory_file_uri().context_marks.document_texts()reduces documents to text.The callable, HTTP, decision and OpenEnv targets use them.
DecisionTargetnow treats an explicitdocuments=[]as given.README. "Writing your own target" describes what a
CallableTargetfunction and anHTTPAppTargetrequest receive.Behaviour changes
CallableTargetfunctions no longer finddocumentsonhistory[0], and getfile_urionly on the turn that attaches it.HTTPAppTargetsends the system prompt when the audit sets one: as part of the leading system message inmessagesmode, or atsystem_field.Tests
Eight new tests in
tests/test_targets.py. They cover:CallableTargetfunction receives in a two-turn audit with marked documents and a file;send;pytest -n auto: 1725 passed.Live checks with
claude-haiku-5-5:message_field="messages",system_field="system"): the template's and the audit's system prompts went out as one top-levelsystemfield, and the model followed them on both turns.CallableTargetwrapping a small RAG function: it built its prompt from the documents as text, with no marks in its requests.