Skip to content

Targets: clean history and documents for CallableTarget, system prompt and template history for HTTPAppTarget - #129

Merged
kelkalot merged 2 commits into
devfrom
fix/target-history-followups
Oct 24, 2026
Merged

kelkalot merged 2 commits into
devfrom
fix/target-history-followups

Conversation

@kelkalot

@kelkalot kelkalot commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #126, #127 and #128. These are the remaining places where a target passed the system under test something other than the scenario's real input.

Changes

  • CallableTarget (CallableTarget passes the scenario's evaluation-only document marks to the wrapped function #126). The wrapped function now gets:

    • the history without the engine's markers (documents, decision, tool_events);
    • the scenario's documents as text through documents, on every turn;
    • its files through file_uri, on the turn that attaches them. That turn keeps its file_uri in the history, so the function can attach the files where they belong.

    No evaluation-only document mark reaches it.

  • HTTPAppTarget system prompt (HTTPAppTarget ignores the audit's system prompt #127). The audit's system_prompt is now sent:

    • With the new system_field, it goes to that request field in either body mode. That covers APIs that take the system prompt at the top level, such as Anthropic's native Messages API. A value the template already sets there comes first; content-block lists get the prompt appended as a text block.
    • Without system_field, in messages mode, it joins the template's leading system message or becomes one. Many chat templates reject two system messages in a row. A messages list in request_template is kept ahead of the conversation; before, it was overwritten.
    • Without system_field, in other modes, it can't be sent, and a warning says so.
  • HTTPAppTarget template history (A history key in HTTPAppTarget's request_template replaces the conversation history #128). A history list in request_template comes before the conversation's earlier turns instead of replacing them. A history value that isn't a list is now an error.

  • Shared helpers.

    • targets/base.py gains ENGINE_MARKERS, plain_history() and history_file_uri().
    • context_marks.document_texts() reduces documents to text.

    The callable, HTTP, decision and OpenEnv targets use them. DecisionTarget now treats an explicit documents=[] as given.

  • README. "Writing your own target" describes what a CallableTarget function and an HTTPAppTarget request receive.

Behaviour changes

  • CallableTarget functions no longer find documents on history[0], and get file_uri only on the turn that attaches it.
  • HTTPAppTarget sends the system prompt when the audit sets one: as part of the leading system message in messages mode, or at system_field.

Tests

Eight new tests in tests/test_targets.py. They cover:

  • what a CallableTarget function receives in a two-turn audit with marked documents and a file;
  • documents passed directly to send;
  • system prompt routing in both body modes, including joining with the template and the warning, also when warnings are errors;
  • template messages and history ahead of the conversation, and a non-list history.

pytest -n auto: 1725 passed.

Live checks with claude-haiku-5-5:

  • Anthropic's native Messages API (message_field="messages", system_field="system"): the template's and the audit's system prompts went out as one top-level system field, and the model followed them on both turns.
  • The OpenAI-compatible endpoint: a template system message and the audit's prompt went out as one system message, with the same result.
  • A CallableTarget wrapping a small RAG function: it built its prompt from the documents as text, with no marks in its requests.

- CallableTarget gives the function the history without the engine's
  markers, the scenario's documents as text through `documents`, and its
  files through `file_uri`, so no evaluation-only document mark reaches
  it (#126).
- HTTPAppTarget sends the audit's system prompt: as a leading system
  message in "messages" mode, after which a `messages` list from
  request_template comes before the conversation; in other modes through
  the new `system_field`, with a warning when there is none (#127).
- A `history` list in HTTPAppTarget's request_template comes before the
  conversation's earlier turns instead of replacing them (#128).
- targets/base.py gains ENGINE_MARKERS, plain_history() and
  history_file_uri(), shared by both targets.

Fixes #126
Fixes #127
Fixes #128
- system_field is honoured in "messages" mode too, so APIs that take the
  system prompt as a top-level field (Anthropic's native Messages API)
  work. A value request_template already sets at that field comes first;
  content-block lists get the prompt appended as a text block.
- Without system_field, the audit's prompt joins the template's leading
  system message instead of adding a second one; many chat templates
  reject two system messages in a row.
- The missing-system_field warning fires on every call when warnings are
  errors, instead of failing once and then dropping the prompt silently.
- A non-list "history" in request_template is an error instead of being
  replaced by the conversation.
- CallableTarget passes file_uri on the turn that attaches it only, and
  that turn keeps its file_uri in the history, so a function can attach
  the files where they belong.
- context_marks.document_texts() replaces the copies in the callable and
  HTTP targets; DecisionTarget treats an explicit documents=[] as given;
  OpenEnvTarget uses the shared history helpers. Stale comments and the
  README are updated.
@kelkalot
kelkalot merged commit cdb95cf into dev Oct 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant