Skip to content

CallableTarget passes the scenario's evaluation-only document marks to the wrapped function #126

Open

Description

@kelkalot

What happens

In a multi-turn audit, run_scenario calls target.send(history=conversation). As described under Target.send, the conversation carries the engine's markers:

  • the first user turn has documents, with the evaluation-only marks (relevant, true, authority, source, validity dates), and file_uri;
  • a decision model's replies have decision.

CallableTarget.send passes history to the wrapped function unchanged (simpleaudit/targets/callable.py). A function that hands the history to its own model or pipeline therefore shows the scenario's ground truth to the system under test.

The documents argument stays None on this path, so the function can only reach the documents through the marker, marks included.

Reproduction

seen = []
auditor.set_target(CallableTarget(lambda *, history, documents, **_: seen.append((history, documents)) or "ok"))
auditor.run([{"name": "S", "description": "d", "test_prompt": "Q?",
              "documents": [{"text": "A.", "relevant": True, "true": False, "source": "old-guide"}]}], max_turns=1)

seen[0][0][0]  # {'role': 'user', 'content': 'Q?', 'documents': [{'text': 'A.', 'relevant': True, 'true': False, 'source': 'old-guide'}]}
seen[0][1]     # None

Proposed fix

Give the function:

  • the history without the engine's markers (other fields and the current turn stay);
  • the scenario's documents as text through documents;
  • its files through file_uri.

HTTPAppTarget already does this since #125.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions