[kelkalot]

Runs SimpleAudit scenario packs as an Inspect AI task, and loads Inspect logs as SimpleAudit results so that rejudge, reframing_check and the stability report work on them. It's a new optional extra, inspect; nothing in simpleaudit imports it.

Usage

pip install 'simpleaudit[inspect]'
inspect eval simpleaudit/audit -T pack=nav_aap -T language=Norwegian \
    --model openai/gpt-4o-mini --model-role grader=openai/gpt-4o --epochs 5

from simpleaudit.integrations.inspect import load_inspect_log

runs = load_inspect_log("logs/<file>.eval")   # one run per epoch
runs.stability(model).summary()

How it works

• Roles. The task model is the target. The grader role is the judge, and it's required: without it, Inspect falls back to the model under test. The auditor role writes the probes and defaults to the grader.
• Model calls. The solver runs the existing SimpleAudit loop for each sample. Model calls go through InspectModelClient, so they appear in Inspect's transcript and usage and follow Inspect's limits and retries. SimpleAudit's own retries are off.
• Failures.
  • A model error or an exceeded limit is re-raised, so Inspect handles it. The result is stored first, with the transcript up to that point.
  • A judge reply that can't be parsed leaves the sample unscored.
• Scores.
  • The score is the severity's points (pass 4 … critical 0). As in AuditResults.score, a severity off the ladder counts as medium, and ERROR and ungraded are left out.
  • Epochs reduce with modal_severity: the most common severity, with a tie going to the more severe one, as in aggregate_severities. Inspect's mode reducer keeps whichever tied epoch came first.
  • The metrics are safety_score (0–100) and a count per severity.
• Logs. load_inspect_log returns RepeatedExperimentResults.
  • Samples from this task come back as stored. One that stopped before storing comes back as ERROR with an empty transcript.
  • Samples from other evals are rebuilt from their messages, with tool calls, results and error messages as tool_events. Their severity is ungraded unless a score or the severity= mapper gives one; those values are normalised.
• Schemas. A judge schema that Inspect's JSONSchema can't express in full ($defs, $ref, const, …) isn't sent, and a warning says so. The reply is parsed as JSON either way.
• Example. examples/inspect_log_robustness.py runs the perturbation check on any Inspect log.

For review

• Judge placement. The judge runs in the solver, not in a scorer, so inspect score doesn't grade again. Re-grading goes through load_inspect_log and rejudge. Moving the judge into a scorer would need run_scenario split into a conversation phase and a judge phase.
• Dependency conflict. any-llm-sdk 1.40.1 requires anthropic<1, while Inspect's Anthropic provider needs anthropic>=2. The README describes the workaround.
• Version floor. The minimum inspect-ai version is the one this was tested with, 0.3.283.
• Editable installs. When running eval("simpleaudit/audit") from Python in the repo root, the editable install's simpleaudit.egg-info hides the metadata Inspect uses to name the task. The inspect CLI and regular installs aren't affected, and the registry test runs in a subprocess for this reason.
• CI. CI installs the extra. OpenEnvTarget: audit agents hosted in OpenEnv environments #123 edits the same install lines.

Tests

tests/test_inspect_integration.py has 23 tests that run offline on Inspect's mockllm. They cover:

• scores, metrics, epochs and tie-breaking;
• the target never seeing the scenario description;
• the auditor role defaulting to the grader;
• a missing grader role;
• model failures and token limits, including the kept transcript;
• unparseable verdicts and scoring of off-ladder severities;
• lookup by registry name;
• schema handling;
• log round trips and unfinished samples;
• an agent transcript from another eval with tool errors, re-graded with rejudge.

Live checks, with anthropic/claude-haiku-5-5 as model and grader:

• inspect eval simpleaudit/audit on 3 nav_aap scenarios × 2 epochs: a medium/low split reduced to medium both in Inspect and in the stability report from load_inspect_log.
• A run with --token-limit: Inspect recorded it as a limit, the transcript up to that point was kept, and it loaded as ERROR.
• examples/inspect_log_robustness.py ran on the log.