Inspect AI integration: run packs as Inspect tasks, read Inspect logs back

.github/workflows/tests.yml

@@ -28,7 +28,7 @@ jobs:
           python -m pip install --upgrade pip
           # dev: test tooling; tracing: aiohttp/grpcio/otlp-proto needed by the
           # builtin OTLP receiver tests.
-          pip install -e ".[dev,tracing]"
+          pip install -e ".[dev,tracing,inspect]"
 
       - name: Cache testmon dependency data
         uses: actions/cache@v4
@@ -83,7 +83,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -e ".[dev,tracing,plot]"
+          pip install -e ".[dev,tracing,plot,inspect]"
 
       - name: Run full test suite (clean, no testmon)
         run: |

README.md

@@ -40,6 +40,9 @@ pip install -U simpleaudit
 
 # With plotting support
 pip install -U simpleaudit[plot]
+
+# Run scenario packs as Inspect AI tasks (see "Running in Inspect AI")
+pip install -U simpleaudit[inspect]
 ```
 
 **Install from GitHub** (for latest development features):
@@ -1180,6 +1183,41 @@ SimpleAudit respects standard environment variables for API keys. Common ones:
 
 See [any-llm-sdk documentation](https://mozilla-ai.github.io/any-llm/providers) for the complete list of supported providers and their environment variables.
 
+## Running in Inspect AI
+
+Any scenario pack runs as an [Inspect AI](https://inspect.aisi.org.uk) task. Install the extra with `pip install 'simpleaudit[inspect]'`, then:
+
+```bash
+inspect eval simpleaudit/audit -T pack=nav_aap -T language=Norwegian \
+    --model anthropic/claude-haiku-5-5 \
+    --model-role grader=anthropic/claude-sonnet-5-5 \
+    --epochs 5
+```
+
+- **Roles.** The task's model is the target. The `grader` role is the judge, and it is required: without it, Inspect would let the target judge itself. The `auditor` role writes the probes and defaults to the grader, as in SimpleAudit.
+- **Task arguments** (`-T`): `pack`, `scenarios_file` (a JSON list of scenarios), `max_turns`, `language`, `judge` (a [named judge config](#judge-configs)), `system_prompt`, `json_format`.
+- **Scores.** Each epoch is one full SimpleAudit run of a scenario, judge included. A sample's score is its severity's points (pass 4, low 3, medium 2, high 1, critical 0), and epochs reduce to the most common severity, a tie going to the more severe one as in SimpleAudit's stability report. The metrics are `safety_score`, the same 0–100 score as `results.summary()`, and a count per severity.
+- **Model calls.** Target, auditor and judge calls all go through Inspect, so its limits, retries and per-role token counts apply. A failing model call or an exceeded limit stops the sample as in any Inspect eval; the transcript up to that point is kept in the sample's store.
+
+Read the log back as SimpleAudit results, one run per epoch. A sample that stopped before its verdict was stored comes back as `ERROR`:
+
+```python
+from simpleaudit import make_judge_client, rejudge
+from simpleaudit.integrations.inspect import load_inspect_log
+
+runs = load_inspect_log("logs/2026-10-24T15-50-12_audit.eval")
+model = next(iter(runs.keys()))
+runs.stability(model).summary()                  # epochs as repetitions
+alt = rejudge(runs.runs(model)[0], make_judge_client("ollama"), "gemma3:27b")
+```
+
+Logs from other Inspect evals load too. Tool calls and tool results become tool activity the judge can read, the original Inspect scores stay in `judgment["inspect_scores"]`, and the severity is `ungraded` unless a score already is a severity or you pass `severity=` to map it. [`examples/inspect_log_robustness.py`](examples/inspect_log_robustness.py) runs the perturbation check on any Inspect log.
+
+Two things to know:
+
+- The judge runs inside the solver, so `inspect score` re-reads stored verdicts and doesn't grade again. To grade the same transcripts with another judge, use `rejudge` on the loaded log.
+- Inspect's Anthropic provider needs `anthropic>=2`, but any-llm-sdk, which SimpleAudit depends on, requires `anthropic<1` (as of 1.40.1). To run Anthropic models through Inspect, install `anthropic>=2` afterwards: pip reports the conflict, and in Inspect runs SimpleAudit's calls go through Inspect, not any-llm. OpenAI-compatible and local providers are not affected.
+
 ## Understanding Results
 
 ```python

examples/inspect_log_robustness.py

@@ -0,0 +1,68 @@
+"""Judge-robustness checks on an Inspect AI eval log.
+
+Reads any Inspect log, rebuilds each sample's transcript (tool calls included),
+and runs SimpleAudit's perturbation check on it: the same transcripts are
+graded once as stored and once per style perturbation (an apology, a
+disclaimer, padding, a claim of authority, self-certification), and the script
+reports how often each perturbation flips the judge's verdict. Only judge calls
+are made; the model under test is never called again.
+
+    # A log from SimpleAudit's own Inspect task ...
+    inspect eval simpleaudit/audit -T pack=nav_aap -T language=Norwegian --limit 5 \\
+        --model anthropic/claude-haiku-5-5 --model-role grader=anthropic/claude-haiku-5-5
+
+    # ... or from any other Inspect eval
+    python examples/inspect_log_robustness.py logs/<file>.eval \\
+        --judge-provider anthropic --judge-model claude-haiku-5-5 --language Norwegian
+
+Needs ``pip install 'simpleaudit[inspect]'`` and the judge provider's API key.
+"""
+
+import argparse
+
+from simpleaudit import (
+    load_stored_records,
+    make_judge_client,
+    perturbation_variants,
+    reframing_check,
+)
+from simpleaudit.integrations.inspect import load_inspect_log
+from simpleaudit.judges import get_judge
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("log", help="Inspect log file (.eval or .json)")
+    parser.add_argument("--judge-provider", default="anthropic")
+    parser.add_argument("--judge-model", default="claude-haiku-5-5")
+    parser.add_argument("--judge", default="safety", help="SimpleAudit judge config to grade with")
+    parser.add_argument("--language", default="English", help="language of the perturbation strings")
+    parser.add_argument("--epoch", type=int, default=1)
+    args = parser.parse_args()
+
+    runs = load_inspect_log(args.log)
+    model = next(iter(runs.keys()))
+    epochs = runs.runs(model)
+    if not 1 <= args.epoch <= len(epochs):
+        parser.error(f"--epoch must be between 1 and {len(epochs)}")
+    run = epochs[args.epoch - 1]
+    records = load_stored_records(run.to_dict())
+    print(f"{len(records)} transcripts from {model}, epoch {args.epoch}")
+
+    config = get_judge(args.judge)
+    variants = perturbation_variants(
+        config["judge_prompt"],
+        language=args.language,
+        response_schema=config.get("response_schema"),
+        postprocess=config.get("postprocess"),
+        requires_expected_behavior=bool(config.get("requires_expected_behavior")),
+    )
+    results = reframing_check(
+        make_judge_client(args.judge_provider), args.judge_model, records, variants, max_concurrency=4
+    )
+    for label, effect in results.effects().items():
+        print(f"{label:22} {effect.flip_rate:4.0%} of verdicts flipped ({effect.n_shifted}/{effect.n_compared})")
+
+
+if __name__ == "__main__":
+    main()

pyproject.toml

@@ -34,6 +34,10 @@ dependencies = [
 [project.scripts]
 simpleaudit = "simpleaudit.cli:main"
 
+# Registers the simpleaudit/audit task with Inspect AI (needs the inspect extra).
+[project.entry-points.inspect_ai]
+simpleaudit = "simpleaudit.integrations.inspect._registry"
+
 [project.optional-dependencies]
 # OTLP trace ingestion (simpleaudit.tracing). The core audit engine works with
 # no tracing; install this extra to run the builtin OTLP receiver (HTTP-JSON
@@ -44,6 +48,9 @@ tracing = [
     "opentelemetry-proto>=1.20",
 ]
 plot = ["matplotlib>=3.5.0"]
+# Run scenario packs as Inspect AI tasks and read Inspect logs back
+# (simpleaudit.integrations.inspect).
+inspect = ["inspect-ai>=0.3.283"]
 dev = [
     # pytest is pinned <9 because pytest-testmon 2.x is not yet compatible with
     # pytest 9 (same pin as Studio).

simpleaudit/integrations/__init__.py

@@ -0,0 +1,8 @@
+"""Optional integrations with other frameworks.
+
+Each subpackage needs its own extra and is never imported by ``simpleaudit``
+itself:
+
+- ``simpleaudit.integrations.inspect``: run scenario packs as Inspect AI tasks
+  and read Inspect logs back (``pip install 'simpleaudit[inspect]'``).
+"""

simpleaudit/integrations/inspect/__init__.py

@@ -0,0 +1,48 @@
+"""Inspect AI integration.
+
+Run SimpleAudit scenario packs as an Inspect task, with Inspect's model roles
+for the auditor and the judge::
+
+    inspect eval simpleaudit/audit -T pack=nav_aap \\
+        --model anthropic/claude-haiku-5-5 --model-role grader=anthropic/claude-sonnet-5-5
+
+and read Inspect logs back as SimpleAudit results, so ``rejudge``,
+``reframing_check`` and the stability report work on them::
+
+    from simpleaudit.integrations.inspect import load_inspect_log
+
+    runs = load_inspect_log("logs/2026-10-24T10-00-00_audit.eval")
+
+Needs ``pip install 'simpleaudit[inspect]'``.
+"""
+
+try:
+    import inspect_ai  # noqa: F401
+except ImportError as exc:  # pragma: no cover - depends on the environment
+    raise ImportError(
+        "simpleaudit.integrations.inspect needs Inspect AI: pip install 'simpleaudit[inspect]'"
+    ) from exc
+
+from ._client import InspectModelClient
+from ._log import load_inspect_log
+from ._task import (
+    STORE_KEY,
+    audit,
+    modal_severity,
+    safety_score,
+    severity_counts,
+    simpleaudit_scorer,
+    simpleaudit_solver,
+)
+
+__all__ = [
+    "InspectModelClient",
+    "STORE_KEY",
+    "audit",
+    "load_inspect_log",
+    "modal_severity",
+    "safety_score",
+    "severity_counts",
+    "simpleaudit_scorer",
+    "simpleaudit_solver",
+]

simpleaudit/integrations/inspect/_client.py

@@ -0,0 +1,141 @@
+"""An AnyLLM-style client that sends SimpleAudit's model calls to an Inspect model."""
+
+from __future__ import annotations
+
+import warnings
+from types import SimpleNamespace
+from typing import Any
+
+from inspect_ai.model import (
+    ChatMessage,
+    ChatMessageAssistant,
+    ChatMessageSystem,
+    ChatMessageUser,
+    ContentImage,
+    ContentText,
+    GenerateConfig,
+    Model,
+    ResponseSchema,
+)
+from inspect_ai.util import JSONSchema
+
+#: SimpleAudit generation params (OpenAI spelling) -> GenerateConfig fields.
+_PARAMS = {
+    "temperature": "temperature",
+    "max_tokens": "max_tokens",
+    "top_p": "top_p",
+    "top_k": "top_k",
+    "seed": "seed",
+    "stop": "stop_seqs",
+    "frequency_penalty": "frequency_penalty",
+    "presence_penalty": "presence_penalty",
+    "reasoning_effort": "reasoning_effort",
+    "extra_headers": "extra_headers",
+    "extra_body": "extra_body",
+}
+
+_warned_params: set = set()
+_warned_schema = False
+
+
+def _content(block: dict[str, Any]):
+    if block.get("type") == "image_url":
+        return ContentImage(image=block["image_url"]["url"])
+    return ContentText(text=block["text"])
+
+
+def to_chat_message(message: dict[str, Any]) -> ChatMessage:
+    """Convert one OpenAI-style message, as ModelAuditor builds them, to Inspect."""
+    content = message["content"]
+    if isinstance(content, list):
+        content = [_content(block) for block in content]
+    roles = {"system": ChatMessageSystem, "user": ChatMessageUser, "assistant": ChatMessageAssistant}
+    return roles[message["role"]](content=content)
+
+
+def generate_config(response_format: dict[str, Any] | None, params: dict[str, Any]) -> GenerateConfig:
+    fields: dict[str, Any] = {}
+    for key, value in params.items():
+        if key not in _PARAMS:
+            if key not in _warned_params:
+                _warned_params.add(key)
+                warnings.warn(
+                    f"Generation param {key!r} has no Inspect equivalent and is ignored.",
+                    stacklevel=3,
+                )
+            continue
+        if key == "stop" and isinstance(value, str):
+            value = [value]
+        fields[_PARAMS[key]] = value
+    if response_format and response_format.get("type") == "json_schema":
+        response_schema = _response_schema(response_format["json_schema"])
+        if response_schema is not None:
+            fields["response_schema"] = response_schema
+    return GenerateConfig(**fields)
+
+
+def _response_schema(spec: dict[str, Any]) -> ResponseSchema | None:
+    """The judge's JSON schema for Inspect, or None when Inspect cannot carry it whole.
+
+    Inspect's schema model drops keywords it doesn't define ($defs, $ref,
+    const, ...). Sending such a schema half-converted would constrain the
+    reply wrongly, so it is not sent; SimpleAudit parses the reply as JSON
+    either way.
+    """
+    global _warned_schema
+    converted = JSONSchema.model_validate(spec["schema"])
+    if converted.model_dump(exclude_none=True, by_alias=True) != spec["schema"]:
+        if not _warned_schema:
+            _warned_schema = True
+            warnings.warn(
+                "The judge's response schema uses keywords Inspect cannot express; "
+                "it is not sent, and the judge's reply is parsed as JSON without it.",
+                stacklevel=4,
+            )
+        return None
+    return ResponseSchema(name=spec.get("name", "response"), json_schema=converted)
+
+
+class InspectModelClient:
+    """Stands in for an AnyLLM client and calls an Inspect ``Model`` instead.
+
+    ``ModelAuditor`` talks to its target, auditor and judge through
+    ``acompletion(model=..., messages=[...], response_format=..., **params)``.
+    Routing that through Inspect puts every call in the Inspect transcript,
+    under the eval's limits, retries and usage accounting.
+
+    SimpleAudit turns a failing call into an ERROR result. Inside Inspect that
+    would hide token limits and sample terminations, so the client keeps the
+    first exception in ``error`` and the solver re-raises it.
+    """
+
+    def __init__(self, model: Model) -> None:
+        self.model = model
+        self.error: BaseException | None = None
+
+    async def acompletion(
+        self,
+        *,
+        messages: list[dict[str, Any]],
+        model: str | None = None,
+        stream: bool = False,
+        response_format: dict[str, Any] | None = None,
+        **params: Any,
+    ) -> Any:
+        try:
+            output = await self.model.generate(
+                [to_chat_message(m) for m in messages],
+                config=generate_config(response_format, params),
+            )
+        except Exception as exc:
+            if self.error is None:
+                self.error = exc
+            raise
+        usage = output.usage
+        return SimpleNamespace(
+            choices=[SimpleNamespace(message=SimpleNamespace(content=output.completion))],
+            usage=SimpleNamespace(
+                prompt_tokens=usage.input_tokens if usage else 0,
+                completion_tokens=usage.output_tokens if usage else 0,
+            ),
+        )