chore(evals): update Python dependencies and align locked E2B runtime

docs/dependency-maintenance.md

@@ -41,7 +41,7 @@ Runtime pins use keyring **2.1.2**, Chalk **6.0.1**, diff **9.0.0**, highlight.j
 
 Keyring's missing-password result is now `null`. The storage adapter still reports a missing session as `undefined`. Socket adapters normalize the new declaration's string-or-buffer input, and syntax highlighting uses highlight.js's public entry point. The extension dependency example pins Chalk 6.0.1.
 
-The standalone sandbox example lock now resolves shell-quote **1.12.0**, addressing both reported shell-quote advisories. Sandbox runtime itself remains at 0.0.26 while the latest 0.0.81's unpatched node-forge signature-verification advisory is reviewed; forcing a Zod major or accepting a new vulnerable dependency is not automatic.
+The standalone sandbox example lock now resolves shell-quote **1.12.0**, addressing both reported shell-quote advisories. Sandbox runtime itself is intentionally retained at **0.0.26**: version 0.0.81 introduces node-forge 1.4.0 with the unpatched [GHSA-86w9-cpqp-85rv](https://github.com/advisories/GHSA-86w9-cpqp-85rv) signature-verification advisory. Revisit this exception when a patched upstream dependency is available; do not force an incompatible Zod major override to suppress the remaining audit findings.
 
 ## Networking review
 
@@ -55,6 +55,12 @@ Google automatic continuation is explicitly disabled for Gemini and Vertex reque
 
 Anthropic adopts the SDK's API version **2026-10-21** and no longer sends the removed sampling parameters. A model-context-window limit is reported as `length`. The Google GenAI install-script allowlist was reviewed and updated: its published `preinstall` is still a no-op, and installs continue to skip scripts.
 
+## Python evaluation dependencies
+
+The shared Terminal-Bench/DeepSWE project pins Harbor **0.24.0**, Daytona **0.226.1**, E2B **2.59.0**, Pydantic **2.14.1**, PyYAML **6.0.3**, and build backend Hatchling **1.32.4**. Runtime guards and live setup guides match the checked-in lock; DeepSWE no longer resolves separate `uv run --with` overlays.
+
+The refreshed Python lock patches the reported certifi, aiohttp, OpenTelemetry API, PyJWT, multidict, and urllib3 advisories. Stable direct dependencies still require beta-only OpenTelemetry instrumentation transitives; `prerelease = "if-necessary"` records that resolver policy. Dependency validation uses published wheels without build hooks and mocked tests, not paid sandbox/model benchmark runs. Linux Docker qualification and live service qualification remain separate release checks.
+
 ## Security review
 
 The initial audit reported seven affected packages, all present in the baseline lockfile. Updating Vitest/coverage to 4.1.12 and refreshing compatible Vite and source-map-js transitives removes five findings. The remaining sandbox-runtime/Zod dependency chain requires further review; do not force a Zod major-version override across consumers without checking their APIs.

evals/deepswe-full-steps-itemized.md

@@ -1,21 +1,21 @@
 # DeepSWE 1.1: complete 113-task evaluation
 
-This workflow schedules all 113 tasks four times, for 452 independent trials in one Harbor job.
+This workflow schedules all 113 tasks once, for 113 independent trials in one Harbor job.
 Concurrency is 8, using E2B.
-Four attempts per task do not impose a synchronization barrier between whole-suite passes.
+This preserves the launcher's one-attempt configuration; it does not reproduce four leaderboard passes.
 
 | Setting | Value |
 | --- | --- |
 | Dataset | [datacurve/deep-swe-1-1](https://hub.harborframework.com/datasets/datacurve/deep-swe-1-1) |
 | Revision | `sha256:5affcd534fd90ac85d202d4c63f8b35ddc942140afdd5d60014a21365440a2f5` |
-| Coverage | 113 distinct tasks, 4 attempts each, 452 trials |
+| Coverage | 113 distinct tasks, 1 attempt each, 113 trials |
 | Diffex | Published 0.3.2 |
 | Model | `openai/gpt-6-luna` |
 | Thinking | `max` |
 | Subagents | Disabled |
 | Self evolution | Disabled by the adapter's fresh per-run settings |
 | Environment | `deepswe_e2b:ResourceCheckedE2B` |
-| Dependencies | Harbor 0.23.0, E2B 2.51.0 |
+| Dependencies | Harbor 0.24.0, E2B 2.59.0, shared Terminal-Bench lock |
 | Concurrent trials | 8 |
 | Automatic Harbor trial retries | 0 |
 
@@ -25,7 +25,7 @@ Diffex 0.3.2 retains its published request retry behavior; the proposed 5, 15, 4
 
 ## Preparation
 
-1. Synchronize the existing evaluation project:
+1. Synchronize the shared locked evaluation project; no `--with` dependency overlays are needed:
 
    ```zsh
    cd /Users/jialeyu/Projects/diffex-new/evals/terminal-bench
@@ -38,7 +38,7 @@ Diffex 0.3.2 retains its published request retry behavior; the proposed 5, 15, 4
    zsh /Users/jialeyu/Projects/diffex-new/evals/deepswe/run-full.zsh --prepare
    ```
 
-   Preparation saves `job.yaml`, `manifest.json`, and `task-cache/` in `.artifacts/deepswe/prepared-113x4-c8/`.
+   Preparation saves `job.yaml`, `manifest.json`, and `task-cache/` in `.artifacts/deepswe/prepared-113x1-c8-runtime-v3/`.
    It verifies the release pins, all 113 task identities, separate verification, agent-only model network access, and offline verifiers.
    Every task is included explicitly; no random subset or seed is used.
    The manifest records each task's original resource requirements and timeouts.
@@ -50,15 +50,15 @@ Diffex 0.3.2 retains its published request retry behavior; the proposed 5, 15, 4
    zsh /Users/jialeyu/Projects/diffex-new/evals/deepswe/run-full.zsh --print-config
    ```
 
-   Confirm four attempts, concurrency eight, the pinned dataset, all 113 task names, Luna at max effort, and disabled subagents.
+   Confirm one attempt, concurrency eight, the pinned dataset, all 113 task names, Luna at max effort, and disabled subagents.
    Evolution is disabled inside the adapter, not by a Harbor YAML setting.
    The upstream artifact overlap warning about `/logs/artifacts/model.patch` is expected; inspect retained patches after execution.
 
 ## Execution and evidence
 
 The launcher revalidates preparation and prompts silently for missing `E2B_API_KEY` and `OPENAI_API_KEY` values.
-It creates a fresh `evals/terminal-bench/results/deepswe-1-1-luna-max-113x4-<timestamp>-<uuid>/` directory.
-The native Harbor job lives in its `full-113x4/` subdirectory, whose absolute path is printed at launch.
+It creates a fresh `evals/terminal-bench/results/deepswe-1-1-luna-max-113x1-<timestamp>-<uuid>/` directory.
+The native Harbor job lives in its `full-113x1/` subdirectory, whose absolute path is printed at launch.
 The parent directory retains the manifest, configuration, launcher, backend, adapter snapshot, Git revision, and adapter diff.
 
 The adapter installs Diffex 0.3.2, uses its evaluation system prompt, disables automatic context files, and starts with fresh state for every attempt.
@@ -75,7 +75,7 @@ Keep the terminal and computer running until verification and cleanup finish.
 ## After completion
 
 1. Set `DS_JOB` to the exact absolute job path printed by the launcher and inspect `result.json`.
-2. Require 452 terminal trial results covering 113 distinct tasks exactly four times each.
+2. Require 113 terminal trial results covering 113 distinct tasks exactly once each.
    Inspect errors separately from failed solutions and retain all attempts.
 3. Inspect verifier rewards, `verifier/test-stdout.txt`, collected `model.patch`, `agent/trajectory.json`, and native `agent/run-*/sessions/`.
    Check requested and effective settings for the model, max thinking, disabled subagents and evolution, delivered instructions, and confirmed shutdown.
@@ -89,14 +89,14 @@ Keep the terminal and computer running until verification and cleanup finish.
      [[ ${#DS_TRACES} -gt 0 ]]
      for DS_TRACE in "${DS_TRACES[@]}"; do
        uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked \
-         --with 'harbor[e2b]==0.23.0' --with 'e2b==2.51.0' \
          python -m harbor.utils.trajectory_validator "$DS_TRACE"
      done
    )
    ```
 
    Separately check missing trajectories and incomplete token accounting.
-   Official pass@1 is successful scored attempts divided by scored attempts; pass@4 is tasks with at least one successful attempt divided by tasks attempted.
+   Official pass@1 is successful scored attempts divided by scored attempts; pass@4 is tasks with at least one successful attempt divided by tasks attempted across four attempts.
+   This one-attempt run cannot measure pass@4.
    Official scoring excludes provider, verifier, and network errors but counts agent timeouts and context-window failures as failures.
    Report coverage and exclusions alongside scores; Harbor's raw mean may use a different denominator.
 
@@ -105,20 +105,19 @@ Keep the terminal and computer running until verification and cleanup finish.
    ```zsh
    cd /Users/jialeyu/Projects/diffex-new/evals/terminal-bench
    uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked \
-     --with 'harbor[e2b]==0.23.0' --with 'e2b==2.51.0' \
      harbor upload "$DS_JOB" --org diffex --private
    ```
 
    If authentication is required, run `uv run --locked harbor auth login` first.
-   Verify ownership, private visibility, all 452 trials, and available traces at the printed Hub link.
+   Verify ownership, private visibility, all 113 trials, and available traces at the printed Hub link.
    Retry a partial upload using the same job directory.
 
 If interrupted, preserve the entire output directory, inspect job and trial logs, and check E2B for remaining sandboxes after Harbor cleanup.
 Each launcher invocation starts a fresh job; it does not resume an interrupted job or replace failed attempts.
 
 ## Final command
 
-Run this command to start all 452 trials:
+Run this command to start all 113 trials:
 
 ```zsh
 zsh /Users/jialeyu/Projects/diffex-new/evals/deepswe/run-full.zsh

evals/deepswe-steps-itemized.md

@@ -15,7 +15,7 @@ The final command starts the evaluation.
 | Subagents | Disabled |
 | Self evolution | Disabled by the adapter's fresh `evolution.enabled=false` settings |
 | Environment | E2B, through `deepswe_e2b:ResourceCheckedE2B` |
-| Harness dependencies | Harbor `0.23.0`, E2B `2.51.0` |
+| Harness dependencies | Harbor `0.24.0`, E2B `2.59.0`, shared Terminal-Bench lock |
 | Sample | Five tasks selected without replacement using a saved random seed |
 | Attempts per task | 2 |
 | Concurrent trials | 5 |
@@ -34,24 +34,24 @@ See the official [leaderboard data](https://deepswe.datacurve.ai/artifacts/v1.1/
     uv --cache-dir /tmp/diffex-e2b-uv-cache sync --locked
     ```
 
-2. Install the pinned E2B extras, select five random tasks, and download and validate their definitions:
+2. Using the shared locked environment without dependency overlays, select five random tasks and download and validate their definitions:
 
     ```zsh
     zsh /Users/jialeyu/Projects/diffex-new/evals/deepswe/run-sample.zsh --prepare
     ```
 
-    Preparation saves `seed.json`, `manifest.json`, `job.yaml`, and a task cache in `.artifacts/deepswe/prepared-5x2/` under the repository root.
+    Preparation saves `seed.json`, `manifest.json`, `job.yaml`, and a task cache in `.artifacts/deepswe/prepared-5x2-runtime-v3/` under the repository root.
     It confirms Diffex 0.3.2, the 113-task dataset digest, five selected task identities, two attempts, concurrency five, and separate verification.
     Repeating preparation retains the existing seed and selected tasks.
-    Harbor 0.23.0's `n_tasks` takes the first entries, so this launcher explicitly samples a sorted task list with the recorded seed and writes `task_names` into the prepared configuration.
+    Harbor's `n_tasks` takes the first entries, so this launcher explicitly samples a sorted task list with the recorded seed and writes `task_names` into the prepared configuration.
     Harbor warns that the original explicit `model.patch` artifact overlaps its standard `/logs/artifacts` collection.
     The upstream task definitions are preserved; check that the patch is retained after execution.
 
 3. Inspect the resolved configuration and selection:
 
     ```zsh
     zsh /Users/jialeyu/Projects/diffex-new/evals/deepswe/run-sample.zsh --print-config
-    python3 -m json.tool /Users/jialeyu/Projects/diffex-new/.artifacts/deepswe/prepared-5x2/manifest.json
+    python3 -m json.tool /Users/jialeyu/Projects/diffex-new/.artifacts/deepswe/prepared-5x2-runtime-v3/manifest.json
     ```
 
     Confirm `n_attempts: 2`, `n_concurrent_trials: 5`, the five task names, the pinned dataset, `thinking: max`, and disabled subagents.
@@ -127,7 +127,6 @@ Harbor collects local logs and deletes sandboxes through its normal lifecycle.
       [[ ${#DS_TRACES} -gt 0 ]]
       for DS_TRACE in "${DS_TRACES[@]}"; do
         uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked \
-          --with 'harbor[e2b]==0.23.0' --with 'e2b==2.51.0' \
           python -m harbor.utils.trajectory_validator "$DS_TRACE"
       done
     )
@@ -146,10 +145,8 @@ Native Harbor jobs can be uploaded without the Terminal-Bench staged export comm
 ```zsh
 cd /Users/jialeyu/Projects/diffex-new/evals/terminal-bench
 uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked \
-  --with 'harbor[e2b]==0.23.0' --with 'e2b==2.51.0' \
   harbor auth login
 uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked \
-  --with 'harbor[e2b]==0.23.0' --with 'e2b==2.51.0' \
   harbor upload "$DS_JOB" --org diffex --private
 ```
 

evals/deepswe/deepswe_e2b.py

@@ -169,7 +169,7 @@ async def _dispatch_command(
         self, command: str, *, cwd: str | None, env: dict[str, str] | None,
         timeout_sec: int | None, user: str,
     ):
-        # E2B 2.51 executes /bin/bash -l -c: login profiles can replace the PATH
+        # E2B executes /bin/bash -l -c: login profiles can replace the PATH
         # supplied through envs. Restore the fully merged PATH inside that shell
         # so startup probes, agent tools, and verifier commands use task binaries.
         # Keep credentials in the env parameter, never in shell command text.
@@ -245,7 +245,7 @@ async def start(self, force_build: bool) -> None:
             raise RuntimeError(f"E2B capacity differs from the task requirements: {resources}")
 
     async def stop(self, delete: bool) -> None:
-        # Harbor 0.23 downloads verifier logs only after verify() returns.
+        # Recover verifier logs before teardown, including when verify() fails.
         # Retain a separate recovery snapshot on timeout/transport failure too,
         # without turning an incomplete reward file into a completed grade.
         try:

evals/deepswe/prepare-full.py

@@ -27,8 +27,8 @@ async def main() -> None:
     output = args.output.resolve()
     output.mkdir(parents=True, exist_ok=True)
     config = JobConfig.model_validate(yaml.safe_load((source / "full.yaml").read_text()))
-    if (PINS["diffex_version"], version("harbor"), version("e2b")) != ("0.3.2", "0.23.0", "2.51.0"):
-        raise RuntimeError("Expected Diffex 0.3.2, Harbor 0.23.0, E2B 2.51.0.")
+    if (PINS["diffex_version"], version("harbor"), version("e2b")) != ("0.3.2", "0.24.0", "2.59.0"):
+        raise RuntimeError("Expected Diffex 0.3.2, Harbor 0.24.0, E2B 2.59.0.")
     if (config.n_attempts, config.n_concurrent_trials, config.retry.max_retries) != (1, 8, 0):
         raise RuntimeError("Expected one attempt, eight concurrent trials, and zero retries.")
     if len(config.agents) != 1 or len(config.datasets) != 1 or config.tasks:

evals/deepswe/prepare-sample.py

@@ -40,8 +40,8 @@ async def main() -> None:
             json.dump({"seed": seed}, stream)
             stream.write("\n")
     config = JobConfig.model_validate(yaml.safe_load((source / "sample.yaml").read_text()))
-    if (PINS["diffex_version"], version("harbor"), version("e2b")) != ("0.3.2", "0.23.0", "2.51.0"):
-        raise RuntimeError("Expected Diffex 0.3.2, Harbor 0.23.0, E2B 2.51.0.")
+    if (PINS["diffex_version"], version("harbor"), version("e2b")) != ("0.3.2", "0.24.0", "2.59.0"):
+        raise RuntimeError("Expected Diffex 0.3.2, Harbor 0.24.0, E2B 2.59.0.")
     if (config.n_attempts, config.n_concurrent_trials, config.retry.max_retries) != (2, 5, 0):
         raise RuntimeError("Expected two attempts, five concurrent trials, and zero retries.")
     if len(config.agents) != 1 or len(config.datasets) != 1 or config.tasks:

evals/deepswe/prepare.py

@@ -25,8 +25,8 @@ async def main() -> None:
     args = parser.parse_args()
     source = Path(__file__).resolve().parent
     config = JobConfig.model_validate(yaml.safe_load((source / "smoke.yaml").read_text()))
-    if (PINS["diffex_version"], version("harbor"), version("e2b")) != ("0.3.2", "0.23.0", "2.51.0"):
-        raise RuntimeError("Expected Diffex 0.3.2, Harbor 0.23.0, and E2B 2.51.0.")
+    if (PINS["diffex_version"], version("harbor"), version("e2b")) != ("0.3.2", "0.24.0", "2.59.0"):
+        raise RuntimeError("Expected Diffex 0.3.2, Harbor 0.24.0, and E2B 2.59.0.")
     if (config.n_attempts, config.n_concurrent_trials, config.retry.max_retries) != (1, 1, 0):
         raise RuntimeError("The smoke trial requires one attempt, concurrency one, and zero retries.")
     agent = config.agents[0]

evals/deepswe/run-full.zsh

@@ -6,7 +6,7 @@ DS_REPO=${DS_SETUP:h:h}
 DS_PREPARED=${DS_PREPARED:-$DS_REPO/.artifacts/deepswe/prepared-113x1-c8-runtime-v3}
 cd "$DS_REPO/evals/terminal-bench"
 export PYTHONPATH="$DS_SETUP${PYTHONPATH:+:$PYTHONPATH}"
-DS_UV=(uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked --with 'harbor[e2b]==0.23.0' --with 'e2b==2.51.0')
+DS_UV=(uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked)
 [[ $# -le 1 ]] || { print -u2 'Too many arguments.'; exit 1; }
 case "${1:-}" in
   --prepare)

evals/deepswe/run-sample.zsh

@@ -6,7 +6,7 @@ DS_REPO=${DS_SETUP:h:h}
 DS_PREPARED=${DS_PREPARED:-$DS_REPO/.artifacts/deepswe/prepared-5x2-runtime-v3}
 cd "$DS_REPO/evals/terminal-bench"
 export PYTHONPATH="$DS_SETUP${PYTHONPATH:+:$PYTHONPATH}"
-DS_UV=(uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked --with 'harbor[e2b]==0.23.0' --with 'e2b==2.51.0')
+DS_UV=(uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked)
 [[ $# -le 1 ]] || { print -u2 'Too many arguments.'; exit 1; }
 case "${1:-}" in
   --prepare)

evals/deepswe/run-smoke.zsh

@@ -5,7 +5,7 @@ DS_SETUP=${0:A:h}
 DS_REPO=${DS_SETUP:h:h}
 cd "$DS_REPO/evals/terminal-bench"
 export PYTHONPATH="$DS_SETUP${PYTHONPATH:+:$PYTHONPATH}"
-DS_UV=(uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked --with 'harbor[e2b]==0.23.0' --with 'e2b==2.51.0')
+DS_UV=(uv --cache-dir /tmp/diffex-e2b-uv-cache run --locked)
 [[ $# -le 1 ]] || { print -u2 'Too many arguments.'; exit 1; }
 case "${1:-}" in
   --print-config)