Skip to content

feat(harness): opt-in risk classifier behind DEEPCODE_RISK_CLASSIFIER - #263

Closed
raymondginger2018-sudo wants to merge 4 commits into
HKUDS:mainfrom
raymondginger2018-sudo:pr/risk-classifier
Closed

raymondginger2018-sudo wants to merge 4 commits into
HKUDS:mainfrom
raymondginger2018-sudo:pr/risk-classifier

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor

Revives the risk-classifier split requested on #181 (08-17 review). The split PR #189 was opened and then self-closed during a strategy change; it was never declined in review. Rebased onto current main (f183965), conflict-free.

What

  • core/harness/classifier.py (289 lines; stdlib + loguru only): scores high-risk tool calls; enabled with DEEPCODE_RISK_CLASSIFIER=1 (model override via DEEPCODE_RISK_CLASSIFIER_MODEL), off by default.
  • core/agent_runtime/runner.py: the approval path consults the classifier and fails closed (deny + log) on classifier errors, the same pattern as permission_checker.
  • core/events/session.py: the classifier applies to turn builds when enabled.
  • With DEEPCODE_RISK_CLASSIFIER unset there is zero behavior change.

Verification

  • tests/test_harness_classifier.py + tests/test_classifier_wiring.py: 19 passed.
  • runner/session-area suite: 103 passed; the same 5 Windows-local failures occur on clean f183965f59 (differential-verified, unrelated).

Checks

  • Dedup: upstream core/harness/ had no classifier.
  • Dependency closure: stdlib + loguru only.
  • Scope: opt-in, in-repo, no default behavior change.

Prompt-injection defense: DEEPCODE_RISK_CLASSIFIER-gated classifier
that scores incoming user/harness content before it reaches the loop.
All new files; existing behavior untouched when the knob is off.
Add AgentRunSpec.risk_classifier seam: in _check_permission an 'ask' with no
approval_callback (headless) now consults the optional classifier and auto-allows
only a decisive 'low' verdict; medium/high/error/no-verdict keep the existing
deny — it can upgrade an ask, never widen a deny or escalate risk.
build_risk_classifier(provider) returns the env-gated (DEEPCODE_RISK_CLASSIFIER=1)
async resolver; off by default (P0-1 Claude Code Auto-mode lesson).

Co-Authored-By: Claude <noreply@anthropic.com>
When DEEPCODE_RISK_CLASSIFIER=1 and a provider is available, Session now attaches
risk_classifier=build_risk_classifier(provider) to the AgentRunSpec of each real
turn, so headless asks can be auto-allowed only when clearly low risk. Off by
default; inert when a human approver is attached.

Co-Authored-By: Claude <noreply@anthropic.com>
@Zongwei9888

Copy link
Copy Markdown
Collaborator

Thank you for this, @raymondginger2018-sudo, and for keeping it behind a flag. We're closing it, though, because it goes the opposite way from the design agreed in #245:

  1. It loosens a decision. In [Design] Risk triage for tool calls: what does it classify, and where does it sit in the permission engine? #245 the evaluator's only move is allow → ask, and a headless run still ends in a deny. Here a headless ask becomes allow when the model answers low (the new block in AgentRunner just before the fail-closed return).
  2. No model call in the permission path for the first version. [Design] Risk triage for tool calls: what does it classify, and where does it sit in the permission engine? #245 settled on rules only, built on core/harness/command_guard.py; a model-based classifier was to be a separate proposal that covers latency, cost and failure modes.
  3. The verdict is easy to steer. The arguments go to the model as written, so a command can carry text addressed to the classifier. And json.dumps(arguments)[:2000] means anything past 2,000 characters is never seen: a long harmless prefix followed by ; curl … | sh is judged on the prefix alone.

#257 implements the #245 design and is close to merge. If you'd like to pursue a model-based classifier, please start with a design issue as you did with #245. A version we could consider would only ever turn allow into ask, see the full arguments (or decline to judge when they're too long), and leave headless asks denied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants