Windows run_shell: prevent nested-shell quoting failures and silent wrong output #475

Problem

Windows shell quoting failures cause CraftBot to spend multiple agent turns changing escape characters and adding shell wrappers. Some attempts return exit code 0 while producing the wrong result.

Confirmed locally on Windows using the actual shell_exec_windows function from commit b50970c3fbf68908a0dde7da2e06bb65896d947e. No VM is required for this investigation or the proposed fix.

Implementation follow-up: Codex. This issue is the handoff ticket for a subsequent implementation task; this investigation does not change production behavior.

Evidence and scope

The supplied screenshots show disk-space queries repeatedly retried as powershell -Command ..., followed by cmd /c powershell -Command .... They truncate the full commands and do not show the shell argument or complete stderr, so the exact historic executions cannot be reconstructed. The reproductions below confirm the same failure classes in the current checkout.

Executed 38 read-only cases with real Windows processes:

• 14 direct-script cases through the current action passed: seven scripts on Windows PowerShell and seven on pwsh. Coverage: single/double quotes, variables, pipelines, multiline input, parentheses in property names, literal backslash-plus-quote, and quoted metacharacters.
• The same 14 scripts passed through a separate UTF-16LE Base64 -EncodedCommand control. Windows PowerShell emitted CLIXML progress on stderr in seven controls, so that transport would require deliberate output/error handling.
• 10 additional routing/nesting cases produced four confirmed mismatches: two execution failures and two silent wrong outputs.
• A raw PowerShell disk query with a calculated property named Free (GB) passed. Removing parentheses or simplifying the calculation is therefore not the underlying fix.
• A simple cmd-to-PowerShell command using backslash-escaped quotes also passed. Do not assume every occurrence of \\\" in a displayed command proves a launcher defect.

To isolate the actual function without installing the entire application, the harness AST-loads its unchanged body, removes the registration decorator, and stubs only Node PATH preparation and cancellation registration. All subprocess launches and output capture execute normally. This is a launcher-level reproduction, not an end-to-end LLM retry-loop test.

Minimal reproductions

These are tool payloads. JSON escaping is decoded once before execution.

1. Variables expanded by the outer PowerShell

{
  "shell": "powershell",
  "command": "powershell -NoProfile -Command \"$n=2; Write-Output $n\"",
  "timeout": 15
}

Expected: 2. Actual: exit 1; stderr includes The term '=2' is not recognized. The outer PowerShell expands $n before the inner PowerShell sees the script.

Working payload:

{
  "shell": "powershell",
  "command": "$n=2; Write-Output $n",
  "timeout": 15
}

2. Single-quoted -Command wrapper returns success without executing

{
  "shell": "cmd",
  "command": "powershell -NoProfile -Command 'Write-Output \"hello world\"'",
  "timeout": 15
}

Expected stdout: hello world. Actual stdout: Write-Output hello world, exit 0, empty stderr. PowerShell evaluates the passed expression as a string rather than performing the intended Write-Output invocation. A retry controller cannot detect this from exit code alone.

Working payload:

{
  "shell": "powershell",
  "command": "Write-Output 'hello world'",
  "timeout": 15
}

3. cmd silently expands a literal percent expression

{
  "shell": "cmd",
  "command": "powershell -NoProfile -Command \"Write-Output '%CRAFTBOT_QUOTE_REPRO%'\"",
  "env": {"CRAFTBOT_QUOTE_REPRO": "expanded-by-cmd"},
  "timeout": 15
}

Expected: literal %CRAFTBOT_QUOTE_REPRO%. Actual: expanded-by-cmd, exit 0. Passing the raw Write-Output script with shell="powershell" preserves the literal.

4. Screenshot-style disk query fails through nested PowerShell

{
  "shell": "powershell",
  "command": "powershell -NoProfile -Command \"Get-CimInstance Win32_LogicalDisk -Filter \\\"DriveType=3\\\" | Select-Object @{Name=\\\"Free (GB)\\\";Expression={[math]::Round($_.FreeSpace/1GB,2)}} | ConvertTo-Json -Compress",
  "timeout": 15
}

Actual: exit 1; stderr includes The term 'GB' is not recognized. The raw script with shell="powershell" succeeds:

Get-CimInstance Win32_LogicalDisk -Filter "DriveType=3" |
  Select-Object @{Name="Free (GB)";Expression={[math]::Round($_.FreeSpace/1GB,2)}} |
  ConvertTo-Json -Compress

Why this happens

Current code:

• Windows default/auto routing selects cmd.
• Explicit PowerShell routing already launches PowerShell directly with an argument list.
• The tool describes supported shells but supplies a cmd-oriented example; it does not clearly tell the model to send raw PowerShell source and omit powershell -Command/cmd /c wrappers.
• Windows diagnostic skips execution in its Linux diagnostic environment.

JSON string escaping, cmd parsing, PowerShell parsing, and nested PowerShell interpolation are separate layers. Guessing alternative quotes can add another layer or turn executable source into a literal string. Direct-script tests provide no evidence that the current PowerShell argument-list launcher universally mangles quotes.

How other agents do it

Primary sources inspected during this investigation:

• OpenCode shell implementation: its Windows candidate order starts with pwsh/PowerShell, then Git Bash and cmd. Its PowerShell argument builder passes ["-NoProfile", "-Command", command] directly. The relevant lesson is explicit shell selection and one script interpreter; it is not a guarantee that all nested/native quoting problems disappear.
• Claude Code Windows setup: documents native Windows operation, a PowerShell tool without Git for Windows, and Git Bash when available. WSL is another option, not required for shell execution.
• Microsoft PowerShell executable documentation: documents UTF-16LE Base64 EncodedCommand for complex quoting.

These references do not support a claim that most agents require a VM. VM/sandbox isolation and faithful command transport solve different problems.

Proposed implementation

1. Improve the Windows run_shell schema and agent guidance first: for PowerShell source, set shell="powershell" or "pwsh" and submit the raw script. Include the disk-space example and explicitly explain that the tool launches the interpreter.
2. Keep cmd available explicitly for cmd syntax and batch workflows. Do not silently change existing default semantics in a compatibility fix; consider a dedicated PowerShell tool or a separately reviewed default migration.
3. Add the resolved shell/executable to results and deterministic, shell-aware recovery guidance for recognized parser failures. Guide the model to remove unintended wrappers and choose one interpreter rather than retrying guessed escapes. Do not automatically execute a rewritten command after a failed command with possible partial side effects.
4. Consider EncodedCommand or a temporary .ps1 plus -File as transport hardening for raw PowerShell source. The runtime should construct this transport, while the agent continues to submit readable source. Encoding the already nested bad command does not fix its semantics. Account for command-line length, Unicode/output decoding, CLIXML stderr, script cleanup, and background-process lifetime.
5. Add real Windows regression coverage and semantic output assertions, since exit code 0 does not establish that a command did the intended work.

Acceptance criteria

• A raw disk-space query with the Free (GB) calculated property executes successfully in one call.
• Direct PowerShell and pwsh execution preserves quotes, variables, pipelines, multiline scripts, literal percent expressions, and parentheses.
• Tool guidance gives the working payloads above and surfaces the actual interpreter.
• Existing explicit cmd behavior remains usable.
• Windows regression checks assert stdout as well as exit status, and include the silent wrong-output reproductions.
• If transport changes, verify nonzero exit propagation, timeout/cancellation, foreground/background behavior, and stderr handling.