Skip to content

Log per-action tool calls in CuaAgent templates - #241

Merged
dprevoznik merged 6 commits into
mainfrom
hypeship/cua-template-action-logging
Sep 8, 2026
Merged

Log per-action tool calls in CuaAgent templates#241
dprevoznik merged 6 commits into
mainfrom
hypeship/cua-template-action-logging

Conversation

@dprevoznik

@dprevoznik dprevoznik commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

The anthropic-computer-use, openai-computer-use, and gemini-computer-use templates build on CuaAgent but never subscribed to its events, so they only printed the final answer — no visibility into the steps the agent took.

This adds a shared logAgentEvent helper per template (logging.ts / lib/logging.ts, matching each template's file layout) wired up with agent.subscribe(logAgentEvent). It logs:

  • the model's narration between steps (agent> ...) from message_end
  • each concrete browser action (→ click (85, 267), → drag (780,385) → (1100,385), → goto https://...) from tool_execution_start
  • failed actions (✗ click failed) so retries are visible

Anthropic batches actions under computer_batch and OpenAI wraps navigation in computer_use_extra; both are unwrapped so every provider's logs read the same. Byte-count/base64 noise from the raw tool results is not logged.

This brings the CuaAgent templates in line with the readable per-step logs the hand-rolled yutori/tzafon templates and the Python openai-computer-use template already produce.

Test plan

Deployed and invoked all three end-to-end against live Kernel browsers (multi-step task: navigate, filter, drag cards). Sample output:

agent> I need to filter by Alice Johnson. Let me click the Filters button.
  → click (638)
  ✗ click failed
  → click (638, 125)
agent> The filter dropdown appeared. Clicking Alice Johnson.
  → click (706, 271)
agent> Now dragging her in-progress tasks to Done.
  → drag (780, 385) → (1100, 385)
  • bun install && bunx tsc --noEmit passes in all three template dirs
  • anthropic (computer_batch), openai (computer_use_extra), and gemini (top-level tools) all verified live

Note

Low Risk
Template-only console logging with no changes to agent behavior, APIs, or security-sensitive paths.

Overview
The anthropic, gemini, and openai TypeScript computer-use templates now subscribe to CuaAgent events so runs emit step-by-step console output instead of only the final answer.

Each template adds a logAgentEvent helper (under logging.ts, or lib/logging.ts for OpenAI) and wires it with agent.subscribe(logAgentEvent). Logs include assistant narration (agent>), each browser/tool action on start (), and failures on tool_execution_end. describe unwraps Anthropic computer_batch and OpenAI computer_use_extra so actions read consistently across providers, with long argument values truncated.

Reviewed by Cursor Bugbot for commit 1184541. Bugbot is set up for automated code reviews on this repo. Configure here.

dprevoznik and others added 5 commits September 1, 2026 14:44
The anthropic/openai/gemini computer-use templates never subscribed to
CuaAgent's tool_execution_start/end events, so they only printed the
final answer. Wire up agent.subscribe() with a small logger so each
click/type/screenshot action streams to stdout, matching the yutori
and tzafon templates.
AgentToolResult.content carries the image bytes sent back to the model,
so logging the whole result dumped truncated base64 on every screenshot
turn. Log result.details — the structured, human-readable summary — to
match the browser-loop reference logger.
The first pass logged raw tool_execution_start/end pairs with screenshot
byte counts — hard to follow. Subscribe to message_end too and print the
model's narration (`agent>`) between steps, then each action as a concise
line (`→ click (85, 267)`, `→ drag (780,385) → (1100,385)`), unwrapping
Anthropic's computer_batch and OpenAI's computer_use_extra so every
provider reads the same. Failed actions are marked so retries are visible.
Replace the per-tool formatters (point/dragPath/quote/keypress/etc.) with a
generic key=value renderer, keeping only the two unwraps that materially help
readability: Anthropic's computer_batch and OpenAI's computer_use_extra.
Collapse narration whitespace so each agent> line stays on one line. ~99 → 45
lines, same clean output across all three providers.
@dprevoznik
dprevoznik requested a review from rgarcia September 3, 2026 19:08
@dprevoznik
dprevoznik merged commit c0816aa into main Sep 8, 2026
7 checks passed
@dprevoznik
dprevoznik deleted the hypeship/cua-template-action-logging branch September 8, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants