feat(browser): batch planned page actions and keep snapshot ids after refused clicks by waleedlatif1 · Pull Request #8214 · simstudioai/sim

waleedlatif1 · 2026-09-23T23:52:13Z

Summary

Keep snapshot element ids valid when an observed action is refused before dispatch (e.g. covered by an overlay), so the agent can click the overlay's own close button without re-snapshotting. A failed observed action now only cancels an in-flight capture, which clears refs itself when it starts
Add browser_batch: the desktop runs 2–8 single-page interactions from one authorized call, in order, stopping at the first failure or after an action navigates, switches tabs, or invalidates snapshot ids. A batch stopped by a failure reports an error status; once an action has run, a cancelled or timed-out batch reports an unknown outcome so it isn't repeated
Companion mothership PR adds the catalog tool, stale-observation pruning, and the benchmark harness changes

Companion: simstudioai/mothership#515

Type of Change

New feature
Bug fix

Testing

Paired A/B on the production browser model (gemini-3.8-flash), staging vs this branch plus the mothership side:

WebGames: 68 → 76 of 104 (exact McNemar p=0.02)
REAL web clones: 66 → 64 of 112 (p=0.82); on shared successes −13% rounds, −25% input tokens, −19% wall time
Capability suite: 55 → 54 of 56 (p=1); the two misses are a pre-existing download-stall flake seen on every arm
Refused-overlay clicks followed by a wasted "ids not valid" round: 11 of 20 → 0 of 20

Desktop 1,878 tests, Sim browser client 1,160 tests, type-check, lint, and all 49 audits pass. New tests were checked to fail without their fix.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…used An observed click refused before dispatch (for example, covered by an overlay) never starts its observation, yet it invalidated every id in the snapshot, so the agent could not click the overlay's own close button without re-observing. A failed observed action with no dispatched outcome now only cancels an in-flight capture, which clears the refs itself when it starts.

The desktop runs 2-8 single-page interactions from one authorized call, in order, stopping at the first failed action or after an action navigates, switches tabs, or invalidates the snapshot's element ids. A batch stopped by a failure reports an error status; one stopped by a page change does not. Once an action has run, a cancelled or timed-out batch reports an unknown outcome so the agent does not repeat it.

vercel · 2026-09-23T23:52:19Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 24, 2026 12:05am UTC

github-actions · 2026-09-23T23:54:35Z

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

❌ simstudioai/mothership#515 — OPEN, not merged (targets staging) — feat(browser): prune stale page observations and add browser_batch

cubic-dev-ai

4 issues found across 10 files

Confidence score: 2/5

apps/desktop/src/main/browser-agent/post-action-observation.ts and apps/desktop/src/main/browser-agent/driver.ts can report a batch as an ordinary error or cancellation after its first nested action has already dispatched, which may cause the model to repeat input or other side effects; mark the first action pending as soon as dispatch begins and preserve the partial-outcome state.
apps/desktop/src/main/browser-agent/driver.ts does not detect same-document SPA navigation via pushState, so later batch actions can run against a changed page and produce incorrect or unsafe results; include URL/history-based navigation detection before continuing the batch.
packages/browser-protocol/src/index.ts exposes browser_batch to older supported shells whose IPC allowlists reject the tool, causing compatibility failures; gate the tool by shell support or raise MIN_DESKTOP_VERSION.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/desktop/src/main/browser-agent/post-action-observation.ts">

<violation number="1" location="apps/desktop/src/main/browser-agent/post-action-observation.ts:13">
P1: An observed batch can be reported as an ordinary error after its first action has dispatched but hangs, inviting the model to repeat it. Mark the first nested action pending once dispatch begins so cancellation reports an unknown outcome.</violation>
</file>

<file name="packages/browser-protocol/src/index.ts">

<violation number="1" location="packages/browser-protocol/src/index.ts:46">
P2: Adding `browser_batch` exposes it to older supported shells, whose unchanged IPC allowlist rejects it as an unknown tool. Gate the tool by shell support or raise `MIN_DESKTOP_VERSION`.</violation>
</file>

<file name="apps/desktop/src/main/browser-agent/driver.ts">

<violation number="1" location="apps/desktop/src/main/browser-agent/driver.ts:3461">
P1: The first action can dispatch input before it returns, but the batch does not mark its outcome pending until action 1. Cancellation after that first dispatch therefore returns an ordinary cancellation instead of `outcomeUnknown`/`doNotRetry`, allowing the agent to repeat an action that may already have run; report pending as soon as the first action can dispatch.</violation>

<violation number="2" location="apps/desktop/src/main/browser-agent/driver.ts:3485">
P1: This check misses same-document SPA navigation: `navigationEpoch` changes only for cross-document loads, so a click that changes the URL with `pushState` can leave it unchanged and the batch will run later actions against stale page state. Also stop when the action reports a URL change or when the tab URL differs from its pre-action URL.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

greptile-apps · 2026-09-23T23:56:31Z

Confidence Score: 5/5

The PR appears safe to merge; the previously reported first-action replay risk is fixed and no new actionable failure remains.

Summary

Adds a desktop browser action-batching capability and preserves snapshot references after pre-dispatch action refusals.

Executes 2–8 authorized single-page actions sequentially and stops after failures, page changes, tab changes, or snapshot invalidation.
Marks an in-flight batch outcome unknown after cancellation or timeout so potentially dispatched actions are not replayed.
Detects same-document URL changes between actions.
Adds shared protocol registration, generated schemas, client completion reporting, UI labels, and focused tests.
The previous first-action replay issue is fixed; the older-shell compatibility finding was withdrawn and its thread resolved.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Authorize browser_batch] --> B[Validate 2–8 actions]
  B --> C[Mark current action pending]
  C --> D[Execute next browser action]
  D -->|Action fails| E[Return partial failure]
  D -->|Cancellation or timeout| F[Return unknown outcome and doNotRetry]
  D -->|Action succeeds| G{Page, tab, URL, or snapshot changed?}
  G -->|Yes| H[Return partial page-change result]
  G -->|No, actions remain| C
  G -->|No, batch complete| I[Return all action results]

_{Reviews (2) · Last reviewed commit: "fix(browser): mark batched actions pendi..."}

…ocument URL changes A batch now reports its outcome as pending before every action, since an action can dispatch input before it returns; a cancelled or timed-out batch therefore never reads as not started. It also stops after an action changes the tab URL within the document, not only on cross-document navigation.

waleedlatif1 · 2026-09-24T00:06:21Z

@greptile

waleedlatif1 · 2026-09-24T00:06:22Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-24T00:06:28Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 10 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 added 2 commits September 23, 2026 16:50

github-actions Bot added the requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep label Sep 23, 2026

cubic-dev-ai Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread apps/desktop/src/main/browser-agent/post-action-observation.ts

Comment thread apps/desktop/src/main/browser-agent/driver.ts Outdated

Comment thread apps/desktop/src/main/browser-agent/driver.ts

Comment thread packages/browser-protocol/src/index.ts

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread apps/desktop/src/main/browser-agent/driver.ts Outdated

Comment thread apps/sim/lib/mothership/generated/tool-catalog-v1.ts

vercel Bot temporarily deployed to Preview September 24, 2026 00:05 Inactive

cubic-dev-ai Bot reviewed Sep 24, 2026

View reviewed changes

waleedlatif1 merged commit 3c3770a into staging Sep 24, 2026
37 checks passed

waleedlatif1 deleted the feat/browser-agent-efficiency branch September 24, 2026 00:53

Navigation Menu

feat(browser): batch planned page actions and keep snapshot ids after refused clicks - #8214

feat(browser): batch planned page actions and keep snapshot ids after refused clicks#8214
waleedlatif1 merged 3 commits into
stagingfrom
feat/browser-agent-efficiency

waleedlatif1 commented Sep 23, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 24, 2026

Uh oh!

waleedlatif1 commented Sep 24, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 24, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

waleedlatif1 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 23, 2026

⚠️ Cross-repo companion check

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 24, 2026

Uh oh!

waleedlatif1 commented Sep 24, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 24, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Sep 23, 2026 •

edited

Loading

vercel Bot commented Sep 23, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 23, 2026 •

edited

Loading