Skip to content

Simulated human input for click/fill without focus stealing - #134

Merged
broven merged 4 commits into
mainfrom
feat/extension-human-input
Oct 8, 2026
Merged

broven merged 4 commits into
mainfrom
feat/extension-human-input

Conversation

@broven

@broven broven commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Summary

Stacked on #133: review and merge that one first.

This PR adds simulated human input to executor click() / fill(). It never activates a tab and never focuses a window: the extension backend drives the user's own Chrome, so focus stealing interrupts them.

  • repl/human_input.py, installed in _Worker._bind_live_surface. Locator, Page and Frame click/fill, plus Mouse wrappers:
    • pointer approach that samples many points on the way to a random point inside the target;
    • randomized press and key timing, and Shift chords;
    • fill() replaces existing text with a physical Ctrl/Cmd+A then Backspace;
    • targets outside the viewport are scrolled into view with native wheel input.
  • Facade. Consecutive mouse moves are forwarded together and finished before any press, so Chrome delivers real coalesced pointer events. Generated clicks are snapped to the device-pixel grid when they are dispatched.
  • Extension. It reports Chrome's real user agent through Browser.getVersion. Without this, Playwright assumes Linux on a Mac and presses Ctrl+A, which does not select all on macOS.
  • No focus stealing. The battle branch's bring_to_front() calls and its Page.bringToFront / Target.activateTarget → chrome.tabs.update({active}) / chrome.windows.update({focused}) handling are not ported. The battle branch is the red-vs-blue detection exercise these changes come from.

Defaults, opt-outs and cost

  • On by default on both backends. Pixel-grid snapping is extension-only.

  • Opt out per call with human=False, or globally with BW_HUMAN_INPUT=0 in the daemon environment.

  • Latency is significant. Measured on a headful background tab:

    Action Time
    click() ~4 s
    9-character fill() ~3 s
    click that needs a wheel scroll ~5.4 s

    Native Playwright takes under 1 s for each. Details are in the "Cost" section of docs/human-input.md.

  • An agent that explicitly calls page.bring_to_front() still gets Chrome's native behaviour. This is unchanged from main and documented.

Test plan

  • mise run lint

  • mise run test: skill 338; daemon 982 passed, 102 skipped; evals 13/13; pi.

  • Real-Chrome E2E, 28 files: 63 passed. These are the new test_human_input.py, test_human_scroll.py, test_human_pointer_layout.py and test_human_input_no_focus.py, plus the Keep the extension backend quiet: isolated framework state, no Runtime/Debugger, no title marker #133 suites.

  • test_human_input_no_focus.py runs click, fill and a wheel-scroll click on a background session tab, while another tab is active and another window has focus. It asserts:

    • every event on the page is trusted;
    • the active tab in every window and the focused window are unchanged;
    • no tabs.onActivated, tabs.onHighlighted or windows.onFocusChanged fired.

    As a sanity check, temporarily adding one bring_to_front() makes this test fail.

  • The diff adds no bring_to_front, bringToFront, activateTarget, windows.update or tabs.update( call outside the docs.

🤖 Generated with Claude Code

broven and others added 4 commits October 7, 2026 22:34
…Version

chrome.debugger cannot answer Browser.getVersion, so the extension backend
returned a fixed "extension-relay" identity and Playwright treated every
browser as Linux. ControlOrMeta then resolved to Control on macOS, where
Control+A moves the caret instead of selecting.

The service worker now sends its own navigator.userAgent in the relay hello
(never read from a page); Browser.getVersion returns it with the product
version parsed from it. Extensions that don't send the field keep the legacy
identity.

Co-Authored-By: Claude <noreply@anthropic.com>
Two extension-facade pieces the executor's human input relies on:

- Consecutive session-scoped Input.dispatchMouseEvent(mouseMoved) frames are
  forwarded without waiting for each ACK, so Chrome can coalesce the samples
  like a real pointer device. The run is drained before any other frame is
  handled, so presses, keys and stateful handshakes stay sequential.
- Browserwright.setPointerGrid, a facade-only method, leases dispatch-time
  rounding of a tab's mouse coordinates to the device-pixel grid and returns
  the last dispatched point on release. The lease is owned by the CDP session
  that took it and dropped on its detach, tab eviction or bridge close.

Co-Authored-By: Claude <noreply@anthropic.com>
…us stealing

The resident executor installs repl/human_input.py on its live surface next
to the screenshot defaults. Locator/Page/Frame click() approaches a random
interior point through densely sampled, coalescable pointer moves and
presses with a randomized duration; fill() focuses through that click,
replaces existing text with a native ControlOrMeta+A chord and Backspace,
and types with randomized key timing and physical Shift chords. Targets
outside their clip are revealed with native wheel input, planned by a
selector engine in Playwright's utility world. Native Playwright still owns
actionability, navigation waiting and error semantics; specialized inputs,
selects and non-editable targets keep native fill.

Ported from the extension-backend-battle exercise with every tab/window
activation removed: no page.bring_to_front(), no foreground navigation.
All steps run on a background tab (the extension keeps attached tabs
rendered).

Opt out per call with human=False or globally with BW_HUMAN_INPUT=0.
See docs/human-input.md.

Co-Authored-By: Claude <noreply@anthropic.com>
…window

Moves the user to another active tab and a newly focused window, then runs
executor click, fill and scroll-into-view on the background session tab.
Asserts trusted input took effect and that Chrome's active tabs, focused
window and tabs.onActivated / windows.onFocusChanged events are untouched.
Verified to fail when click() calls page.bring_to_front().

Co-Authored-By: Claude <noreply@anthropic.com>
@broven
broven changed the base branch from feat/extension-quiet-attach to main October 8, 2026 08:03
@broven
broven merged commit b172bee into main Oct 8, 2026
1 check passed
@broven
broven deleted the feat/extension-human-input branch October 8, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant