Repository navigation
Simulated human input for click/fill without focus stealing - #134
Merged
Merged
Conversation
…Version chrome.debugger cannot answer Browser.getVersion, so the extension backend returned a fixed "extension-relay" identity and Playwright treated every browser as Linux. ControlOrMeta then resolved to Control on macOS, where Control+A moves the caret instead of selecting. The service worker now sends its own navigator.userAgent in the relay hello (never read from a page); Browser.getVersion returns it with the product version parsed from it. Extensions that don't send the field keep the legacy identity. Co-Authored-By: Claude <noreply@anthropic.com>
Two extension-facade pieces the executor's human input relies on: - Consecutive session-scoped Input.dispatchMouseEvent(mouseMoved) frames are forwarded without waiting for each ACK, so Chrome can coalesce the samples like a real pointer device. The run is drained before any other frame is handled, so presses, keys and stateful handshakes stay sequential. - Browserwright.setPointerGrid, a facade-only method, leases dispatch-time rounding of a tab's mouse coordinates to the device-pixel grid and returns the last dispatched point on release. The lease is owned by the CDP session that took it and dropped on its detach, tab eviction or bridge close. Co-Authored-By: Claude <noreply@anthropic.com>
…us stealing The resident executor installs repl/human_input.py on its live surface next to the screenshot defaults. Locator/Page/Frame click() approaches a random interior point through densely sampled, coalescable pointer moves and presses with a randomized duration; fill() focuses through that click, replaces existing text with a native ControlOrMeta+A chord and Backspace, and types with randomized key timing and physical Shift chords. Targets outside their clip are revealed with native wheel input, planned by a selector engine in Playwright's utility world. Native Playwright still owns actionability, navigation waiting and error semantics; specialized inputs, selects and non-editable targets keep native fill. Ported from the extension-backend-battle exercise with every tab/window activation removed: no page.bring_to_front(), no foreground navigation. All steps run on a background tab (the extension keeps attached tabs rendered). Opt out per call with human=False or globally with BW_HUMAN_INPUT=0. See docs/human-input.md. Co-Authored-By: Claude <noreply@anthropic.com>
…window Moves the user to another active tab and a newly focused window, then runs executor click, fill and scroll-into-view on the background session tab. Asserts trusted input took effect and that Chrome's active tabs, focused window and tabs.onActivated / windows.onFocusChanged events are untouched. Verified to fail when click() calls page.bring_to_front(). Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #133: review and merge that one first.
This PR adds simulated human input to executor
click()/fill(). It never activates a tab and never focuses a window: the extension backend drives the user's own Chrome, so focus stealing interrupts them.repl/human_input.py, installed in_Worker._bind_live_surface. Locator, Page and Frameclick/fill, plusMousewrappers:fill()replaces existing text with a physical Ctrl/Cmd+A then Backspace;Browser.getVersion. Without this, Playwright assumes Linux on a Mac and presses Ctrl+A, which does not select all on macOS.bring_to_front()calls and itsPage.bringToFront/Target.activateTarget→chrome.tabs.update({active})/chrome.windows.update({focused})handling are not ported. The battle branch is the red-vs-blue detection exercise these changes come from.Defaults, opt-outs and cost
On by default on both backends. Pixel-grid snapping is extension-only.
Opt out per call with
human=False, or globally withBW_HUMAN_INPUT=0in the daemon environment.Latency is significant. Measured on a headful background tab:
click()fill()Native Playwright takes under 1 s for each. Details are in the "Cost" section of
docs/human-input.md.An agent that explicitly calls
page.bring_to_front()still gets Chrome's native behaviour. This is unchanged from main and documented.Test plan
mise run lintmise run test: skill 338; daemon 982 passed, 102 skipped; evals 13/13; pi.Real-Chrome E2E, 28 files: 63 passed. These are the new
test_human_input.py,test_human_scroll.py,test_human_pointer_layout.pyandtest_human_input_no_focus.py, plus the Keep the extension backend quiet: isolated framework state, no Runtime/Debugger, no title marker #133 suites.test_human_input_no_focus.pyruns click, fill and a wheel-scroll click on a background session tab, while another tab is active and another window has focus. It asserts:tabs.onActivated,tabs.onHighlightedorwindows.onFocusChangedfired.As a sanity check, temporarily adding one
bring_to_front()makes this test fail.The diff adds no
bring_to_front,bringToFront,activateTarget,windows.updateortabs.update(call outside the docs.🤖 Generated with Claude Code