diff --git a/.agents/skills/add-connector/SKILL.md b/.agents/skills/add-connector/SKILL.md index 6d7ef0b2930..4633b0e5f38 100644 --- a/.agents/skills/add-connector/SKILL.md +++ b/.agents/skills/add-connector/SKILL.md @@ -6,6 +6,12 @@ argument-hint: [api-docs-url] # Add Connector Skill +## Choose the connector runtime first + +For **Sim Search**, use the live provider workflow in [the federated Search developer guide](../../../apps/sim/lib/sim-search/live/README.md#adding-a-live-search-connector). Its browser-safe provider catalog owns provider IDs, API origins, credential aliases, and account modes; its typed runtime registry requires both search and read handlers. `ConnectorMeta` remains the owner of logos and setup fields. Member mode has no admin resource filters. Service mode requires independent live source verification and shared selectors. Do not implement a Search source by adding a crawler, embeddings, or a scheduled ACL build. + +The ingestion instructions below apply to **ordinary knowledge-base connectors** and the explicit legacy Search backend (`SIM_SEARCH_LIVE=false`). If a provider supports both, implement and test both runtimes; adding `search: true` to metadata alone does not implement federated search. Preserve indexing documentation and behavior for those KB/legacy callers. + You are an expert at adding knowledge base connectors to Sim. A connector syncs documents from an external source (Confluence, Google Drive, Notion, etc.) into a knowledge base. ## Your Task diff --git a/.agents/skills/validate-connector/SKILL.md b/.agents/skills/validate-connector/SKILL.md index 9e25db59753..a65759287a8 100644 --- a/.agents/skills/validate-connector/SKILL.md +++ b/.agents/skills/validate-connector/SKILL.md @@ -6,6 +6,12 @@ argument-hint: [api-docs-url] # Validate Connector Skill +## Identify the runtime under review + +For **Sim Search**, validate the [live provider registration and access pipeline](../../../apps/sim/lib/sim-search/live/README.md#adding-a-live-search-connector): catalog and metadata parity, both search/read handlers, current member grants, service-source restrictions, safe scoped references, pagination, provenance, and provider failure behavior. Test real localhost setup/search/read with authorized fixtures when available, and distinguish those results from mocked provider tests. Live Search must not enqueue content indexing or background ACL/directory builds; GitLab still computes request-time permissions or uses current CSV grants. + +The ingestion-specific checks below apply to ordinary workspace KB connectors and legacy Search selected with `SIM_SEARCH_LIVE=false`. Keep those checks for providers supporting both runtimes; do not require a live-only provider to implement content hashes, ingestion cursors, embeddings, or stored ACL snapshots. + You are an expert auditor for Sim knowledge base connectors. Your job is to thoroughly validate that an existing connector is correct, complete, and follows all conventions. ## Your Task diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 71708399892..9e2d86607e9 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -256,6 +256,8 @@ If you prefer not to use Docker. **All commands run from the repository root unl For ad-hoc schema iteration during development you can also use `bun run db:push` from `packages/db`, but `db:migrate` is the canonical command for staging and production. `db:push` reconciles directly to the current schema without running versioned migration guards. For disposable local/dev databases, `bun run db:push --force` accepts Drizzle's data-loss prompts, including column drops. + `db:push` treats added and removed columns, tables, and other schema objects as separate creations and deletions. It never infers a rename. For an intentional rename during local development, run `bun run db:push --interactive-renames` in a terminal and select the old object in Drizzle's chooser. This flag does not approve data loss; `--force` controls that separately. After schema reconciliation succeeds, the wrapper reconciles credential policies and OAuth providers, then backfills search vectors. A failure stops subsequent steps. Staging and production changes still use reviewed versioned migrations with expand/contract deployment steps. + 4. **Run the Development Servers:** ```bash diff --git a/.github/actions/docker-build/action.yml b/.github/actions/docker-build/action.yml index 72d90e8e8bb..7a880bc768c 100644 --- a/.github/actions/docker-build/action.yml +++ b/.github/actions/docker-build/action.yml @@ -19,6 +19,10 @@ inputs: tags: description: Comma-separated list of tags to push. required: true + build-args: + description: Newline-separated Docker build arguments. + required: false + default: '' max-cache-size-mb: description: >- Layer cache to retain after this action prunes, in MB. Must stay above one @@ -72,6 +76,7 @@ runs: platforms: ${{ inputs.platforms }} push: true tags: ${{ inputs.tags }} + build-args: ${{ inputs.build-args }} provenance: false sbom: false @@ -177,5 +182,6 @@ runs: platforms: ${{ inputs.platforms }} push: true tags: ${{ inputs.tags }} + build-args: ${{ inputs.build-args }} provenance: false sbom: false diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 92b44041af3..50ceca03939 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -103,7 +103,7 @@ jobs: echo "ℹ️ No comparable base commit; skipping desktop prerelease" exit 0 fi - if git diff --name-only "$BEFORE" HEAD | grep -qE '^(apps/desktop/|packages/desktop-bridge/|packages/browser-protocol/)'; then + if git diff --name-only "$BEFORE" HEAD | grep -qE '^(apps/desktop/|packages/desktop-bridge/|packages/browser-protocol/|\.github/workflows/(ci|desktop-release)\.yml$)'; then echo "changed=true" >> "$GITHUB_OUTPUT" echo "✅ Desktop shell code changed" else @@ -221,15 +221,18 @@ jobs: file: ${{ matrix.dockerfile }} platforms: linux/amd64 tags: ${{ steps.login-ecr.outputs.registry }}/${{ steps.ecr-repo.outputs.name }}:${{ github.sha }}-dev + build-args: | + SIM_SEARCH_LIVE_DEFAULT=true + MSHIP_PLAN_MODE_DEFAULT=true max-cache-size-mb: ${{ matrix.cache_mb }} - # Build and upload tasks alongside tests and images. The unpromoted version - # cannot serve new runs; promote-images waits for it and successful migrations. + # Staging/production coordinate task releases with app traffic cutover. + # Dev tasks deploy independently in deploy-trigger-dev.yml. prepare-trigger: name: Prepare Trigger.dev if: >- github.event_name == 'push' && - (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging' || github.ref == 'refs/heads/dev') + (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} timeout-minutes: 30 outputs: @@ -265,7 +268,6 @@ jobs: case "$GITHUB_REF" in refs/heads/main) TRIGGER_ENV=prod; TRIGGER_BRANCH='' ;; refs/heads/staging) TRIGGER_ENV=staging; TRIGGER_BRANCH='' ;; - refs/heads/dev) TRIGGER_ENV=preview; TRIGGER_BRANCH=dev-sim ;; *) echo "ERROR: unsupported Trigger release ref: $GITHUB_REF" >&2; exit 1 ;; esac echo "environment=$TRIGGER_ENV" >> "$GITHUB_OUTPUT" @@ -426,8 +428,9 @@ jobs: tags: ${{ steps.meta.outputs.tags }} max-cache-size-mb: ${{ matrix.cache_mb }} - # Promote the sha-tagged ECR images once tests, migrations, and the Trigger - # upload pass. Pushing the ECR latest/staging tag is what triggers + # Promote the sha-tagged ECR images once their build and migrations pass. + # Staging/production also require the Trigger upload; dev never waits for it. + # Pushing the ECR latest/staging/dev tag is what triggers # CodePipeline, so this seconds-long manifest retag is the deploy gate — # the image builds themselves run in parallel with the tests. A single job # (not a matrix) so all four sha manifests are verified before any tag @@ -438,9 +441,9 @@ jobs: # Explicit results: see migrate's comment. if: >- !cancelled() && github.event_name == 'push' && - needs.prepare-trigger.result == 'success' && ( ((github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') && + needs.prepare-trigger.result == 'success' && needs.migrate.result == 'success' && needs.build-amd64.result == 'success') || (github.ref == 'refs/heads/dev' && @@ -546,7 +549,7 @@ jobs: fi done - # Promote the parked Trigger.dev version after observing the ECS + # Staging/production: promote the parked Trigger.dev version after observing the ECS # traffic cutover (CodeDeploy AllowTraffic on every target). The image retag # triggers the ECS pipeline; this job correlates it via the digest + retag epoch # (rejecting a stale execution reusing the digest) and promotes at cutover. @@ -560,14 +563,13 @@ jobs: if: >- !cancelled() && github.event_name == 'push' && - (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging' || github.ref == 'refs/heads/dev') && + (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') && needs.promote-images.result == 'success' && needs.prepare-trigger.result == 'success' && needs.promote-images.outputs.promoted == 'true' runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} - # Leave setup/promotion headroom above the cutover poll (dev: 20 min; - # staging/prod: 70 min, including a deploy queued behind a long bake). - timeout-minutes: ${{ github.ref == 'refs/heads/dev' && 40 || 90 }} + # Leave setup/promotion headroom above the 70-minute cutover poll. + timeout-minutes: 90 permissions: contents: read id-token: write @@ -599,14 +601,13 @@ jobs: with: role-to-assume: ${{ github.ref == 'refs/heads/main' && secrets.AWS_ROLE_TO_ASSUME || github.ref == 'refs/heads/dev' && secrets.DEV_AWS_ROLE_TO_ASSUME || secrets.STAGING_AWS_ROLE_TO_ASSUME }} aws-region: ${{ github.ref == 'refs/heads/main' && secrets.AWS_REGION || github.ref == 'refs/heads/dev' && secrets.DEV_AWS_REGION || secrets.STAGING_AWS_REGION }} - # Match each environment's session budget; both outlast their polls. - role-duration-seconds: ${{ github.ref == 'refs/heads/dev' && 2400 || 5400 }} + role-duration-seconds: 5400 # An unchanged tag may belong to a failed or still-running earlier deploy. # Verify its latest cutover rather than treating tag equality as success. - name: Wait for ECS traffic cutover env: - OVERALL_TIMEOUT: ${{ github.ref == 'refs/heads/dev' && 1200 || 4200 }} + OVERALL_TIMEOUT: 4200 APP_IMAGE_CHANGED: ${{ needs.promote-images.outputs.app_image_changed }} DIGEST: ${{ needs.promote-images.outputs.app_image_digest }} PIPELINE: sim-${{ github.ref == 'refs/heads/main' && 'production' || github.ref == 'refs/heads/dev' && 'dev' || 'staging' }}-us-east-1-app-deployment @@ -1311,24 +1312,30 @@ jobs: name: Prune Desktop Prereleases runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }} timeout-minutes: 5 - needs: [publish-desktop-prerelease] + needs: [create-desktop-prerelease, publish-desktop-prerelease] permissions: contents: read env: GH_TOKEN: ${{ secrets.DESKTOP_RELEASE_TOKEN }} GH_REPO: simstudioai/sim-desktop-releases + CURRENT_TAG: ${{ needs.create-desktop-prerelease.outputs.version }} steps: - name: Delete stale prereleases run: | + set -euo pipefail + : "${CURRENT_TAG:?Current publication tag is required before pruning}" if [ -z "$GH_TOKEN" ]; then echo "::error::DESKTOP_RELEASE_TOKEN is required to prune desktop prereleases." exit 1 fi if [ "$GITHUB_REF" = "refs/heads/dev" ]; then CHANNELS='(dev|alpha)'; else CHANNELS='(staging|beta)'; fi - gh release list --limit 100 --json tagName,isPrerelease,isDraft,createdAt \ - --jq "[.[] | select(.isPrerelease and (.isDraft | not) and (.tagName | test(\"-${CHANNELS}\\\\.\")))] | sort_by(.createdAt) | reverse | .[5:] | .[].tagName" | + # createdAt follows the tag's commit: published releases in the release-only + # repository can all share it. Retain by publication time, with deterministic ties. + gh release list --limit 100 --json tagName,isPrerelease,isDraft,publishedAt \ + --jq "[.[] | select(.isPrerelease and (.isDraft | not) and (.tagName | test(\"-${CHANNELS}\\\\.\")))] | sort_by(.publishedAt, .tagName) | reverse | .[5:] | .[].tagName" | while read -r TAG; do [ -n "$TAG" ] || continue + [ "$TAG" != "$CURRENT_TAG" ] || continue echo "Deleting stale prerelease $TAG" gh release delete "$TAG" --cleanup-tag --yes done diff --git a/.github/workflows/deploy-trigger-dev.yml b/.github/workflows/deploy-trigger-dev.yml new file mode 100644 index 00000000000..c9738f51a78 --- /dev/null +++ b/.github/workflows/deploy-trigger-dev.yml @@ -0,0 +1,75 @@ +name: Deploy Dev Tasks + +# Independent of app CI: task packaging and deployment must never delay dev images. +on: + push: + branches: [dev] + +permissions: + contents: read + +# Serialize promotions without cancelling an external deployment in flight. +# Pending pushes coalesce to the newest run while the current run finishes. +concurrency: + group: deploy-trigger-dev + cancel-in-progress: false + +jobs: + deploy: + name: Deploy Trigger.dev preview + runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }} + timeout-minutes: 30 + steps: + - name: Checkout code + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + + - name: Setup Bun + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 + with: + bun-version: 1.4.1 + + - name: Cache Bun dependencies + uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5 + with: + path: | + ~/.bun/install/cache + node_modules + **/node_modules + key: ${{ runner.os }}-bun-${{ hashFiles('**/bun.lock') }} + restore-keys: | + ${{ runner.os }}-bun- + + - name: Install dependencies + run: bun install --frozen-lockfile --ignore-scripts + + - name: Upload preview version + id: deploy + working-directory: ./apps/sim + env: + TRIGGER_ACCESS_TOKEN: ${{ secrets.TRIGGER_ACCESS_TOKEN }} + TRIGGER_PROJECT_ID: ${{ secrets.TRIGGER_PROJECT_ID }} + run: | + set -euo pipefail + : "${TRIGGER_ACCESS_TOKEN:?TRIGGER_ACCESS_TOKEN must be configured}" + : "${TRIGGER_PROJECT_ID:?TRIGGER_PROJECT_ID must be configured}" + bunx trigger.dev@4.5.16 deploy --env preview --branch dev-sim --skip-promotion + + - name: Promote current dev preview + working-directory: ./apps/sim + env: + GH_TOKEN: ${{ github.token }} + TRIGGER_ACCESS_TOKEN: ${{ secrets.TRIGGER_ACCESS_TOKEN }} + TRIGGER_PROJECT_ID: ${{ secrets.TRIGGER_PROJECT_ID }} + VERSION: ${{ steps.deploy.outputs.deploymentVersion }} + run: | + set -euo pipefail + if ! [[ "$VERSION" =~ ^[0-9]{8}\.[0-9]+$ ]]; then + echo "ERROR: Trigger.dev did not report a valid deploymentVersion output" >&2 + exit 1 + fi + CURRENT_SHA=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/dev" --jq '.object.sha') + if [ "$CURRENT_SHA" != "$GITHUB_SHA" ]; then + echo "::notice::Skipping superseded dev task version $VERSION" + exit 0 + fi + bunx trigger.dev@4.5.16 promote "$VERSION" --env preview --branch dev-sim diff --git a/.github/workflows/migrations.yml b/.github/workflows/migrations.yml index 5b06c42bc14..0afdddeff97 100644 --- a/.github/workflows/migrations.yml +++ b/.github/workflows/migrations.yml @@ -64,6 +64,8 @@ jobs: MIGRATION_DATABASE_URL: ${{ inputs.environment == 'production' && secrets.MIGRATION_DATABASE_URL || inputs.environment == 'staging' && secrets.STAGING_MIGRATION_DATABASE_URL || '' }} ENVIRONMENT: ${{ inputs.environment }} run: | + set -euo pipefail + if [ -z "$DATABASE_URL" ]; then echo "ERROR: no database URL secret resolved for environment '${ENVIRONMENT}'" >&2 exit 1 @@ -73,16 +75,7 @@ jobs: echo "Dev environment — pushing schema directly (db:push)" # Dev deliberately forces direct schema reconciliation; staging and # production use guarded versioned migrations in the other branch. - # drizzle-kit push needs a TTY to resolve ambiguous renames (--force only - # covers data-loss). In CI it throws "Interactive prompts require a TTY - # terminal" but still exits 0, so the job goes green without applying the - # change. tee keeps the output live in the log; we then fail on drizzle's - # own TTY error. A genuine non-zero exit already fails via `set -e`. - bun run db:push --force < /dev/null 2>&1 | tee /tmp/db-push.log - if grep -q "Interactive prompts require a TTY terminal" /tmp/db-push.log; then - echo "ERROR: db:push needs an interactive rename decision; land it as a versioned migration instead of relying on push." >&2 - exit 1 - fi + SIM_DEV_DB_PUSH=1 bun run db:push --force < /dev/null else echo "Applying versioned migrations (db:migrate)" bun run ./scripts/migrate.ts diff --git a/.github/workflows/test-build.yml b/.github/workflows/test-build.yml index 1688bcf267f..aafa67327d2 100644 --- a/.github/workflows/test-build.yml +++ b/.github/workflows/test-build.yml @@ -83,6 +83,12 @@ jobs: - name: Install dependencies run: bun install --frozen-lockfile --ignore-scripts + - name: Verify direct schema push compatibility + working-directory: packages/db + env: + DB_PUSH_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/postgres + run: bunx vitest run scripts/push.postgres.test.ts + - name: Provision a fresh database through the supported command working-directory: packages/db run: | @@ -233,14 +239,16 @@ jobs: if-no-files-found: ignore retention-days: 7 - - name: Verify durable provenance, concurrent memory writes, and attachment replay + - name: Verify durable provenance, concurrent memory writes, and browser download admission working-directory: apps/sim env: + BROWSER_FILE_TRANSFER_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim TABLE_PROVENANCE_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim MEMORY_PROVENANCE_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim AGENT_MEMORY_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim run: >- bunx vitest run + lib/mothership/async-runs/browser-download-claim.postgres.test.ts lib/table/rows/secret-provenance.postgres.test.ts lib/memory/message-provenance.postgres.test.ts lib/memory/conversation-store.postgres.test.ts @@ -256,6 +264,7 @@ jobs: script-migrations/0016_backfill_search_vectors.postgres.test.ts script-migrations/0018_repair_workspace_file_content_revision.postgres.test.ts script-migrations/0019_tin_keyword_projection.postgres.test.ts + member-sync-status-migration.postgres.test.ts - name: Verify Search progress, pagination, and outbox scheduling in PostgreSQL working-directory: apps/sim diff --git a/apps/desktop/README.md b/apps/desktop/README.md index 7420b414215..ebb50f9a412 100644 --- a/apps/desktop/README.md +++ b/apps/desktop/README.md @@ -156,6 +156,26 @@ const desktop = useDesktop() Good fits for the bridge: OS notifications + dock badge on workflow completion, global shortcuts, "reveal in Finder", tray, secure OS-keychain storage. Anything that touches the server/DB still goes through normal APIs — the bridge is only for **native** capability. This same bridge is also the robust way to retire the web-app couplings in the table above: have the web app *tell* the shell (`signalLogout()`, `markAuthSurface()`) instead of the shell inferring from URLs. +### Browser authentication and Sim previews + +Browser tabs for the exact configured Sim origin share the desktop app's existing +Electron session. This includes dev: authenticated file previews and deployed chat +pages use the current login without copying cookies or exposing tokens to the model. +Normal resource permissions and any separate deployed-chat password still apply. +External websites use the independent `persist:sim-browser-agent` partition. + +Crossing between Sim and an external site opens a tab in the destination session and +preserves the source tab's history; an unused blank tab adopts its first destination's +session. A cross-session form POST is blocked instead of replayed as a GET. Restored +tabs and popups select their session from the destination origin. Sign-out and account +or server changes use the existing browser teardown. Browser views retain their own +permission/download policy, SSRF guards, and minimal preload with no `simDesktop` API. + +Generated HTML remains inside its `allow-scripts` sandbox. The driver can inspect and +interact with inline frames through isolated Chromium worlds, including out-of-process +frames; it does not grant those pages access to the parent app. These shell changes +require a desktop update, not just a hosted web deployment. + ### Local filesystem access Copilot can inspect user-selected local directories through the ordinary VFS tools. Granted folders appear beneath the top-level `user-local/` namespace, and `glob`, `grep`, and `read` are routed to Electron only when their path/pattern is explicitly scoped there. This capability is: @@ -188,8 +208,8 @@ Raw local file bytes are never exposed through the preload bridge and cannot be ## Known caveats -- The hosted Sim renderer may request microphone access for voice input from the configured app origin; camera access remains denied. On macOS the shell also requires the operating-system microphone grant. Separately, a page in the isolated agent browser may request microphone or camera only from its main frame after a recent native user gesture; Sim then requires an explicit document-scoped prompt and the operating-system grant where applicable. -- The built-in agent browser is not a general-purpose download manager. Its dedicated partition applies the same bounded policy to every download, including one started by a direct user click: at most 2 GiB per file, two active downloads per task, six app-wide, and a 1 GiB free-disk reserve. A rejected download appears in the browser's downloads menu; use a normal browser for an intentionally larger transfer. +- The hosted Sim renderer may request microphone access for voice input from the configured app origin; camera access remains denied. On macOS the shell also requires the operating-system microphone grant. Separately, a page in the agent browser may request microphone or camera only from its main frame after a recent native user gesture; Sim then requires an explicit document-scoped prompt and the operating-system grant where applicable. +- The built-in agent browser is not a general-purpose download manager. Both browser session types apply the same bounded policy to every download, including one started by a direct user click: at most 2 GiB per file, two active downloads per task, six app-wide, and a 1 GiB free-disk reserve. A rejected download appears in the browser's downloads menu; use a normal browser for an intentionally larger transfer. - Default Electron ships H.264/AAC/MP3 — do not swap in the codec-free ffmpeg build. - Third-party web analytics (GTM/GA) are blocked at the network layer by default (`blockThirdPartyAnalytics`); first-party PostHog `/ingest` is untouched. - `Cmd+F` opens the native find overlay in built-in browser tabs. The hosted Sim workspace continues to use Monaco- and table-specific find surfaces. diff --git a/apps/desktop/e2e/browser-tools.spec.ts b/apps/desktop/e2e/browser-tools.spec.ts index 2f4eaf9b148..ab5f150f2e4 100644 --- a/apps/desktop/e2e/browser-tools.spec.ts +++ b/apps/desktop/e2e/browser-tools.spec.ts @@ -52,16 +52,59 @@ const CLICK_FIXTURE = `Click fixture }); ` +const CAPABILITIES_FIXTURE = `Capabilities fixture + +
Report row
+
Drop a receipt
+ +` + +const POPUP_FIXTURE = `Authorize fixture +` + +const UPLOAD_TARGET_FIXTURE = `Upload target fixture +
+` + test.describe('browser tools', () => { const calls = new Map< string, - { chatId: string; toolName: BrowserToolName; args: Record } + { chatId: string; toolName: BrowserToolName | 'terminal'; args: Record } >() + const claimed = new Map }>() let server: Server + let popupServer: Server let origin: string + let site: string + let popupOrigin: string let app: ElectronApplication let window: Page let callCount = 0 + let beforeUploadResponse: (() => Promise) | undefined test.beforeAll(async () => { server = createServer(async (request, response) => { @@ -71,27 +114,124 @@ test.describe('browser tools', () => { response.end() return } + if (path === '/enter-sim') { + response.writeHead(302, { Location: `${origin}/private-chat` }) + response.end() + return + } + if (path === '/api/auth/get-session') { + response.writeHead(200, { 'Content-Type': 'application/json' }) + response.end( + JSON.stringify( + request.headers.cookie?.includes('better-auth.session_token=fixture') + ? { user: { id: 'browser-auth-fixture' }, session: { id: 'fixture-session' } } + : null + ) + ) + return + } + if (path === '/api/auth/sign-out') { + response.writeHead(200, { + 'Content-Type': 'application/json', + 'Set-Cookie': 'better-auth.session_token=; HttpOnly; SameSite=Lax; Path=/; Max-Age=0', + }) + response.end('{}') + return + } + if (path === '/private-chat' || path === '/private-preview') { + if (!request.headers.cookie?.includes('better-auth.session_token=fixture')) { + response.writeHead(302, { Location: '/login' }) + response.end() + return + } + response.writeHead(200, { 'Content-Type': 'text/html' }) + response.end( + path === '/private-chat' + ? 'Private deployed chat

Authenticated deployed chat

' + : `Private HTML preview

Authenticated HTML preview

` + ) + return + } + if (path === '/api/desktop/tool/file') { + let body = '' + for await (const chunk of request) body += chunk.toString() + const { toolCallId, index } = JSON.parse(body) + const reference = claimed.get(toolCallId)?.args.paths + const found = Array.isArray(reference) && reference[index] === 'files/receipt.txt' + const beforeResponse = beforeUploadResponse + beforeUploadResponse = undefined + try { + await beforeResponse?.() + } catch (error) { + response.writeHead(500, { 'Content-Type': 'text/plain' }) + response.end(String(error)) + return + } + response.writeHead(found ? 200 : 404, { + 'Content-Type': found ? 'application/octet-stream' : 'application/json', + ...(found ? { 'Content-Disposition': 'attachment; filename="receipt.txt"' } : {}), + }) + response.end(found ? 'receipt-bytes' : JSON.stringify({ error: 'File not found' })) + return + } + if (path === '/upload-target' || path === '/upload-host') { + const params = new URL(request.url ?? '/', 'http://127.0.0.1').searchParams + const frameOrigin = params.get('kind') === 'oopif' ? origin : site + response.writeHead(200, { 'Content-Type': 'text/html' }) + response.end( + path === '/upload-target' + ? UPLOAD_TARGET_FIXTURE + : `Upload frame fixture` + ) + return + } + if (path === '/doc.pdf') { + response.writeHead(200, { 'Content-Type': 'application/pdf' }) + response.end( + '%PDF-1.4\n1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj\n2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 >> endobj\n3 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] >> endobj\ntrailer << /Root 1 0 R >>\n%%EOF\n' + ) + return + } if (path === '/api/desktop/tool/authorize') { let body = '' for await (const chunk of request) body += chunk.toString() const authorization = calls.get(JSON.parse(body).toolCallId) + calls.delete(JSON.parse(body).toolCallId) + if (authorization) claimed.set(JSON.parse(body).toolCallId, authorization) response.writeHead(authorization ? 200 : 403, { 'Content-Type': 'application/json' }) response.end(JSON.stringify(authorization ?? {})) return } - response.writeHead(200, { 'Content-Type': 'text/html' }) + response.writeHead(200, { + 'Content-Type': 'text/html', + ...(path === '/workspace' || path === '/home' || path === '/' + ? { 'Set-Cookie': 'better-auth.session_token=fixture; HttpOnly; SameSite=Lax; Path=/' } + : {}), + }) response.end( path === '/click' ? CLICK_FIXTURE - : path === '/form' - ? FORM - : 'Sim fixture

Browser tools fixture

' + : path === '/capabilities' + ? CAPABILITIES_FIXTURE + : path === '/form' + ? FORM + : 'Sim fixture

Browser tools fixture

' ) }) await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)) const address = server.address() if (!address || typeof address === 'string') throw new Error('Missing fixture address') origin = `http://127.0.0.1:${address.port}` + /** Pages outside the app origin browse in the agent partition, like any third-party site. */ + site = origin.replace('127.0.0.1', 'localhost') + popupServer = createServer((_request, response) => { + response.writeHead(200, { 'Content-Type': 'text/html' }) + response.end(POPUP_FIXTURE) + }) + await new Promise((resolve) => popupServer.listen(0, '127.0.0.1', resolve)) + const popupAddress = popupServer.address() + if (!popupAddress || typeof popupAddress === 'string') throw new Error('Missing popup address') + popupOrigin = `http://localhost:${popupAddress.port}` }) test.beforeEach(async () => { @@ -104,6 +244,11 @@ test.describe('browser tools', () => { SIM_DESKTOP_USER_DATA: mkdtempSync(join(tmpdir(), 'sim-browser-tools-e2e-')), }, }) + // A dialog listener stops Playwright auto-dismissing page dialogs, so the desktop's own CDP + // dialog handling decides their outcome exactly as it does in production. + const leaveDialogsToDesktop = (page: Page) => page.on('dialog', () => {}) + app.context().pages().forEach(leaveDialogsToDesktop) + app.context().on('page', leaveDialogsToDesktop) window = await app.firstWindow() await app.evaluate(({ app, BrowserWindow }) => { const host = BrowserWindow.getAllWindows()[0] @@ -130,14 +275,18 @@ test.describe('browser tools', () => { }) test.afterEach(async () => { + beforeUploadResponse = undefined await app?.close() calls.clear() + claimed.clear() }) test.afterAll(async () => { - await new Promise((resolve, reject) => - server.close((error) => (error ? reject(error) : resolve())) - ) + for (const listener of [server, popupServer]) { + await new Promise((resolve, reject) => + listener.close((error) => (error ? reject(error) : resolve())) + ) + } }) async function execute(tool: BrowserToolName, args: Record) { @@ -167,6 +316,433 @@ test.describe('browser tools', () => { } } + async function openCapabilities() { + const url = `${site}/capabilities?popup=${encodeURIComponent(`${popupOrigin}/authorize`)}` + const response = await execute('browser_open_url', { url }) + expect(response.ok, response.error).toBe(true) + const outline = (response.result as { snapshot: { outline: string } }).snapshot.outline + const ref = (name: string) => { + const match = outline + .split('\n') + .find((line) => line.includes(`"${name}"`) && /\[ref=\d+\]/.test(line)) + ?.match(/\[ref=(\d+)\]/) + if (!match) throw new Error(`No reference for ${name}: ${outline}`) + return Number(match[1]) + } + const dataset = () => + app.evaluate( + ({ webContents }, url) => + webContents + .getAllWebContents() + .find((contents) => contents.getURL() === url) + ?.executeJavaScript('({ ...document.body.dataset })'), + url + ) + return { ref, dataset } + } + + test('answers confirm dialogs only when the action asks and right-clicks reach the page', async () => { + const { ref, dataset } = await openCapabilities() + + const dismissed = await execute('browser_click', { elementId: ref('Delete report') }) + expect(JSON.stringify(dismissed.result)).toContain('which was dismissed') + expect(await dataset()).toMatchObject({ deleted: 'false' }) + + const accepted = await execute('browser_click', { + elementId: ref('Delete report'), + dialog: { accept: true }, + }) + expect(JSON.stringify(accepted.result)).toContain('accepted as requested') + expect(await dataset()).toMatchObject({ deleted: 'true' }) + + const menu = await execute('browser_click', { + elementId: ref('Report row'), + button: 'right', + modifiers: ['Shift'], + }) + expect(menu.ok, menu.error).toBe(true) + expect(await dataset()).toMatchObject({ menu: '2:true' }) + }) + + test('uploads a workspace file into the hidden input behind a drop zone', async () => { + const { ref, dataset } = await openCapabilities() + + const upload = await execute('browser_upload_file', { + elementId: ref('Attach receipt'), + paths: ['files/receipt.txt'], + }) + + expect(upload.ok, upload.error).toBe(true) + expect(upload.result).toMatchObject({ + uploaded: [{ name: 'receipt.txt', size: 13 }], + effectObserved: true, + }) + await expect.poll(dataset).toMatchObject({ upload: 'receipt.txt:receipt-bytes' }) + }) + + async function openUploadTarget( + kind: 'root' | 'same-origin' | 'oopif' | 'shadow', + steal = false + ) { + const framed = kind === 'same-origin' || kind === 'oopif' + const url = framed + ? `${site}/upload-host?kind=${kind}` + : `${site}/upload-target?shadow=${kind === 'shadow' ? '1' : '0'}&steal=${steal ? '1' : '0'}` + const response = await execute('browser_open_url', { url }) + expect(response.ok, response.error).toBe(true) + const evaluate = (expression: string, inTopFrame = false) => + app.evaluate( + ({ webContents }, { url, framed, expression }) => { + const contents = webContents + .getAllWebContents() + .find((contents) => contents.getURL() === url) + const frame = framed ? contents?.mainFrame.frames[0] : contents?.mainFrame + if (!frame) throw new Error('Missing upload fixture frame') + return frame.executeJavaScript(expression) + }, + { url, framed: framed && !inTopFrame, expression } + ) + await expect.poll(() => evaluate('typeof window.uploadTestState')).toBe('function') + if (kind === 'oopif') { + expect( + await app.evaluate(({ webContents }, url) => { + const contents = webContents + .getAllWebContents() + .find((contents) => contents.getURL() === url) + const child = contents?.mainFrame.frames[0] + return child && child.processId !== contents?.mainFrame.processId + }, url) + ).toBe(true) + } + const snapshot = await execute('browser_snapshot', {}) + expect(snapshot.ok, snapshot.error).toBe(true) + const outline = (snapshot.result as { outline: string }).outline + const match = outline + .split('\n') + .find((line) => line.includes('"Upload original"')) + ?.match(/\[ref=(\d+)\]/) + expect(match, outline).toBeTruthy() + return { elementId: Number(match?.[1]), evaluate } + } + + test('pins uploads to the original input when page code steals a DOM marker', async () => { + const { elementId, evaluate } = await openUploadTarget('root', true) + const upload = await execute('browser_upload_file', { elementId, paths: ['files/receipt.txt'] }) + + expect(upload.ok, upload.error).toBe(true) + expect(await evaluate('window.uploadTestState()')).toEqual({ + original: [{ name: 'receipt.txt', text: 'receipt-bytes' }], + decoy: [], + currentCount: 1, + }) + }) + + test('reports an unconfirmed upload when cancelled before Chromium acknowledgement arrives', async () => { + const { elementId, evaluate } = await openUploadTarget('root') + await app.evaluate(({ webContents }, site) => { + const contents = webContents + .getAllWebContents() + .find((contents) => contents.getURL() === `${site}/upload-target?shadow=0&steal=0`) + if (!contents) throw new Error('Missing upload acknowledgement fixture') + const original = contents.debugger.sendCommand + let release = () => {} + const acknowledgement = new Promise((resolve) => { + release = resolve + }) + const state = { + count: 0, + applied: false, + release, + restore: () => { + contents.debugger.sendCommand = original + }, + } + const globals = globalThis as typeof globalThis & { heldUploadAcknowledgement?: typeof state } + globals.heldUploadAcknowledgement = state + contents.debugger.sendCommand = async (method, params, sessionId) => { + if (method !== 'DOM.setFileInputFiles') { + return original.call(contents.debugger, method, params, sessionId) + } + state.count++ + const result = await original.call(contents.debugger, method, params, sessionId) + state.applied = true + await acknowledgement + return result + } + }, site) + const pending = execute('browser_upload_file', { elementId, paths: ['files/receipt.txt'] }) + const toolCallId = `browser-fixture-${callCount}` + try { + await expect + .poll(() => evaluate('window.uploadTestState()')) + .toEqual({ + original: [{ name: 'receipt.txt', text: 'receipt-bytes' }], + decoy: [], + currentCount: 1, + }) + await expect + .poll(() => + app.evaluate( + () => + ( + globalThis as typeof globalThis & { + heldUploadAcknowledgement?: { applied: boolean } + } + ).heldUploadAcknowledgement?.applied + ) + ) + .toBe(true) + await window.evaluate( + async ({ toolCallId, scope }) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + if (!api.browserAgent.cancelTool) throw new Error('Browser cancellation is unavailable') + await api.browserAgent.cancelTool(toolCallId, scope) + }, + { toolCallId, scope: SCOPE } + ) + + const upload = await pending + expect(upload.ok, JSON.stringify(upload)).toBe(true) + expect(upload.result).toMatchObject({ + outcomeUnknown: true, + doNotRetry: true, + note: expect.stringContaining('Inspect the page'), + }) + expect(upload.result).not.toHaveProperty('dispatched', true) + expect((await execute('browser_list_tabs', {})).ok).toBe(true) + expect( + await app.evaluate( + () => + ( + globalThis as typeof globalThis & { + heldUploadAcknowledgement?: { count: number } + } + ).heldUploadAcknowledgement?.count + ) + ).toBe(1) + } finally { + await app.evaluate(() => { + const globals = globalThis as typeof globalThis & { + heldUploadAcknowledgement?: { release: () => void; restore: () => void } + } + globals.heldUploadAcknowledgement?.release() + globals.heldUploadAcknowledgement?.restore() + globals.heldUploadAcknowledgement = undefined + }) + await pending + } + }) + + for (const mutation of ['replace', 'disable', 'navigate-frame'] as const) { + test(`refuses uploads when the target changes during staging: ${mutation}`, async () => { + const { elementId, evaluate } = await openUploadTarget( + mutation === 'navigate-frame' ? 'same-origin' : 'root' + ) + let mutated = false + beforeUploadResponse = async () => { + if (mutation === 'navigate-frame') { + await evaluate('parent.previousUploadInput = document.querySelector("#original")') + await evaluate('location.replace("/upload-target?after=1")') + await expect.poll(() => evaluate('location.search')).toBe('?after=1') + await expect.poll(() => evaluate('typeof window.uploadTestState')).toBe('function') + } else { + await evaluate(`window.mutateUploadTarget(${JSON.stringify(mutation)})`) + } + mutated = true + } + const upload = await execute('browser_upload_file', { + elementId, + paths: ['files/receipt.txt'], + }) + + expect(mutated).toBe(true) + expect(upload.ok, JSON.stringify(upload)).toBe(false) + expect(await evaluate('window.uploadTestState()')).toEqual({ + original: [], + decoy: [], + currentCount: 0, + }) + if (mutation === 'navigate-frame') { + expect(await evaluate('parent.previousUploadInput.files.length')).toBe(0) + } + }) + } + + test('refuses uploads after their same-origin frame is removed during staging', async () => { + const { elementId, evaluate } = await openUploadTarget('same-origin') + let removed = false + beforeUploadResponse = async () => { + await evaluate( + 'window.removedUploadInput = document.querySelector("iframe").contentDocument.querySelector("#original"); document.querySelector("iframe").remove()', + true + ) + removed = true + } + const upload = await execute('browser_upload_file', { + elementId, + paths: ['files/receipt.txt'], + }) + + expect(removed).toBe(true) + expect(upload.ok, JSON.stringify(upload)).toBe(false) + expect(await evaluate('window.removedUploadInput.files.length', true)).toBe(0) + }) + + for (const kind of ['same-origin', 'oopif', 'shadow'] as const) { + test(`uploads through a pinned input in a ${kind} context`, async () => { + const { elementId, evaluate } = await openUploadTarget(kind) + const upload = await execute('browser_upload_file', { + elementId, + paths: ['files/receipt.txt'], + }) + + expect(upload.ok, upload.error).toBe(true) + expect(upload.result).toMatchObject({ uploaded: [{ name: 'receipt.txt', size: 13 }] }) + expect(await evaluate('window.uploadTestState()')).toEqual({ + original: [{ name: 'receipt.txt', text: 'receipt-bytes' }], + decoy: [], + currentCount: 1, + }) + }) + } + + test('keeps window.opener for page popups and returns to the opener when they close', async () => { + const { ref, dataset } = await openCapabilities() + + expect((await execute('browser_click', { elementId: ref('Connect') })).ok).toBe(true) + const popup = await execute('browser_snapshot', {}) + const allow = (popup.result as { outline: string }).outline.match( + /button "Allow" \[ref=(\d+)\]/ + )?.[1] + expect(allow, JSON.stringify(popup.result)).toBeTruthy() + expect((await execute('browser_click', { elementId: Number(allow) })).ok).toBe(true) + + await expect.poll(dataset).toMatchObject({ connected: 'granted' }) + await expect + .poll( + async () => ((await execute('browser_list_tabs', {})).result as { tabs: unknown[] }).tabs + ) + .toHaveLength(1) + }) + + test('renders PDFs in the built-in viewer', async () => { + const response = await execute('browser_open_url', { url: `${site}/doc.pdf` }) + expect(response.ok, response.error).toBe(true) + + await expect + .poll(() => + app.evaluate( + ({ webContents }, url) => + webContents + .getAllWebContents() + .find((contents) => contents.getURL() === url) + ?.mainFrame.framesInSubtree.some((frame) => + frame.url.startsWith('chrome-extension://mhjfbmdgcfjbbpaeojofohoefgiehjai/') + ), + `${site}/doc.pdf` + ) + ) + .toBe(true) + }) + + test('shares desktop authentication for private HTML previews and deployed chats', async () => { + for (const path of ['/private-preview', '/private-chat']) { + const result = await execute('browser_open_url', { url: `${origin}${path}` }) + expect(result.ok, result.error).toBe(true) + expect(result.result).toMatchObject({ url: `${origin}${path}` }) + const state = await app.evaluate(async ({ webContents, BrowserWindow }, url) => { + const host = BrowserWindow.getAllWindows()[0].webContents + const page = webContents.getAllWebContents().find((contents) => contents.getURL() === url) + if (!page) throw new Error('Missing protected page') + return { + sharedSession: page.session === host.session, + hasDesktopBridge: await page.executeJavaScript( + 'typeof window.simDesktop !== "undefined"' + ), + heading: await page.executeJavaScript('document.querySelector("h1").textContent'), + escaped: await page.executeJavaScript('document.body.dataset.escaped === "true"'), + } + }, `${origin}${path}`) + expect(state.sharedSession).toBe(true) + expect(state.hasDesktopBridge).toBe(false) + expect(state.heading).toContain('Authenticated') + expect(state.escaped).toBe(false) + if (path === '/private-preview') { + const outline = (result.result as { snapshot: { outline: string } }).snapshot.outline + const button = outline + .split('\n') + .find((line) => line.includes('"Test quiz"') && /\[ref=\d+\]/.test(line)) + ?.match(/\[ref=(\d+)\]/)?.[1] + expect(button, JSON.stringify(result.result)).toBeTruthy() + const click = await execute('browser_click', { elementId: Number(button) }) + expect(click.ok, click.error).toBe(true) + const frameState = await app.evaluate(async ({ webContents }, url) => { + const page = webContents.getAllWebContents().find((contents) => contents.getURL() === url) + const frame = page?.mainFrame.frames.find((frame) => frame.url === 'about:srcdoc') + if (!frame) throw new Error('Missing sandboxed preview') + return frame.executeJavaScript( + '({ clicked: document.body.dataset.clicked, isolated: document.body.dataset.isolated, bridge: typeof window.simDesktop })' + ) + }, `${origin}${path}`) + expect(frameState).toEqual({ clicked: 'true', isolated: 'true', bridge: 'undefined' }) + } + } + const typed = await execute('browser_open_url', { url: `${origin}/private-chat` }) + expect(typed.ok, typed.error).toBe(true) + const result = typed.result as { snapshot: { outline: string } } + const message = result.snapshot.outline + .split('\n') + .find((line) => line.includes('"Message"') && /\[ref=\d+\]/.test(line)) + ?.match(/\[ref=(\d+)\]/)?.[1] + const send = result.snapshot.outline + .split('\n') + .find((line) => line.includes('"Send"') && /\[ref=\d+\]/.test(line)) + ?.match(/\[ref=(\d+)\]/)?.[1] + expect(message, result.snapshot.outline).toBeTruthy() + expect(send, result.snapshot.outline).toBeTruthy() + expect( + (await execute('browser_type', { elementId: Number(message), text: 'Session works' })).ok + ).toBe(true) + expect((await execute('browser_click', { elementId: Number(send) })).ok).toBe(true) + expect(JSON.stringify(await execute('browser_snapshot', {}))).toContain('Session works') + }) + + test('keeps external navigation isolated and authenticates redirects back into Sim', async () => { + expect((await execute('browser_open_url', { url: `${origin}/private-chat` })).ok).toBe(true) + const external = origin.replace('127.0.0.1', 'localhost') + const result = await execute('browser_open_url', { url: `${origin}/redirect` }) + expect(result.ok, result.error).toBe(true) + expect(result.result).toMatchObject({ url: `${external}/landing` }) + expect( + await app.evaluate(({ webContents, BrowserWindow }, url) => { + const page = webContents.getAllWebContents().find((contents) => contents.getURL() === url) + return page?.session === BrowserWindow.getAllWindows()[0].webContents.session + }, `${external}/landing`) + ).toBe(false) + const back = await execute('browser_open_url', { url: `${external}/enter-sim` }) + expect(back.ok, back.error).toBe(true) + expect(back.result).toMatchObject({ url: `${origin}/private-chat` }) + }) + + test('sign-out clears authenticated browser pages with the desktop session', async () => { + const opened = await execute('browser_open_url', { url: `${origin}/private-chat` }) + expect(opened.ok, opened.error).toBe(true) + expect(opened.result).toMatchObject({ url: `${origin}/private-chat` }) + await window.evaluate(async () => { + await fetch('/api/auth/sign-out', { method: 'POST' }) + }) + await expect + .poll(() => + app.evaluate( + ({ webContents }, url) => + webContents.getAllWebContents().some((contents) => contents.getURL() === url), + `${origin}/private-chat` + ) + ) + .toBe(false) + await expect(window).toHaveURL(`${origin}/login`) + }) + async function formState() { return app.evaluate(async ({ webContents }, origin) => { const page = webContents @@ -619,6 +1195,28 @@ test.describe('browser tools', () => { expect(await formState()).toMatchObject({ scrollLeft: 0 }) }) + test('batches an action with a fresh observation without replaying form fields', async () => { + const ref = await openForm() + const fill = await execute('browser_fill_form', { + fields: [{ elementId: ref('Name'), kind: 'text', text: 'Observed value' }], + observe: { query: 'Name' }, + }) + expect(fill.ok, fill.error).toBe(true) + expect(fill.result).toMatchObject({ + completed: true, + completedCount: 1, + observation: { ok: true, result: { totalMatches: 1 } }, + }) + expect(await formState()).toMatchObject({ name: 'Observed value' }) + const result = fill.result as { observation: { result: { matches: { elementId: number }[] } } } + const freshId = result.observation.result.matches[0].elementId + expect(freshId).not.toBe(ref('Name')) + const clear = await execute('browser_type', { elementId: freshId, text: '', observe: {} }) + expect(clear.ok, clear.error).toBe(true) + expect(clear.result).toMatchObject({ observation: { ok: true } }) + expect(await formState()).toMatchObject({ name: '' }) + }) + test('stops after a route change without writing the next field', async () => { const ref = await openForm() const fill = await execute('browser_fill_form', { @@ -708,4 +1306,37 @@ test.describe('browser tools', () => { expect(fill.result).toMatchObject({ completed: false, completedCount: 0 }) expect(await formState()).toMatchObject({ name: '', password: '' }) }) + test('local terminal executes through its native PTY and refuses repeated authorization', async () => { + await window.evaluate(async (scope) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + await api.terminal.activateScope(scope) + await api.terminal.openTerminal(undefined, scope) + }, SCOPE) + calls.set('local-cwd', { + chatId: SCOPE, + toolName: 'terminal', + args: { operation: 'cwd', args: {} }, + }) + const cwd = await window.evaluate(async (scope) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + return api.terminal.executeTool('local-cwd', 'cwd', {}, scope) + }, SCOPE) + expect(cwd.ok).toBe(true) + calls.set('local-run', { + chatId: SCOPE, + toolName: 'terminal', + args: { operation: 'run', args: { command: "printf 'SIM_NATIVE_TERMINAL_VERIFIED\\n'" } }, + }) + const result = await window.evaluate(async (scope) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + return api.terminal.executeTool('local-run', 'run', {}, scope) + }, SCOPE) + expect(result.ok).toBe(true) + expect(JSON.stringify(result)).toContain('SIM_NATIVE_TERMINAL_VERIFIED') + const replay = await window.evaluate(async (scope) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + return api.terminal.executeTool('local-run', 'run', {}, scope) + }, SCOPE) + expect(replay.ok).toBe(false) + }) }) diff --git a/apps/desktop/e2e/local-files.spec.ts b/apps/desktop/e2e/local-files.spec.ts new file mode 100644 index 00000000000..d2645297598 --- /dev/null +++ b/apps/desktop/e2e/local-files.spec.ts @@ -0,0 +1,144 @@ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { createServer, type Server } from 'node:http' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { type ElectronApplication, _electron as electron, expect, test } from '@playwright/test' +import type { DesktopLocalFileRequest, SimDesktopApi } from '@sim/desktop-bridge' + +const DESKTOP_DIR = fileURLToPath(new URL('..', import.meta.url)) + +test('native file tools read and import through the installed preload without Sim folder grants', async () => { + const root = mkdtempSync(join(tmpdir(), 'sim-native-files-e2e-')) + const source = join(root, 'Reports') + mkdirSync(join(source, 'empty'), { recursive: true }) + writeFileSync(join(source, 'report.txt'), 'native file contents') + const png = + 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Y9Zl1sAAAAASUVORK5CYII=' + writeFileSync(join(source, 'image.png'), Buffer.from(png, 'base64')) + let claimed = false + const calls: Record }> = { + text: { toolName: 'read_local_file', args: { path: join(source, 'report.txt') } }, + image: { toolName: 'read_local_file', args: { path: join(source, 'image.png') } }, + import: { + toolName: 'import_local_files', + args: { path: source, targetWorkspaceId: 'target-workspace' }, + }, + } + let server: Server | undefined + let app: ElectronApplication | undefined + try { + server = createServer(async (request, response) => { + const path = new URL(request.url ?? '/', 'http://127.0.0.1').pathname + if (path === '/api/auth/get-session') { + response.writeHead(200, { 'Content-Type': 'application/json' }).end( + JSON.stringify({ + user: { id: 'local-file-user' }, + session: { id: 'local-file-session' }, + }) + ) + return + } + if (path === '/api/desktop/tool/authorize') { + let body = '' + for await (const chunk of request) body += chunk.toString() + const input = JSON.parse(body) + const call = calls[input.toolCallId] + if (!call || (input.claim && claimed)) { + response.writeHead(call ? 409 : 403, { 'Content-Type': 'application/json' }).end('{}') + return + } + if (input.claim) claimed = true + response + .writeHead(200, { 'Content-Type': 'application/json' }) + .end(JSON.stringify({ ...call, chatId: 'org-chat' })) + return + } + response + .writeHead(200, { + 'Content-Type': 'text/html', + 'Set-Cookie': 'better-auth.session_token=fixture; HttpOnly; SameSite=Lax; Path=/', + }) + .end('Local file fixture

Local files

') + }) + await new Promise((resolve) => server?.listen(0, '127.0.0.1', resolve)) + const address = server.address() + if (!address || typeof address === 'string') throw new Error('Missing fixture address') + app = await electron.launch({ + args: ['.'], + cwd: DESKTOP_DIR, + env: { + ...process.env, + SIM_DESKTOP_ORIGIN: `http://127.0.0.1:${address.port}`, + SIM_DESKTOP_USER_DATA: join(root, 'profile'), + }, + }) + const window = await app.firstWindow() + await expect(window.getByRole('heading')).toHaveText('Local files') + const invoke = (input: DesktopLocalFileRequest) => + window.evaluate(async (request) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + if (!api.localFiles) throw new Error('Native file bridge missing') + return api.localFiles(request) + }, input) + await expect + .poll(async () => (await invoke({ operation: 'read', toolCallId: 'text' })).ok) + .toBe(true) + expect(await invoke({ operation: 'read', toolCallId: 'text' })).toMatchObject({ + ok: true, + data: { representation: 'text', text: 'native file contents' }, + }) + expect(await invoke({ operation: 'read', toolCallId: 'image' })).toMatchObject({ + ok: true, + data: { observations: [{ mediaType: 'image/png', data: png }] }, + }) + const result = await invoke({ operation: 'manifest', toolCallId: 'import' }) + if (!result.ok || result.data.kind !== 'manifest') throw new Error(JSON.stringify(result)) + expect(result.data.targetWorkspaceId).toBe('target-workspace') + expect(result.data.entries.map((entry) => entry.relativePath)).toEqual([ + '', + 'empty', + 'image.png', + 'report.txt', + ]) + const file = result.data.entries.find((entry) => entry.relativePath === 'report.txt') + if (!file) throw new Error('Missing import file') + const chunk = await invoke({ + operation: 'chunk', + toolCallId: 'import', + relativePath: file.relativePath, + revision: file.revision, + offset: 0, + }) + if (!chunk.ok || chunk.data.kind !== 'chunk') throw new Error(JSON.stringify(chunk)) + expect(Object.values(chunk.data.bytes)).toEqual([...Buffer.from('native file contents')]) + expect(chunk.data.eof).toBe(true) + expect( + await window.evaluate( + async (input) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + const response = await api.localFiles?.(input) + if (!response?.ok || response.data.kind !== 'chunk') + throw new Error('Missing native chunk') + const file = new File([new Uint8Array(response.data.bytes)], 'report.txt') + return file.text() + }, + { + operation: 'chunk' as const, + toolCallId: 'import', + relativePath: file.relativePath, + revision: file.revision, + offset: 0, + } + ) + ).toBe('native file contents') + expect(await invoke({ operation: 'manifest', toolCallId: 'import' })).toMatchObject({ + ok: false, + code: 'ALREADY_STARTED', + }) + } finally { + await app?.close() + server?.close() + rmSync(root, { recursive: true, force: true }) + } +}) diff --git a/apps/desktop/e2e/password-autofill.spec.ts b/apps/desktop/e2e/password-autofill.spec.ts new file mode 100644 index 00000000000..949581bce3c --- /dev/null +++ b/apps/desktop/e2e/password-autofill.spec.ts @@ -0,0 +1,527 @@ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { createServer, type Server } from 'node:http' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { + type ElectronApplication, + _electron as electron, + expect, + type Page, + test, +} from '@playwright/test' +import type { SimDesktopApi } from '@sim/desktop-bridge' + +const DESKTOP_DIR = fileURLToPath(new URL('..', import.meta.url)) +const SCOPE = 'password-fixture' +const SCREENSHOTS = process.env.SIM_PASSWORD_SCREENSHOTS +const LOGIN = `Account sign in

Sign in to your account

` + +test.describe('saved password autofill', () => { + let server: Server + let origin: string + let site: string + let app: ElectronApplication + let host: Page + let userData: string + const calls = new Map< + string, + { chatId: string; toolName: string; args: Record } + >() + let serial = 0 + + test.beforeAll(async () => { + server = createServer(async (request, response) => { + const path = new URL(request.url ?? '/', 'http://localhost').pathname + if (path === '/api/auth/get-session') { + response.writeHead(200, { 'Content-Type': 'application/json' }) + response.end( + JSON.stringify({ user: { id: 'fixture-user' }, session: { id: 'fixture-session' } }) + ) + return + } + if (path === '/api/desktop/tool/authorize') { + let body = '' + for await (const chunk of request) body += chunk.toString() + const call = calls.get(JSON.parse(body).toolCallId) + response.writeHead(call ? 200 : 403, { 'Content-Type': 'application/json' }) + response.end(JSON.stringify(call ?? {})) + return + } + if (path.startsWith('/api/')) { + response.writeHead(200, { 'Content-Type': 'application/json' }) + response.end('{}') + return + } + response.writeHead(200, { + 'Content-Type': 'text/html', + 'Set-Cookie': 'better-auth.session_token=fixture; HttpOnly; SameSite=Lax; Path=/', + }) + response.end( + path === '/login' + ? LOGIN + : 'Sim fixture

Password fixture

' + ) + }) + await new Promise((resolve) => server.listen(0, resolve)) + const address = server.address() + if (!address || typeof address === 'string') throw new Error('Missing fixture address') + origin = `http://127.0.0.1:${address.port}` + site = `http://localhost:${address.port}` + }) + + test.beforeEach(async () => { + userData = mkdtempSync(join(tmpdir(), 'sim-password-e2e-')) + app = await electron.launch({ + args: ['.'], + cwd: DESKTOP_DIR, + env: { ...process.env, SIM_DESKTOP_ORIGIN: origin, SIM_DESKTOP_USER_DATA: userData }, + }) + host = await app.firstWindow() + await expect(host.getByRole('heading')).toHaveText('Password fixture') + await app.evaluate(({ app, BrowserWindow }) => { + app.focus({ steal: true }) + BrowserWindow.getAllWindows()[0].focus() + }) + await expect + .poll(() => app.evaluate(({ BrowserWindow }) => BrowserWindow.getAllWindows()[0].isFocused())) + .toBe(true) + await host.evaluate(async (scope) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + await api.browserAgent.activateScope(scope) + api.browserAgent.setPanelBounds( + { x: 0, y: 130, width: innerWidth, height: innerHeight - 130 }, + null, + scope + ) + setInterval( + () => + api.browserAgent.setPanelBounds( + { x: 0, y: 130, width: innerWidth, height: innerHeight - 130 }, + null, + scope + ), + 200 + ) + }, SCOPE) + await seed(3) + await navigate(`${site}/login`) + }) + + test.afterEach(async () => { + await app?.close() + rmSync(userData, { recursive: true, force: true }) + calls.clear() + }) + test.afterAll(async () => { + await new Promise((resolve, reject) => + server.close((error) => (error ? reject(error) : resolve())) + ) + }) + + async function seed(count: number, prefix = 'account') { + const ciphertext = await app.evaluate( + ({ safeStorage }, { site, count, prefix }) => { + const records = Array.from({ length: count }, (_, index) => ({ + id: `account-${index}`, + origin: site, + username: `${prefix}${index + 1}@example.test`, + password: `fixture-secret-${index}`, + createdAt: '2026-01-01T00:00:00.000Z', + updatedAt: '2026-01-01T00:00:00.000Z', + source: 'manual', + })) + return safeStorage.encryptString(JSON.stringify(records)).toString('base64') + }, + { site, count, prefix } + ) + writeFileSync( + join(userData, 'browser-credentials.json'), + JSON.stringify({ version: 1, ciphertext }), + { mode: 0o600 } + ) + } + + async function navigate(url: string) { + const id = `fixture-${++serial}` + calls.set(id, { chatId: SCOPE, toolName: 'browser_open_url', args: { url } }) + const result = await host.evaluate( + async ({ id, url, scope }) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + return api.browserAgent.executeTool(id, 'browser_open_url', { url }, scope) + }, + { id, url, scope: SCOPE } + ) + expect(result.ok, result.error).toBe(true) + } + + async function pageScript(script: string): Promise { + return app.evaluate( + async ({ webContents }, { site, script }) => { + const page = webContents + .getAllWebContents() + .find((contents) => contents.getURL().startsWith(`${site}/login`)) + if (!page) throw new Error('Missing fixture page') + return page.executeJavaScript(script) + }, + { site, script } + ) + } + + async function clickField(id = 'user') { + await app.evaluate( + async ({ webContents }, { site, id }) => { + const page = webContents + .getAllWebContents() + .find((contents) => contents.getURL().startsWith(`${site}/login`))! + const point = await page.executeJavaScript( + `(() => {const r=document.getElementById(${JSON.stringify(id)}).getBoundingClientRect();return {x:Math.round(r.x+12),y:Math.round(r.y+12)}})()` + ) + page.focus() + page.sendInputEvent({ type: 'mouseDown', button: 'left', clickCount: 1, ...point }) + page.sendInputEvent({ type: 'mouseUp', button: 'left', clickCount: 1, ...point }) + }, + { site, id } + ) + } + + async function picker() { + await expect + .poll(() => app.windows().some((page) => page.url().includes('credential-picker.html'))) + .toBe(true) + const page = app.windows().find((page) => page.url().includes('credential-picker.html'))! + await expect(page.getByRole('menu')).toBeVisible() + return page + } + + async function pickerKey(keyCode: string) { + await app.evaluate(({ BrowserWindow }, keyCode) => { + const contents = BrowserWindow.getAllWindows().find((window) => + window.webContents.getURL().includes('credential-picker.html') + )!.webContents + contents.sendInputEvent({ type: 'keyDown', keyCode }) + contents.sendInputEvent({ type: 'keyUp', keyCode }) + }, keyCode) + } + + test('uses shared emcn styling, fills a selected account, and never submits', async () => { + await clickField() + const menu = await picker() + await expect(menu.getByRole('menuitem')).toHaveCount(3) + const denied = await menu.evaluate(async () => { + const api = ( + window as Window & { simCredentialPicker?: { select(id: string): Promise } } + ).simCredentialPicker + return api?.select('account-0') + }) + expect(denied).toBe('failed') + expect(await pageScript('document.getElementById("pass").value')).toBe('') + const placement = await app.evaluate(async ({ BrowserWindow, WebContentsView }) => { + const picker = BrowserWindow.getAllWindows().find((window) => + window.webContents.getURL().includes('credential-picker.html') + )! + const parent = picker.getParentWindow()! + const view = parent.contentView.children.find( + (view) => view instanceof WebContentsView && view.webContents.getURL().includes('/login') + ) as InstanceType + const field = await view.webContents.executeJavaScript( + '(() => { const r=document.getElementById("user").getBoundingClientRect(); return {x:r.x,y:r.y,height:r.height} })()' + ) + const bounds = picker.getBounds() + const zoom = view.webContents.getZoomFactor() + return { + x: bounds.x, + y: bounds.y, + height: bounds.height, + fieldX: parent.getContentBounds().x + view.getBounds().x + field.x * zoom, + fieldY: parent.getContentBounds().y + view.getBounds().y + field.y * zoom, + fieldHeight: field.height * zoom, + } + }) + expect(Math.abs(placement.x - placement.fieldX)).toBeLessThanOrEqual(1) + expect( + Math.min( + Math.abs(placement.y - placement.fieldY - placement.fieldHeight - 4), + Math.abs(placement.y + placement.height + 4 - placement.fieldY) + ) + ).toBeLessThanOrEqual(1) + await menu.evaluate(async () => { + await document.fonts.ready + await new Promise(requestAnimationFrame) + await Promise.all(document.getAnimations().map((animation) => animation.finished)) + }) + const geometry = await menu.getByRole('menu').evaluate((element) => ({ + x: element.getBoundingClientRect().x, + width: element.getBoundingClientRect().width, + overflow: element.scrollHeight > element.clientHeight, + font: getComputedStyle(element).fontFamily, + weight: getComputedStyle(element).fontWeight, + })) + expect(geometry).toMatchObject({ x: 0, width: 320, overflow: false, weight: '400' }) + expect(geometry.font).toContain('Season Sans') + if (SCREENSHOTS) { + mkdirSync(SCREENSHOTS, { recursive: true }) + for (const theme of ['light', 'dark']) { + await host.evaluate((theme) => { + document.documentElement.className = theme + document.documentElement.style.colorScheme = theme + }, theme) + await expect(menu.locator('html')).toHaveClass(theme) + await menu.screenshot({ path: join(SCREENSHOTS, `password-picker-${theme}.png`) }) + } + } + await menu.getByRole('menuitem', { name: 'account2@example.test', exact: true }).click() + await expect + .poll(() => + pageScript( + '({user:document.getElementById("user").value,pass:document.getElementById("pass").value,submitted:document.body.dataset.submitted ?? null})' + ) + ) + .toEqual({ user: 'account2@example.test', pass: 'fixture-secret-1', submitted: null }) + await expect + .poll(() => app.windows().some((page) => page.url().includes('credential-picker.html'))) + .toBe(false) + }) + + test('supports keyboard selection and dismissal without unnecessary scrolling', async () => { + await seed(15) + await clickField() + const menu = await picker() + await app.evaluate(({ webContents }, site) => { + const page = webContents + .getAllWebContents() + .find((contents) => contents.getURL().startsWith(`${site}/login`))! + page.sendInputEvent({ type: 'keyDown', keyCode: 'Down' }) + page.sendInputEvent({ type: 'keyUp', keyCode: 'Down' }) + }, site) + await expect + .poll(() => + app.evaluate(({ BrowserWindow }) => + BrowserWindow.getFocusedWindow()?.webContents.getURL().includes('credential-picker.html') + ) + ) + .toBe(true) + await expect(menu.getByRole('menuitem')).toHaveCount(15) + await pickerKey('End') + await expect(menu.getByRole('menuitem').last()).toBeFocused() + await pickerKey('Enter') + await expect + .poll(() => pageScript('document.getElementById("user").value')) + .toBe('account9@example.test') + await clickField() + await picker() + await pickerKey('Escape') + await expect + .poll(() => app.windows().some((page) => page.url().includes('credential-picker.html'))) + .toBe(false) + }) + + for (const count of [1, 3]) { + test(`keeps full account tooltips inside a ${count}-account picker`, async () => { + const prefix = 'person.with.a.long.name.and.department.for.signin' + await seed(count, prefix) + await clickField() + const menu = await picker() + await menu.getByRole('menuitem').first().hover() + const tooltip = menu.locator('body > [data-native-surface-overlay][aria-hidden="true"]') + await expect(tooltip).toBeVisible() + await expect(tooltip).toHaveText(`${prefix}1@example.test`) + await expect + .poll(() => + tooltip.evaluate((element) => { + const rect = element.getBoundingClientRect() + return ( + rect.left >= 0 && + rect.top >= 0 && + rect.right <= innerWidth && + rect.bottom <= innerHeight + ) + }) + ) + .toBe(true) + if (SCREENSHOTS) { + await menu.screenshot({ path: join(SCREENSHOTS, `password-picker-long-${count}.png`) }) + } + await app.evaluate(({ BrowserWindow }) => { + BrowserWindow.getAllWindows() + .find((window) => window.webContents.getURL().includes('credential-picker.html'))! + .setContentSize(800, 600) + }) + await menu + .getByRole('menuitem') + .first() + .hover({ position: { x: 100, y: 14 } }) + await expect + .poll(() => + tooltip.evaluate((element) => { + const rect = element.getBoundingClientRect() + return ( + rect.left >= 100 && + rect.top >= 0 && + rect.right <= innerWidth && + rect.bottom <= innerHeight + ) + }) + ) + .toBe(true) + }) + } + + test('dismisses an inactive picker on outside input and hides it when its page disappears', async () => { + await clickField() + await picker() + await host.getByRole('button', { name: 'Outside the browser' }).click() + await expect + .poll(() => app.windows().some((page) => page.url().includes('credential-picker.html'))) + .toBe(false) + await expect(host.getByRole('button', { name: 'Outside the browser' })).toBeFocused() + await clickField() + await picker() + await host.evaluate((scope) => { + ;( + globalThis as typeof globalThis & { simDesktop: SimDesktopApi } + ).simDesktop.browserAgent.setPanelBounds(null, null, scope) + }, SCOPE) + await expect + .poll(() => app.windows().some((page) => page.url().includes('credential-picker.html'))) + .toBe(false) + expect(await pageScript('document.getElementById("pass").value')).toBe('') + }) + + test('returns keyboard focus to the login after dismissal and selection', async () => { + await clickField() + await picker() + await app.evaluate(({ webContents }, site) => { + const page = webContents + .getAllWebContents() + .find((contents) => contents.getURL().startsWith(`${site}/login`))! + page.sendInputEvent({ type: 'keyDown', keyCode: 'Down' }) + page.sendInputEvent({ type: 'keyUp', keyCode: 'Down' }) + }, site) + await expect + .poll(() => + app.evaluate(({ BrowserWindow }) => + BrowserWindow.getFocusedWindow()?.webContents.getURL().includes('credential-picker.html') + ) + ) + .toBe(true) + await pickerKey('Escape') + await expect + .poll(() => + app.evaluate(({ BrowserWindow, webContents }) => ({ + window: BrowserWindow.getFocusedWindow()?.webContents.getURL(), + page: webContents.getFocusedWebContents()?.getURL(), + })) + ) + .toEqual({ window: host.url(), page: `${site}/login` }) + await app.evaluate(({ webContents }) => { + const page = webContents.getFocusedWebContents()! + page.sendInputEvent({ type: 'keyDown', keyCode: 'Tab' }) + page.sendInputEvent({ type: 'keyUp', keyCode: 'Tab' }) + }) + await expect.poll(() => pageScript('document.activeElement.id')).toBe('pass') + const menu = await picker() + await menu.getByRole('menuitem').first().click() + await expect + .poll(() => app.evaluate(({ webContents }) => webContents.getFocusedWebContents()?.getURL())) + .toBe(`${site}/login`) + }) + + test('rejects replaced fields and excludes account creation', async () => { + await clickField() + await picker() + await pageScript( + 'document.querySelector("form").innerHTML = \'\'' + ) + await expect + .poll(() => app.windows().some((page) => page.url().includes('credential-picker.html'))) + .toBe(false) + await clickField('pass') + expect(await pageScript('document.getElementById("pass").value')).toBe('') + expect(app.windows().some((page) => page.url().includes('credential-picker.html'))).toBe(false) + }) + test('offers a focused manual chooser with one account', async () => { + await seed(1) + await host.evaluate(async (scope) => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + await api.browserAgent.capturePanelSnapshot(scope) + if (!(await api.browserAgent.setPanelOccluded(true, scope))) { + throw new Error('Could not open the toolbar overlay') + } + }, SCOPE) + await host.evaluate((scope) => { + document.getElementById('outside')!.onclick = async () => { + const api = (globalThis as typeof globalThis & { simDesktop: SimDesktopApi }).simDesktop + await api.browserAgent.setPanelOccluded(false, scope) + await api.browserCredentials.showChooser({ x: 20, y: 80 }, scope) + } + }, SCOPE) + await host.getByRole('button', { name: 'Outside the browser' }).click() + const menu = await picker() + await expect(menu.getByRole('menuitem')).toHaveCount(1) + await expect + .poll(() => + app.evaluate(({ BrowserWindow }) => + BrowserWindow.getFocusedWindow()?.webContents.getURL().includes('credential-picker.html') + ) + ) + .toBe(true) + await menu.getByRole('menuitem').click() + await expect + .poll(() => pageScript('document.getElementById("pass").value')) + .toBe('fixture-secret-0') + }) + + test('shows a fill failure when the page rejects the value', async () => { + await pageScript( + 'document.getElementById("pass").addEventListener("input", (event) => { event.target.value = "" })' + ) + await clickField() + const menu = await picker() + await menu.getByRole('menuitem').first().click() + await expect(menu.getByRole('alert')).toHaveText( + 'Could not fill this form. Select the field again.' + ) + expect(await pageScript('document.getElementById("pass").value')).toBe('') + if (SCREENSHOTS) await menu.screenshot({ path: join(SCREENSHOTS, 'password-picker-error.png') }) + }) + + test('closes on navigation and never fills the destination', async () => { + await clickField() + await picker() + await navigate(`${site}/login?next=1`) + await expect + .poll(() => app.windows().some((page) => page.url().includes('credential-picker.html'))) + .toBe(false) + expect(await pageScript('document.getElementById("pass").value')).toBe('') + }) + + test('fills the focused form in an open shadow root with composed events', async () => { + await pageScript(`document.body.innerHTML = '
'; + const root = document.getElementById('shadow').attachShadow({mode:'open'}); + root.innerHTML = '
'; + document.addEventListener('input', () => document.body.dataset.input = 'yes')`) + await app.evaluate(async ({ webContents }, site) => { + const page = webContents + .getAllWebContents() + .find((contents) => contents.getURL().startsWith(`${site}/login`))! + const point = await page.executeJavaScript( + '(() => {const r=document.getElementById("shadow").shadowRoot.getElementById("pass").getBoundingClientRect();return {x:Math.round(r.x+4),y:Math.round(r.y+4)}})()' + ) + page.focus() + page.sendInputEvent({ type: 'mouseDown', button: 'left', clickCount: 1, ...point }) + page.sendInputEvent({ type: 'mouseUp', button: 'left', clickCount: 1, ...point }) + }, site) + const menu = await picker() + await menu.getByRole('menuitem').first().click() + await expect + .poll(() => + pageScript('document.getElementById("shadow").shadowRoot.getElementById("pass").value') + ) + .toBe('fixture-secret-0') + expect(await pageScript('document.body.dataset.input')).toBe('yes') + }) +}) diff --git a/apps/desktop/package.json b/apps/desktop/package.json index f4c24dee88c..206f288200a 100644 --- a/apps/desktop/package.json +++ b/apps/desktop/package.json @@ -42,7 +42,8 @@ "@xterm/headless": "6.0.0", "electron-updater": "6.8.9", "micromatch": "4.0.8", - "safe-regex2": "5.1.0" + "safe-regex2": "5.1.0", + "pdf-lib": "1.17.1" }, "devDependencies": { "@electron/fuses": "1.8.0", diff --git a/apps/desktop/scripts/build.ts b/apps/desktop/scripts/build.ts index 2f009c58087..b15dc2fb354 100644 --- a/apps/desktop/scripts/build.ts +++ b/apps/desktop/scripts/build.ts @@ -102,6 +102,7 @@ const renderer: BuildOptions = { server: 'src/renderer/server/index.tsx', offline: 'src/renderer/offline/index.tsx', dialog: 'src/renderer/dialog/index.tsx', + 'credential-picker': 'src/renderer/credential-picker/index.tsx', }, outdir: 'dist/renderer', bundle: true, @@ -146,6 +147,11 @@ async function run(): Promise { entryPoints: ['src/preload/shell.ts'], outfile: 'dist/shell-preload.cjs', }) + const credentialPickerPreloadCtx = await context({ + ...common, + entryPoints: ['src/preload/credential-picker.ts'], + outfile: 'dist/credential-picker-preload.cjs', + }) const mainCtx = await context({ ...common, entryPoints: ['src/main/index.ts'], @@ -169,11 +175,17 @@ async function run(): Promise { browserPreloadCtx.watch(), rendererCtx.watch(), shellPreloadCtx.watch(), + credentialPickerPreloadCtx.watch(), ]) return } await Promise.all([ build(renderer), + build({ + ...common, + entryPoints: ['src/preload/credential-picker.ts'], + outfile: 'dist/credential-picker-preload.cjs', + }), build({ ...common, entryPoints: ['src/preload/shell.ts'], outfile: 'dist/shell-preload.cjs' }), build({ ...common, entryPoints: ['src/main/index.ts'], outfile: 'dist/main.cjs' }), build({ ...common, entryPoints: ['src/preload/index.ts'], outfile: 'dist/preload.cjs' }), diff --git a/apps/desktop/src/main/browser-agent/cdp.test.ts b/apps/desktop/src/main/browser-agent/cdp.test.ts index fdc0884e3f7..36fb408ae4d 100644 --- a/apps/desktop/src/main/browser-agent/cdp.test.ts +++ b/apps/desktop/src/main/browser-agent/cdp.test.ts @@ -1,3 +1,4 @@ +import { getErrorMessage } from '@sim/utils/errors' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' vi.mock('electron', () => import('@/test/electron-mock')) @@ -15,7 +16,10 @@ import { ensureInstrumented, evaluateInIsolatedFrame, insertText, + releaseFileInput, + resolveFileInput, setColorScheme, + setFileInputFiles, } from '@/main/browser-agent/cdp' function createOopifFrameFixture() { @@ -45,6 +49,7 @@ function createOopifFrameFixture() { { frame: { id: 'child', + parentId: 'top', name: 'account-menu', url: 'https://accounts.example/menu', }, @@ -58,7 +63,7 @@ describe('browser-agent CDP instrumentation', () => { it('leaves file chooser dialogs native so users can upload files', async () => { const contents = new WebContentsView().webContents - await ensureInstrumented(contents, { onDialog: vi.fn() }) + await ensureInstrumented(contents, { onDialog: vi.fn(), dialogResponse: () => null }) expect(contents.debugger.sendCommand).toHaveBeenCalledWith('Page.enable', undefined) expect(contents.debugger.sendCommand).not.toHaveBeenCalledWith( @@ -78,10 +83,12 @@ describe('browser-agent CDP instrumentation', () => { return Promise.resolve({}) }) - await expect(ensureInstrumented(contents, { onDialog: vi.fn() })).rejects.toThrow( - 'setup acknowledgement lost' - ) - await expect(ensureInstrumented(contents, { onDialog: vi.fn() })).resolves.toBeUndefined() + await expect( + ensureInstrumented(contents, { onDialog: vi.fn(), dialogResponse: () => null }) + ).rejects.toThrow('setup acknowledgement lost') + await expect( + ensureInstrumented(contents, { onDialog: vi.fn(), dialogResponse: () => null }) + ).resolves.toBeUndefined() expect(autoAttachAttempts).toBe(2) }) @@ -89,7 +96,7 @@ describe('browser-agent CDP instrumentation', () => { it('dismisses an OOPIF dialog on the flattened child session', async () => { const contents = new WebContentsView().webContents const onDialog = vi.fn() - await ensureInstrumented(contents, { onDialog }) + await ensureInstrumented(contents, { onDialog, dialogResponse: () => null }) const listener = vi .mocked(contents.debugger.on) .mock.calls.find(([event]) => event === 'message')?.[1] as @@ -111,13 +118,18 @@ describe('browser-agent CDP instrumentation', () => { { accept: false }, 'child-session' ) - expect(onDialog).toHaveBeenCalledWith({ type: 'alert', message: 'Hello', handled: true }) + expect(onDialog).toHaveBeenCalledWith({ + type: 'alert', + message: 'Hello', + handled: true, + accepted: false, + }) }) it('accepts an OOPIF beforeunload dialog on the flattened child session', async () => { const contents = new WebContentsView().webContents const onDialog = vi.fn() - await ensureInstrumented(contents, { onDialog }) + await ensureInstrumented(contents, { onDialog, dialogResponse: () => null }) const listener = vi .mocked(contents.debugger.on) .mock.calls.find(([event]) => event === 'message')?.[1] as @@ -143,13 +155,43 @@ describe('browser-agent CDP instrumentation', () => { type: 'beforeunload', message: 'Leave this page?', handled: true, + accepted: true, + }) + }) + + it('answers dialogs with the running action requested response', async () => { + const contents = new WebContentsView().webContents + const onDialog = vi.fn() + const dialogResponse = vi.fn(() => ({ accept: true })) + await ensureInstrumented(contents, { onDialog, dialogResponse }) + const listener = vi + .mocked(contents.debugger.on) + .mock.calls.find(([event]) => event === 'message')?.[1] as + | ((event: unknown, method: string, params: unknown, sessionId?: string) => void) + | undefined + vi.mocked(contents.debugger.sendCommand).mockClear() + + listener?.({}, 'Page.javascriptDialogOpening', { type: 'alert', message: 'Saved' }) + await vi.waitFor(() => expect(onDialog).toHaveBeenCalledTimes(1)) + listener?.({}, 'Page.javascriptDialogOpening', { type: 'confirm', message: 'Delete?' }) + await vi.waitFor(() => expect(onDialog).toHaveBeenCalledTimes(2)) + + expect(vi.mocked(contents.debugger.sendCommand).mock.calls).toEqual([ + ['Page.handleJavaScriptDialog', { accept: true }], + ['Page.handleJavaScriptDialog', { accept: true }], + ]) + expect(onDialog).toHaveBeenLastCalledWith({ + type: 'confirm', + message: 'Delete?', + handled: true, + accepted: true, }) }) it('reports an OOPIF dialog as unhandled when child and root commands fail', async () => { const contents = new WebContentsView().webContents const onDialog = vi.fn() - await ensureInstrumented(contents, { onDialog }) + await ensureInstrumented(contents, { onDialog, dialogResponse: () => null }) const listener = vi .mocked(contents.debugger.on) .mock.calls.find(([event]) => event === 'message')?.[1] as @@ -175,6 +217,7 @@ describe('browser-agent CDP instrumentation', () => { type: 'confirm', message: 'Continue?', handled: false, + accepted: false, }) }) @@ -193,6 +236,7 @@ describe('browser-agent CDP instrumentation', () => { y: 240, button: 'left', buttons: 1, + modifiers: 0, clickCount: 1, }, ], @@ -204,6 +248,7 @@ describe('browser-agent CDP instrumentation', () => { y: 240, button: 'left', buttons: 0, + modifiers: 0, clickCount: 1, }, ], @@ -228,6 +273,7 @@ describe('browser-agent CDP instrumentation', () => { y: 24, button: 'left', buttons: 0, + modifiers: 0, clickCount: 1, }, ]) @@ -252,6 +298,7 @@ describe('browser-agent CDP instrumentation', () => { y: 48, button: 'left', buttons: 1, + modifiers: 0, clickCount: 1, }, ], @@ -263,6 +310,7 @@ describe('browser-agent CDP instrumentation', () => { y: 48, button: 'left', buttons: 0, + modifiers: 0, clickCount: 1, }, ], @@ -310,81 +358,115 @@ describe('browser-agent CDP instrumentation', () => { } }) - it('routes OOPIF isolated-world creation and evaluation through its flattened session', async () => { - const contents = new WebContentsView().webContents - const { child, frameTree } = createOopifFrameFixture() - await ensureInstrumented(contents, { onDialog: vi.fn() }) - const listener = vi - .mocked(contents.debugger.on) - .mock.calls.find(([event]) => event === 'message')?.[1] as - | ((event: unknown, method: string, params: unknown, sessionId?: string) => void) - | undefined - expect(listener).toBeTypeOf('function') - - listener?.( - {}, - 'Target.attachedToTarget', - { - sessionId: 'child-session', - targetInfo: { targetId: 'child', type: 'iframe' }, - }, - undefined - ) - expect(contents.debugger.sendCommand).toHaveBeenCalledWith( - 'Target.setAutoAttach', - { autoAttach: true, waitForDebuggerOnStart: false, flatten: true }, - 'child-session' - ) - vi.mocked(contents.debugger.sendCommand).mockClear() - vi.mocked(contents.debugger.sendCommand).mockImplementation((method) => { - if (method === 'Page.getFrameTree') { - return Promise.resolve({ frameTree }) - } - if (method === 'Page.createIsolatedWorld') { - return Promise.resolve({ executionContextId: 42 }) - } - if (method === 'Runtime.evaluate') { - return Promise.resolve({ result: { type: 'number', value: 4 } }) + it.each(['complete', 'split', 'worker', 'detached', 'ambiguous'])( + 'routes OOPIF evaluation through its session (%s tree)', + async (treeKind) => { + const contents = new WebContentsView().webContents + const { child, frameTree } = createOopifFrameFixture() + await ensureInstrumented(contents, { onDialog: vi.fn(), dialogResponse: () => null }) + const listener = vi + .mocked(contents.debugger.on) + .mock.calls.find(([event]) => event === 'message')?.[1] as + | ((event: unknown, method: string, params: unknown, sessionId?: string) => void) + | undefined + expect(listener).toBeTypeOf('function') + + listener?.( + {}, + 'Target.attachedToTarget', + { + sessionId: 'child-session', + targetInfo: { targetId: 'child', type: 'iframe' }, + }, + undefined + ) + expect(contents.debugger.sendCommand).toHaveBeenCalledWith( + 'Target.setAutoAttach', + { autoAttach: true, waitForDebuggerOnStart: false, flatten: true }, + 'child-session' + ) + if (treeKind === 'worker' || treeKind === 'detached') { + listener?.({}, 'Target.attachedToTarget', { + sessionId: 'unavailable-session', + targetInfo: { + targetId: 'unavailable', + type: treeKind === 'worker' ? 'worker' : 'iframe', + }, + }) } - return Promise.resolve({}) - }) - - await expect(evaluateInIsolatedFrame(contents, child, '2 + 2')).resolves.toBe(4) + vi.mocked(contents.debugger.sendCommand).mockClear() + vi.mocked(contents.debugger.sendCommand).mockImplementation((method, _params, sessionId) => { + if (method === 'Page.getFrameTree') { + if (sessionId === 'unavailable-session') + return Promise.reject(new Error('Target unavailable')) + return Promise.resolve({ + frameTree: + treeKind === 'complete' + ? frameTree + : sessionId + ? frameTree.childFrames[0] + : { frame: frameTree.frame }, + }) + } + if (method === 'Page.createIsolatedWorld') { + return Promise.resolve({ executionContextId: 42 }) + } + if (method === 'Runtime.evaluate') { + return Promise.resolve({ result: { type: 'number', value: 4 } }) + } + return Promise.resolve({}) + }) - expect( - vi - .mocked(contents.debugger.sendCommand) - .mock.calls.filter(([method]) => - ['Page.createIsolatedWorld', 'Runtime.evaluate'].includes(method) + if (treeKind === 'ambiguous') { + /** An omitted twin must not be mistaken for the only frame in a partial tree. */ + child.parent?.frames.push(createOopifFrameFixture().child) + await expect(evaluateInIsolatedFrame(contents, child, '2 + 2')).rejects.toThrow( + 'Could not map' ) - ).toEqual([ - [ - 'Page.createIsolatedWorld', - { - frameId: 'child', - worldName: 'sim-browser-agent', - grantUniveralAccess: false, - }, - 'child-session', - ], - [ - 'Runtime.evaluate', - { - expression: '2 + 2', - contextId: 42, - returnByValue: true, - awaitPromise: true, - userGesture: false, - }, - 'child-session', - ], - ]) - }) + expect( + vi + .mocked(contents.debugger.sendCommand) + .mock.calls.some(([method]) => method === 'Runtime.evaluate') + ).toBe(false) + return + } + await expect(evaluateInIsolatedFrame(contents, child, '2 + 2')).resolves.toBe(4) + + expect( + vi + .mocked(contents.debugger.sendCommand) + .mock.calls.filter(([method]) => + ['Page.createIsolatedWorld', 'Runtime.evaluate'].includes(method) + ) + ).toEqual([ + [ + 'Page.createIsolatedWorld', + { + frameId: 'child', + worldName: 'sim-browser-agent', + grantUniveralAccess: false, + }, + 'child-session', + ], + [ + 'Runtime.evaluate', + { + expression: '2 + 2', + contextId: 42, + returnByValue: true, + awaitPromise: true, + userGesture: false, + }, + 'child-session', + ], + ]) + } + ) it('falls back to the root target when OOPIF isolated-world creation fails', async () => { const contents = new WebContentsView().webContents const { child, frameTree } = createOopifFrameFixture() - await ensureInstrumented(contents, { onDialog: vi.fn() }) + await ensureInstrumented(contents, { onDialog: vi.fn(), dialogResponse: () => null }) const listener = vi .mocked(contents.debugger.on) .mock.calls.find(([event]) => event === 'message')?.[1] as @@ -460,6 +542,313 @@ describe('browser-agent CDP instrumentation', () => { }) }) +describe('browser-agent file input handles', () => { + async function fileInputFixture(childSession = false) { + const contents = new WebContentsView().webContents + const { child, frameTree } = createOopifFrameFixture() + await ensureInstrumented(contents, { onDialog: vi.fn(), dialogResponse: () => null }) + if (childSession) { + const onMessage = vi.mocked(contents.debugger.on).mock.calls[0]?.[1] as + | ((event: unknown, method: string, params: unknown, sessionId?: string) => void) + | undefined + onMessage?.({}, 'Target.attachedToTarget', { + sessionId: 'child-session', + targetInfo: { targetId: 'child', type: 'iframe' }, + }) + } + const document: { defaultView: { document: unknown } | null } = { defaultView: null } + document.defaultView = { document } + const input = { + tagName: 'INPUT', + type: 'file', + isConnected: true, + ownerDocument: document, + multiple: true, + accept: 'application/pdf', + matches: vi.fn(() => false), + files: [] as Array<{ name: string; size: number }>, + } + const wrapper = { input, document } + const behavior = { + rejectEvaluation: false, + rejectSet: false, + rejectReadback: false, + beforeSet: async () => {}, + beforeReadback: async () => {}, + afterInputValidation: () => {}, + afterSet: () => {}, + } + const send = vi.mocked(contents.debugger.sendCommand) + send.mockClear().mockImplementation(async (method, params) => { + if (method === 'Page.getFrameTree') return { frameTree } + if (method === 'Page.createIsolatedWorld') return { executionContextId: 42 } + if (method === 'Runtime.evaluate') { + if (behavior.rejectEvaluation) { + return { + result: { objectId: 'exception' }, + exceptionDetails: { exception: { objectId: 'exception', description: 'Ref expired' } }, + } + } + return { result: { objectId: 'wrapper' } } + } + if (method === 'Runtime.callFunctionOn') { + expect(params?.objectId).toBe('wrapper') + const args = params?.arguments as Array<{ value: unknown }> + if (args[0].value === 'files') await behavior.beforeReadback() + if (behavior.rejectReadback && args[0].value === 'files') { + throw new Error('Execution context was destroyed') + } + try { + const inspect = new Function(`return (${params?.functionDeclaration})`)() as ( + ...args: unknown[] + ) => unknown + const result = inspect.apply( + wrapper, + args.map((arg) => arg.value) + ) + if (result === input) { + behavior.afterInputValidation() + return { result: { objectId: 'original-input' } } + } + return { result: { value: result } } + } catch (error) { + return { + result: { objectId: 'exception' }, + exceptionDetails: { + exception: { objectId: 'exception', description: getErrorMessage(error) }, + }, + } + } + } + if (method === 'DOM.setFileInputFiles') { + await behavior.beforeSet() + if (behavior.rejectSet) throw new Error('Input target disappeared') + expect(params).toEqual({ files: ['/staged/a.pdf'], objectId: 'original-input' }) + input.files = [{ name: 'a.pdf', size: 12 }] + behavior.afterSet() + } + return {} + }) + return { + contents, + frame: childSession ? child : child.parent!, + input, + document, + send, + behavior, + } + } + + it.each([false, true])( + 'keeps capture, dispatch, readback and release in the original session (OOPIF: %s)', + async (childSession) => { + const { contents, frame, input, send } = await fileInputFixture(childSession) + const handle = await resolveFileInput(contents, frame, 'captureUploadInput(4)') + expect(handle).toMatchObject({ multiple: true, accept: 'application/pdf' }) + expect(input.matches).toHaveBeenCalledWith(':disabled') + + try { + await expect(setFileInputFiles(contents, handle, ['/staged/a.pdf'])).resolves.toEqual({ + files: [{ name: 'a.pdf', size: 12 }], + }) + } finally { + await releaseFileInput(contents, handle) + } + + const sessionId = childSession ? 'child-session' : undefined + const protocolCalls = send.mock.calls.filter(([method]) => method !== 'Page.getFrameTree') + expect(protocolCalls.every((call) => call[2] === sessionId)).toBe(true) + expect(send.mock.calls.some(([method]) => /Search|DOM.getDocument/.test(method))).toBe(false) + expect(send.mock.calls.find(([method]) => method === 'Runtime.evaluate')?.[1]).toEqual({ + expression: 'captureUploadInput(4)', + contextId: 42, + returnByValue: false, + awaitPromise: true, + userGesture: false, + }) + expect( + send.mock.calls + .filter(([method]) => method === 'Runtime.releaseObject') + .map(([, params]) => params?.objectId) + ).toEqual(['original-input', 'wrapper']) + } + ) + + it.each([ + 'detached', + 'disabled', + 'adopted', + 'document-replaced', + 'document-closed', + 'type', + 'multiple', + ])('refuses a captured input changed before dispatch (%s)', async (change) => { + const { contents, frame, input, document, send } = await fileInputFixture() + const handle = await resolveFileInput(contents, frame, 'captureUploadInput(4)') + if (change === 'detached') input.isConnected = false + if (change === 'disabled') input.matches.mockReturnValue(true) + if (change === 'adopted') input.ownerDocument = { defaultView: null } + if (change === 'document-replaced') document.defaultView = { document: {} } + if (change === 'document-closed') document.defaultView = null + if (change === 'type') input.type = 'text' + if (change === 'multiple') input.multiple = false + const onDispatch = vi.fn() + try { + await expect( + setFileInputFiles( + contents, + handle, + ['/staged/a.pdf', '/staged/b.pdf'], + undefined, + onDispatch + ) + ).rejects.toThrow(/upload input|upload target/) + expect(onDispatch).not.toHaveBeenCalled() + expect(send.mock.calls.some(([method]) => method === 'DOM.setFileInputFiles')).toBe(false) + } finally { + await releaseFileInput(contents, handle) + } + expect(send).toHaveBeenCalledWith('Runtime.releaseObject', { objectId: 'wrapper' }) + expect( + send.mock.calls.filter( + ([method, params]) => method === 'Runtime.releaseObject' && params?.objectId === 'exception' + ) + ).toHaveLength(1) + }) + + it('supports a same-origin child document and XHTML input captured by its parent world', async () => { + const { contents, frame, input } = await fileInputFixture() + input.tagName = 'input' + const handle = await resolveFileInput(contents, frame, 'captureSameOriginChildInput()') + try { + await expect(setFileInputFiles(contents, handle, ['/staged/a.pdf'])).resolves.toEqual({ + files: [{ name: 'a.pdf', size: 12 }], + }) + } finally { + await releaseFileInput(contents, handle) + } + }) + + it.each(['evaluation', 'metadata'] as const)('releases handles when %s fails', async (phase) => { + const { contents, frame, input, behavior, send } = await fileInputFixture() + if (phase === 'evaluation') behavior.rejectEvaluation = true + else input.matches.mockReturnValue(true) + + await expect(resolveFileInput(contents, frame, 'captureUploadInput(4)')).rejects.toThrow() + const released = send.mock.calls + .filter(([method]) => method === 'Runtime.releaseObject') + .map(([, params]) => params?.objectId) + expect(released).toEqual(phase === 'evaluation' ? ['exception'] : ['exception', 'wrapper']) + }) + + it('releases the transient node when cancellation arrives during validation', async () => { + const { contents, frame, behavior, send } = await fileInputFixture() + const handle = await resolveFileInput(contents, frame, 'captureUploadInput(4)') + const controller = new AbortController() + const onDispatch = vi.fn() + behavior.afterInputValidation = () => controller.abort() + try { + await expect( + setFileInputFiles(contents, handle, ['/staged/a.pdf'], controller.signal, onDispatch) + ).rejects.toThrow() + expect(send.mock.calls.some(([method]) => method === 'DOM.setFileInputFiles')).toBe(false) + expect(onDispatch).not.toHaveBeenCalled() + } finally { + await releaseFileInput(contents, handle) + } + expect(send).toHaveBeenCalledWith('Runtime.releaseObject', { objectId: 'original-input' }) + }) + + it('releases the transient node when Chromium rejects the file assignment', async () => { + const { contents, frame, behavior, send } = await fileInputFixture(true) + const handle = await resolveFileInput(contents, frame, 'captureUploadInput(4)') + behavior.rejectSet = true + const onDispatch = vi.fn() + try { + await expect( + setFileInputFiles(contents, handle, ['/staged/a.pdf'], undefined, onDispatch) + ).rejects.toThrow('disappeared') + expect(onDispatch.mock.calls).toEqual([['pending']]) + } finally { + await releaseFileInput(contents, handle) + } + expect(send).toHaveBeenCalledWith( + 'Runtime.releaseObject', + { objectId: 'original-input' }, + 'child-session' + ) + expect(send).toHaveBeenCalledWith( + 'Runtime.releaseObject', + { objectId: 'wrapper' }, + 'child-session' + ) + }) + + it('reads the original input even when its change handler removes it', async () => { + const { contents, frame, input, behavior } = await fileInputFixture() + const handle = await resolveFileInput(contents, frame, 'captureUploadInput(4)') + behavior.afterSet = () => { + input.isConnected = false + } + try { + await expect(setFileInputFiles(contents, handle, ['/staged/a.pdf'])).resolves.toEqual({ + files: [{ name: 'a.pdf', size: 12 }], + }) + } finally { + await releaseFileInput(contents, handle) + } + }) + + it('reports pending dispatch while acknowledgement is held, then acknowledges before readback', async () => { + const { contents, frame, behavior, send } = await fileInputFixture() + const handle = await resolveFileInput(contents, frame, 'captureUploadInput(4)') + let acknowledge: () => void = () => {} + let releaseReadback: () => void = () => {} + const acknowledgement = new Promise((resolve) => { + acknowledge = resolve + }) + const readback = new Promise((resolve) => { + releaseReadback = resolve + }) + behavior.beforeSet = () => acknowledgement + behavior.beforeReadback = () => readback + const onDispatch = vi.fn() + const pending = setFileInputFiles(contents, handle, ['/staged/a.pdf'], undefined, onDispatch) + try { + await vi.waitFor(() => + expect(send.mock.calls.some(([method]) => method === 'DOM.setFileInputFiles')).toBe(true) + ) + expect(onDispatch.mock.calls).toEqual([['pending']]) + acknowledge() + await vi.waitFor(() => expect(onDispatch.mock.calls).toEqual([['pending'], ['acknowledged']])) + expect(send.mock.calls.some(([method]) => method === 'Runtime.releaseObject')).toBe(false) + releaseReadback() + await expect(pending).resolves.toEqual({ files: [{ name: 'a.pdf', size: 12 }] }) + expect(onDispatch.mock.calls).toEqual([['pending'], ['acknowledged']]) + } finally { + acknowledge() + releaseReadback() + await pending + await releaseFileInput(contents, handle) + } + }) + + it('reports readback failure separately once Chromium has acknowledged the upload', async () => { + const { contents, frame, behavior, send } = await fileInputFixture() + const handle = await resolveFileInput(contents, frame, 'captureUploadInput(4)') + behavior.rejectReadback = true + try { + await expect(setFileInputFiles(contents, handle, ['/staged/a.pdf'])).resolves.toEqual({ + readbackError: 'Execution context was destroyed', + }) + } finally { + await releaseFileInput(contents, handle) + } + expect(send.mock.calls.filter(([method]) => method === 'DOM.setFileInputFiles')).toHaveLength(1) + expect(send).toHaveBeenCalledWith('Runtime.releaseObject', { objectId: 'original-input' }) + }) +}) + describe('browser-agent CDP theme', () => { it('emulates explicit light and dark preferences', async () => { const contents = new WebContentsView().webContents diff --git a/apps/desktop/src/main/browser-agent/cdp.ts b/apps/desktop/src/main/browser-agent/cdp.ts index cef27dbd389..8a542b46cad 100644 --- a/apps/desktop/src/main/browser-agent/cdp.ts +++ b/apps/desktop/src/main/browser-agent/cdp.ts @@ -10,7 +10,9 @@ */ import type { BrowserTheme } from '@sim/browser-protocol' import { createLogger } from '@sim/logger' +import { getErrorMessage } from '@sim/utils/errors' import { sleep } from '@sim/utils/helpers' +import { isRecordLike } from '@sim/utils/object' import type { NativeImage, WebContents, WebFrameMain } from 'electron' const logger = createLogger('BrowserAgentCdp') @@ -25,11 +27,19 @@ export interface PageDialog { type: string message: string handled: boolean + accepted: boolean +} + +/** How an agent action asked its JavaScript dialogs to be answered. Electron removes prompt(). */ +export interface DialogResponse { + accept: boolean } export interface CdpCallbacks { - /** A JS dialog was auto-handled; the driver surfaces it to the model. */ + /** A JS dialog was handled; the driver surfaces it to the model. */ onDialog: (dialog: PageDialog) => void + /** The running action's requested answer; dialogs are dismissed when it has none. */ + dialogResponse: () => DialogResponse | null } /** Per-tab callbacks, so a background tab's events reach ITS driver, not the @@ -182,34 +192,29 @@ function handleDebuggerEvent( if (method === 'Page.javascriptDialogOpening') { const type = String(params.type ?? 'dialog') const message = String(params.message ?? '').slice(0, 500) - // beforeunload is accepted (navigation proceeds); everything else is - // dismissed — the model reacts to the recorded message instead of a - // dialog that would block the page. + // Dialogs never stay open: beforeunload is accepted (navigation proceeds), + // and alert/confirm follow the running action's requested answer, defaulting + // to dismissal so an unexpected dialog can never block the page. + const accept = type === 'beforeunload' || callbacks?.dialogResponse()?.accept === true + const answer = { accept } void (async () => { let handled = false try { - await send( - contents, - 'Page.handleJavaScriptDialog', - { accept: type === 'beforeunload' }, - parentSessionId - ) + await send(contents, 'Page.handleJavaScriptDialog', answer, parentSessionId) handled = true } catch { // Some Chromium builds surface an OOPIF's tab-modal dialog on its - // flattened session but accept the dismissal only on the root target. + // flattened session but accept the answer only on the root target. if (parentSessionId) { try { - await send(contents, 'Page.handleJavaScriptDialog', { - accept: type === 'beforeunload', - }) + await send(contents, 'Page.handleJavaScriptDialog', answer) handled = true } catch {} } } - if (handled) logger.info('Auto-handled page dialog', { type }) - else logger.warn('Could not auto-handle page dialog', { type }) - callbacks?.onDialog({ type, message, handled }) + if (handled) logger.info('Handled page dialog', { type, accept }) + else logger.warn('Could not handle page dialog', { type }) + callbacks?.onDialog({ type, message, handled, accepted: handled && accept }) })() return } @@ -255,7 +260,11 @@ export function sameWebFrame(left: WebFrameMain, right: WebFrameMain): boolean { return false } -function locateProtocolFrame(root: ProtocolFrameTree, target: WebFrameMain): ProtocolFrame | null { +function locateProtocolFrame( + root: ProtocolFrameTree, + target: WebFrameMain, + ordered = true +): ProtocolFrame | null { const path: WebFrameMain[] = [] for (let current: WebFrameMain | null = target; current?.parent; current = current.parent) { path.push(current) @@ -269,11 +278,13 @@ function locateProtocolFrame(root: ProtocolFrameTree, target: WebFrameMain): Pro while (electronParent.parent) electronParent = electronParent.parent for (const frame of path) { const children = tree.childFrames ?? [] + /** A partial tree cannot distinguish an omitted sibling with the same URL. */ + if (children.length !== electronParent.frames.length) return null const siblingIndex = electronParent.frames.findIndex((candidate) => sameWebFrame(candidate, frame) ) const indexed = siblingIndex >= 0 ? children[siblingIndex] : undefined - if (indexed && frameMatches(indexed.frame, frame)) { + if (ordered && indexed && frameMatches(indexed.frame, frame)) { tree = indexed } else { const matches = children.filter((candidate) => frameMatches(candidate.frame, frame)) @@ -285,20 +296,47 @@ function locateProtocolFrame(root: ProtocolFrameTree, target: WebFrameMain): Pro return tree.frame } -/** - * Executes code in a persistent isolated world belonging to one child frame. - * WebFrameMain.executeJavaScript runs in the untrusted page's main world, - * where the page can replace the ref registry and built-ins between tools. - */ -export async function evaluateInIsolatedFrame( +async function isolatedFrameContext( contents: WebContents, - frame: WebFrameMain, - expression: string, - userGesture = false -): Promise { + frame: WebFrameMain +): Promise<{ contextId: number; sessionId?: string }> { const { frameTree } = await send<{ frameTree?: ProtocolFrameTree }>(contents, 'Page.getFrameTree') if (!frameTree) throw new Error('Chromium did not return a frame tree') - const protocolFrame = locateProtocolFrame(frameTree, frame) + let protocolFrame = locateProtocolFrame(frameTree, frame) + if (!protocolFrame) { + /** Chromium omits out-of-process frames from the root target's tree. */ + const childSessions = [...(childSessionsByContents.get(contents)?.values() ?? [])] + const results = await Promise.allSettled( + childSessions.map(async (sessionId) => { + const result = await send<{ frameTree?: ProtocolFrameTree }>( + contents, + 'Page.getFrameTree', + undefined, + sessionId + ) + return result.frameTree + }) + ) + const trees = results.flatMap((result) => + result.status === 'fulfilled' && result.value ? [result.value] : [] + ) + const nodes = new Map() + const visit = (tree: ProtocolFrameTree) => { + nodes.set(tree.frame.id, tree) + tree.childFrames?.forEach(visit) + } + visit(frameTree) + for (const tree of trees) if (tree) visit(tree) + for (const tree of trees) { + const parent = tree?.frame.parentId ? nodes.get(tree.frame.parentId) : undefined + if (!tree || !parent) continue + parent.childFrames ??= [] + if (!parent.childFrames.some((child) => child.frame.id === tree.frame.id)) + parent.childFrames.push(tree) + } + /** Target attachment order is not DOM order; ambiguous siblings must not be guessed. */ + protocolFrame = locateProtocolFrame(frameTree, frame, false) + } if (!protocolFrame) throw new Error('Could not map the Electron frame to Chromium') const childSession = childSessionsByContents.get(contents)?.get(protocolFrame.id) @@ -331,7 +369,21 @@ export async function evaluateInIsolatedFrame( if (contextId === undefined) { throw lastError instanceof Error ? lastError : new Error('Could not create an isolated world') } + return { contextId, sessionId: selectedSession } +} +/** + * Executes code in a persistent isolated world belonging to one frame. + * WebFrameMain.executeJavaScript runs in the untrusted page's main world, + * where the page can replace the ref registry and built-ins between tools. + */ +export async function evaluateInIsolatedFrame( + contents: WebContents, + frame: WebFrameMain, + expression: string, + userGesture = false +): Promise { + const { contextId, sessionId } = await isolatedFrameContext(contents, frame) const evaluation = await send<{ result?: { type?: string; value?: unknown; unserializableValue?: string } exceptionDetails?: { text?: string; exception?: { description?: string } } @@ -345,7 +397,7 @@ export async function evaluateInIsolatedFrame( awaitPromise: true, userGesture, }, - selectedSession + sessionId ) if (evaluation.exceptionDetails) { throw new Error( @@ -671,6 +723,200 @@ export async function captureScreenshot( } } +/** Opaque isolated-world wrapper retaining one input and its original owner document. */ +export interface FileInputHandle { + readonly objectId: string + readonly sessionId?: string + readonly multiple: boolean + readonly accept?: string +} + +interface RemoteObject { + objectId?: string + value?: unknown +} + +interface RemoteEvaluation { + result?: RemoteObject + exceptionDetails?: { text?: string; exception?: { description?: string; objectId?: string } } +} + +interface CapturedFileInput { + input: HTMLInputElement + document: Document +} + +/** Runs in the wrapper's isolated world; owner documents may belong to same-origin child frames. */ +function inspectFileInput( + this: CapturedFileInput, + mode: 'metadata' | 'input' | 'files', + fileCount: number +): unknown { + const { input, document: capturedDocument } = this + if (mode === 'files') { + return { + files: Array.from(input.files ?? [], (file) => ({ name: file.name, size: file.size })), + } + } + if (!input || String(input.tagName).toUpperCase() !== 'INPUT' || input.type !== 'file') { + throw new Error('The upload target is no longer a file input') + } + if ( + !input.isConnected || + input.ownerDocument !== capturedDocument || + !capturedDocument.defaultView || + capturedDocument.defaultView.document !== capturedDocument + ) { + throw new Error('The upload input or its document changed. Inspect the page before uploading.') + } + if (input.matches(':disabled')) throw new Error('The upload input is disabled') + if (fileCount > 1 && !input.multiple) { + throw new Error('The upload input no longer accepts multiple files') + } + return mode === 'input' ? input : { multiple: input.multiple, accept: input.accept || undefined } +} + +async function releaseRemoteObject( + contents: WebContents, + objectId: string | undefined, + sessionId?: string +): Promise { + if (objectId) { + await send(contents, 'Runtime.releaseObject', { objectId }, sessionId).catch(() => {}) + } +} + +async function checkedRemoteResult( + contents: WebContents, + evaluation: RemoteEvaluation, + sessionId?: string +): Promise { + if (evaluation.exceptionDetails) { + const ids = new Set([ + evaluation.result?.objectId, + evaluation.exceptionDetails.exception?.objectId, + ]) + await Promise.all([...ids].map((id) => releaseRemoteObject(contents, id, sessionId))) + throw new Error( + evaluation.exceptionDetails.exception?.description || + evaluation.exceptionDetails.text || + 'File input evaluation failed' + ) + } + if (!evaluation.result) throw new Error('Chromium returned no file input evaluation result') + return evaluation.result +} + +async function callFileInput( + contents: WebContents, + handle: Pick, + mode: 'metadata' | 'input' | 'files', + fileCount = 0 +): Promise { + const evaluation = await send( + contents, + 'Runtime.callFunctionOn', + { + objectId: handle.objectId, + functionDeclaration: inspectFileInput.toString(), + arguments: [{ value: mode }, { value: fileCount }], + returnByValue: mode !== 'input', + }, + handle.sessionId + ) + return checkedRemoteResult(contents, evaluation, handle.sessionId) +} + +/** Captures a trusted isolated-world expression's input/document wrapper without exposing a DOM marker. */ +export async function resolveFileInput( + contents: WebContents, + frame: WebFrameMain, + expression: string +): Promise { + const { contextId, sessionId } = await isolatedFrameContext(contents, frame) + const evaluation = await send( + contents, + 'Runtime.evaluate', + { expression, contextId, returnByValue: false, awaitPromise: true, userGesture: false }, + sessionId + ) + const remote = await checkedRemoteResult(contents, evaluation, sessionId) + if (!remote.objectId) throw new Error('Chromium did not retain the upload input') + const handle = { objectId: remote.objectId, sessionId } + try { + const { value } = await callFileInput(contents, handle, 'metadata') + if ( + !isRecordLike(value) || + typeof value.multiple !== 'boolean' || + (value.accept !== undefined && typeof value.accept !== 'string') + ) { + throw new Error('Chromium did not return valid upload input metadata') + } + return { ...handle, multiple: value.multiple, accept: value.accept } + } catch (error) { + await releaseRemoteObject(contents, handle.objectId, sessionId) + throw error + } +} + +/** Releases the captured input/document wrapper after upload preparation or dispatch finishes. */ +export async function releaseFileInput( + contents: WebContents, + handle: FileInputHandle +): Promise { + await releaseRemoteObject(contents, handle.objectId, handle.sessionId) +} + +/** + * Reports assignment dispatch and acknowledgement separately before reading the captured input. + * Stopping a wait cannot revoke a CDP command, so its outcome becomes uncertain before the send. + */ +export async function setFileInputFiles( + contents: WebContents, + handle: FileInputHandle, + files: readonly string[], + signal?: AbortSignal, + onDispatch?: (status: 'pending' | 'acknowledged') => void +): Promise<{ files: Array<{ name: string; size: number }> } | { readbackError: string }> { + signal?.throwIfAborted() + const input = await callFileInput(contents, handle, 'input', files.length) + if (!input.objectId) throw new Error('Chromium did not retain the upload input node') + try { + signal?.throwIfAborted() + onDispatch?.('pending') + await send( + contents, + 'DOM.setFileInputFiles', + { files, objectId: input.objectId }, + handle.sessionId + ) + onDispatch?.('acknowledged') + try { + const { value } = await callFileInput(contents, handle, 'files') + if (!isRecordLike(value) || !Array.isArray(value.files)) { + throw new Error('Chromium did not confirm the uploaded files') + } + const uploaded = value.files.map((file: unknown) => { + if ( + !isRecordLike(file) || + typeof file.name !== 'string' || + typeof file.size !== 'number' || + !Number.isFinite(file.size) || + file.size < 0 + ) { + throw new Error('Chromium did not confirm the uploaded files') + } + return { name: file.name, size: file.size } + }) + return { files: uploaded } + } catch (error) { + return { readbackError: getErrorMessage(error) } + } + } finally { + await releaseRemoteObject(contents, input.objectId, handle.sessionId) + } +} + /** One half of a trusted key press (`Input.dispatchKeyEvent` params). */ export interface CdpKeyEvent { type: 'keyDown' | 'rawKeyDown' | 'keyUp' @@ -702,14 +948,44 @@ export async function moveMouse(contents: WebContents, x: number, y: number): Pr }) } +/** One trusted click gesture: which button, how many presses, and held modifiers. */ +export interface PointerClick { + button: 'left' | 'right' | 'middle' + clickCount: 1 | 2 | 3 + /** CDP modifier bitmask (Alt=1, Ctrl=2, Meta=4, Shift=8). */ + modifiers: number +} + +export const PRIMARY_CLICK: PointerClick = { button: 'left', clickCount: 1, modifiers: 0 } + +const BUTTON_MASKS: Record = { left: 1, right: 2, middle: 4 } +const agentContextClicks = new WeakMap() +const AGENT_CONTEXT_CLICK_WINDOW_MS = 1000 + +/** Consumes the single context menu echo expected from an agent right-click. */ +export function consumeAgentContextMenu(contents: WebContents): boolean { + const at = agentContextClicks.get(contents) + agentContextClicks.delete(contents) + if (at === undefined) return false + const elapsed = Date.now() - at + return elapsed >= 0 && elapsed < AGENT_CONTEXT_CLICK_WINDOW_MS +} + +/** A real user gesture supersedes an agent click whose page prevented its native menu. */ +export function clearAgentContextMenu(contents: WebContents): void { + agentContextClicks.delete(contents) +} + export async function clickAt( contents: WebContents, x: number, y: number, moveBeforePress = true, - clickCount = 1 + click: PointerClick = PRIMARY_CLICK ): Promise { if (moveBeforePress) await moveMouse(contents, x, y) + const { button, clickCount, modifiers } = click + const buttons = BUTTON_MASKS[button] let pressed = false try { // Set before awaiting: CDP can deliver the press and then lose/reject the @@ -719,20 +995,23 @@ export async function clickAt( // A multi-click is a sequence of press/release pairs with an increasing // clickCount — Blink synthesizes dblclick from the pair whose count is 2. for (let count = 1; count <= clickCount; count++) { + if (button === 'right') agentContextClicks.set(contents, Date.now()) await sendInput(contents, 'Input.dispatchMouseEvent', { type: 'mousePressed', x, y, - button: 'left', - buttons: 1, + button, + buttons, + modifiers, clickCount: count, }) await sendInput(contents, 'Input.dispatchMouseEvent', { type: 'mouseReleased', x, y, - button: 'left', + button, buttons: 0, + modifiers, clickCount: count, }) } @@ -746,8 +1025,9 @@ export async function clickAt( type: 'mouseReleased', x, y, - button: 'left', + button, buttons: 0, + modifiers, clickCount: 1, }).catch(() => {}) } diff --git a/apps/desktop/src/main/browser-agent/context-menu.test.ts b/apps/desktop/src/main/browser-agent/context-menu.test.ts index b3a011ed03c..2dafb8d0593 100644 --- a/apps/desktop/src/main/browser-agent/context-menu.test.ts +++ b/apps/desktop/src/main/browser-agent/context-menu.test.ts @@ -4,6 +4,7 @@ import { describe, expect, it, vi } from 'vitest' vi.mock('electron', () => import('@/test/electron-mock')) import { Menu, WebContentsView } from 'electron' +import { clickAt } from '@/main/browser-agent/cdp' import { attachAgentContextMenu, BASE_ZOOM_FACTOR, @@ -215,6 +216,51 @@ describe('buildAgentContextMenuTemplate', () => { describe('attachAgentContextMenu', () => { type ContextMenuListener = (event: unknown, params: Params) => void + it('suppresses one agent context menu and immediately allows the next menu', async () => { + const contents = new WebContentsView().webContents + attachAgentContextMenu(contents, { + addToChat: vi.fn(), + openTab: vi.fn(), + defaultZoomFactor: () => BASE_ZOOM_FACTOR, + }) + const listeners = vi.mocked(contents.on).mock.calls as unknown as [ + string, + ContextMenuListener, + ][] + const onContextMenu = listeners.find(([event]) => event === 'context-menu')![1] + await clickAt(contents, 10, 20, false, { button: 'right', clickCount: 1, modifiers: 0 }) + vi.mocked(Menu.buildFromTemplate).mockClear() + + onContextMenu({}, params()) + expect(Menu.buildFromTemplate).not.toHaveBeenCalled() + onContextMenu({}, params()) + expect(Menu.buildFromTemplate).toHaveBeenCalledTimes(1) + }) + + it.each(['mouseDown', 'keyDown', 'touchStart'])( + 'allows human %s when a page prevents the agent context menu event', + async (inputEvent) => { + const contents = new WebContentsView().webContents + attachAgentContextMenu(contents, { + addToChat: vi.fn(), + openTab: vi.fn(), + defaultZoomFactor: () => BASE_ZOOM_FACTOR, + }) + const listeners = vi.mocked(contents.on).mock.calls as unknown as [ + string, + (event: unknown, params: unknown) => void, + ][] + const onInput = listeners.find(([event]) => event === 'input-event')?.[1] + const onContextMenu = listeners.find(([event]) => event === 'context-menu')![1] + await clickAt(contents, 10, 20, false, { button: 'right', clickCount: 1, modifiers: 0 }) + vi.mocked(Menu.buildFromTemplate).mockClear() + + onInput?.({}, { type: inputEvent }) + onContextMenu({}, params()) + expect(Menu.buildFromTemplate).toHaveBeenCalledTimes(1) + } + ) + it('pops a menu built from the page that was right-clicked', () => { const contents = new WebContentsView().webContents vi.mocked(contents.navigationHistory.canGoBack).mockReturnValue(true) diff --git a/apps/desktop/src/main/browser-agent/context-menu.ts b/apps/desktop/src/main/browser-agent/context-menu.ts index 3ac8b19e82b..de9688e6592 100644 --- a/apps/desktop/src/main/browser-agent/context-menu.ts +++ b/apps/desktop/src/main/browser-agent/context-menu.ts @@ -17,6 +17,11 @@ import { resolveDesktopZoom } from '@sim/desktop-bridge' import type { ContextMenuParams, MenuItemConstructorOptions, WebContents } from 'electron' import { clipboard, Menu } from 'electron' +import { + clearAgentContextMenu, + consumeAgentContextMenu, + isDispatchingAgentInput, +} from '@/main/browser-agent/cdp' /** * Page-zoom ladder for the embedded browser, in Chromium's absolute zoom @@ -178,9 +183,18 @@ export function buildAgentContextMenuTemplate( return template } -/** Gives one agent tab its page menu. */ +/** Gives one agent tab its page menu; an agent right-click reaches only the page's own handlers. */ export function attachAgentContextMenu(contents: WebContents, host: AgentContextMenuHost): void { + contents.on('input-event', (_event, input) => { + if ( + ['mouseDown', 'keyDown', 'rawKeyDown', 'char', 'touchStart'].includes(input.type) && + !isDispatchingAgentInput(contents) + ) { + clearAgentContextMenu(contents) + } + }) contents.on('context-menu', (_event, params) => { + if (consumeAgentContextMenu(contents)) return const template = buildAgentContextMenuTemplate( params, { diff --git a/apps/desktop/src/main/browser-agent/driver.test.ts b/apps/desktop/src/main/browser-agent/driver.test.ts index 94ef24f1bba..a471258d051 100644 --- a/apps/desktop/src/main/browser-agent/driver.test.ts +++ b/apps/desktop/src/main/browser-agent/driver.test.ts @@ -1,9 +1,19 @@ import { BROWSER_TOOL_QUEUE_WAIT_TIMEOUT_MS } from '@sim/browser-protocol' import type { MenuItemConstructorOptions, WebContents } from 'electron' -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' vi.mock('electron', () => import('@/test/electron-mock')) +const { stageUploadFiles, saveDownloadToWorkspace } = vi.hoisted(() => ({ + stageUploadFiles: vi.fn(), + saveDownloadToWorkspace: vi.fn(), +})) +vi.mock('@/main/browser-agent/file-transfer', () => ({ + stageUploadFiles, + saveDownloadToWorkspace, + discardStagedUploads: vi.fn(async () => {}), +})) + import { BrowserWindow, Menu, type nativeImage } from 'electron' import * as cdp from '@/main/browser-agent/cdp' import * as driverModule from '@/main/browser-agent/driver' @@ -278,6 +288,9 @@ describe('executeTool', () => { code: -102, description: 'ERR_CONNECTION_REFUSED', } + vi.mocked(tab.view.webContents.loadURL).mockImplementationOnce(async () => { + session.notePageLoadStarted(tab.view.webContents) + }) vi.useFakeTimers() try { const result = driver.executeTool('chat-test', 'browser_reload', {}) @@ -1046,9 +1059,11 @@ describe('executeTool', () => { beforeMouse({}, { type: 'mouseDown' }) const openWindow = vi.mocked(source.setWindowOpenHandler).mock.calls[0]?.[0] as (details: { url: string - }) => { action: string } + }) => { action: string; createWindow?: (options: object) => unknown } - expect(openWindow({ url: 'https://user-popup.example/' })).toEqual({ action: 'deny' }) + const decision = openWindow({ url: 'https://user-popup.example/' }) + expect(decision.action).toBe('allow') + decision.createWindow?.({}) const popup = session.activeTab()?.view.webContents if (!popup) throw new Error('Expected user popup tab') expect(session.automationTab()?.view.webContents).toBe(source) @@ -1062,9 +1077,11 @@ describe('executeTool', () => { session.setAutomationActive(true) const openWindow = vi.mocked(source.setWindowOpenHandler).mock.calls[0]?.[0] as (details: { url: string - }) => { action: string } + }) => { action: string; createWindow?: (options: object) => unknown } - expect(openWindow({ url: 'https://agent-popup.example/' })).toEqual({ action: 'deny' }) + const decision = openWindow({ url: 'https://agent-popup.example/' }) + expect(decision.action).toBe('allow') + decision.createWindow?.({}) const popup = session.requireAutomationTab().view.webContents expect(session.activeTab()?.view.webContents).toBe(source) session.setAutomationActive(false) @@ -1113,7 +1130,14 @@ describe('executeTool', () => { | MenuItemConstructorOptions[] | undefined const labels = template?.filter((item) => item.type !== 'separator').map((item) => item.label) - expect(labels).toEqual(['Find in Page', 'Zoom (110%)', 'Import Passwords', 'Browser Settings']) + expect(labels).toEqual([ + 'Find in Page', + 'Zoom (110%)', + 'Fill Saved Password', + 'Passwords', + 'Import Passwords', + 'Browser Settings', + ]) const settings = template?.find((item) => item.label === 'Browser Settings') const openSettings = settings?.click as (() => void) | undefined @@ -1756,7 +1780,8 @@ describe('executeTool', () => { } ) - it('merges cross-origin structure and routes its refs through production frame isolation', async () => { + const frameUrls = ['https://ogs.google.com/u/0/widget/app', 'about:srcdoc', 'about:blank'] + it.each(frameUrls)('inspects and interacts with isolated frame %s', async (frameUrl) => { const win = new BrowserWindow() driver.initDriver( { @@ -1846,8 +1871,8 @@ describe('executeTool', () => { detached: false, isDestroyed: vi.fn(() => false), name: 'google-apps', - origin: 'https://ogs.google.com', - url: 'https://ogs.google.com/u/0/widget/app', + origin: frameUrl.startsWith('about:') ? 'null' : 'https://ogs.google.com', + url: frameUrl, parent: mainFrame, executeJavaScript: vi.fn((expression: string) => { if (isPageCall(expression, 'collectSnapshot')) { @@ -1918,7 +1943,11 @@ describe('executeTool', () => { const isolatedFrameEval = vi .spyOn(cdp, 'evaluateInIsolatedFrame') .mockImplementation((_contents, frame, expression) => { - if ((frame as unknown) === mainFrame) return mainFrame.executeJavaScript(expression) + if ((frame as unknown) === mainFrame) { + return isPageCall(expression, 'readChildFrameElementState') + ? mainFrame.executeJavaScript(expression) + : contents.executeJavaScript(expression) + } if ((frame as unknown) === crossFrame) return crossFrame.executeJavaScript(expression) return Promise.reject(new Error('unexpected isolated frame target')) }) @@ -2389,6 +2418,56 @@ describe('credential protection', () => { expect(cdpCalls(contents, 'Input.dispatchKeyEvent').length).toBeGreaterThan(0) }) + it('repeats a keystroke with a trusted down/up pair per press', async () => { + const contents = await openPage() + respondWith(contents, { activeElementSecrecy: 'safe', readActiveElementState: {} }) + + const result = await driver.executeTool('chat-test', 'browser_press_key', { + key: 'ArrowRight', + repeat: 3, + }) + + expect(result).toMatchObject({ ok: true, result: { pressed: 'ArrowRight', repeat: 3 } }) + const downs = cdpCalls(contents, 'Input.dispatchKeyEvent').filter( + ([, event]) => (event as { type?: string }).type === 'rawKeyDown' + ) + expect(downs).toHaveLength(3) + }) + + it('rejects an out-of-range repeat before dispatch', async () => { + const contents = await openPage() + + const result = await driver.executeTool('chat-test', 'browser_press_key', { + key: 'Tab', + repeat: 51, + }) + + expect(result).toMatchObject({ ok: false, error: expect.stringContaining('repeat must') }) + expect(cdpCalls(contents, 'Input.dispatchKeyEvent')).toHaveLength(0) + }) + + it('stops repeating once focus reaches a password field', async () => { + const contents = await openPage() + let secrecyChecks = 0 + vi.mocked(contents.executeJavaScript).mockImplementation(async (expression: string) => { + if (isPageCall(expression, 'activeElementSecrecy')) + return ++secrecyChecks > 1 ? 'secret' : 'safe' + if (isPageCall(expression, 'readActiveElementState')) return {} + return undefined + }) + + const result = await driver.executeTool('chat-test', 'browser_press_key', { + key: 'Tab', + repeat: 5, + }) + + expect(result).toMatchObject({ ok: false, error: expect.stringContaining('1 of 5 times') }) + const downs = cdpCalls(contents, 'Input.dispatchKeyEvent').filter( + ([, event]) => (event as { type?: string }).type === 'rawKeyDown' + ) + expect(downs).toHaveLength(1) + }) + it('sends the keystroke when nothing sensitive is focused', async () => { const contents = await openPage() respondWith(contents, { activeElementSecrecy: 'safe', readActiveElementState: {} }) @@ -2399,6 +2478,75 @@ describe('credential protection', () => { expect(cdpCalls(contents, 'Input.dispatchKeyEvent').length).toBeGreaterThan(0) }) + it.each(['cancelled', 'timed out'] as const)( + 'retains a completed action when its observation is %s', + async (stop) => { + const contents = await openPage() + respondWith(contents, { + activeElementSecrecy: 'safe', + readActiveElementState: {}, + readPageActionState: {}, + }) + const pageCall = vi.mocked(contents.executeJavaScript).getMockImplementation() + let releaseObservation: (value: unknown) => void = () => {} + const observation = new Promise((resolve) => { + releaseObservation = resolve + }) + let observing = false + vi.mocked(contents.executeJavaScript).mockImplementation((expression, ...args) => { + if (isPageCall(expression, 'collectSnapshot')) { + observing = true + return observation + } + return pageCall?.(expression, ...args) ?? Promise.resolve(undefined) + }) + vi.useFakeTimers() + try { + const timersBefore = vi.getTimerCount() + const pending = driver.executeTool( + 'chat-test', + 'browser_press_key', + { key: 'a', observe: {} }, + 'observed-action' + ) + await vi.advanceTimersByTimeAsync(200) + expect(observing).toBe(true) + + if (stop === 'cancelled') driver.cancelTool('chat-test', 'observed-action') + else + await vi.advanceTimersByTimeAsync(driver.browserToolWatchdogMs('browser_press_key', {})!) + + await expect(pending).resolves.toMatchObject({ + ok: true, + result: { + pressed: 'a', + trusted: true, + observation: { + ok: false, + doNotRetry: true, + note: expect.stringContaining('The action was dispatched'), + }, + }, + }) + await expect( + driver.executeTool('chat-test', 'browser_list_tabs', {}) + ).resolves.toMatchObject({ + ok: true, + }) + expect(vi.getTimerCount()).toBe(timersBefore) + expect( + cdpCalls(contents, 'Input.dispatchKeyEvent').filter(([, event]) => + ['keyDown', 'rawKeyDown'].includes((event as { type: string }).type) + ) + ).toHaveLength(1) + } finally { + releaseObservation({ outline: 'Late snapshot', refIds: [], nextElementId: 1 }) + await vi.advanceTimersByTimeAsync(0) + vi.useRealTimers() + } + } + ) + it('reports when a platform-mismatched shortcut produces no observable effect', async () => { const contents = await openPage() respondWith(contents, { @@ -3271,7 +3419,7 @@ describe('credential protection', () => { expect(first).toMatchObject({ ok: true, result: { - notices: [expect.stringContaining('alert dialog ("Heads up") which was auto-dismissed')], + notices: [expect.stringContaining('alert dialog ("Heads up") which was dismissed')], }, }) expect(second).not.toMatchObject({ result: { notices: expect.anything() } }) @@ -3485,6 +3633,535 @@ describe('credential protection', () => { expect(counts).toEqual([1, 2]) }) + it('dispatches right-clicks and modifier clicks through the same trusted gesture', async () => { + const contents = await openPage() + respondWith(contents, { + describePointTarget: { found: true, element: 'row "notes.pdf"', editable: false }, + readActiveElementState: {}, + readPageActionState: {}, + }) + + const result = await driver.executeTool('chat-test', 'browser_click_at', { + x: 10, + y: 20, + button: 'right', + modifiers: ['Shift'], + }) + + expect(result).toMatchObject({ ok: true, result: { dispatched: true } }) + const presses = cdpCalls(contents, 'Input.dispatchMouseEvent').filter( + ([, event]) => (event as { type?: string }).type === 'mousePressed' + ) + expect(presses.map(([, event]) => event)).toEqual([ + expect.objectContaining({ button: 'right', buttons: 2, modifiers: 8, clickCount: 1 }), + ]) + }) + + it('rejects unknown click buttons and modifiers before dispatch', async () => { + const contents = await openPage() + + const badButton = await driver.executeTool('chat-test', 'browser_click_at', { + x: 10, + y: 20, + button: 'back', + }) + const badModifier = await driver.executeTool('chat-test', 'browser_click_at', { + x: 10, + y: 20, + modifiers: ['Hyper'], + }) + + expect(badButton).toMatchObject({ ok: false, error: expect.stringContaining('button must be') }) + expect(badModifier).toMatchObject({ + ok: false, + error: expect.stringContaining('Unrecognized modifier'), + }) + expect(cdpCalls(contents, 'Input.dispatchMouseEvent')).toHaveLength(0) + }) + + it('answers a dialog opened by an action with that action dialog response only', async () => { + const contents = await openPage() + respondWith(contents, { + describePointTarget: { found: true, element: 'button "Delete"', editable: false }, + readActiveElementState: {}, + readPageActionState: {}, + }) + const listener = vi + .mocked(contents.debugger.on) + .mock.calls.find(([event]) => event === 'message')?.[1] as + | ((event: unknown, method: string, params: unknown, sessionId?: string) => void) + | undefined + const send = vi.mocked(contents.debugger.sendCommand) + const dispatch = send.getMockImplementation() + send.mockImplementation(async (method, params, ...rest) => { + if (method === 'Input.dispatchMouseEvent' && params?.type === 'mouseReleased') { + listener?.({}, 'Page.javascriptDialogOpening', { type: 'confirm', message: 'Delete?' }) + } + return dispatch?.(method, params, ...rest) + }) + + const accepted = await driver.executeTool('chat-test', 'browser_click_at', { + x: 10, + y: 20, + dialog: { accept: true }, + }) + const dismissed = await driver.executeTool('chat-test', 'browser_click_at', { x: 10, y: 20 }) + await driver.executeTool('chat-test', 'browser_list_tabs', {}) + + const answers = cdpCalls(contents, 'Page.handleJavaScriptDialog').map(([, answer]) => answer) + expect(answers).toEqual([{ accept: true }, { accept: false }]) + expect(accepted).toMatchObject({ ok: true }) + expect(dismissed).toMatchObject({ ok: true }) + }) + + it('dismisses background tab dialogs while the action target accepts its dialog', async () => { + const background = await openPage() + await driver.executeTool('chat-test', 'browser_open_tab', {}) + const contents = session.requireAutomationTab().view.webContents + vi.mocked(contents.getURL).mockReturnValue('https://example.com/target') + respondWith(contents, { + describePointTarget: { found: true, element: 'button "Delete"', editable: false }, + readActiveElementState: {}, + readPageActionState: {}, + }) + const listeners = [background, contents].map( + (tab) => + vi.mocked(tab.debugger.on).mock.calls.find(([event]) => event === 'message')?.[1] as + | ((event: unknown, method: string, params: unknown) => void) + | undefined + ) + vi.mocked(contents.debugger.sendCommand).mockImplementation(async (method, params) => { + if (method === 'Input.dispatchMouseEvent' && params?.type === 'mouseReleased') { + for (const listener of listeners) { + listener?.({}, 'Page.javascriptDialogOpening', { type: 'confirm', message: 'Delete?' }) + } + } + return {} + }) + + await expect( + driver.executeTool('chat-test', 'browser_click_at', { + x: 10, + y: 20, + dialog: { accept: true }, + }) + ).resolves.toMatchObject({ ok: true }) + + expect(cdpCalls(background, 'Page.handleJavaScriptDialog').map(([, answer]) => answer)).toEqual( + [{ accept: false }] + ) + expect(cdpCalls(contents, 'Page.handleJavaScriptDialog').map(([, answer]) => answer)).toEqual([ + { accept: true }, + ]) + }) + + it('rejects a malformed dialog response before dispatch', async () => { + const contents = await openPage() + + const result = await driver.executeTool('chat-test', 'browser_click_at', { + x: 10, + y: 20, + dialog: { accept: 'yes' }, + }) + + expect(result).toMatchObject({ ok: false, error: expect.stringContaining('dialog must be') }) + expect(cdpCalls(contents, 'Input.dispatchMouseEvent')).toHaveLength(0) + }) + + describe('file uploads', () => { + afterEach(() => vi.restoreAllMocks()) + + it('retains one input handle through staging and releases it after uploading', async () => { + const contents = await openPage() + respondWith(contents, { readPageActionState: {} }) + const input = { objectId: 'isolated-input', multiple: true, accept: '.pdf' } + const resolve = vi.spyOn(cdp, 'resolveFileInput').mockResolvedValue(input) + const setFiles = vi + .spyOn(cdp, 'setFileInputFiles') + .mockResolvedValue({ files: [{ name: 'a.pdf', size: 3 }] }) + const release = vi.spyOn(cdp, 'releaseFileInput').mockResolvedValue() + stageUploadFiles.mockResolvedValue(['/staged/a.pdf']) + + const result = await driver.executeTool( + 'chat-test', + 'browser_upload_file', + { elementId: 0, paths: ['files/a.pdf'] }, + 'call-upload' + ) + + expect(result).toMatchObject({ + ok: true, + result: { uploaded: [{ name: 'a.pdf', size: 3 }], effectObserved: true, accept: '.pdf' }, + }) + expect(resolve).toHaveBeenCalledWith( + contents, + contents.mainFrame, + expect.stringContaining('function resolveFileInputTarget(') + ) + expect(stageUploadFiles).toHaveBeenCalledWith( + expect.objectContaining({ toolCallId: 'call-upload', paths: ['files/a.pdf'] }) + ) + expect(setFiles).toHaveBeenCalledWith( + contents, + input, + ['/staged/a.pdf'], + expect.any(AbortSignal), + expect.any(Function) + ) + expect(release).toHaveBeenCalledWith(contents, input) + expect(resolve).toHaveBeenCalledTimes(1) + }) + + it('refuses several files for a single-file input and releases its handle without staging', async () => { + const contents = await openPage() + const input = { objectId: 'isolated-input', multiple: false } + vi.spyOn(cdp, 'resolveFileInput').mockResolvedValue(input) + const release = vi.spyOn(cdp, 'releaseFileInput').mockResolvedValue() + stageUploadFiles.mockClear() + + const result = await driver.executeTool( + 'chat-test', + 'browser_upload_file', + { elementId: 0, paths: ['files/a.pdf', 'files/b.pdf'] }, + 'call-single' + ) + + expect(result).toMatchObject({ + ok: false, + error: expect.stringContaining('accepts one file'), + }) + expect(stageUploadFiles).not.toHaveBeenCalled() + expect(release).toHaveBeenCalledWith(contents, input) + }) + + it.each(['staging', 'attachment'])( + 'releases the pinned input after %s fails before dispatch', + async (failure) => { + const contents = await openPage() + const input = { objectId: 'isolated-input', multiple: false } + vi.spyOn(cdp, 'resolveFileInput').mockResolvedValue(input) + const setFiles = vi + .spyOn(cdp, 'setFileInputFiles') + .mockRejectedValue(new Error('attachment failed')) + const release = vi.spyOn(cdp, 'releaseFileInput').mockResolvedValue() + stageUploadFiles.mockReset() + if (failure === 'staging') stageUploadFiles.mockRejectedValue(new Error('staging failed')) + else stageUploadFiles.mockResolvedValue(['/staged/a.pdf']) + + const result = await driver.executeTool( + 'chat-test', + 'browser_upload_file', + { elementId: 0, paths: ['files/a.pdf'] }, + 'call-failed' + ) + + expect(result).toEqual({ + ok: false, + error: expect.stringContaining(`${failure} failed`), + }) + if (failure === 'staging') expect(setFiles).not.toHaveBeenCalled() + expect(release).toHaveBeenCalledTimes(1) + expect(release).toHaveBeenCalledWith(contents, input) + } + ) + + it('reports an acknowledged upload with unavailable readback without inviting a retry', async () => { + const contents = await openPage() + const input = { objectId: 'isolated-input', multiple: false } + vi.spyOn(cdp, 'resolveFileInput').mockResolvedValue(input) + vi.spyOn(cdp, 'setFileInputFiles').mockResolvedValue({ + readbackError: 'Execution context destroyed', + }) + const release = vi.spyOn(cdp, 'releaseFileInput').mockResolvedValue() + stageUploadFiles.mockResolvedValue(['/staged/a.pdf']) + + const result = await driver.executeTool( + 'chat-test', + 'browser_upload_file', + { elementId: 0, paths: ['files/a.pdf'] }, + 'call-navigated' + ) + + expect(result).toMatchObject({ + ok: true, + result: { + dispatched: true, + observation: { ok: false, doNotRetry: true, error: 'Execution context destroyed' }, + }, + }) + expect(release).toHaveBeenCalledWith(contents, input) + }) + + it.each(['cancelled', 'timed out'] as const)( + 'does not retry a dispatched upload when acknowledgement is %s', + async (stop) => { + const contents = await openPage() + const input = { objectId: 'isolated-input', multiple: false } + vi.spyOn(cdp, 'resolveFileInput').mockResolvedValue(input) + stageUploadFiles.mockResolvedValue(['/staged/a.pdf']) + let acknowledge: () => void = () => {} + const acknowledgement = new Promise((resolve) => { + acknowledge = resolve + }) + const send = vi.mocked(contents.debugger.sendCommand) + send.mockImplementation(async (method, params) => { + if (method === 'Runtime.callFunctionOn') { + const mode = (params?.arguments as Array<{ value: string }>)[0].value + return mode === 'input' + ? { result: { objectId: 'original-input' } } + : { result: { value: { files: [{ name: 'a.pdf', size: 3 }] } } } + } + if (method === 'DOM.setFileInputFiles') await acknowledgement + return {} + }) + vi.useFakeTimers() + try { + const pending = driver.executeTool( + 'chat-test', + 'browser_upload_file', + { elementId: 0, paths: ['files/a.pdf'] }, + 'unacknowledged-upload' + ) + await vi.advanceTimersByTimeAsync(200) + expect(cdpCalls(contents, 'DOM.setFileInputFiles')).toHaveLength(1) + expect(cdpCalls(contents, 'Runtime.releaseObject')).toHaveLength(0) + + if (stop === 'cancelled') driver.cancelTool('chat-test', 'unacknowledged-upload') + else + await vi.advanceTimersByTimeAsync( + driver.browserToolWatchdogMs('browser_upload_file', {})! + ) + + await expect(pending).resolves.toMatchObject({ + ok: true, + result: { + outcomeUnknown: true, + doNotRetry: true, + }, + }) + await expect( + driver.executeTool('chat-test', 'browser_list_tabs', {}, 'after-unacknowledged-upload') + ).resolves.toMatchObject({ ok: true }) + } finally { + acknowledge() + await vi.advanceTimersByTimeAsync(200) + vi.useRealTimers() + } + expect(cdpCalls(contents, 'DOM.setFileInputFiles')).toHaveLength(1) + expect(cdpCalls(contents, 'Runtime.releaseObject').map(([, params]) => params)).toEqual([ + { objectId: 'original-input' }, + { objectId: 'isolated-input' }, + ]) + } + ) + + it.each(['cancelled', 'timed out'] as const)( + 'retains an acknowledged upload when readback is %s and releases its handle when readback settles', + async (stop) => { + const contents = await openPage() + respondWith(contents, { readPageActionState: {} }) + const input = { objectId: 'isolated-input', multiple: false } + vi.spyOn(cdp, 'resolveFileInput').mockResolvedValue(input) + let releaseReadback: (value: { files: Array<{ name: string; size: number }> }) => void = + () => {} + const readback = new Promise<{ files: Array<{ name: string; size: number }> }>( + (resolve) => { + releaseReadback = resolve + } + ) + const setFiles = vi + .spyOn(cdp, 'setFileInputFiles') + .mockImplementation(async (_contents, _handle, _files, _signal, onDispatch) => { + onDispatch?.('pending') + onDispatch?.('acknowledged') + return readback + }) + const release = vi.spyOn(cdp, 'releaseFileInput').mockResolvedValue() + stageUploadFiles.mockResolvedValue(['/staged/a.pdf']) + vi.useFakeTimers() + try { + const timersBefore = vi.getTimerCount() + const pending = driver.executeTool( + 'chat-test', + 'browser_upload_file', + { elementId: 0, paths: ['files/a.pdf'] }, + 'interrupted-upload' + ) + await vi.advanceTimersByTimeAsync(200) + expect(setFiles).toHaveBeenCalledTimes(1) + expect(release).not.toHaveBeenCalled() + const queued = driver.executeTool('chat-test', 'browser_list_tabs', {}, 'after-upload') + + if (stop === 'cancelled') driver.cancelTool('chat-test', 'interrupted-upload') + else + await vi.advanceTimersByTimeAsync( + driver.browserToolWatchdogMs('browser_upload_file', {})! + ) + + await expect(pending).resolves.toMatchObject({ + ok: true, + result: { + dispatched: true, + observation: { + ok: false, + doNotRetry: true, + note: expect.stringContaining('The action was dispatched'), + }, + }, + }) + await expect(queued).resolves.toMatchObject({ ok: true }) + expect(vi.getTimerCount()).toBe(timersBefore) + expect(setFiles).toHaveBeenCalledTimes(1) + expect(release).not.toHaveBeenCalled() + } finally { + releaseReadback({ files: [{ name: 'a.pdf', size: 3 }] }) + await vi.advanceTimersByTimeAsync(200) + vi.useRealTimers() + } + expect(release).toHaveBeenCalledExactlyOnceWith(contents, input) + expect(setFiles).toHaveBeenCalledTimes(1) + } + ) + + it.each(['cancelled', 'timed out'] as const)( + 'reports an unconfirmed upload when its acknowledgment is %s without replaying it or affecting queued work', + async (stop) => { + const contents = await openPage() + const input = { objectId: 'isolated-input', multiple: false } + vi.spyOn(cdp, 'resolveFileInput').mockResolvedValue(input) + let acknowledgeUpload: () => void = () => {} + const acknowledgment = new Promise((resolve) => { + acknowledgeUpload = resolve + }) + let appliedUploads = 0 + const setFiles = vi + .spyOn(cdp, 'setFileInputFiles') + .mockImplementation(async (_contents, _handle, _files, _signal, onDispatch) => { + onDispatch?.('pending') + appliedUploads++ + await acknowledgment + onDispatch?.('acknowledged') + return { files: [{ name: 'a.pdf', size: 3 }] } + }) + const release = vi.spyOn(cdp, 'releaseFileInput').mockResolvedValue() + stageUploadFiles.mockResolvedValue(['/staged/a.pdf']) + let releaseSnapshot: (value: unknown) => void = () => {} + const snapshot = new Promise((resolve) => { + releaseSnapshot = resolve + }) + let snapshotStarted = false + vi.mocked(contents.executeJavaScript).mockImplementation((expression) => { + if (isPageCall(expression, 'collectSnapshot')) { + snapshotStarted = true + return snapshot + } + return Promise.resolve({}) + }) + + vi.useFakeTimers() + try { + const timersBefore = vi.getTimerCount() + const pending = driver.executeTool( + 'chat-test', + 'browser_upload_file', + { elementId: 0, paths: ['files/a.pdf'] }, + 'unconfirmed-upload' + ) + await vi.advanceTimersByTimeAsync(200) + expect(appliedUploads).toBe(1) + const queued = driver.executeTool('chat-test', 'browser_snapshot', {}, 'next-snapshot') + + if (stop === 'cancelled') driver.cancelTool('chat-test', 'unconfirmed-upload') + else + await vi.advanceTimersByTimeAsync( + driver.browserToolWatchdogMs('browser_upload_file', {})! + ) + + const result = await pending + expect(result).toMatchObject({ + ok: true, + result: { + outcomeUnknown: true, + doNotRetry: true, + error: expect.any(String), + note: expect.stringContaining('The action may already have run'), + }, + }) + expect(result.result).not.toHaveProperty('dispatched') + await vi.advanceTimersByTimeAsync(0) + expect(snapshotStarted).toBe(true) + expect(release).not.toHaveBeenCalled() + + acknowledgeUpload() + await vi.advanceTimersByTimeAsync(200) + expect(release).toHaveBeenCalledExactlyOnceWith(contents, input) + expect(appliedUploads).toBe(1) + expect(setFiles).toHaveBeenCalledTimes(1) + expect(result.result).toMatchObject({ outcomeUnknown: true, doNotRetry: true }) + expect(result.result).not.toHaveProperty('dispatched') + + driver.cancelTool('chat-test', 'next-snapshot') + await expect(queued).resolves.toEqual({ + ok: false, + error: expect.stringContaining('cancelled'), + }) + expect(vi.getTimerCount()).toBe(timersBefore) + } finally { + acknowledgeUpload() + releaseSnapshot({ outline: 'Late snapshot', refIds: [], nextElementId: 1 }) + await vi.advanceTimersByTimeAsync(200) + vi.useRealTimers() + } + } + ) + }) + + it('validates upload paths before touching the page', async () => { + await openPage() + + const result = await driver.executeTool( + 'chat-test', + 'browser_upload_file', + { elementId: 0, paths: [] }, + 'call-empty' + ) + + expect(result).toMatchObject({ ok: false, error: expect.stringContaining('paths must list') }) + }) + + it('saves only a completed download, bound to its tool call', async () => { + await openPage() + const completed = vi + .spyOn(session, 'completedBrowserDownload') + .mockReturnValueOnce({ filename: 'report.csv', savePath: '/downloads/report.csv' }) + .mockReturnValueOnce(null) + saveDownloadToWorkspace.mockResolvedValue({ + path: 'files/report.csv', + name: 'report.csv', + size: 8, + }) + + const saved = await driver.executeTool( + 'chat-test', + 'browser_save_download', + { downloadId: 'd1' }, + 'call-save' + ) + const missing = await driver.executeTool( + 'chat-test', + 'browser_save_download', + { downloadId: 'd2' }, + 'call-save-2' + ) + + expect(saved).toMatchObject({ ok: true, result: { path: 'files/report.csv' } }) + expect(saveDownloadToWorkspace).toHaveBeenCalledWith( + expect.objectContaining({ toolCallId: 'call-save', filePath: '/downloads/report.csv' }) + ) + expect(missing).toMatchObject({ ok: false, error: expect.stringContaining('not a completed') }) + completed.mockRestore() + }) + it('refuses a coordinate click on a file input', async () => { const contents = await openPage() respondWith(contents, { @@ -3569,7 +4246,8 @@ describe('credential protection', () => { }) const isolatedFrameEval = vi .spyOn(cdp, 'evaluateInIsolatedFrame') - .mockImplementation((_contents, _frame, expression) => { + .mockImplementation((_contents, frame, expression) => { + if ((frame as unknown) === mainFrame) return contents.executeJavaScript(expression) if (isPageCall(expression, 'activeElementSecrecy')) return Promise.resolve('safe') if (isPageCall(expression, 'describeFocusedEditable')) { return Promise.resolve({ editable: true, kind: 'input' }) diff --git a/apps/desktop/src/main/browser-agent/driver.ts b/apps/desktop/src/main/browser-agent/driver.ts index f593a1884ed..9a049a8fcb0 100644 --- a/apps/desktop/src/main/browser-agent/driver.ts +++ b/apps/desktop/src/main/browser-agent/driver.ts @@ -20,6 +20,7 @@ import { BROWSER_DATA_KINDS, BROWSER_NAVIGATION_NATIVE_WATCHDOG_MS, BROWSER_TOOL_QUEUE_WAIT_TIMEOUT_MS, + BROWSER_UPLOAD_MAX_FILES, type BrowserDataKind, type BrowserKnownSessionsState, type BrowserPageState, @@ -39,10 +40,18 @@ import * as cdp from '@/main/browser-agent/cdp' import { steppedZoomFactor, zoomPercentOf } from '@/main/browser-agent/context-menu' import { ToolError } from '@/main/browser-agent/errors' import { + discardStagedUploads, + type LocalFileSource, + saveDownloadToWorkspace, + stageUploadFiles, +} from '@/main/browser-agent/file-transfer' +import { + cdpModifiers, comboTouchesClipboard, dispatchKeyCombo, KeyDispatchError, parseKeyCombo, + parseModifiers, } from '@/main/browser-agent/keyboard' import { BrowserKnownSessionRegistry } from '@/main/browser-agent/known-sessions' import { @@ -64,15 +73,22 @@ import { readPageActionState, readPageText, readSelectElementState, + resolveFileInputTarget, scrollPage, selectOptionInElement, setFocusedInputValue, typeIntoElement, } from '@/main/browser-agent/page-functions' +import { isPanelVisible, panelWindow } from '@/main/browser-agent/panel' +import { + withFailedPostActionObservation, + withPostActionObservation, +} from '@/main/browser-agent/post-action-observation' import * as session from '@/main/browser-agent/session' import { checkAgentUrl } from '@/main/browser-agent/url-guard' import { clearCredentials, fillCoordinator, initFillCoordinator } from '@/main/browser-credentials' import type { ConfigStore } from '@/main/config' +import { trackInputActivity } from '@/main/input-activity' const logger = createLogger('BrowserAgentDriver') @@ -97,7 +113,6 @@ export const BROWSER_TOOL_ADMISSION_LIMITS = Object.freeze({ const MAX_CROSS_ORIGIN_SNAPSHOT_FRAMES = 8 const MAX_CROSS_ORIGIN_SCAN_FRAMES = 32 const COMBINED_SNAPSHOT_LINE_CAP = 900 -const BROWSER_AGENT_ISOLATED_WORLD_ID = 1001 const BROWSER_WAIT_ELEMENT_STATES = [ 'attached', 'detached', @@ -122,6 +137,8 @@ function isBrowserWaitElementState(value: string): value is BrowserWaitElementSt type PageExecutionTarget = WebContents | WebFrameMain +type BrowserActionOutcome = { status: 'pending' } | { status: 'acknowledged'; result: unknown } + type FormField = | { elementId: number; kind: 'text'; text: string } | { elementId: number; kind: 'select'; value: string } @@ -202,6 +219,9 @@ export interface DriverCallbacks { } let driverCallbacks: DriverCallbacks | null = null +/** The app origin and authenticated session file transfers use; absent in headless tests. */ +let driverAppSession: session.BrowserAppSession | undefined +let driverLocalFiles: LocalFileSource | undefined let knownSessions: BrowserKnownSessionRegistry | null = null /** Kept so teardown can force its erasures past the settings write debounce. */ let configStore: ConfigStore | null = null @@ -215,6 +235,8 @@ interface DriverScopeState { /** Unique state generation so teardown cannot suffer an epoch ABA race. */ generation: number pendingNotices: string[] + /** Answer for JavaScript dialogs from the running action's target tab only. */ + dialogResponse: { contents: WebContents; response: cdp.DialogResponse } | null takeoverActive: boolean takeoverDone: boolean takeoverResponse: string | null @@ -266,6 +288,7 @@ function createDriverScopeState(): DriverScopeState { return { generation: nextDriverScopeGeneration++, pendingNotices: [], + dialogResponse: null, takeoverActive: false, takeoverDone: false, takeoverResponse: null, @@ -505,20 +528,28 @@ function pushTabsState(): void { /** Instruments a fresh tab: CDP dialog handling + page-state pushes. */ function instrumentTab(contents: WebContents): void { + trackInputActivity(contents) const scopeId = session.browserScopeIdForContents(contents) ?? session.getBrowserScopeId() const inScope = (fn: (...args: Args) => void) => (...args: Args) => session.withBrowserScope(scopeId, () => fn(...args)) - const callbacks = { + const callbacks: cdp.CdpCallbacks = { onDialog: inScope((dialog: cdp.PageDialog) => { recordNotice( - dialog.handled - ? `The page showed a ${dialog.type} dialog ("${dialog.message}") which was auto-dismissed.` - : `The page showed a ${dialog.type} dialog ("${dialog.message}") which could not be dismissed and may still be blocking the page.` + !dialog.handled + ? `The page showed a ${dialog.type} dialog ("${dialog.message}") which could not be answered and may still be blocking the page.` + : dialog.accepted + ? `The page showed a ${dialog.type} dialog ("${dialog.message}") which was accepted as requested.` + : `The page showed a ${dialog.type} dialog ("${dialog.message}") which was dismissed. If the task needs it accepted, repeat the same action with dialog: {"accept": true}.` ) }), + dialogResponse: () => + session.withBrowserScope(scopeId, () => { + const requested = driverScopeState().dialogResponse + return requested?.contents === contents ? requested.response : null + }), } void (async () => { let lastError: unknown @@ -607,9 +638,14 @@ export function initDriver( getMainWindow: () => BrowserWindow | null, config?: ConfigStore, persistence?: BrowserSessionPersistence, - downloadSettings?: session.BrowserDownloadSettings + downloadSettings?: session.BrowserDownloadSettings, + appSession?: session.BrowserAppSession, + localFiles?: LocalFileSource ): void { driverCallbacks = callbacks + driverAppSession = appSession + driverLocalFiles = localFiles + void discardStagedUploads().catch(() => {}) knownSessions = config ? new BrowserKnownSessionRegistry(config) : null configStore = config ?? null // The rest of this module's state is per-session too. Left behind, a new @@ -623,6 +659,35 @@ export function initDriver( // leaving the old chain head in place would queue the new session's first // tool call behind a promise nothing can ever settle. initFillCoordinator({ + pickerHost: (contents, bounds) => { + const scopeId = session.getActiveBrowserScopeId() + const window = panelWindow() + if (!scopeId || !window || window.isDestroyed() || !window.isVisible() || !isPanelVisible()) + return null + return session.withBrowserScope(scopeId, () => { + const tab = session.activeTab() + if (!tab || tab.view.webContents !== contents || !tab.view.getVisible()) return null + const panel = tab.view.getBounds() + const content = window.getContentBounds() + const zoom = contents.getZoomFactor() + if ( + bounds.x + bounds.width <= 0 || + bounds.y + bounds.height <= 0 || + bounds.x * zoom >= panel.width || + bounds.y * zoom >= panel.height + ) + return null + return { + window, + anchor: { + x: content.x + panel.x + bounds.x * zoom, + y: content.y + panel.y + bounds.y * zoom, + width: bounds.width * zoom, + height: bounds.height * zoom, + }, + } + }) + }, getActiveContents: (scopeId) => { const activeScopeId = session.getActiveBrowserScopeId() if (!activeScopeId) return null @@ -644,6 +709,7 @@ export function initDriver( }) session.initSession( { + onPanelGeometryChanged: () => fillCoordinator()?.dismissPicker(), onSessionClosed: () => { driverCallbacks?.onSessionStatus(false, session.getBrowserScopeId()) }, @@ -670,7 +736,8 @@ export function initDriver( }, getMainWindow, persistence, - downloadSettings + downloadSettings, + appSession ) } @@ -745,6 +812,14 @@ export function showToolbarMenu( ], }, { type: 'separator' }, + { + label: 'Fill Saved Password', + enabled: pageAvailable, + click: () => { + void fillCoordinator()?.showChooser(ownerWindow, anchor, resolved) + }, + }, + { label: 'Passwords', click: () => sendCommand('passwords') }, { label: 'Import Passwords', click: () => sendCommand('import') }, { type: 'separator' }, { label: 'Browser Settings', click: () => sendCommand('browser-settings') }, @@ -840,6 +915,7 @@ export function disposeBrowserScope(scopeId: string): void { const state = driverScopeStates.get(resolved) if (state) retireDriverScopeState(state) driverScopeStates.delete(resolved) + void discardStagedUploads(resolved).catch(() => {}) for (const [alias, target] of driverScopeAliases) { if (alias === resolved || resolveDriverScopeId(target) === resolved) { driverScopeAliases.delete(alias) @@ -925,6 +1001,7 @@ export async function clearBrowserProfile( export function closeBrowserSession(): void { retireAllDriverScopeStates() session.closeSession() + void discardStagedUploads().catch(() => {}) } function str(params: Record, key: string): string | undefined { @@ -959,7 +1036,9 @@ export function browserToolWatchdogMs( tool === 'browser_go_forward' || tool === 'browser_reload' || tool === 'browser_open_tab' || - tool === 'browser_switch_tab' + tool === 'browser_switch_tab' || + tool === 'browser_upload_file' || + tool === 'browser_save_download' ) { return BROWSER_NAVIGATION_NATIVE_WATCHDOG_MS } @@ -982,6 +1061,82 @@ function requireNum(params: Record, key: string): number { return value } +const POINTER_BUTTONS: ReadonlySet = new Set(['left', 'right', 'middle']) +/** Enough to walk a slider or list by keyboard in one call without flooding the page. */ +const MAX_KEY_REPEAT = 50 + +/** The optional click gesture shared by `browser_click` and `browser_click_at`. */ +function pointerClick(params: Record): cdp.PointerClick { + const button = str(params, 'button') ?? 'left' + if (!POINTER_BUTTONS.has(button)) throw new ToolError('button must be left, right, or middle.') + const clickCount = num(params, 'clickCount') ?? 1 + if (clickCount !== 1 && clickCount !== 2 && clickCount !== 3) { + throw new ToolError('clickCount must be 1 (click), 2 (double-click), or 3 (triple-click).') + } + const names = params.modifiers ?? [] + if (!Array.isArray(names) || names.length > 4 || names.some((name) => typeof name !== 'string')) { + throw new ToolError('modifiers must be a list of modifier names such as ["Shift"] or ["Mod"].') + } + return { + button: button as cdp.PointerClick['button'], + clickCount, + modifiers: cdpModifiers(parseModifiers(names)), + } +} + +/** The exact client tool call executing now; the app binds file transfers to it. */ +function requireActiveToolCallId(): string { + const toolCallId = driverScopeState().activeToolCallId + if (!toolCallId) throw new ToolError('This browser action has no tool call to authorize it.') + return toolCallId +} + +function uploadPaths(params: Record): string[] { + const paths = params.paths + if ( + !Array.isArray(paths) || + paths.length === 0 || + paths.length > BROWSER_UPLOAD_MAX_FILES || + paths.some((path) => typeof path !== 'string' || path.trim() === '' || path.length > 1024) + ) { + throw new ToolError( + `paths must list 1 to ${BROWSER_UPLOAD_MAX_FILES} file paths (files/…, uploads/…, or user-local/…).` + ) + } + return paths as string[] +} + +function isPrimaryClick(click: cdp.PointerClick): boolean { + return click.button === 'left' && click.clickCount === 1 && click.modifiers === 0 +} + +const DIALOG_ANSWERING_TOOLS: ReadonlySet = new Set([ + 'browser_click', + 'browser_click_at', + 'browser_press_key', + 'browser_type', +]) + +/** The optional answer for JavaScript dialogs an action opens; absent means dismiss. */ +function dialogResponse( + tool: BrowserToolName, + params: Record +): cdp.DialogResponse | null { + const value = params.dialog + if (value === undefined) return null + if ( + !DIALOG_ANSWERING_TOOLS.has(tool) || + !isRecordLike(value) || + typeof value.accept !== 'boolean' || + Object.keys(value).some((key) => key !== 'accept') + ) { + throw new ToolError( + 'dialog must be {accept: boolean} on browser_click, browser_click_at, browser_press_key, or browser_type.' + ) + } + return { accept: value.accept } +} + function browserElementStateMatches( targetState: Record, requestedState: BrowserWaitElementState @@ -1045,10 +1200,10 @@ function browserElementStateMatches( /** * Serializes a self-contained page function with JSON-encoded arguments. - * WebContents runs it in a persistent isolated world so page scripts cannot - * replace the ref registry or built-ins. Child WebFrameMain targets use a CDP - * isolated world mapped to that exact Chromium frame; test doubles without a - * frameTreeNodeId alone retain the legacy executeJavaScript fallback. + * Root and child frames share a persistent CDP isolated world, so snapshot refs + * and retained DOM handles have the same execution context. Page scripts cannot + * replace its registry or built-ins. Test doubles without an immutable frame id + * retain the executeJavaScript fallback. */ async function execInPage( target: PageExecutionTarget, @@ -1058,7 +1213,7 @@ async function execInPage( notAfter?: number ): Promise { const url = 'getURL' in target ? target.getURL() : target.url - if (url === '' || url === 'about:blank') { + if ('getURL' in target && (url === '' || url === 'about:blank')) { throw new ToolError( 'The active tab is blank. Call browser_navigate before using page inspection or interaction tools.' ) @@ -1069,19 +1224,9 @@ async function execInPage( ? `(Date.now() >= ${Math.floor(notAfter)} ? ({error: "expired"}) : ${invocation})` : invocation try { - if ( - 'executeJavaScriptInIsolatedWorld' in target && - typeof target.executeJavaScriptInIsolatedWorld === 'function' - ) { - return (await target.executeJavaScriptInIsolatedWorld( - BROWSER_AGENT_ISOLATED_WORLD_ID, - [{ code: expression }], - userGesture - )) as Result - } - if ('frameTreeNodeId' in target && typeof target.frameTreeNodeId === 'number') { - const contents = session.automationTab()?.view.webContents - const frame = target as WebFrameMain + const frame = 'getURL' in target ? target.mainFrame : target + if (frame && typeof frame.frameTreeNodeId === 'number') { + const contents = 'getURL' in target ? target : session.automationTab()?.view.webContents if ( !contents || contents.isDestroyed() || @@ -1091,8 +1236,7 @@ async function execInPage( } return (await cdp.evaluateInIsolatedFrame(contents, frame, expression, userGesture)) as Result } - // Unit-test WebFrame mocks omit Electron's immutable frameTreeNodeId. Real - // WebFrameMain instances always take the isolated CDP branch above. + /** Unit-test frame doubles omit Electron's immutable frameTreeNodeId. */ return (await target.executeJavaScript(expression, userGesture)) as Result } catch (error) { const message = getErrorMessage(error) @@ -1180,6 +1324,9 @@ const PASSWORD_REFUSAL = 'Refusing to act on a password field. Ask the user to enter their credentials in the visible browser, then take a fresh browser_snapshot.' /** Maps sentinel `{ error: ... }` results from injected functions to ToolErrors. */ +const FILE_INPUT_REFUSAL = + 'Clicking a file input opens a native file chooser the browser agent cannot complete. Use browser_upload_file with this element and the files to attach.' + function unwrapPageResult(result: unknown): unknown { if (isRecordLike(result) && 'error' in result) { const code = (result as { error: string }).error @@ -1196,8 +1343,16 @@ function unwrapPageResult(result: unknown): unknown { throw new ToolError(PASSWORD_REFUSAL) } if (code === 'file-input') { + throw new ToolError(FILE_INPUT_REFUSAL) + } + if (code === 'no-file-input') { throw new ToolError( - 'Refusing to click a file input because it opens a native chooser the browser agent cannot inspect or complete. Ask the user to upload the file themselves.' + 'No file input belongs to that element. Target the file input, its label, or the upload button or drop zone that contains it.' + ) + } + if (code === 'ambiguous-file-input') { + throw new ToolError( + 'That element contains several file inputs. Target the specific upload control or its file input.' ) } if (code === 'expired') { @@ -1359,8 +1514,17 @@ async function navigationResult( completion = waitForLoadComplete(contents, NAVIGATION_TIMEOUT_MS) ): Promise> { await completion + const target = session.navigationTarget(contents) + if (target !== contents) { + contents = target + await waitForLoadComplete(contents, NAVIGATION_TIMEOUT_MS) + } await sleep(NAVIGATION_SETTLE_MS) if (contents.isDestroyed()) throw new ToolError('The tab was closed during navigation.') + const issue = session.pageIssueForContents(contents) + if (issue?.kind === 'load-error') { + throw new ToolError(`The page failed to load (${issue.description || issue.code}).`) + } return { url: contents.getURL(), title: contents.getTitle() } } @@ -1368,6 +1532,7 @@ async function loadAgentCheckedUrlAndGetResult( contents: WebContents, url: string ): Promise> { + contents = session.tabForNavigation(contents, url, { agentOwned: true }) session.prepareExplicitNavigation(contents) if (contents.isDestroyed()) { throw new ToolError('The tab was closed before navigation could start.') @@ -1383,13 +1548,16 @@ async function loadAgentCheckedUrlAndGetResult( candidate.code === 'ERR_ABORTED' || candidate.errno === -3 || /ERR_ABORTED/i.test(getErrorMessage(error)) - if (!routineAbort) { + if (!routineAbort && session.navigationTarget(contents) === contents) { throw new ToolError(`The page failed to load (${getErrorMessage(error)}).`) } // Redirect/client-abort races can reject the initiating promise even // though a replacement navigation committed. Accept only concrete URL // progress; an unchanged URL is a real failure, not a successful load. await sleep(100) + contents = session.navigationTarget(contents) + if (!contents.getURL() && contents.isLoading()) + await waitForLoadComplete(contents, NAVIGATION_TIMEOUT_MS) if (!contents.getURL() || contents.getURL() === beforeUrl) { throw new ToolError(`The navigation was aborted (${getErrorMessage(error)}).`) } @@ -1776,6 +1944,11 @@ function pageEffect( } } +/** Inline sandboxed previews are real browser frames even though they have no HTTP URL. */ +function isInspectableFrameUrl(url: string): boolean { + return /^https?:\/\//i.test(url) || /^about:(?:srcdoc|blank)(?:[?#]|$)/i.test(url) +} + function crossOriginBoundaryFrames(contents: WebContents): WebFrameMain[] { const mainFrame = contents.mainFrame if (!mainFrame) return [] @@ -1783,8 +1956,8 @@ function crossOriginBoundaryFrames(contents: WebContents): WebFrameMain[] { if (sameWebFrame(frame, mainFrame) || frame.detached || frame.isDestroyed() || !frame.parent) { return false } - if (!/^https?:\/\//i.test(frame.url)) return false - return frame.origin !== frame.parent.origin + if (!isInspectableFrameUrl(frame.url)) return false + return frame.origin === 'null' || frame.origin !== frame.parent.origin }) } @@ -1897,7 +2070,7 @@ async function visibleFrameTargets( !frame.detached && !frame.isDestroyed() && Boolean(frame.parent) && - /^https?:\/\//i.test(frame.url) + isInspectableFrameUrl(frame.url) ) .slice(0, MAX_CROSS_ORIGIN_SCAN_FRAMES) const targets: WebFrameMain[] = [] @@ -2281,7 +2454,8 @@ async function executeToolInner( assertCurrentExecution: () => void, executionDeadline: number | undefined, invocationEpoch: number, - signal?: AbortSignal + signal?: AbortSignal, + onActionOutcome?: (outcome: BrowserActionOutcome) => void ): Promise { switch (tool) { case 'browser_navigate': { @@ -2321,7 +2495,10 @@ async function executeToolInner( // A failed snapshot (browser-internal page, injection error) should not // fail the open itself — the page is on screen either way. assertCurrentExecution() - const snapshot = await captureSnapshot(contents, executionDeadline).catch(() => null) + const snapshot = await captureSnapshot( + session.requireAutomationTab().view.webContents, + executionDeadline + ).catch(() => null) return snapshot === null ? { ...nav, note: 'The page loaded but a snapshot could not be captured.' } : { ...nav, snapshot } @@ -2368,12 +2545,12 @@ async function executeToolInner( } } assertCurrentExecution() - const tab = session.addAutomationTab() + const tab = session.addAutomationTab(url) const contents = tab.view.webContents if (url) { assertCurrentExecution() const result = await loadAgentCheckedUrlAndGetResult(contents, url) - return { tabId: tab.id, ...result } + return { tabId: session.requireAutomationTab().id, ...result } } return { tabId: tab.id, url: '', title: '' } } @@ -2415,6 +2592,80 @@ async function executeToolInner( return session.getBrowserDownloadsState(session.getBrowserScopeId()) } + case 'browser_save_download': { + const download = session.completedBrowserDownload( + session.getBrowserScopeId(), + requireStr(params, 'downloadId') + ) + if (!download) { + throw new ToolError( + 'That download is not a completed file. Check browser_list_downloads for its id and state.' + ) + } + assertCurrentExecution() + return saveDownloadToWorkspace({ + appSession: driverAppSession, + toolCallId: requireActiveToolCallId(), + filePath: download.savePath, + filename: download.filename, + signal, + }) + } + + case 'browser_upload_file': { + const contents = session.requireAutomationTab().view.webContents + const elementId = requireNum(params, 'elementId') + const paths = uploadPaths(params) + const target = pageTargetForElement(contents, elementId) + assertCurrentExecution() + const frame = 'getURL' in target ? target.mainFrame : target + const expression = `(${String(resolveFileInputTarget)})(${elementId})` + const input = await cdp.resolveFileInput(contents, frame, expression) + let attachment: Awaited> + try { + assertCurrentExecution() + if (!input.multiple && paths.length > 1) { + throw new ToolError('That file input accepts one file. Upload the files one at a time.') + } + const files = await stageUploadFiles({ + scopeId: session.getBrowserScopeId(), + toolCallId: requireActiveToolCallId(), + paths, + appSession: driverAppSession, + localFiles: driverLocalFiles, + signal, + }) + assertCurrentExecution() + assertActiveContents(contents) + attachment = await cdp.setFileInputFiles(contents, input, files, signal, (status) => { + onActionOutcome?.( + status === 'pending' ? { status } : { status, result: { dispatched: true } } + ) + }) + } finally { + await cdp.releaseFileInput(contents, input) + } + if ('readbackError' in attachment) { + return withFailedPostActionObservation({ dispatched: true }, attachment.readbackError) + } + const result = { + dispatched: true, + uploaded: attachment.files, + effectObserved: attachment.files.length === paths.length, + ...(input.accept ? { accept: input.accept } : {}), + } + try { + await sleep(150) + const afterPage = await pageActionState(contents) + return { + ...result, + dialogs: Array.isArray(afterPage.dialogs) ? afterPage.dialogs.map(String) : [], + } + } catch (error) { + return withFailedPostActionObservation(result, error) + } + } + case 'browser_wait_for': { const text = str(params, 'text') const urlContains = str(params, 'urlContains') @@ -2738,6 +2989,7 @@ async function executeToolInner( const clickedTab = session.requireAutomationTab() const contents = clickedTab.view.webContents const elementId = requireNum(params, 'elementId') + const click = pointerClick(params) const target = pageTargetForElement(contents, elementId) const targetFrame = frameExecutionTarget(target, contents) let trusted = false @@ -2896,7 +3148,7 @@ async function executeToolInner( try { assertCurrentExecution() assertElementActionCurrent(contents, elementId, target) - await cdp.clickAt(contents, x, y, false) + await cdp.clickAt(contents, x, y, false, click) trusted = true activation = 'native-pointer' } catch (error) { @@ -2936,7 +3188,7 @@ async function executeToolInner( try { assertCurrentExecution() assertElementActionCurrent(contents, elementId, target) - await cdp.clickAt(contents, finalTopPoint.x, finalTopPoint.y, false) + await cdp.clickAt(contents, finalTopPoint.x, finalTopPoint.y, false, click) trusted = true activation = 'native-pointer' prepared = finalSurface @@ -2951,6 +3203,11 @@ async function executeToolInner( ) } } else { + if (!isPrimaryClick(click)) { + throw new ToolError( + 'This framed control has no reliable pointer position, so only a plain left click can activate it. Use browser_screenshot and browser_click_at for other buttons, click counts, or modifiers.' + ) + } const activationKey = prepared.activationKey if (prepared.focusSucceeded === true && typeof activationKey === 'string') { // Observation probes above give the app time to open a modal, move a @@ -3652,6 +3909,10 @@ async function executeToolInner( case 'browser_press_key': { const requestedKey = requireStr(params, 'key') const combo = parseKeyCombo(requestedKey) + const repeat = num(params, 'repeat') ?? 1 + if (!Number.isInteger(repeat) || repeat < 1 || repeat > MAX_KEY_REPEAT) { + throw new ToolError(`repeat must be a whole number from 1 to ${MAX_KEY_REPEAT}.`) + } const contents = session.requireAutomationTab().view.webContents const pressedNavigationEpoch = navigationEpoch(contents) let target: PageExecutionTarget = focusedPageTarget(contents) @@ -3710,12 +3971,28 @@ async function executeToolInner( } let trusted = true let fallbackState: Record = {} + let pressesDispatched = 0 try { - assertCurrentExecution() - assertActiveContents(contents, pressedNavigationEpoch) - assertFocusedTargetUnchanged(contents, target, pressedFrameEpoch) - await dispatchKeyCombo(contents, combo) + for (; pressesDispatched < repeat; pressesDispatched++) { + // A repeated key must never carry on into a credential field focus has reached. + if ( + pressesDispatched > 0 && + (await execInPage(target, activeElementSecrecy, []).catch(() => 'opaque')) === 'secret' + ) { + throw new ToolError(PASSWORD_REFUSAL) + } + assertCurrentExecution() + assertActiveContents(contents, pressedNavigationEpoch) + assertFocusedTargetUnchanged(contents, target, pressedFrameEpoch) + await dispatchKeyCombo(contents, combo) + } } catch (error) { + if (pressesDispatched > 0) { + throw new ToolError( + `Pressed ${requestedKey} ${pressesDispatched} of ${repeat} times before the page stopped accepting it (${getErrorMessage(error)}). Inspect the page before continuing.` + ) + } + if (error instanceof ToolError) throw error if (error instanceof KeyDispatchError && error.keyDownDispatched) { throw new ToolError( 'The key-down may have reached the page but dispatch did not complete. The keystroke was not retried to avoid a duplicate action; inspect the page before continuing.' @@ -3815,6 +4092,7 @@ async function executeToolInner( return { ...fallbackState, pressed: requestedKey, + ...(repeat > 1 ? { repeat: trusted ? repeat : 1 } : {}), primaryModifier: process.platform === 'darwin' ? 'Cmd' : 'Control', trusted, ...state, @@ -4206,10 +4484,7 @@ async function executeToolInner( const contents = clickedTab.view.webContents const x = requireNum(params, 'x') const y = requireNum(params, 'y') - const clickCount = num(params, 'clickCount') ?? 1 - if (![1, 2, 3].includes(clickCount)) { - throw new ToolError('clickCount must be 1 (click), 2 (double-click), or 3 (triple-click).') - } + const click = pointerClick(params) const clickNavigationEpoch = navigationEpoch(contents) const urlAtDispatch = contents.getURL() assertCurrentExecution() @@ -4223,16 +4498,14 @@ async function executeToolInner( ) } if (pointTarget.fileInput === true) { - throw new ToolError( - 'Refusing to click a file input because it opens a native chooser the browser agent cannot inspect or complete. Ask the user to upload the file themselves.' - ) + throw new ToolError(FILE_INPUT_REFUSAL) } const beforePage = await pageActionState(contents, true) const beforeElement = await activeElementState(contents) assertCurrentExecution() assertActiveContents(contents, clickNavigationEpoch) try { - await cdp.clickAt(contents, x, y, true, clickCount) + await cdp.clickAt(contents, x, y, true, click) } catch (error) { const rescued = navigationRescue(contents, clickNavigationEpoch, urlAtDispatch, { trusted: true, @@ -4282,7 +4555,7 @@ async function executeToolInner( trusted: true, activation: 'native-pointer', clickedAt: { x, y }, - clickCount, + clickCount: click.clickCount, target: pointTarget.element, targetCursor: pointTarget.cursor, effectObserved, @@ -4667,6 +4940,10 @@ export async function executeTool( session.setAutomationActive(true) } try { + const response = dialogResponse(tool, params) + state.dialogResponse = response + ? { contents: session.requireAutomationTab().view.webContents, response } + : null const executionEpoch = ++state.toolExecutionEpoch const watchdogMs = browserToolWatchdogMs(tool, params) const executionDeadline = watchdogMs === null ? undefined : Date.now() + watchdogMs @@ -4675,29 +4952,69 @@ export async function executeTool( throw new ToolError('This browser action expired before it could dispatch input.') } } - const execution = executeToolInner( + let actionOutcome: BrowserActionOutcome | undefined + const execution = withPostActionObservation( tool, params, - assertCurrentExecution, - executionDeadline, - invocationEpoch, - executionController.signal + async (actionParams) => { + const result = await executeToolInner( + tool, + actionParams, + assertCurrentExecution, + executionDeadline, + invocationEpoch, + executionController.signal, + (outcome) => { + actionOutcome = outcome + } + ) + if (params.observe !== undefined) actionOutcome = { status: 'acknowledged', result } + return result + }, + (query) => + executeToolInner( + query === undefined ? 'browser_snapshot' : 'browser_find', + query === undefined ? {} : { query }, + assertCurrentExecution, + executionDeadline, + invocationEpoch, + executionController.signal + ), + assertCurrentExecution ) + const cancellableExecution = Promise.race([execution, cancellation]) const guardedExecution = watchdogMs === null - ? execution - : raceAgainstWatchdog(execution, watchdogMs, () => { + ? cancellableExecution + : raceAgainstWatchdog(cancellableExecution, watchdogMs, () => { executionController.abort() if (state.toolExecutionEpoch === executionEpoch) state.toolExecutionEpoch++ if ( tool === 'browser_snapshot' || tool === 'browser_open_url' || - tool === 'browser_find' + tool === 'browser_find' || + params.observe !== undefined ) { invalidateSnapshot(state) } }) - const result = withNotices(await Promise.race([guardedExecution, cancellation])) + let observedResult: unknown + try { + observedResult = await guardedExecution + } catch (error) { + if (!actionOutcome) throw error + invalidateSnapshot(state) + observedResult = + actionOutcome.status === 'pending' + ? { + outcomeUnknown: true, + doNotRetry: true, + error: getErrorMessage(error), + note: 'The action may already have run. Inspect the page before repeating it.', + } + : withFailedPostActionObservation(actionOutcome.result, error) + } + const result = withNotices(observedResult) logger.info('Browser tool completed', { tool, toolCallId, @@ -4707,6 +5024,7 @@ export async function executeTool( }) return result } finally { + state.dialogResponse = null executionController.abort() if (keepHiddenPageActive && !state.disposed) { session.setAutomationActive(false) @@ -4731,7 +5049,12 @@ export async function executeTool( // The watchdog cannot cancel an in-flight renderer promise. Invalidate its // capture token before releasing the queue so a late snapshot cannot // overwrite refs belonging to a newer tab or snapshot. - if (tool === 'browser_snapshot' || tool === 'browser_open_url' || tool === 'browser_find') { + if ( + tool === 'browser_snapshot' || + tool === 'browser_open_url' || + tool === 'browser_find' || + params.observe !== undefined + ) { invalidateSnapshot(state) } const message = String(sanitizeBrowserResult(getErrorMessage(error), undefined, 0, 'error')) @@ -4823,7 +5146,11 @@ export async function handlePanelAction( if (action.action === 'navigate') { if (typeof action.url === 'string' && /^https?:\/\//i.test(action.url)) { session.claimActiveTabForUser() - const contents = session.ensureTab().view.webContents + const contents = session.tabForNavigation( + session.ensureTab().view.webContents, + action.url, + { agentOwned: false } + ) session.prepareExplicitNavigation(contents) void contents.loadURL(action.url).catch(() => {}) } diff --git a/apps/desktop/src/main/browser-agent/file-transfer.test.ts b/apps/desktop/src/main/browser-agent/file-transfer.test.ts new file mode 100644 index 00000000000..16c648da340 --- /dev/null +++ b/apps/desktop/src/main/browser-agent/file-transfer.test.ts @@ -0,0 +1,306 @@ +import { existsSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs' +import { open, rename, rm, truncate } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { BROWSER_FILE_TRANSFER_MAX_BYTES } from '@sim/browser-protocol' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('electron', () => import('@/test/electron-mock')) + +import { app, type Session } from 'electron' +import { + discardStagedUploads, + saveDownloadToWorkspace, + stageUploadFiles, +} from '@/main/browser-agent/file-transfer' +import { LocalFilesystemService } from '@/main/local-filesystem' + +let temp: string +const signal = new AbortController().signal + +function appSession(fetch: (url: string, init: RequestInit) => Promise) { + return { origin: 'https://sim.test', session: { fetch } as unknown as Session } +} + +beforeEach(() => { + temp = mkdtempSync(join(tmpdir(), 'sim-file-transfer-')) + vi.mocked(app.getPath).mockReturnValue(temp) +}) + +afterEach(async () => { + await rm(temp, { recursive: true, force: true }) +}) + +describe('stageUploadFiles', () => { + it('stages workspace files from the claimed call and local files by copy, in order', async () => { + const local = join(temp, 'granted.txt') + writeFileSync(local, 'local bytes') + const fetch = vi.fn( + async () => + new Response('workspace bytes', { + headers: { 'content-disposition': "attachment; filename*=UTF-8''Q3%20plan.pdf" }, + }) + ) + + const staged = await stageUploadFiles({ + scopeId: 'chat-1', + toolCallId: 'call-1', + paths: ['files/Q3 plan.pdf', 'user-local/Docs--m1/granted.txt'], + appSession: appSession(fetch), + localFiles: { + resolveGrantedFile: vi.fn(async () => ({ + handle: await open(local, 'r'), + name: 'granted.txt', + size: 11, + })), + }, + signal, + }) + + expect(fetch).toHaveBeenCalledWith('https://sim.test/api/desktop/tool/file', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ toolCallId: 'call-1', index: 0 }), + signal, + }) + expect(staged.map((path) => path.split('/').pop())).toEqual(['Q3 plan.pdf', 'granted.txt']) + expect(readFileSync(staged[0], 'utf8')).toBe('workspace bytes') + expect(readFileSync(staged[1], 'utf8')).toBe('local bytes') + expect(staged.every((path) => path.startsWith(join(temp, 'sim-browser-uploads')))).toBe(true) + }) + + it('keeps a hostile server file name inside the staging directory', async () => { + const fetch = vi.fn( + async () => + new Response('x', { + headers: { 'content-disposition': 'attachment; filename="../../evil"' }, + }) + ) + + const [staged] = await stageUploadFiles({ + scopeId: '../scope', + toolCallId: '../call', + paths: ['files/a'], + appSession: appSession(fetch), + localFiles: undefined, + signal, + }) + + expect(staged.startsWith(join(temp, 'sim-browser-uploads'))).toBe(true) + expect(staged.endsWith('/evil')).toBe(true) + }) + + it('reports an app refusal and discards partial staging', async () => { + const fetch = vi.fn(async () => Response.json({ error: 'File not found' }, { status: 404 })) + + await expect( + stageUploadFiles({ + scopeId: 'chat-1', + toolCallId: 'call-2', + paths: ['files/missing.pdf'], + appSession: appSession(fetch), + localFiles: undefined, + signal, + }) + ).rejects.toThrow('Could not read that workspace file (File not found).') + }) + + it('refuses a workspace file over the transfer ceiling', async () => { + const oversized = new Uint8Array(BROWSER_FILE_TRANSFER_MAX_BYTES + 1) + const fetch = vi.fn(async () => new Response(oversized)) + + await expect( + stageUploadFiles({ + scopeId: 'chat-1', + toolCallId: 'call-3', + paths: ['files/huge.bin'], + appSession: appSession(fetch), + localFiles: undefined, + signal, + }) + ).rejects.toThrow(/upload limit/) + }) + + it('bounds a local file that grows after validation and discards every staged file', async () => { + const local = join(temp, 'growing.bin') + writeFileSync(local, 'initial') + const handle = await open(local, 'r') + const size = (await handle.stat()).size + await truncate(local, BROWSER_FILE_TRANSFER_MAX_BYTES + 1) + + try { + await expect( + stageUploadFiles({ + scopeId: 'chat-growing', + toolCallId: 'call-growing', + paths: ['files/first.txt', 'user-local/Docs--m1/growing.bin'], + appSession: appSession(async () => new Response('already staged')), + localFiles: { + resolveGrantedFile: async () => ({ handle, name: 'growing.bin', size }), + }, + signal, + }) + ).rejects.toThrow(/upload limit/) + expect(existsSync(join(temp, 'sim-browser-uploads/chat-growing/call-growing'))).toBe(false) + expect(handle.fd).toBe(-1) + } finally { + await handle.close() + } + }) + + it('copies the pinned local file when its path is replaced after validation', async () => { + const local = join(temp, 'granted.txt') + writeFileSync(local, 'granted bytes') + const handle = await open(local, 'r') + await rename(local, join(temp, 'original.txt')) + writeFileSync(local, 'replacement bytes') + + try { + const [staged] = await stageUploadFiles({ + scopeId: 'chat-pinned', + toolCallId: 'call-pinned', + paths: ['user-local/Docs--m1/granted.txt'], + appSession: undefined, + localFiles: { + resolveGrantedFile: async () => ({ handle, name: 'granted.txt', size: 13 }), + }, + signal, + }) + + expect(readFileSync(staged, 'utf8')).toBe('granted bytes') + expect(handle.fd).toBe(-1) + } finally { + await handle.close() + } + }) + + it('closes a local handle and discards staging when the upload is cancelled', async () => { + const local = join(temp, 'cancelled.txt') + writeFileSync(local, 'local bytes') + const handle = await open(local, 'r') + const controller = new AbortController() + controller.abort() + + try { + await expect( + stageUploadFiles({ + scopeId: 'chat-cancelled', + toolCallId: 'call-cancelled', + paths: ['user-local/Docs--m1/cancelled.txt'], + appSession: undefined, + localFiles: { + resolveGrantedFile: async () => ({ handle, name: 'cancelled.txt', size: 11 }), + }, + signal: controller.signal, + }) + ).rejects.toThrow(/aborted/) + expect(handle.fd).toBe(-1) + expect(existsSync(join(temp, 'sim-browser-uploads/chat-cancelled/call-cancelled'))).toBe( + false + ) + } finally { + await handle.close() + } + }) + + it.runIf(process.platform !== 'win32')( + 'preserves a granted POSIX backslash filename when staging', + async () => { + const name = 'report\\draft.txt' + writeFileSync(join(temp, name), 'draft bytes') + const localFiles = new LocalFilesystemService({ chooseDirectory: async () => temp }) + const grant = await localFiles.handle({ operation: 'mount_directory' }) + if (!grant.ok || !('mount' in grant.data) || !grant.data.mount) { + throw new Error('Expected a granted directory') + } + const mount = grant.data.mount + + try { + const [staged] = await stageUploadFiles({ + scopeId: 'chat-backslash', + toolCallId: 'call-backslash', + paths: [ + `user-local/${encodeURIComponent(mount.name)}--${mount.id}/${encodeURIComponent(name)}`, + ], + appSession: undefined, + localFiles, + signal, + }) + + expect(staged).toBe(join(temp, 'sim-browser-uploads/chat-backslash/call-backslash/0', name)) + expect(readFileSync(staged, 'utf8')).toBe('draft bytes') + } finally { + localFiles.close() + } + } + ) + + it('requires a granted-folder source for user-local paths', async () => { + await expect( + stageUploadFiles({ + scopeId: 'chat-1', + toolCallId: 'call-4', + paths: ['user-local/Docs--m1/a.txt'], + appSession: undefined, + localFiles: undefined, + signal, + }) + ).rejects.toThrow('Local folders are unavailable') + }) + + it('discards a scope staging directory', async () => { + const fetch = vi.fn(async () => new Response('x')) + const [staged] = await stageUploadFiles({ + scopeId: 'chat-9', + toolCallId: 'call-9', + paths: ['files/a.txt'], + appSession: appSession(fetch), + localFiles: undefined, + signal, + }) + + await discardStagedUploads('chat-9') + + expect(() => readFileSync(staged)).toThrow() + }) +}) + +describe('saveDownloadToWorkspace', () => { + it('stores the download under its claimed call and returns the workspace path', async () => { + const file = join(temp, 'report.csv') + writeFileSync(file, 'a,b\n1,2\n') + const fetch = vi.fn(async () => + Response.json({ path: 'files/report.csv', name: 'report.csv', size: 8 }) + ) + + const saved = await saveDownloadToWorkspace({ + appSession: appSession(fetch), + toolCallId: 'call-5', + filePath: file, + filename: 'report.csv', + signal, + }) + + expect(saved).toEqual({ path: 'files/report.csv', name: 'report.csv', size: 8 }) + const [url, init] = fetch.mock.calls[0] as unknown as [string, RequestInit] + expect(url).toBe('https://sim.test/api/desktop/tool/file?toolCallId=call-5&name=report.csv') + expect(init.method).toBe('PUT') + expect(await new Response(init.body).text()).toBe('a,b\n1,2\n') + }) + + it('surfaces the app error message', async () => { + const file = join(temp, 'report.csv') + writeFileSync(file, 'x') + const fetch = vi.fn(async () => Response.json({ error: 'Storage limit' }, { status: 402 })) + + await expect( + saveDownloadToWorkspace({ + appSession: appSession(fetch), + toolCallId: 'call-6', + filePath: file, + filename: 'report.csv', + signal, + }) + ).rejects.toThrow('Could not save the download (Storage limit).') + }) +}) diff --git a/apps/desktop/src/main/browser-agent/file-transfer.ts b/apps/desktop/src/main/browser-agent/file-transfer.ts new file mode 100644 index 00000000000..ddc72c6b2da --- /dev/null +++ b/apps/desktop/src/main/browser-agent/file-transfer.ts @@ -0,0 +1,207 @@ +/** + * Moves browser-agent files between the Sim app and this machine. + * + * Uploads are staged into a private temporary directory before a page sees them: workspace files + * stream from the app for the exact claimed tool call, and granted local files stream from a + * pinned handle after their containment check. Both streams enforce the upload byte limit, and + * later path replacements cannot redirect a local copy. Chromium reads a chosen file lazily, so + * staged copies live until their browser scope is disposed. Saved downloads travel the other way, + * bound to their own claimed tool call. + */ +import { createWriteStream, openAsBlob } from 'node:fs' +import { type FileHandle, mkdir, rm } from 'node:fs/promises' +import { basename, join } from 'node:path' +import { Readable, Transform } from 'node:stream' +import { pipeline } from 'node:stream/promises' +import { BROWSER_FILE_TRANSFER_MAX_BYTES, BROWSER_FILE_TRANSFER_PATH } from '@sim/browser-protocol' +import { getErrorMessage } from '@sim/utils/errors' +import { app } from 'electron' +import { ToolError } from '@/main/browser-agent/errors' +import type { BrowserAppSession } from '@/main/browser-agent/session' + +/** Granted local folders; the caller owns and must close each returned file handle. */ +export interface LocalFileSource { + resolveGrantedFile( + vfsPath: string, + maxBytes: number + ): Promise<{ handle: FileHandle; name: string; size: number }> +} + +const LOCAL_PATH_PREFIX = 'user-local/' +const MAX_MB = BROWSER_FILE_TRANSFER_MAX_BYTES / 1024 / 1024 + +function stagingRoot(): string { + return join(app.getPath('temp'), 'sim-browser-uploads') +} + +/** One directory name per scope or call; identifiers are opaque, so encode rather than trust. */ +function directoryName(value: string): string { + return encodeURIComponent(value).replaceAll('.', '%2E') +} + +/** A file name that cannot leave the staging directory. */ +function stagedFileName(name: string): string { + const base = basename(name) + .replace(/[\x00-\x1f\x7f]/g, '') + .trim() + return base && base !== '.' && base !== '..' ? base.slice(0, 200) : 'upload' +} + +function filenameFromDisposition(header: string | null): string | null { + const encoded = header?.match(/filename\*=UTF-8''([^;]+)/i)?.[1] + if (encoded) { + try { + return decodeURIComponent(encoded) + } catch {} + } + return header?.match(/filename="([^"]+)"/i)?.[1] ?? null +} + +async function appErrorMessage(response: Response): Promise { + const body = (await response.json().catch(() => null)) as { error?: unknown } | null + return typeof body?.error === 'string' ? body.error : `HTTP ${response.status}` +} + +function limitBytes(maxBytes: number): Transform { + let total = 0 + return new Transform({ + transform(chunk: Buffer, _encoding, callback) { + total += chunk.length + if (total > maxBytes) + callback(new ToolError(`The file exceeds the ${MAX_MB} MB upload limit.`)) + else callback(null, chunk) + }, + }) +} + +async function stageWorkspaceFile( + appSession: BrowserAppSession, + toolCallId: string, + index: number, + directory: string, + signal: AbortSignal | undefined +): Promise { + const response = await appSession.session.fetch( + `${appSession.origin}${BROWSER_FILE_TRANSFER_PATH}`, + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ toolCallId, index }), + signal, + } + ) + if (!response.ok || !response.body) { + throw new ToolError(`Could not read that workspace file (${await appErrorMessage(response)}).`) + } + const name = stagedFileName( + filenameFromDisposition(response.headers.get('content-disposition')) ?? 'upload' + ) + const destination = join(directory, name) + await pipeline( + Readable.fromWeb(response.body as import('node:stream/web').ReadableStream), + limitBytes(BROWSER_FILE_TRANSFER_MAX_BYTES), + createWriteStream(destination, { flags: 'wx' }), + { signal } + ) + return destination +} + +/** + * Stages every requested file for one `browser_upload_file` call and returns their local paths in + * request order. `user-local/` paths come from granted folders; every other path is a workspace + * reference the app resolves from the claimed call's persisted arguments. + */ +export async function stageUploadFiles({ + scopeId, + toolCallId, + paths, + appSession, + localFiles, + signal, +}: { + scopeId: string + toolCallId: string + paths: readonly string[] + appSession: BrowserAppSession | undefined + localFiles: LocalFileSource | undefined + signal?: AbortSignal +}): Promise { + const directory = join(stagingRoot(), directoryName(scopeId), directoryName(toolCallId)) + await rm(directory, { recursive: true, force: true }) + const staged: string[] = [] + for (const [index, path] of paths.entries()) { + const fileDirectory = join(directory, String(index)) + try { + await mkdir(fileDirectory, { recursive: true }) + if (path.startsWith(LOCAL_PATH_PREFIX)) { + if (!localFiles) throw new ToolError('Local folders are unavailable in this desktop app.') + const local = await localFiles.resolveGrantedFile(path, BROWSER_FILE_TRANSFER_MAX_BYTES) + try { + const destination = join(fileDirectory, stagedFileName(local.name)) + await pipeline( + local.handle.createReadStream({ autoClose: false }), + limitBytes(BROWSER_FILE_TRANSFER_MAX_BYTES), + createWriteStream(destination, { flags: 'wx' }), + { signal } + ) + staged.push(destination) + } finally { + await local.handle.close() + } + } else { + if (!appSession) throw new ToolError('Workspace files are unavailable in this desktop app.') + staged.push(await stageWorkspaceFile(appSession, toolCallId, index, fileDirectory, signal)) + } + } catch (error) { + await rm(directory, { recursive: true, force: true }) + if (error instanceof ToolError) throw error + throw new ToolError(`Could not prepare "${path}" for upload (${getErrorMessage(error)}).`) + } + } + return staged +} + +/** Deletes one scope's staged uploads, or every scope's when no id is given. */ +export async function discardStagedUploads(scopeId?: string): Promise { + const target = scopeId ? join(stagingRoot(), directoryName(scopeId)) : stagingRoot() + await rm(target, { recursive: true, force: true }) +} + +/** Stores a completed download as a workspace file for one claimed `browser_save_download` call. */ +export async function saveDownloadToWorkspace({ + appSession, + toolCallId, + filePath, + filename, + signal, +}: { + appSession: BrowserAppSession | undefined + toolCallId: string + filePath: string + filename: string + signal?: AbortSignal +}): Promise<{ path: string; name: string; size: number }> { + if (!appSession) throw new ToolError('Workspace files are unavailable in this desktop app.') + const content = await openAsBlob(filePath) + if (content.size > BROWSER_FILE_TRANSFER_MAX_BYTES) { + throw new ToolError(`The download exceeds the ${MAX_MB} MB workspace save limit.`) + } + const query = new URLSearchParams({ toolCallId, name: filename }) + const response = await appSession.session.fetch( + `${appSession.origin}${BROWSER_FILE_TRANSFER_PATH}?${query}`, + { + method: 'PUT', + headers: { 'content-type': 'application/octet-stream' }, + body: content, + signal, + } + ) + if (!response.ok) { + throw new ToolError(`Could not save the download (${await appErrorMessage(response)}).`) + } + const saved = (await response.json().catch(() => null)) as Record | null + if (typeof saved?.path !== 'string' || typeof saved.name !== 'string') { + throw new ToolError('The app did not confirm where the download was saved.') + } + return { path: saved.path, name: saved.name, size: Number(saved.size) || content.size } +} diff --git a/apps/desktop/src/main/browser-agent/keyboard.test.ts b/apps/desktop/src/main/browser-agent/keyboard.test.ts index 5b56e2a1b9e..26d1566abc3 100644 --- a/apps/desktop/src/main/browser-agent/keyboard.test.ts +++ b/apps/desktop/src/main/browser-agent/keyboard.test.ts @@ -5,9 +5,11 @@ vi.mock('electron', () => import('@/test/electron-mock')) import { WebContentsView } from 'electron' import { buildKeyDispatchPlan, + cdpModifiers, dispatchKeyCombo, KeyDispatchError, parseKeyCombo, + parseModifiers, } from '@/main/browser-agent/keyboard' describe('parseKeyCombo', () => { @@ -34,6 +36,26 @@ describe('parseKeyCombo', () => { expect(parseKeyCombo('ControlOrMeta+K', 'darwin')).toMatchObject({ meta: true }) }) + it('parses function keys and Insert without inserting text', () => { + expect(parseKeyCombo('F2')).toMatchObject({ key: 'F2', code: 'F2', keyCode: 113 }) + expect(parseKeyCombo('Shift+F12')).toMatchObject({ key: 'F12', keyCode: 123, shift: true }) + expect(parseKeyCombo('Insert')).toMatchObject({ key: 'Insert', keyCode: 45 }) + const [down] = buildKeyDispatchPlan(parseKeyCombo('F2'), 'linux') + expect(down).toMatchObject({ type: 'rawKeyDown', key: 'F2' }) + expect(down.text).toBeUndefined() + }) + + it('parses modifier lists shared with pointer input', () => { + expect(parseModifiers(['Shift', 'Mod'], 'darwin')).toEqual({ + ctrl: false, + meta: true, + shift: true, + alt: false, + }) + expect(cdpModifiers(parseModifiers(['Mod', 'Alt'], 'linux'))).toBe(3) + expect(() => parseModifiers(['Hyper'])).toThrow(/Unrecognized modifier/) + }) + it('rejects unknown keys and modifiers', () => { expect(() => parseKeyCombo('Hyper+X')).toThrow(/Unrecognized modifier/) expect(() => parseKeyCombo('NotAKey')).toThrow(/Unrecognized key/) diff --git a/apps/desktop/src/main/browser-agent/keyboard.ts b/apps/desktop/src/main/browser-agent/keyboard.ts index 6038cc68313..bb72ce448b8 100644 --- a/apps/desktop/src/main/browser-agent/keyboard.ts +++ b/apps/desktop/src/main/browser-agent/keyboard.ts @@ -50,6 +50,13 @@ const NAMED_KEYS: Record = { '=': { key: '=', code: 'Equal', keyCode: 187 }, '`': { key: '`', code: 'Backquote', keyCode: 192 }, plus: { key: '+', code: 'Equal', keyCode: 187 }, + insert: { key: 'Insert', code: 'Insert', keyCode: 45 }, + ...Object.fromEntries( + Array.from({ length: 12 }, (_, index) => [ + `f${index + 1}`, + { key: `F${index + 1}`, code: `F${index + 1}`, keyCode: 112 + index }, + ]) + ), } const SHIFTED_CHARACTERS: Record = { @@ -93,13 +100,15 @@ const BASE_FOR_SHIFTED_CHARACTER: Record = Object.fromEntries( Object.entries(SHIFTED_CHARACTERS).map(([base, shifted]) => [shifted, base]) ) -export interface ParsedCombo extends KeyDescriptor { +export interface KeyModifiers { ctrl: boolean meta: boolean shift: boolean alt: boolean } +export interface ParsedCombo extends KeyDescriptor, KeyModifiers {} + export class KeyDispatchError extends Error { constructor( message: string, @@ -132,24 +141,8 @@ export function parseKeyCombo( .map((part) => part.trim()) .filter(Boolean) if (parts.length === 0) throw new ToolError(`Unrecognized key: "${combo}"`) - const modifiers = { ctrl: false, meta: false, shift: false, alt: false } + const modifiers = parseModifiers(parts.slice(0, -1), platform) const keyPart = parts[parts.length - 1] - for (const part of parts.slice(0, -1)) { - const lower = part.toLowerCase() - if (lower === 'control' || lower === 'ctrl') modifiers.ctrl = true - else if (lower === 'meta' || lower === 'cmd' || lower === 'command') modifiers.meta = true - else if ( - lower === 'mod' || - lower === 'primary' || - lower === 'controlormeta' || - lower === 'commandorcontrol' - ) { - if (platform === 'darwin') modifiers.meta = true - else modifiers.ctrl = true - } else if (lower === 'shift') modifiers.shift = true - else if (lower === 'alt' || lower === 'option') modifiers.alt = true - else throw new ToolError(`Unrecognized modifier: "${part}"`) - } const named = NAMED_KEYS[keyPart.toLowerCase()] if (named) { const key = modifiers.shift ? (SHIFTED_CHARACTERS[named.key] ?? named.key) : named.key @@ -175,9 +168,42 @@ export function parseKeyCombo( throw new ToolError(`Unrecognized key: "${keyPart}"`) } +/** + * Parses modifier names ("Shift", "Cmd", "Mod", …). `Mod` is the platform's primary + * shortcut modifier: Meta on macOS, Control elsewhere. + */ +export function parseModifiers( + names: readonly string[], + platform: NodeJS.Platform = process.platform +): KeyModifiers { + const modifiers = { ctrl: false, meta: false, shift: false, alt: false } + for (const name of names) { + const lower = name.trim().toLowerCase() + if (lower === 'control' || lower === 'ctrl') modifiers.ctrl = true + else if (lower === 'meta' || lower === 'cmd' || lower === 'command') modifiers.meta = true + else if ( + lower === 'mod' || + lower === 'primary' || + lower === 'controlormeta' || + lower === 'commandorcontrol' + ) { + if (platform === 'darwin') modifiers.meta = true + else modifiers.ctrl = true + } else if (lower === 'shift') modifiers.shift = true + else if (lower === 'alt' || lower === 'option') modifiers.alt = true + else throw new ToolError(`Unrecognized modifier: "${name}"`) + } + return modifiers +} + /** CDP `Input` modifier bitmask: Alt=1, Ctrl=2, Meta=4, Shift=8. */ -function cdpModifiers(combo: ParsedCombo): number { - return (combo.alt ? 1 : 0) | (combo.ctrl ? 2 : 0) | (combo.meta ? 4 : 0) | (combo.shift ? 8 : 0) +export function cdpModifiers(modifiers: KeyModifiers): number { + return ( + (modifiers.alt ? 1 : 0) | + (modifiers.ctrl ? 2 : 0) | + (modifiers.meta ? 4 : 0) | + (modifiers.shift ? 8 : 0) + ) } /** diff --git a/apps/desktop/src/main/browser-agent/page-functions.test.ts b/apps/desktop/src/main/browser-agent/page-functions.test.ts index d7226734454..e2b62480cbb 100644 --- a/apps/desktop/src/main/browser-agent/page-functions.test.ts +++ b/apps/desktop/src/main/browser-agent/page-functions.test.ts @@ -19,6 +19,7 @@ import { readPageActionState, readPageText, readSelectElementState, + resolveFileInputTarget, scrollPage, selectOptionInElement, setFocusedInputValue, @@ -806,17 +807,118 @@ describe('collectSnapshot', () => { } }) - it('marks file inputs unsupported and refuses to open a native chooser', () => { + it('labels file inputs and refuses to open a native chooser', () => { document.body.innerHTML = '' visible(document.querySelector('input') as HTMLInputElement) const outline = outlineOf(collectSnapshot()) const ref = refFor(outline, 'Upload receipt') expect(outline).toContain('file-input "Upload receipt"') - expect(outline).toContain('upload-unsupported') expect(clickElement(ref)).toEqual({ error: 'file-input' }) }) + it('pins the file input behind a drop zone, label, or the input itself', () => { + document.body.innerHTML = `
Drop files
+ +
` + register( + document.getElementById('zone') as HTMLElement, + document.getElementById('label') as HTMLElement, + document.getElementById('labelled') as HTMLElement, + document.getElementById('two') as HTMLElement + ) + + expect(resolveFileInputTarget(0)).toEqual({ + input: document.getElementById('hidden'), + document, + }) + expect(resolveFileInputTarget(1)).toEqual({ + input: document.getElementById('labelled'), + document, + }) + expect(resolveFileInputTarget(2)).toEqual({ + input: document.getElementById('labelled'), + document, + }) + expect(() => resolveFileInputTarget(3)).toThrow('multiple file inputs') + expect(document.querySelector('[data-sim-agent-upload]')).toBeNull() + }) + + it('reports an element with no nearby file input', () => { + document.body.innerHTML = + '
' + register(document.getElementById('b') as HTMLElement) + + expect(() => resolveFileInputTarget(0)).toThrow('no nearby file input') + }) + + it.each([ + ['disabled input', ''], + ['disabled fieldset', '
'], + [ + 'second legend of a disabled fieldset', + '
First
', + ], + [ + 'enabled fieldset within a disabled fieldset', + '
', + ], + [ + 'disabled fieldset within an exempt legend', + '
', + ], + ])('refuses to resolve an upload in a %s', (_label, html) => { + document.body.innerHTML = html + const input = document.querySelector('input') as HTMLInputElement + register(input) + + expect(() => runSerialized(resolveFileInputTarget, [0])).toThrow('disabled') + expect(input.hasAttribute('data-sim-agent-upload')).toBe(false) + }) + + it('allows the first legend exemption in a disabled fieldset', () => { + document.body.innerHTML = + '
' + const input = document.querySelector('input') as HTMLInputElement + register(document.querySelector('label') as HTMLLabelElement) + + expect(runSerialized(resolveFileInputTarget, [0])).toEqual({ input, document }) + }) + + it('pins the original input inside an open shadow root without modifying the DOM', () => { + const host = document.createElement('div') + document.body.append(host) + const input = document.createElement('input') + input.type = 'file' + host.attachShadow({ mode: 'open' }).append(input) + register(host) + + expect(runSerialized(resolveFileInputTarget, [0])).toEqual({ input, document }) + expect(input.hasAttribute('data-sim-agent-upload')).toBe(false) + }) + + it('captures the actual owner document for an input reached through a same-origin frame', () => { + const frame = document.createElement('iframe') + document.body.append(frame) + const childDocument = frame.contentDocument as Document + childDocument.body.innerHTML = '' + const input = childDocument.querySelector('input') as HTMLInputElement + register(input) + + const captured = resolveFileInputTarget(0) + expect(captured).toEqual({ input, document: childDocument }) + document.body.append(input) + expect(captured.document).toBe(childDocument) + expect(captured.input.ownerDocument).toBe(document) + }) + + it('refuses a disconnected or stale upload reference', () => { + const input = document.createElement('input') + input.type = 'file' + register(input) + expect(() => resolveFileInputTarget(0)).toThrow('stale') + }) + it('sets a complete multiple selection atomically and can clear it', () => { document.body.innerHTML = '' diff --git a/apps/desktop/src/main/browser-agent/page-functions.ts b/apps/desktop/src/main/browser-agent/page-functions.ts index f2e4ff93a4e..30b81638104 100644 --- a/apps/desktop/src/main/browser-agent/page-functions.ts +++ b/apps/desktop/src/main/browser-agent/page-functions.ts @@ -1,5 +1,5 @@ /** - * Functions injected into automated pages via `webContents.executeJavaScript`. + * Functions injected into automated pages through a CDP isolated execution world. * The driver serializes each function's source (`String(fn)`) and calls it * with JSON-encoded arguments, so every function here MUST be fully * self-contained: no imports, no closed-over variables, only its own @@ -463,9 +463,8 @@ export function collectSnapshot(startingElementId = 0, elementId?: number): unkn // widening this cannot expose a credential field. const value = (el as HTMLInputElement).value const inputType = tag === 'INPUT' ? (el as HTMLInputElement).type : '' - if (inputType === 'file') { - parts.push('upload-unsupported') - } else if (inputType !== 'checkbox' && inputType !== 'radio') { + // A chosen file input reads as Chromium's C:\fakepath\, which confirms an upload. + if (inputType !== 'checkbox' && inputType !== 'radio') { if (value && isSensitiveValueField(el)) parts.push('value-withheld') else if (value) parts.push(`value=${quote(cut(String(value), 120))}`) } @@ -3299,6 +3298,63 @@ export function readPageText(id?: number): unknown { } } +/** + * Resolves the exact file input and its document for an isolated CDP object handle. The ref may be the + * input itself, its label, or a visible upload control (button, dropzone) whose hidden input sits + * inside it or within a few ancestors — the nearest level with exactly one file input wins. + */ +export function resolveFileInputTarget(id: number): { + input: HTMLInputElement + document: Document +} { + const resolver = window.__simAgentResolveElement + const resolved = resolver?.(id) + const el = resolver ? resolved?.element : (window.__simAgentElements || [])[id] + if (!el || !el.isConnected) { + throw new Error( + window.__simAgentStaleReason || 'The upload target is stale. Take a fresh snapshot.' + ) + } + const isFileInput = (node: Element | null | undefined): node is HTMLInputElement => + Boolean( + node && + String(node.tagName || '').toUpperCase() === 'INPUT' && + String((node as HTMLInputElement).type || '').toLowerCase() === 'file' + ) + const fileInputsWithin = (root: Element): HTMLInputElement[] => { + const found: HTMLInputElement[] = [] + const visit = (scope: Element | ShadowRoot) => { + for (const node of Array.from(scope.querySelectorAll('*'))) { + if (isFileInput(node)) found.push(node) + if (node.shadowRoot) visit(node.shadowRoot) + } + } + if (isFileInput(root)) return [root] + visit(root) + return found + } + let input: HTMLInputElement | null = null + if (isFileInput(el)) input = el + else if (String(el.tagName || '').toUpperCase() === 'LABEL') { + const control = (el as HTMLLabelElement).control + if (isFileInput(control)) input = control + } + let scope: Element | null = el + for (let depth = 0; !input && scope && depth <= 3; depth++) { + const candidates = fileInputsWithin(scope) + if (candidates.length > 1) + throw new Error( + 'The upload target contains multiple file inputs. Select one input explicitly.' + ) + if (candidates.length === 1) input = candidates[0] + const root = scope.getRootNode() + scope = scope.parentElement ?? (root instanceof ShadowRoot ? root.host : null) + } + if (!input) throw new Error('The selected element has no nearby file input.') + if (input.matches(':disabled')) throw new Error('The file input is disabled.') + return { input, document: input.ownerDocument } +} + export function pageContainsText(text: string): boolean { return Boolean(document.body?.innerText.includes(text)) } diff --git a/apps/desktop/src/main/browser-agent/panel.test.ts b/apps/desktop/src/main/browser-agent/panel.test.ts index 5aa024772c7..cb35b99cfc7 100644 --- a/apps/desktop/src/main/browser-agent/panel.test.ts +++ b/apps/desktop/src/main/browser-agent/panel.test.ts @@ -30,7 +30,7 @@ function freshPanel(): PanelModule { const PANEL_RECT = { x: 400, y: 64, width: 600, height: 800 } /** A panel showing one tab. */ -function showPanel(panel: PanelModule) { +function showPanel(panel: PanelModule, onGeometryChanged?: () => void) { const win = new BrowserWindow() const view = new WebContentsView() const active = { id: 'tab-1', scopeId: 'chat-test', view } @@ -40,6 +40,7 @@ function showPanel(panel: PanelModule) { backgroundColor: () => '#0c0c0c', restoreActiveScope: () => {}, onViewDetached: () => {}, + onGeometryChanged, }) panel.activatePanelScope('chat-test') panel.setPanelBounds(PANEL_RECT, win) @@ -278,6 +279,20 @@ describe('panel chat scope', () => { expect(view.webContents.invalidate).toHaveBeenCalledTimes(2) }) + it('dismisses field-anchored UI when the browser resource is hidden', () => { + const onGeometryChanged = vi.fn() + const { win, view } = showPanel(panel, onGeometryChanged) + onGeometryChanged.mockClear() + + panel.setPanelBounds(null, win) + + expect(view.setVisible).toHaveBeenLastCalledWith(false) + expect(onGeometryChanged).toHaveBeenCalledOnce() + expect(win.contentView.removeChildView).not.toHaveBeenCalled() + panel.layout() + expect(onGeometryChanged).toHaveBeenCalledOnce() + }) + it('reports the exact applied native rectangle in renderer viewport coordinates', async () => { const { win, view } = showPanel(panel) const scopeId = panel.getActivePanelScopeId() diff --git a/apps/desktop/src/main/browser-agent/panel.ts b/apps/desktop/src/main/browser-agent/panel.ts index 42ca72e4f89..93ad3538169 100644 --- a/apps/desktop/src/main/browser-agent/panel.ts +++ b/apps/desktop/src/main/browser-agent/panel.ts @@ -45,6 +45,8 @@ export interface PanelHost { restoreActiveScope: () => void /** Lets the session drop focus tracking for a view that is no longer attached. */ onViewDetached: (view: WebContentsView | null) => void + /** Invalidates field-anchored UI when the page moves, hides, or detaches. */ + onGeometryChanged?: () => void } let host: PanelHost = { @@ -292,6 +294,7 @@ function detachAttachedView(): void { occludableFrame = null unbindHostResize() host.onViewDetached(view) + host.onGeometryChanged?.() if (!view || !win) return try { @@ -331,6 +334,7 @@ function hideAttachedView(): void { error: getErrorMessage(error, 'unknown'), }) } + host.onGeometryChanged?.() } /** @@ -428,11 +432,13 @@ export function layout(): void { lastAppliedBounds = boundsKey occludableFrame = null active.view.setBounds(bounds) + host.onGeometryChanged?.() } const visible = !panelOccluded if (lastAppliedVisibility !== visible) { lastAppliedVisibility = visible active.view.setVisible(visible) + host.onGeometryChanged?.() if (visible && !active.view.webContents.isDestroyed()) { // invalidate() recomposites the LAST frame — which is blank when the // page finished loading while this view was hidden and background diff --git a/apps/desktop/src/main/browser-agent/post-action-observation.test.ts b/apps/desktop/src/main/browser-agent/post-action-observation.test.ts new file mode 100644 index 00000000000..f5fa4dc01a1 --- /dev/null +++ b/apps/desktop/src/main/browser-agent/post-action-observation.test.ts @@ -0,0 +1,152 @@ +import { describe, expect, it, vi } from 'vitest' +import { withPostActionObservation } from '@/main/browser-agent/post-action-observation' + +describe('post-action observation', () => { + it('validates observation arguments before dispatching input', async () => { + for (const observe of [ + null, + true, + [], + { query: '' }, + { query: 'x'.repeat(4097) }, + { extra: 1 }, + ]) { + const action = vi.fn() + await expect( + withPostActionObservation('browser_click', { observe }, action, vi.fn(), vi.fn()) + ).rejects.toThrow('observe must') + expect(action).not.toHaveBeenCalled() + } + }) + + it('observes after scrolling and hovering, which reveal new content', async () => { + for (const tool of ['browser_scroll', 'browser_hover'] as const) { + const result = await withPostActionObservation( + tool, + { direction: 'down', observe: {} }, + async () => ({ movedBy: 400 }), + async () => ({ outline: '- row "Next" [ref=9]' }), + vi.fn() + ) + expect(result).toMatchObject({ observation: { ok: true } }) + } + }) + + it('keeps standalone actions unchanged and does not capture unrequested state', async () => { + const result = { dispatched: true, effectObserved: false } + const observe = vi.fn() + expect( + await withPostActionObservation('browser_click', {}, async () => result, observe, vi.fn()) + ).toBe(result) + expect(observe).not.toHaveBeenCalled() + }) + + it('observes after the action, preserving partial form results and replacing refs', async () => { + const order: string[] = [] + const result = await withPostActionObservation( + 'browser_fill_form', + { observe: { query: 'Save' } }, + async () => { + order.push('action') + return { completed: false, completedCount: 1, doNotRetry: true } + }, + async (query) => { + order.push('observe') + expect(query).toBe('Save') + return { matches: [{ elementId: 24, line: 'button Save [ref=24]' }] } + }, + vi.fn() + ) + expect(order).toEqual(['action', 'observe']) + expect(result).toMatchObject({ + completed: false, + completedCount: 1, + doNotRetry: true, + observation: { ok: true, result: { matches: [{ elementId: 24 }] } }, + }) + }) + + it('does not repeat or misreport a dispatched action when observation fails', async () => { + const action = vi.fn(async () => ({ dispatched: true, effectObserved: false })) + const result = await withPostActionObservation( + 'browser_click', + { observe: {} }, + action, + async () => { + throw new Error('Page changed') + }, + vi.fn() + ) + expect(action).toHaveBeenCalledTimes(1) + expect(action).toHaveBeenCalledWith({}) + expect(result).toMatchObject({ + dispatched: true, + effectObserved: false, + observation: { ok: false, error: 'Page changed' }, + }) + }) + + it('never observes after failed actions', async () => { + const observe = vi.fn() + await expect( + withPostActionObservation( + 'browser_type', + { observe: {} }, + async () => { + throw new Error('Ref expired') + }, + observe, + vi.fn() + ) + ).rejects.toThrow('Ref expired') + expect(observe).not.toHaveBeenCalled() + }) + + it('preserves completed actions when execution expires before observation', async () => { + const observe = vi.fn() + await expect( + withPostActionObservation( + 'browser_type', + { observe: {} }, + async () => ({ typed: true }), + observe, + () => { + throw new Error('Cancelled') + } + ) + ).resolves.toMatchObject({ + typed: true, + observation: { ok: false, error: 'Cancelled', doNotRetry: true }, + }) + expect(observe).not.toHaveBeenCalled() + }) + + it.each([false, true])( + 'preserves completed actions when execution expires during observation (observation fails: %s)', + async (fails) => { + let expired = false + await expect( + withPostActionObservation( + 'browser_click', + { observe: {} }, + async () => ({ dispatched: true }), + async () => { + expired = true + if (fails) throw new Error('Page changed') + return { outline: 'Stale snapshot' } + }, + () => { + if (expired) throw new Error('Cancelled') + } + ) + ).resolves.toMatchObject({ + dispatched: true, + observation: { + ok: false, + error: fails ? 'Page changed' : 'Cancelled', + doNotRetry: true, + }, + }) + } + ) +}) diff --git a/apps/desktop/src/main/browser-agent/post-action-observation.ts b/apps/desktop/src/main/browser-agent/post-action-observation.ts new file mode 100644 index 00000000000..91e8aa4b616 --- /dev/null +++ b/apps/desktop/src/main/browser-agent/post-action-observation.ts @@ -0,0 +1,62 @@ +import type { BrowserToolName } from '@sim/browser-protocol' +import { getErrorMessage } from '@sim/utils/errors' +import { isRecordLike } from '@sim/utils/object' +import { ToolError } from '@/main/browser-agent/errors' + +const OBSERVABLE_ACTIONS: ReadonlySet = new Set([ + 'browser_click', + 'browser_type', + 'browser_press_key', + 'browser_fill_form', + 'browser_scroll', + 'browser_hover', +]) + +/** Preserves a dispatched action when its acknowledgement or observation is interrupted. */ +export function withFailedPostActionObservation(result: unknown, error: unknown): unknown { + const observation = { + ok: false, + error: getErrorMessage(error), + doNotRetry: true, + note: 'The action was dispatched. Inspect its result; do not repeat it just because confirmation failed.', + } + return isRecordLike(result) ? { ...result, observation } : { result, observation } +} + +/** Validate before dispatch; an observation failure must never invite replay of a completed action. */ +export async function withPostActionObservation( + tool: BrowserToolName, + params: Record, + action: (params: Record) => Promise, + observe: (query: string | undefined) => Promise, + assertCurrent: () => void +): Promise { + const request = params.observe + if (request === undefined) return action(params) + if ( + !OBSERVABLE_ACTIONS.has(tool) || + !isRecordLike(request) || + Object.keys(request).some((key) => key !== 'query') || + (request.query !== undefined && + (typeof request.query !== 'string' || + request.query.length === 0 || + request.query.length > 4096)) + ) { + throw new ToolError( + 'observe must be {} or {query: nonempty text up to 4096 characters} on a supported action.' + ) + } + const query = typeof request.query === 'string' ? request.query : undefined + const { observe: _observe, ...actionParams } = params + const result = await action(actionParams) + let observation: unknown + try { + assertCurrent() + const observed = await observe(query) + assertCurrent() + observation = { ok: true, result: observed } + } catch (error) { + return withFailedPostActionObservation(result, error) + } + return isRecordLike(result) ? { ...result, observation } : { result, observation } +} diff --git a/apps/desktop/src/main/browser-agent/registry.ts b/apps/desktop/src/main/browser-agent/registry.ts index eeabc9de405..c51c2a76ed5 100644 --- a/apps/desktop/src/main/browser-agent/registry.ts +++ b/apps/desktop/src/main/browser-agent/registry.ts @@ -1,4 +1,5 @@ -import type { WebContents } from 'electron' +import type { Session, WebContents } from 'electron' +import { isAppOrigin } from '@/main/navigation' /** * Registry of WebContents that belong to the agent browser (the browser-agent @@ -10,11 +11,61 @@ import type { WebContents } from 'electron' * navigation time, so the post-construction registration races nothing. */ const agentContents = new WeakSet() +const agentSessions = new WeakMap() +const appOrigins = new WeakMap() +const navigations = new WeakMap boolean>() +const permissions = new WeakMap() -export function registerAgentWebContents(contents: WebContents): void { +export interface BrowserPermissionHandlers { + request: NonNullable[0]> + check: NonNullable[0]> +} + +export function registerAgentWebContents( + contents: WebContents, + appOrigin?: string, + handlers?: BrowserPermissionHandlers +): void { agentContents.add(contents) + agentSessions.set(contents.session, appOrigin) + if (appOrigin) appOrigins.set(contents, appOrigin) + if (handlers) permissions.set(contents, handlers) +} + +/** + * Unattributed workers retain network guards, but a shared session's configured + * app origin remains reachable for ordinary app workers on self-hosted networks. + */ +export function shouldGuardUnownedAgentRequest(session: Session, url: string): boolean { + if (!agentSessions.has(session)) return false + const appOrigin = agentSessions.get(session) + return !appOrigin || !isAppOrigin(url, appOrigin) } export function isAgentWebContents(contents: WebContents): boolean { return agentContents.has(contents) } + +/** Only first-party browser views may use the app's authenticated session. */ +export function agentAppOrigin(contents: WebContents): string | undefined { + return appOrigins.get(contents) +} + +/** Browser views keep browser permission policy even when sharing Sim authentication. */ +export function agentPermissionHandlers( + contents: WebContents | null +): BrowserPermissionHandlers | undefined { + return contents ? permissions.get(contents) : undefined +} + +/** Session routing also runs at the request boundary, before redirected requests send cookies. */ +export function registerAgentNavigation( + contents: WebContents, + route: (url: string, method: string) => boolean +): void { + navigations.set(contents, route) +} + +export function routeAgentNavigation(contents: WebContents, url: string, method = 'GET'): boolean { + return navigations.get(contents)?.(url, method) ?? false +} diff --git a/apps/desktop/src/main/browser-agent/request-policy-network.test.ts b/apps/desktop/src/main/browser-agent/request-policy-network.test.ts new file mode 100644 index 00000000000..5fb4f42fc6e --- /dev/null +++ b/apps/desktop/src/main/browser-agent/request-policy-network.test.ts @@ -0,0 +1,79 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('electron', () => import('@/test/electron-mock')) + +const { mockLookup } = vi.hoisted(() => ({ mockLookup: vi.fn() })) +vi.mock('node:dns/promises', () => ({ default: { lookup: mockLookup } })) + +import { allowBrowserRequest } from '@/main/browser-agent/request-policy' +import { clearHostVerdictCache } from '@/main/browser-agent/url-guard' + +type ResourceType = Parameters[0]['resourceType'] + +describe('browser requests with the network guards', () => { + beforeEach(() => { + vi.clearAllMocks() + clearHostVerdictCache() + mockLookup.mockResolvedValue([{ address: '93.184.216.34', family: 4 }]) + }) + + it.each(['script', 'font'] as const)( + 'blocks non-web %s URLs independently of DNS', + async (type) => { + for (const url of [ + 'file:///etc/passwd', + 'chrome://settings/', + 'chrome-extension://aaaabbbbccccddddeeeeffffgggghhhh/x.js', + ]) { + expect(await allowBrowserRequest({ url, method: 'GET', resourceType: type }), url).toBe( + false + ) + } + expect(mockLookup).not.toHaveBeenCalled() + } + ) + + it.each<[string, ResourceType]>([ + ['http://example.com/', 'mainFrame'], + ['https://example.com/frame', 'subFrame'], + ['https://example.com/app.js', 'script'], + ['https://example.com/image.png', 'image'], + ['https://example.com/font.woff2', 'font'], + ['http://localhost:3000/app.js', 'script'], + ['http://127.0.0.1:3000/app.css', 'stylesheet'], + ['ws://example.com/socket', 'webSocket'], + ['wss://example.com/socket', 'webSocket'], + ['data:image/png;base64,aGVsbG8=', 'image'], + ['blob:https://example.com/image-id', 'image'], + ['chrome-extension://mhjfbmdgcfjbbpaeojofohoefgiehjai/index.html', 'subFrame'], + ['chrome://resources/js/assert.js', 'script'], + ])('preserves %s as a %s', async (url, resourceType) => { + expect(await allowBrowserRequest({ url, method: 'GET', resourceType })).toBe(true) + }) + + it.each(['mainFrame', 'subFrame'] as const)( + 'keeps data, blob, and websocket URLs out of %s navigations', + async (resourceType) => { + for (const url of [ + 'data:text/html,hello', + 'blob:https://example.com/id', + 'ws://example.com', + ]) { + expect(await allowBrowserRequest({ url, method: 'GET', resourceType })).toBe(false) + } + expect(mockLookup).not.toHaveBeenCalled() + } + ) + + it.each<[string, ResourceType]>([ + ['http://169.254.169.254/latest/meta-data', 'xhr'], + ['http://169.254.169.254/font.woff2', 'font'], + ['http://[fd00::1]/image.png', 'image'], + ['ws://10.0.0.1/socket', 'webSocket'], + ['wss://internal.example/socket', 'webSocket'], + ['https://internal.example/app.js', 'script'], + ])('still blocks private network request %s', async (url, resourceType) => { + mockLookup.mockResolvedValue([{ address: '10.0.0.1', family: 4 }]) + expect(await allowBrowserRequest({ url, method: 'GET', resourceType })).toBe(false) + }) +}) diff --git a/apps/desktop/src/main/browser-agent/request-policy.test.ts b/apps/desktop/src/main/browser-agent/request-policy.test.ts new file mode 100644 index 00000000000..2d5636a606f --- /dev/null +++ b/apps/desktop/src/main/browser-agent/request-policy.test.ts @@ -0,0 +1,141 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('electron', () => import('@/test/electron-mock')) +const guards = vi.hoisted(() => ({ + checkAgentUrl: vi.fn(async () => ({ ok: true })), + isBlockedRequestUrl: vi.fn(() => false), + isBlockedSubresourceUrl: vi.fn(async () => false), + subresourceNeedsResolution: vi.fn(() => true), +})) +vi.mock('@/main/browser-agent/url-guard', () => guards) + +import { WebContentsView } from 'electron' +import { registerAgentNavigation, registerAgentWebContents } from '@/main/browser-agent/registry' +import { allowBrowserRequest, handleBrowserRequest } from '@/main/browser-agent/request-policy' + +describe('authenticated browser request policy', () => { + beforeEach(() => { + vi.clearAllMocks() + guards.checkAgentUrl.mockResolvedValue({ ok: true }) + guards.isBlockedRequestUrl.mockReturnValue(false) + guards.isBlockedSubresourceUrl.mockResolvedValue(false) + guards.subresourceNeedsResolution.mockReturnValue(true) + }) + + it.each(['script', 'font', 'image', 'xhr'] as const)( + 'rejects unsupported schemes before the %s host guard', + async (resourceType) => { + guards.subresourceNeedsResolution.mockReturnValue(resourceType !== 'font') + for (const url of [ + 'file:///etc/passwd', + 'chrome://settings/', + 'chrome-extension://aaaabbbbccccddddeeeeffffgggghhhh/x.js', + 'chrome-extension://mhjfbmdgcfjbbpaeojofohoefgiehjai.evil.test/x.js', + 'chrome://resources.evil.test/x.js', + 'javascript:alert(1)', + 'ftp://example.com/file', + 'not a url', + ]) { + expect(await allowBrowserRequest({ url, method: 'GET', resourceType }), url).toBe(false) + } + expect(guards.isBlockedRequestUrl).not.toHaveBeenCalled() + expect(guards.isBlockedSubresourceUrl).not.toHaveBeenCalled() + } + ) + + it('admits only the built-in PDF viewer and its packaged resources outside http(s)', async () => { + const contents = new WebContentsView().webContents + guards.checkAgentUrl.mockResolvedValue({ ok: false }) + guards.isBlockedSubresourceUrl.mockResolvedValue(true) + const request = (url: string, resourceType: 'subFrame' | 'script') => + allowBrowserRequest({ webContents: contents, url, method: 'GET', resourceType }) + + expect( + await request('chrome-extension://mhjfbmdgcfjbbpaeojofohoefgiehjai/index.html', 'subFrame') + ).toBe(true) + expect(await request('chrome://resources/js/assert.js', 'script')).toBe(true) + expect( + await request('chrome-extension://aaaabbbbccccddddeeeeffffgggghhhh/x.js', 'script') + ).toBe(false) + expect(await request('chrome://settings/', 'subFrame')).toBe(false) + }) + + it('confines the shared session to the exact configured origin', async () => { + const contents = new WebContentsView().webContents + registerAgentWebContents(contents, 'https://www.dev.sim.ai') + for (const url of [ + 'https://www.sim.ai/home', + 'https://www.dev.sim.ai.evil.test/', + 'https://www.dev.sim.ai:444/home', + ]) { + expect( + await allowBrowserRequest({ + webContents: contents, + url, + method: 'GET', + resourceType: 'mainFrame', + }) + ).toBe(false) + } + expect(guards.checkAgentUrl).not.toHaveBeenCalled() + expect( + await allowBrowserRequest({ + webContents: contents, + url: 'https://www.dev.sim.ai/home', + method: 'GET', + resourceType: 'mainFrame', + }) + ).toBe(true) + expect(guards.checkAgentUrl).toHaveBeenCalledOnce() + }) + + it('routes session changes before allowing their network request', async () => { + const contents = new WebContentsView().webContents + registerAgentWebContents(contents) + const route = vi.fn(() => true) + registerAgentNavigation(contents, route) + expect( + await allowBrowserRequest({ + webContents: contents, + url: 'https://www.dev.sim.ai/home', + method: 'GET', + resourceType: 'mainFrame', + }) + ).toBe(false) + expect(route).toHaveBeenCalledWith('https://www.dev.sim.ai/home', 'GET') + expect(guards.checkAgentUrl).not.toHaveBeenCalled() + }) + + it('keeps frame and subresource network checks on authenticated pages', async () => { + const contents = new WebContentsView().webContents + registerAgentWebContents(contents, 'https://www.dev.sim.ai') + guards.checkAgentUrl.mockResolvedValue({ ok: false }) + guards.isBlockedSubresourceUrl.mockResolvedValue(true) + const url = 'http://169.254.169.254/' + expect( + await allowBrowserRequest({ + webContents: contents, + url, + method: 'GET', + resourceType: 'subFrame', + }) + ).toBe(false) + expect( + await allowBrowserRequest({ webContents: contents, url, method: 'GET', resourceType: 'xhr' }) + ).toBe(false) + expect(guards.checkAgentUrl).toHaveBeenCalledWith(url) + expect(guards.isBlockedSubresourceUrl).toHaveBeenCalledWith(url) + }) + + it('fails closed on guard errors and tolerates a loader disappearing during DNS', async () => { + guards.checkAgentUrl.mockRejectedValue(new Error('DNS failed')) + const callback = vi.fn(() => { + throw new Error('Loader gone') + }) + handleBrowserRequest( + { url: 'https://example.com', method: 'GET', resourceType: 'mainFrame' }, + callback + ) + await vi.waitFor(() => expect(callback).toHaveBeenCalledExactlyOnceWith({ cancel: true })) + }) +}) diff --git a/apps/desktop/src/main/browser-agent/request-policy.ts b/apps/desktop/src/main/browser-agent/request-policy.ts new file mode 100644 index 00000000000..50470f088d4 --- /dev/null +++ b/apps/desktop/src/main/browser-agent/request-policy.ts @@ -0,0 +1,77 @@ +import { createLogger } from '@sim/logger' +import type { OnBeforeRequestListenerDetails } from 'electron' +import { agentAppOrigin, routeAgentNavigation } from '@/main/browser-agent/registry' +import { + checkAgentUrl, + isBlockedRequestUrl, + isBlockedSubresourceUrl, + subresourceNeedsResolution, +} from '@/main/browser-agent/url-guard' +import { isAppOrigin } from '@/main/navigation' + +const logger = createLogger('BrowserRequestPolicy') +type BrowserRequest = Pick< + OnBeforeRequestListenerDetails, + 'url' | 'resourceType' | 'webContents' | 'method' +> + +/** + * Chromium's bundled PDF viewer and the shared UI resources it loads. Both are packaged local + * resources rather than network requests, and Chromium never lets web content load them, so the + * network guard, which admits only http(s), would otherwise block every PDF from rendering. + */ +const PDF_VIEWER_RESOURCE_PREFIXES = [ + 'chrome-extension://mhjfbmdgcfjbbpaeojofohoefgiehjai/', + 'chrome://resources/', +] as const + +/** Web subresources include WebSocket connections and page-created data/blob assets. */ +const SUBRESOURCE_PROTOCOLS: ReadonlySet = new Set([ + 'http:', + 'https:', + 'ws:', + 'wss:', + 'data:', + 'blob:', +]) + +/** The same network guard applies to isolated and authenticated browser views. */ +export async function allowBrowserRequest(details: BrowserRequest): Promise { + if (PDF_VIEWER_RESOURCE_PREFIXES.some((prefix) => details.url.startsWith(prefix))) return true + if ( + details.resourceType === 'mainFrame' && + details.webContents && + routeAgentNavigation(details.webContents, details.url, details.method) + ) + return false + const appOrigin = details.webContents && agentAppOrigin(details.webContents) + if (appOrigin && details.resourceType === 'mainFrame' && !isAppOrigin(details.url, appOrigin)) { + return false + } + if (details.resourceType === 'mainFrame' || details.resourceType === 'subFrame') { + return (await checkAgentUrl(details.url)).ok + } + try { + if (!SUBRESOURCE_PROTOCOLS.has(new URL(details.url).protocol)) return false + } catch { + return false + } + return subresourceNeedsResolution(details.resourceType) + ? !(await isBlockedSubresourceUrl(details.url)) + : !isBlockedRequestUrl(details.url) +} + +/** A loader may disappear while DNS resolves; answer once without an unhandled rejection. */ +export function handleBrowserRequest( + details: BrowserRequest, + callback: (response: { cancel: boolean }) => void +): void { + const finish = (allowed: boolean) => { + try { + callback({ cancel: !allowed }) + } catch { + logger.warn('Browser request ended before its guard completed') + } + } + void allowBrowserRequest(details).then(finish, () => finish(false)) +} diff --git a/apps/desktop/src/main/browser-agent/session.test.ts b/apps/desktop/src/main/browser-agent/session.test.ts index a8f5603f594..f08aa52c767 100644 --- a/apps/desktop/src/main/browser-agent/session.test.ts +++ b/apps/desktop/src/main/browser-agent/session.test.ts @@ -18,9 +18,11 @@ import { Menu, shell, systemPreferences, + WebContentsView, } from 'electron' import { BASE_ZOOM_FACTOR, steppedZoomFactor } from '@/main/browser-agent/context-menu' import * as panel from '@/main/browser-agent/panel' +import { agentAppOrigin, routeAgentNavigation } from '@/main/browser-agent/registry' import * as sessionModule from '@/main/browser-agent/session' import type { BrowserSessionSnapshot } from '@/main/desktop-chat-session-store' @@ -32,6 +34,12 @@ function setPlatform(platform: NodeJS.Platform): void { Object.defineProperty(process, 'platform', { configurable: true, value: platform }) } +type PopupHandler = (details: { url: string }) => { + action: string + outlivesOpener?: boolean + createWindow?: (options: Record) => unknown +} + interface MockView { webContents: { session: { @@ -97,7 +105,8 @@ function freshSession( win: BrowserWindow | null | (() => BrowserWindow | null), eventOverrides: Partial = {}, browserPersistence?: sessionModule.BrowserSessionPersistence, - downloadSettings?: sessionModule.BrowserDownloadSettings + downloadSettings?: sessionModule.BrowserDownloadSettings, + appSession?: sessionModule.BrowserAppSession ): SessionModule { const mainWindowProvider = typeof win === 'function' ? win : () => win const session = sessionModule @@ -115,7 +124,8 @@ function freshSession( }, mainWindowProvider, browserPersistence, - downloadSettings + downloadSettings, + appSession ) session.activateBrowserScope('chat-test') return session @@ -2358,6 +2368,26 @@ describe('browser-agent session', () => { expect(session.listTabs()).toHaveLength(13) }) + it('gives background automation a viewport without taking panel ownership', () => { + const tab = session.withBrowserScope('background-chat', () => session.ensureTab()) + + expect(tab.view.setBounds).toHaveBeenCalledWith({ + x: 0, + y: 0, + width: 1180, + height: 850, + }) + expect(win.contentView.addChildView).not.toHaveBeenCalledWith(tab.view) + expect(session.getActiveBrowserScopeId()).toBe('chat-test') + }) + + it('initializes a detached viewport when no application window exists', () => { + const headlessSession = freshSession(null) + const tab = headlessSession.ensureTab() + + expect(tab.view.setBounds).toHaveBeenCalledWith({ x: 0, y: 0, width: 1280, height: 720 }) + }) + it('embeds the active view in the MAIN window only while panel bounds are reported', () => { const tab = session.ensureTab() const view = tab.view as unknown as MockView @@ -2643,14 +2673,16 @@ describe('browser-agent session', () => { expect(contents.session.setPermissionRequestHandler).toHaveBeenCalled() expect(contents.session.setPermissionCheckHandler).toHaveBeenCalled() - const openHandler = contents.setWindowOpenHandler.mock.calls[0][0] as (details: { - url: string - }) => { action: string } - expect(openHandler({ url: 'https://example.com/popup' })).toEqual({ action: 'deny' }) + const openHandler = contents.setWindowOpenHandler.mock.calls[0][0] as PopupHandler + const popup = openHandler({ url: 'https://example.com/popup' }) + expect(popup).toMatchObject({ action: 'allow', outlivesOpener: true }) + const adopted = popup.createWindow?.({ webContents: {} as never }) expect(session.listTabs()).toHaveLength(2) const popupContents = (session.activeTab()?.view as unknown as MockView | undefined) ?.webContents - expect(popupContents?.loadURL).toHaveBeenCalledWith('https://example.com/popup') + expect(adopted).toBe(popupContents) + // Chromium already navigates an adopted popup, which is what keeps window.opener. + expect(popupContents?.loadURL).not.toHaveBeenCalled() expect(contents.loadURL).not.toHaveBeenCalledWith('https://example.com/popup') // Non-http(s) popups are denied without navigating anywhere. contents.loadURL.mockClear() @@ -2658,6 +2690,41 @@ describe('browser-agent session', () => { expect(contents.loadURL).not.toHaveBeenCalled() }) + it('returns agent work to the opener when an adopted popup closes itself', () => { + const opener = session.ensureTab() + session.setAutomationActive(true) + const source = (opener.view as unknown as MockView).webContents + const openWindow = source.setWindowOpenHandler.mock.calls[0]?.[0] as PopupHandler + openWindow({ url: 'https://accounts.example/authorize' }).createWindow?.({ + webContents: {}, + }) + const popup = session.automationTab() + expect(popup).not.toBe(opener) + const popupView = popup?.view as unknown as { webContents?: MockView['webContents'] } + const destroyed = popupView.webContents?.on.mock.calls.find( + ([event]) => event === 'destroyed' + )?.[1] as (() => void) | undefined + + // Electron drops a view's contents once the page closes itself. + popupView.webContents = undefined + destroyed?.() + + expect(session.listTabs().map((tab) => tab.tabId)).toEqual([opener.id]) + expect(session.automationTab()).toBe(opener) + }) + + it('opens a background-disposition popup by URL and keeps cross-scheme popups denied', () => { + const source = (session.ensureTab().view as unknown as MockView).webContents + const openWindow = source.setWindowOpenHandler.mock.calls[0]?.[0] as PopupHandler + + openWindow({ url: 'https://example.com/later' }).createWindow?.({}) + const popup = (session.activeTab()?.view as unknown as MockView).webContents + expect(popup).not.toBe(source) + expect(popup.loadURL).toHaveBeenCalledWith('https://example.com/later') + expect(openWindow({ url: 'javascript:alert(1)' })).toEqual({ action: 'deny' }) + expect(session.listTabs()).toHaveLength(2) + }) + it('opens agent working tabs behind the visible page', () => { // browser_open_tab is the agent choosing a page to work in. Which page is // shown is the renderer's decision, so the native selection never moves. @@ -2681,10 +2748,8 @@ describe('browser-agent session', () => { onTabCreated.mockClear() session.setAutomationActive(true) - const openWindow = source.setWindowOpenHandler.mock.calls[0]?.[0] as (details: { - url: string - }) => { action: string } - openWindow({ url: 'https://agent-popup.example/' }) + const openWindow = source.setWindowOpenHandler.mock.calls[0]?.[0] as PopupHandler + openWindow({ url: 'https://agent-popup.example/' }).createWindow?.({}) const agentPopup = session.automationTab() expect(agentPopup).not.toBeNull() expect(session.activeTab()).toBe(sourceTab) @@ -2718,12 +2783,10 @@ describe('browser-agent session', () => { it('lets internal page popups navigate after the network check', async () => { panel.setPanelBounds({ x: 100, y: 50, width: 800, height: 600 }) const source = (session.ensureTab().view as unknown as MockView).webContents - const openWindow = source.setWindowOpenHandler.mock.calls[0]?.[0] as (details: { - url: string - }) => { action: string } + const openWindow = source.setWindowOpenHandler.mock.calls[0]?.[0] as PopupHandler const destination = 'http://127.0.0.1:4099/private?token=secret' - openWindow({ url: destination }) + openWindow({ url: destination }).createWindow?.({}) const popup = (session.activeTab()?.view as unknown as MockView).webContents const request = beginMainFrameRequest(popup, destination) @@ -3990,8 +4053,9 @@ describe('browser-agent session', () => { const retainedDownload = mockDownloadItem({ filename: 'retained.bin', totalBytes: 100 }) startMockDownload(suspendedContents, suspendedDownload) - startMockDownload(retainedContents, retainedDownload) + await vi.waitFor(() => expect(getFreeDiskBytes).toHaveBeenCalledOnce()) await vi.waitFor(() => expect(suspendedDownload.item.setSavePath).toHaveBeenCalledOnce()) + startMockDownload(retainedContents, retainedDownload) await vi.waitFor(() => expect(retainedDownload.item.resume).toHaveBeenCalledOnce()) onDownloadsChanged.mockClear() @@ -4308,3 +4372,80 @@ describe('importAgentCookies', () => { await expect(session.importAgentCookies([cookie('a')])).rejects.toThrow('Disk unavailable') }) }) + +describe('first-party browser sessions', () => { + const origin = 'https://www.dev.sim.ai' + function initialize(persistence?: sessionModule.BrowserSessionPersistence) { + return freshSession(null, {}, persistence, undefined, { + origin, + session: new WebContentsView().webContents.session, + }) + } + + it('adopts the app session for a blank tab without changing its identity', () => { + const session = initialize() + const tab = session.addAutomationTab() + const original = tab.view.webContents + vi.mocked(original.getURL).mockReturnValue('about:blank') + const contents = session.tabForNavigation(original, `${origin}/home`, { agentOwned: true }) + expect(contents).not.toBe(original) + expect(agentAppOrigin(contents)).toBe(origin) + expect(session.automationTab()?.id).toBe(tab.id) + expect(session.listTabs()).toHaveLength(1) + expect(original.close).toHaveBeenCalledOnce() + expect(contents.session.setPermissionRequestHandler).not.toHaveBeenCalled() + expect(contents.session.webRequest.onBeforeRequest).not.toHaveBeenCalled() + }) + + it('keeps a populated tab and its history when crossing the session boundary', () => { + const session = initialize() + const tab = session.addAutomationTab(`${origin}/home`) + const original = tab.view.webContents + vi.mocked(original.getURL).mockReturnValue(`${origin}/home`) + const destination = session.tabForNavigation(original, 'https://example.com/', { + agentOwned: true, + }) + expect(destination).not.toBe(original) + expect(agentAppOrigin(destination)).toBeUndefined() + expect(session.listTabs()).toHaveLength(2) + expect(original.close).not.toHaveBeenCalled() + expect(session.navigationTarget(original)).toBe(destination) + expect(session.automationTab()?.view.webContents).toBe(destination) + session.recordPageLoadFailure(original, { + kind: 'load-error', + code: -2, + description: 'ERR_FAILED', + url: 'https://example.com/', + }) + expect(session.pageIssueForContents(original)).toBeUndefined() + expect(routeAgentNavigation(original, `${origin}/home`)).toBe(false) + expect(session.navigationTarget(original)).toBe(original) + }) + + it('does not replay cross-session form submissions as GET requests', () => { + const session = initialize() + const tab = session.addAutomationTab(`${origin}/home`) + const contents = tab.view.webContents + expect(routeAgentNavigation(contents, 'https://example.com/submit', 'POST')).toBe(true) + expect(session.listTabs()).toHaveLength(1) + expect(contents.loadURL).not.toHaveBeenCalled() + expect(routeAgentNavigation(contents, `${origin}/submit`, 'POST')).toBe(false) + }) + + it('restores Sim and external tabs into their respective sessions', () => { + const { persistence } = memoryBrowserPersistence({ + 'chat-test': { + v: 1, + tabs: [{ url: `${origin}/home` }, { url: 'https://example.com/' }], + activeIndex: 0, + downloads: [], + }, + }) + const session = initialize(persistence) + session.restoreBrowserSession() + const first = session.switchTab('1').view.webContents + const second = session.switchTab('2').view.webContents + expect(agentAppOrigin(first)).toBe(origin) + expect(agentAppOrigin(second)).toBeUndefined() + }) +}) diff --git a/apps/desktop/src/main/browser-agent/session.ts b/apps/desktop/src/main/browser-agent/session.ts index 1e106142eae..c9fed5fed65 100644 --- a/apps/desktop/src/main/browser-agent/session.ts +++ b/apps/desktop/src/main/browser-agent/session.ts @@ -26,12 +26,14 @@ import { getErrorMessage } from '@sim/utils/errors' import { generateId } from '@sim/utils/id' import type { BrowserWindow, + BrowserWindowConstructorOptions, CookiesSetDetails, DownloadItem, Input, MenuItemConstructorOptions, Session, WebContents, + WebContentsViewConstructorOptions, } from 'electron' import { app, @@ -59,16 +61,18 @@ import { panelUpdateAllowed, panelWindow, } from '@/main/browser-agent/panel' -import { registerAgentWebContents } from '@/main/browser-agent/registry' import { - checkAgentUrl, - clearHostVerdictCache, - isBlockedRequestUrl, - isBlockedSubresourceUrl, - subresourceNeedsResolution, -} from '@/main/browser-agent/url-guard' + agentAppOrigin, + type BrowserPermissionHandlers, + isAgentWebContents, + registerAgentNavigation, + registerAgentWebContents, +} from '@/main/browser-agent/registry' +import { handleBrowserRequest } from '@/main/browser-agent/request-policy' +import { clearHostVerdictCache } from '@/main/browser-agent/url-guard' import type { BrowserSessionSnapshot } from '@/main/desktop-chat-session-store' import { suggestedFilename, uniqueDownloadPath } from '@/main/downloads' +import { isAppOrigin } from '@/main/navigation' import { type FocusedResourceShortcut, isResourceTabSelectionShortcut, @@ -81,6 +85,16 @@ const logger = createLogger('BrowserAgentSession') /** Dedicated cookie jar for the agent browser; `persist:` = survives restarts. */ const AGENT_PARTITION = 'persist:sim-browser-agent' +/** The existing app session is borrowed only by views confined to this origin. */ +export interface BrowserAppSession { + origin: string + session: Session +} + +let browserAppSession: BrowserAppSession | undefined +const configuredDownloadSessions = new WeakSet() +const routedNavigations = new WeakMap() + class SessionError extends Error {} export interface AgentTab { @@ -96,6 +110,8 @@ export interface AgentTab { pendingMediaPermission?: PendingMediaPermission mediaPermissionGrant?: MediaPermissionGrant lastRealUserGestureAt?: number + /** The tab whose page opened this one; agent work returns there when this tab closes. */ + openerTabId?: string } interface PendingMediaPermission { @@ -127,6 +143,8 @@ export interface BrowserDownloadSettings { } export interface AgentSessionEvents { + /** The native page moved or its visibility changed. */ + onPanelGeometryChanged?: () => void /** The browser session ended (all tabs gone). */ onSessionClosed: () => void /** A newly created tab's WebContents, for the driver to instrument. */ @@ -277,6 +295,10 @@ function liveBrowserTabCount(): number { return count } +function hasTabCapacity(): boolean { + return tabs.length < MAX_LIVE_TABS_PER_SCOPE && liveBrowserTabCount() < MAX_LIVE_TABS_GLOBAL +} + function assertTabCapacity(): void { if (tabs.length >= MAX_LIVE_TABS_PER_SCOPE) { throw new SessionError(`A task browser can have at most ${MAX_LIVE_TABS_PER_SCOPE} open tabs.`) @@ -779,11 +801,13 @@ export function showBrowserDownloadsMenu( return true } -/** Reveals a completed download without launching the downloaded file. */ -export function showBrowserDownloadInFolder(scopeId: string, downloadId: string): boolean { - const resolved = resolveBrowserScopeId(scopeId) +/** A finished download of this scope whose file is still on disk. */ +export function completedBrowserDownload( + scopeId: string, + downloadId: string +): { filename: string; savePath: string } | null { const download = browserDownloadsByScope - .get(resolved) + .get(resolveBrowserScopeId(scopeId)) ?.find((candidate) => candidate.id === downloadId) if ( !download || @@ -791,8 +815,15 @@ export function showBrowserDownloadInFolder(scopeId: string, downloadId: string) typeof download.savePath !== 'string' || !existsSync(download.savePath) ) { - return false + return null } + return { filename: download.filename, savePath: download.savePath } +} + +/** Reveals a completed download without launching the downloaded file. */ +export function showBrowserDownloadInFolder(scopeId: string, downloadId: string): boolean { + const download = completedBrowserDownload(scopeId, downloadId) + if (!download) return false shell.showItemInFolder(download.savePath) return true } @@ -837,14 +868,17 @@ export function initSession( handlers: AgentSessionEvents, mainWindowProvider: () => BrowserWindow | null, persistence?: BrowserSessionPersistence, - downloadSettings?: BrowserDownloadSettings + downloadSettings?: BrowserDownloadSettings, + appSession?: BrowserAppSession ): void { resetSessionState() + browserAppSession = appSession events = handlers getMainWindow = mainWindowProvider browserSessionPersistence = persistence ?? null browserDownloadSettings = downloadSettings ?? null initPanel({ + onGeometryChanged: () => events?.onPanelGeometryChanged?.(), getMainWindow: () => getMainWindow(), activeTab: () => { const scopeId = getActiveBrowserScopeId() @@ -1318,18 +1352,8 @@ export async function respondToMediaPermission(requestId: string, allowed: boole publishPageIssue(tab) } -/** - * Default-deny hardening for the agent partition. Site permissions remain - * denied apart from ALLOWED_SITE_PERMISSIONS. Media is granted only after a - * renderer-owned, document-scoped prompt validates the requesting origin, - * active visible tab, recent native user input, and operating-system grant. - * Uploads use Chromium's native file chooser and downloads are saved into the - * device-level browser download directory. - */ -function configureAgentPartition(ses: Session): void { - if (configuredPartitions.has(ses)) return - configuredPartitions.add(ses) - ses.setPermissionRequestHandler((contents, permission, callback, details) => { +const browserPermissions: BrowserPermissionHandlers = { + request: (contents, permission, callback, details) => { if (permission === 'media') { const scoped = scopedTabForContents(contents) const request = details as { @@ -1380,8 +1404,8 @@ function configureAgentPartition(ses: Session): void { return } callback(ALLOWED_SITE_PERMISSIONS.has(permission)) - }) - ses.setPermissionCheckHandler((contents, permission, requestingOrigin, details) => { + }, + check: (contents, permission, requestingOrigin, details) => { if (permission === 'media') { if (!contents || details.isMainFrame !== true) return false const scoped = scopedTabForContents(contents) @@ -1411,56 +1435,34 @@ function configureAgentPartition(ses: Session): void { ) } return ALLOWED_SITE_PERMISSIONS.has(permission) - }) - // SSRF choke point for the agent partition. Document navigations (top-level + - // iframes) get the full DNS-resolving check — the one seam every navigation - // passes through, including page-initiated ones the driver never sees (server - // redirects, link clicks, location.href, meta-refresh) — so an internal host - // can't slip in that way. - // - // Subresources that come back readable, render into screenshots, or execute - // get the resolving check too, cached per host; fonts keep the cheap - // synchronous path. See isBlockedSubresourceUrl and - // subresourceNeedsResolution for why each way round. + }, +} + +/** + * Default-deny hardening for the agent partition. Site permissions remain + * denied apart from ALLOWED_SITE_PERMISSIONS. Media is granted only after a + * renderer-owned, document-scoped prompt validates the requesting origin, + * active visible tab, recent native user input, and operating-system grant. + * Uploads use Chromium's native file chooser and downloads are saved into the + * device-level browser download directory. + */ +function configureAgentPartition(ses: Session): void { + if (configuredPartitions.has(ses)) return + configuredPartitions.add(ses) + ses.setPermissionRequestHandler(browserPermissions.request) + ses.setPermissionCheckHandler(browserPermissions.check) ses.webRequest.onBeforeRequest((details, callback) => { - // Answered exactly once, and never throwing. A throw inside the `then` - // below would otherwise land in the `catch` and answer a second time, and - // by the time an async check settles the request's loader may be gone — - // now the case for most subresources, not just the odd navigation. - let settled = false - const settle = (cancel: boolean) => { - if (settled) return - settled = true - try { - callback({ cancel }) - } catch (error) { - logger.warn('Could not answer an agent request', { error: getErrorMessage(error) }) - } - } - if (details.resourceType === 'mainFrame' || details.resourceType === 'subFrame') { - void checkAgentUrl(details.url) - .then((guard) => { - if (!guard.ok) logger.warn('Blocked agent document navigation to a private host') - settle(!guard.ok) - }) - .catch((error) => { - logger.error('Agent SSRF check failed; cancelling request', { error }) - settle(true) - }) - return - } - if (!subresourceNeedsResolution(details.resourceType)) { - settle(isBlockedRequestUrl(details.url)) - return - } - void isBlockedSubresourceUrl(details.url) - .then((blocked) => settle(blocked)) - .catch((error) => { - logger.error('Agent subresource SSRF check failed; cancelling request', { error }) - settle(true) - }) + handleBrowserRequest(details, callback) }) + configureBrowserDownloads(ses) +} + +/** Borrowing app authentication must not replace its permission or request handlers. */ +function configureBrowserDownloads(ses: Session): void { + if (configuredDownloadSessions.has(ses)) return + configuredDownloadSessions.add(ses) ses.on('will-download', (_event, item, contents) => { + if (!isAgentWebContents(contents)) return const directory = browserDownloadSettings?.getDirectory() if (!directory) { logger.warn('Agent browser download has no configured destination') @@ -1682,6 +1684,7 @@ export function recordPageLoadFailure( contents: WebContents, issue: Extract ): void { + if ((issue.code === -2 || issue.code === -3) && routedNavigations.has(contents)) return const tab = tabForContents(contents) if (!tab) return tab.pageIssue = issue @@ -1904,6 +1907,34 @@ export function stopFindInActiveTab(focusPage: boolean): void { if (tab) tab.view.webContents.focus() } +/** + * Whether a page's popup can become a tab that keeps its opener: http(s), within the tab + * limits, and in the opener's own session (a session boundary needs a separate, opener-less tab). + */ +function canAdoptPopup(opener: WebContents, url: string): boolean { + if (!/^https?:\/\//i.test(url) || !hasTabCapacity()) return false + return ( + !browserAppSession || + isAppOrigin(url, browserAppSession.origin) === Boolean(agentAppOrigin(opener)) + ) +} + +/** Registers a page-opened window as a tab that remembers its opener. */ +function adoptPopupTab( + opener: WebContents, + url: string, + popup: PopupWindowOptions, + agentOwned: boolean +): WebContents { + const openerTabId = tabForContents(opener)?.id + const tab = agentOwned ? addAutomationTab(url, popup) : addTab(url, popup) + tab.openerTabId = openerTabId + const contents = tab.view.webContents + // A background-tab disposition defers creation, so Chromium supplies no contents to adopt. + if (!popup.webContents) void contents.loadURL(url).catch(() => {}) + return contents +} + /** * Opens a link from a page in another tab of this browser. Shared by the * window.open interception and the page's right-click menu — both have to stay @@ -1913,7 +1944,7 @@ export function stopFindInActiveTab(focusPage: boolean): void { function openTabWithUrl(url: string, { agentOwned }: { agentOwned: boolean }): void { if (!/^https?:\/\//i.test(url)) return try { - const tab = agentOwned ? addAutomationTab() : addTab() + const tab = agentOwned ? addAutomationTab(url) : addTab(url) void tab.view.webContents.loadURL(url).catch(() => {}) } catch (error) { logger.warn('Could not open a link in a new browser tab', { @@ -1922,16 +1953,47 @@ function openTabWithUrl(url: string, { agentOwned }: { agentOwned: boolean }): v } } -function createTabView(): WebContentsView { +/** + * The options Electron hands `createWindow`. Its typings omit the Chromium-created `webContents` + * the documented adoption pattern forwards (absent only for a deferred background-tab popup). + */ +type PopupWindowOptions = BrowserWindowConstructorOptions & WebContentsViewConstructorOptions + +/** + * Creates a tab's view. `popup` adopts a page-opened window's WebContents, which Chromium created + * with the opener's (identical) web preferences, so the opener relationship and its session stay + * intact; callers only adopt popups whose URL belongs to the opener's session. + */ +function createTabView(url?: string, popup?: PopupWindowOptions): WebContentsView { + const appSession = + url && browserAppSession && isAppOrigin(url, browserAppSession.origin) + ? browserAppSession + : undefined const scopeId = getBrowserScopeId() - const view = new WebContentsView({ + const view = popup?.webContents ? new WebContentsView(popup) : createFreshTabView(appSession) + try { + /** Detached tabs must lay out before a foreground panel owns their native view. */ + const [width, height] = getMainWindow()?.getContentSize() ?? [1280, 720] + view.setBounds({ x: 0, y: 0, width: Math.max(1, width), height: Math.max(1, height) }) + return initializeTabView(view, scopeId, appSession?.origin) + } catch (error) { + if (!view.webContents.isDestroyed()) view.webContents.close() + throw error + } +} + +function createFreshTabView(appSession: BrowserAppSession | undefined): WebContentsView { + return new WebContentsView({ webPreferences: { - partition: AGENT_PARTITION, + ...(appSession ? { session: appSession.session } : { partition: AGENT_PARTITION }), contextIsolation: true, nodeIntegration: false, sandbox: true, webSecurity: true, webviewTag: false, + // Electron's plugin switch only admits its internal plugins; the built-in + // PDF viewer is one, and without it a PDF renders as an empty frame. + plugins: true, // A minimal, isolated preload that reports login-form presence and // performs user-authorized credential fills. It exposes nothing to the // page, and runs in the top-level frame only. @@ -1946,19 +2008,42 @@ function createTabView(): WebContentsView { zoomFactor: getBrowserDefaultZoomFactor(), }, }) - try { - return initializeTabView(view, scopeId) - } catch (error) { - if (!view.webContents.isDestroyed()) view.webContents.close() - throw error - } } -function initializeTabView(view: WebContentsView, scopeId: string): WebContentsView { +function initializeTabView( + view: WebContentsView, + scopeId: string, + appOrigin?: string +): WebContentsView { view.setBackgroundColor(browserBackgroundColor()) const contents = view.webContents - registerAgentWebContents(contents) - configureAgentPartition(contents.session) + registerAgentWebContents(contents, appOrigin, browserPermissions) + if (appOrigin) configureBrowserDownloads(contents.session) + else configureAgentPartition(contents.session) + const routeNavigation = (url: string, method: string): boolean => { + if (!/^https?:\/\//i.test(url) || !browserAppSession) return false + const wantsAppSession = isAppOrigin(url, browserAppSession.origin) + if (wantsAppSession === Boolean(appOrigin)) { + routedNavigations.delete(contents) + return false + } + /** Never turn a form POST or a method-preserving redirect into a GET in another session. */ + if (method !== 'GET') return true + try { + return withBrowserScope(scopeId, () => { + const target = tabForNavigation(contents, url, { reuseBlank: false }) + if (target === contents) return false + void target.loadURL(url).catch(() => {}) + return true + }) + } catch (error) { + logger.warn('Could not route browser navigation to its session', { + error: getErrorMessage(error), + }) + return true + } + } + registerAgentNavigation(contents, routeNavigation) attachAgentContextMenu(contents, { addToChat: (text) => withBrowserScope(scopeId, () => addPageSelectionToChat(contents, text)), openTab: (url) => withBrowserScope(scopeId, () => openTabWithUrl(url, { agentOwned: false })), @@ -2017,14 +2102,25 @@ function initializeTabView(view: WebContentsView, scopeId: string): WebContentsV // Keep popups inside the browser resource: http(s) window.open and // target=_blank requests become a new internal tab, never a native window. - contents.setWindowOpenHandler((details) => { - withBrowserScope(scopeId, () => - openTabWithUrl(details.url, { - agentOwned: agentOwnsPopupFrom(contents), - }) - ) - return { action: 'deny' } - }) + // A same-session popup adopts Chromium's own WebContents so window.opener + // survives (sign-in and "connect" popups post their result back to it). + contents.setWindowOpenHandler((details) => + withBrowserScope(scopeId, () => { + const agentOwned = agentOwnsPopupFrom(contents) + if (!canAdoptPopup(contents, details.url)) { + openTabWithUrl(details.url, { agentOwned }) + return { action: 'deny' } + } + return { + action: 'allow', + outlivesOpener: true, + createWindow: (options) => + withBrowserScope(scopeId, () => + adoptPopupTab(contents, details.url, options, agentOwned) + ), + } + }) + ) // A page can call window.resizeTo/window.moveTo, and Electron otherwise // applies that request to the BrowserWindow which owns this view. Controlled @@ -2191,10 +2287,11 @@ function initializeTabView(view: WebContentsView, scopeId: string): WebContentsV contents.on( 'destroyed', bindToBrowserScope(scopeId, () => { + // closeTab unlists a tab before closing it, so a listed tab here closed itself + // (window.close() at the end of a sign-in popup). Electron has already dropped + // the view's contents, so remove it without touching them. const tab = tabs.find((entry) => entry.view === view) - if (tab) { - revokeTabMediaPermissions(tab, false) - } + if (tab) removeTab(tab, null) events?.onTabClosed(contents) }) ) @@ -2363,9 +2460,16 @@ export function requireTab(): AgentTab { interface AddTabOptions { activate?: boolean notify?: boolean + url?: string + popup?: PopupWindowOptions } -function addTabInternal({ activate = true, notify = true }: AddTabOptions = {}): AgentTab { +function addTabInternal({ + activate = true, + notify = true, + url, + popup, +}: AddTabOptions = {}): AgentTab { assertTabCapacity() const previousActiveTab = activeTab() const transferBrowserFocus = @@ -2375,7 +2479,7 @@ function addTabInternal({ activate = true, notify = true }: AddTabOptions = {}): const tab: AgentTab = { id: String(currentScope.nextTabId++), scopeId: getBrowserScopeId(), - view: createTabView(), + view: createTabView(url, popup), } tabs.push(tab) if (currentScope.automationTabId === null) currentScope.automationTabId = tab.id @@ -2646,8 +2750,49 @@ export async function waitForPendingTabRestore(tab: AgentTab): Promise return pending ? await pending.ready : true } +/** + * A session boundary opens a separate tab, preserving the source's native history. + * A blank tab can adopt the target session before its first navigation instead. + */ +export function tabForNavigation( + contents: WebContents, + url: string, + options: { agentOwned?: boolean; reuseBlank?: boolean } = {} +): WebContents { + const tab = tabForContents(contents) + if (!tab || !browserAppSession) return contents + const wantsAppSession = isAppOrigin(url, browserAppSession.origin) + if (wantsAppSession === Boolean(agentAppOrigin(contents))) return contents + if (options.reuseBlank !== false && (!contents.getURL() || contents.getURL() === 'about:blank')) { + const view = createTabView(url) + detachIfAttached(tab.view) + tab.view = view + contents.close() + applyActiveTabThrottling() + layout() + events?.onActiveTabChanged(view.webContents) + return view.webContents + } + const target = + (options.agentOwned ?? agentOwnsPopupFrom(contents)) ? addAutomationTab(url) : addTab(url) + routedNavigations.set(contents, target) + return target.view.webContents +} + +/** Follows an intercepted redirect without treating its cancelled source load as a failure. */ +export function navigationTarget(contents: WebContents): WebContents { + let current = contents + for (let count = 0; count < 20; count++) { + const target = routedNavigations.get(current) + if (!target || target.view.webContents.isDestroyed()) return current + current = target.view.webContents + } + throw new SessionError('Too many browser session redirects.') +} + /** Prevents a delayed restore slot from overwriting a newer explicit navigation. */ export function prepareExplicitNavigation(contents: WebContents): void { + routedNavigations.delete(contents) const tab = tabForContents(contents) if (!tab) return tab.pendingRestoreUrl = undefined @@ -2734,7 +2879,7 @@ export function restoreBrowserSession(): void { snapshot.downloads.map((download) => ({ ...download })) ) for (const { entry } of selectedEntries) { - const tab = addTabInternal({ activate: false, notify: false }) + const tab = addTabInternal({ activate: false, notify: false, url: entry.url }) tab.pendingRestoreUrl = entry.url restoredTabs.push(tab) restoredLoads.push({ tab, url: entry.url }) @@ -2784,10 +2929,10 @@ export function restoreBrowserSession(): void { } } -export function addTab(): AgentTab { +export function addTab(url?: string, popup?: PopupWindowOptions): AgentTab { restoreBrowserSession() currentScope.visibleTabUserSelected = true - return addTabInternal() + return addTabInternal({ url, popup }) } /** @@ -2795,9 +2940,9 @@ export function addTab(): AgentTab { * renderer's decision: every page is a resource tab there, and it shows the * agent's tab or badges it depending on what the user is doing. */ -export function addAutomationTab(): AgentTab { +export function addAutomationTab(url?: string, popup?: PopupWindowOptions): AgentTab { restoreBrowserSession() - const tab = addTabInternal({ activate: false, notify: false }) + const tab = addTabInternal({ activate: false, notify: false, url, popup }) currentScope.automationTabId = tab.id applyActiveTabThrottling() persistBrowserSession() @@ -2837,7 +2982,7 @@ export function reopenClosedTab(): AgentTab | null { if (!url) return null currentScope.visibleTabUserSelected = true - const tab = addTabInternal() + const tab = addTabInternal({ url }) if (url !== 'about:blank') { // No checkAgentUrl here, unlike the tool-driven navigations: the stored // URL was already sanitized to http(s) on close, and the partition's @@ -2926,23 +3071,39 @@ export function closeTab( { adoptNeighborForAgent = false }: { adoptNeighborForAgent?: boolean } = {} ): void { restoreBrowserSession() - const index = tabs.findIndex((entry) => entry.id === tabId) - if (index < 0) throw new SessionError(`No tab with id ${tabId} — call browser_list_tabs.`) + const tab = tabs.find((entry) => entry.id === tabId) + if (!tab) throw new SessionError(`No tab with id ${tabId} — call browser_list_tabs.`) + removeTab(tab, tab.view.webContents, adoptNeighborForAgent) +} + +/** + * Unlists a tab and moves selection off it. `contents` is null when the page already + * closed itself; the tab then leaves no reopenable history entry. + */ +function removeTab( + tab: AgentTab, + contents: WebContents | null, + adoptNeighborForAgent = false +): void { + const index = tabs.indexOf(tab) // Before the splice, while the tab is still resolvable, stop page-owned UI. - dismissFind(tabId) - clearAutomationIndicatorsForTab(tabId) - const [tab] = tabs.splice(index, 1) + if (contents) dismissFind(tab.id) + clearAutomationIndicatorsForTab(tab.id) + tabs.splice(index, 1) + if (!contents) dismissFind(tab.id) discardPendingTabRestore(tab) revokeTabMediaPermissions(tab, false) - recentlyClosedTabUrls.unshift(sanitizeRestorableUrl(tabUrl(tab)) ?? 'about:blank') - if (recentlyClosedTabUrls.length > MAX_RECENTLY_CLOSED_TABS) { - recentlyClosedTabUrls.length = MAX_RECENTLY_CLOSED_TABS + if (contents) { + recentlyClosedTabUrls.unshift(sanitizeRestorableUrl(tabUrl(tab)) ?? 'about:blank') + if (recentlyClosedTabUrls.length > MAX_RECENTLY_CLOSED_TABS) { + recentlyClosedTabUrls.length = MAX_RECENTLY_CLOSED_TABS + } } const transferBrowserFocus = - currentScope.focusedBrowserTabId === tab.id || tab.view.webContents.isFocused() + currentScope.focusedBrowserTabId === tab.id || Boolean(contents?.isFocused()) clearFocusedBrowserTab(tab.id) detachIfAttached(tab.view) - tab.view.webContents.close() + if (contents && !contents.isDestroyed()) contents.close() if (currentScope.activeTabId === tab.id) { currentScope.activeTabId = (tabs[index] ?? tabs[index - 1])?.id ?? null layout() @@ -2952,9 +3113,9 @@ export function closeTab( } } if (currentScope.automationTabId === tab.id) { - currentScope.automationTabId = adoptNeighborForAgent - ? ((tabs[index] ?? tabs[index - 1])?.id ?? null) - : null + const opener = tabs.find((entry) => entry.id === tab.openerTabId) + currentScope.automationTabId = + opener?.id ?? (adoptNeighborForAgent ? ((tabs[index] ?? tabs[index - 1])?.id ?? null) : null) applyActiveTabThrottling() } if (transferBrowserFocus) currentScope.focusedBrowserTabId = currentScope.activeTabId diff --git a/apps/desktop/src/main/browser-credentials/fill.test.ts b/apps/desktop/src/main/browser-credentials/fill.test.ts index ba70332204e..3326d0510e1 100644 --- a/apps/desktop/src/main/browser-credentials/fill.test.ts +++ b/apps/desktop/src/main/browser-credentials/fill.test.ts @@ -3,20 +3,48 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' vi.mock('electron', () => import('@/test/electron-mock')) import { sleep } from '@sim/utils/helpers' -import type { BrowserWindow, WebContents } from 'electron' -import { Menu } from 'electron' -import { FillCoordinator } from '@/main/browser-credentials/fill' +import { BrowserWindow, type WebContents } from 'electron' +import { FillCoordinator, type FillCoordinatorDeps } from '@/main/browser-credentials/fill' +import type { CredentialPicker } from '@/main/browser-credentials/picker' import type { CredentialVault } from '@/main/browser-credentials/vault' +import type { CredentialFormReport } from '@/shared/browser-credentials' const ORIGIN = 'https://example.com' const SCOPE = 'chat-a' -const WINDOW = {} as BrowserWindow +const WINDOW = Object.assign(new BrowserWindow(), { + getContentBounds: () => ({ x: 0, y: 0, width: 1200, height: 800 }), + isDestroyed: () => false, + isVisible: () => true, + isMinimized: () => false, + isFocused: () => true, + focus: vi.fn(), +}) + +const { pickerOptions, pickerFocus } = vi.hoisted(() => ({ + pickerOptions: vi.fn(), + pickerFocus: vi.fn(), +})) +vi.mock('@/main/browser-credentials/picker', () => ({ + CredentialPicker: class { + constructor(private readonly options: ConstructorParameters[0]) { + pickerOptions(options) + } + close() { + this.options.closed() + } + position() {} + focus() { + pickerFocus() + } + }, +})) function fakeContents(url = `${ORIGIN}/login`) { return { getURL: vi.fn(() => url), isDestroyed: vi.fn(() => false), send: vi.fn(), + focus: vi.fn(), } } @@ -40,7 +68,19 @@ function fakeVault(overrides: Partial> = {}) { type Contents = ReturnType -function setup(contents: Contents = fakeContents(), vault = fakeVault()) { +function deferred() { + let resolve!: (value: T) => void + const promise = new Promise((done) => { + resolve = done + }) + return { promise, resolve } +} + +function setup( + contents: Contents = fakeContents(), + vault = fakeVault(), + pickerHost?: FillCoordinatorDeps['pickerHost'] +) { const onAvailabilityChanged = vi.fn() let active: Contents | null = contents let activeScope = SCOPE @@ -52,6 +92,7 @@ function setup(contents: Contents = fakeContents(), vault = fakeVault()) { !scopeId || scopeId === activeScope ? (active as unknown as WebContents | null) : null, scopeOwnsContents: (scopeId, candidate) => contentsScopes.get(candidate) === scopeId, onAvailabilityChanged, + pickerHost, }) return { coordinator, @@ -66,15 +107,11 @@ function setup(contents: Contents = fakeContents(), vault = fakeVault()) { } } -function loginFormState( - overrides: Partial<{ - origin: string - hasLoginForm: boolean - hasPasswordField: boolean - }> = {} -) { +function loginFormState(overrides: Partial = {}) { return { origin: ORIGIN, + targetId: 'target-1', + bounds: null, hasLoginForm: true, hasPasswordField: true, ...overrides, @@ -85,10 +122,13 @@ function loginFormState( async function openChooser(context: ReturnType) { context.coordinator.noteFormState(context.contents as unknown as WebContents, loginFormState()) await context.coordinator.showChooser(WINDOW, { x: 10, y: 20 }) - const template = vi.mocked(Menu.buildFromTemplate).mock.calls.at(-1)?.[0] as - | Array<{ label: string; click: () => void }> - | undefined - return template ?? [] + const options = pickerOptions.mock.calls.at(-1)?.[0] as ConstructorParameters< + typeof CredentialPicker + >[0] + return options.configuration.accounts.map((account) => ({ + label: account.username, + click: () => options.select(account.id), + })) } /** @@ -106,7 +146,8 @@ async function settle(): Promise { } beforeEach(() => { - vi.mocked(Menu.buildFromTemplate).mockClear() + pickerOptions.mockClear() + pickerFocus.mockClear() }) describe('fill availability', () => { @@ -242,6 +283,91 @@ describe('fill availability', () => { }) describe('credential chooser', () => { + it.each(['hidden', 'minimized', 'unfocused'])( + 'does not open when the parent becomes %s during a metadata lookup', + async (state) => { + const context = setup() + context.coordinator.noteFormState( + context.contents as unknown as WebContents, + loginFormState() + ) + const matches = await context.vault.listForOrigin() + const pending = deferred() + context.vault.listForOrigin.mockReturnValueOnce(pending.promise) + let available = true + const window = Object.assign(new BrowserWindow(), { + ...WINDOW, + isVisible: () => state !== 'hidden' || available, + isMinimized: () => state === 'minimized' && !available, + isFocused: () => state !== 'unfocused' || available, + }) + const opened = context.coordinator.showChooser(window, { x: 0, y: 0 }) + available = false + pending.resolve(matches) + await expect(opened).resolves.toBe(false) + expect(pickerOptions).not.toHaveBeenCalled() + } + ) + + it.each(['disappears', 'moves'])( + 'rechecks the field host when it %s during a metadata lookup', + async (change) => { + const anchor = { x: 10, y: 20, width: 200, height: 30 } + const pickerHost = vi.fn>(() => ({ + window: WINDOW, + anchor, + })) + const context = setup(fakeContents(), fakeVault(), pickerHost) + context.coordinator.noteFormState( + context.contents as unknown as WebContents, + loginFormState({ bounds: anchor }) + ) + const matches = await context.vault.listForOrigin() + const pending = deferred() + context.vault.listForOrigin.mockReturnValueOnce(pending.promise) + const opened = context.coordinator.showChooser(WINDOW, { x: 0, y: 0 }) + const moved = { ...anchor, x: 90, y: 100 } + pickerHost.mockReturnValue(change === 'disappears' ? null : { window: WINDOW, anchor: moved }) + pending.resolve(matches) + await expect(opened).resolves.toBe(change === 'moves') + if (change === 'disappears') expect(pickerOptions).not.toHaveBeenCalled() + else expect(pickerOptions).toHaveBeenCalledWith(expect.objectContaining({ anchor: moved })) + } + ) + + it.each(['navigation', 'tab change'])('does not restore focus after %s', async (change) => { + const context = setup() + await openChooser(context) + const options = pickerOptions.mock.calls.at(-1)![0] as ConstructorParameters< + typeof CredentialPicker + >[0] + if (change === 'navigation') { + context.coordinator.noteNavigation(context.contents as unknown as WebContents) + } else context.setActive(fakeContents()) + options.restoreFocus() + expect(context.contents.focus).not.toHaveBeenCalled() + }) + + it('does not let a superseded keyboard request focus a newer picker', async () => { + const anchor = { x: 0, y: 0, width: 200, height: 30 } + const context = setup(fakeContents(), fakeVault(), () => ({ window: WINDOW, anchor })) + const contents = context.contents as unknown as WebContents + context.coordinator.noteFormState(contents, loginFormState({ bounds: anchor })) + const matches = await context.vault.listForOrigin() + const pending = deferred() + context.vault.listForOrigin.mockReturnValueOnce(pending.promise) + const first = context.coordinator.requestPicker(contents, 'focus') + context.coordinator.noteFormState( + contents, + loginFormState({ bounds: anchor, targetId: 'next' }) + ) + await context.coordinator.requestPicker(contents, 'open') + expect(pickerOptions).toHaveBeenCalledOnce() + pending.resolve(matches) + await first + expect(pickerFocus).not.toHaveBeenCalled() + }) + it('lists usernames without reading any password', async () => { const context = setup() const template = await openChooser(context) @@ -290,8 +416,17 @@ describe('renderer credential chooser', () => { context.coordinator.noteFormState(context.contents as unknown as WebContents, loginFormState()) await context.coordinator.listFillOptions(SCOPE) - await expect(context.coordinator.fillCredential('c1', SCOPE)).resolves.toBe(true) + const result = context.coordinator.fillCredential('c1', SCOPE) + await settle() + const request = context.contents.send.mock.calls.at(-1)?.[1] + context.coordinator.noteFillResult(context.contents as unknown as WebContents, { + requestId: request.requestId, + status: 'filled', + }) + await expect(result).resolves.toBe(true) expect(context.contents.send).toHaveBeenCalledWith('browser-credentials:fill', { + requestId: expect.any(String), + targetId: 'target-1', origin: ORIGIN, username: 'ada', password: 'hunter2', @@ -351,10 +486,12 @@ describe('performing a fill', () => { const context = setup() const template = await openChooser(context) - template[0].click() + void template[0].click() await settle() expect(context.contents.send).toHaveBeenCalledWith('browser-credentials:fill', { + requestId: expect.any(String), + targetId: 'target-1', origin: ORIGIN, username: 'ada', password: 'hunter2', @@ -368,15 +505,16 @@ describe('performing a fill', () => { loginFormState({ hasPasswordField: false }) ) await context.coordinator.showChooser(WINDOW, { x: 10, y: 20 }) - const template = vi.mocked(Menu.buildFromTemplate).mock.calls.at(-1)?.[0] as Array<{ - click: () => void - }> - - template[0].click() + const options = pickerOptions.mock.calls.at(-1)![0] as ConstructorParameters< + typeof CredentialPicker + >[0] + void options.select('c1') await settle() // The page has nowhere to put a password, so it does not get one. expect(context.contents.send).toHaveBeenCalledWith('browser-credentials:fill', { + requestId: expect.any(String), + targetId: 'target-1', origin: ORIGIN, username: 'ada', password: undefined, @@ -388,7 +526,7 @@ describe('performing a fill', () => { const template = await openChooser(context) context.coordinator.noteNavigation(context.contents as unknown as WebContents) - template[0].click() + void template[0].click() await settle() expect(context.vault.readForFill).not.toHaveBeenCalled() @@ -402,7 +540,7 @@ describe('performing a fill', () => { const template = await openChooser(context) context.contents.getURL.mockReturnValue('https://evil.test/login') - template[0].click() + void template[0].click() await settle() expect(context.vault.readForFill).not.toHaveBeenCalled() @@ -414,7 +552,7 @@ describe('performing a fill', () => { const template = await openChooser(context) context.setActive(fakeContents()) - template[0].click() + void template[0].click() await settle() expect(context.contents.send).not.toHaveBeenCalled() @@ -425,7 +563,7 @@ describe('performing a fill', () => { const template = await openChooser(context) context.contents.isDestroyed.mockReturnValue(true) - template[0].click() + void template[0].click() await settle() expect(context.contents.send).not.toHaveBeenCalled() @@ -448,7 +586,7 @@ describe('performing a fill', () => { const context = setup(fakeContents(), vault) const template = await openChooser(context) - template[0].click() + void template[0].click() await settle() context.coordinator.noteNavigation(context.contents as unknown as WebContents) releaseRead() @@ -462,9 +600,141 @@ describe('performing a fill', () => { const context = setup(fakeContents(), fakeVault({ readForFill: vi.fn(async () => null) })) const template = await openChooser(context) - template[0].click() + void template[0].click() await settle() expect(context.contents.send).not.toHaveBeenCalled() }) }) + +describe('fill acknowledgements and target freshness', () => { + async function start() { + const context = setup() + context.coordinator.noteFormState(context.contents as unknown as WebContents, loginFormState()) + await context.coordinator.listFillOptions(SCOPE) + const result = context.coordinator.fillCredential('c1', SCOPE) + await settle() + const request = context.contents.send.mock.calls.at(-1)![1] + return { ...context, result, request } + } + + it('does not report success merely because IPC was sent', async () => { + const context = await start() + let settled = false + void context.result.then(() => { + settled = true + }) + context.coordinator.noteFillResult(context.contents as unknown as WebContents, { + requestId: 'wrong-request', + status: 'filled', + }) + context.coordinator.noteFillResult(fakeContents() as unknown as WebContents, { + requestId: context.request.requestId, + status: 'filled', + }) + await settle() + expect(settled).toBe(false) + context.coordinator.noteFillResult(context.contents as unknown as WebContents, { + requestId: context.request.requestId, + status: 'failed', + }) + await expect(context.result).resolves.toBe(false) + }) + + it('invalidates an in-flight fill when the selected form changes', async () => { + const context = await start() + context.coordinator.noteFormState( + context.contents as unknown as WebContents, + loginFormState({ targetId: 'replacement' }) + ) + context.coordinator.noteFillResult(context.contents as unknown as WebContents, { + requestId: context.request.requestId, + status: 'filled', + }) + await expect(context.result).resolves.toBe(false) + }) + + it('expires an unacknowledged fill', async () => { + vi.useFakeTimers() + try { + const context = setup() + context.coordinator.noteFormState( + context.contents as unknown as WebContents, + loginFormState() + ) + await context.coordinator.listFillOptions(SCOPE) + const result = context.coordinator.fillCredential('c1', SCOPE) + await vi.advanceTimersByTimeAsync(2_000) + await expect(result).resolves.toBe(false) + } finally { + vi.useRealTimers() + } + }) + + it('invalidates selection before reading secrets when a form is replaced', async () => { + const context = setup() + await openChooser(context) + context.coordinator.noteFormState( + context.contents as unknown as WebContents, + loginFormState({ targetId: 'replacement' }) + ) + await expect(context.coordinator.fillCredential('c1', SCOPE)).resolves.toBe(false) + expect(context.vault.readForFill).not.toHaveBeenCalled() + }) +}) + +describe('native picker lifetime', () => { + it('survives unchanged browser visibility heartbeats', async () => { + const context = setup() + await openChooser(context) + await context.coordinator.refreshAvailability(true) + const options = pickerOptions.mock.calls.at(-1)![0] as ConstructorParameters< + typeof CredentialPicker + >[0] + const result = options.select('c1') + await settle() + const request = context.contents.send.mock.calls.at(-1)![1] + context.coordinator.noteFillResult(context.contents as unknown as WebContents, { + requestId: request.requestId, + status: 'filled', + }) + await expect(result).resolves.toBe('filled') + }) + + it.each(['dismiss', 'dispose', 'closed'] as const)( + 'cancels a vault read after %s', + async (action) => { + let resolve!: (value: { username: string; password: string }) => void + const promise = new Promise<{ username: string; password: string }>((done) => { + resolve = done + }) + const context = setup(fakeContents(), fakeVault({ readForFill: vi.fn(() => promise) })) + await openChooser(context) + const options = pickerOptions.mock.calls.at(-1)![0] as ConstructorParameters< + typeof CredentialPicker + >[0] + const result = options.select('c1') + if (action === 'dispose') context.coordinator.dispose() + else if (action === 'closed') options.closed() + else context.coordinator.dismissPicker() + resolve({ username: 'ada', password: 'fixture-secret' }) + await expect(result).resolves.toBe('stale-target') + expect(context.contents.send).not.toHaveBeenCalled() + } + ) + + it('preserves older hosted chooser authorization while its native panel is occluded', async () => { + const context = setup() + context.coordinator.noteFormState(context.contents as unknown as WebContents, loginFormState()) + await context.coordinator.listFillOptions(SCOPE) + context.coordinator.dismissPicker() + const result = context.coordinator.fillCredential('c1', SCOPE) + await settle() + const request = context.contents.send.mock.calls.at(-1)![1] + context.coordinator.noteFillResult(context.contents as unknown as WebContents, { + requestId: request.requestId, + status: 'filled', + }) + await expect(result).resolves.toBe(true) + }) +}) diff --git a/apps/desktop/src/main/browser-credentials/fill.ts b/apps/desktop/src/main/browser-credentials/fill.ts index b05e6fba3d7..ec5483a7e7d 100644 --- a/apps/desktop/src/main/browser-credentials/fill.ts +++ b/apps/desktop/src/main/browser-credentials/fill.ts @@ -1,71 +1,66 @@ import type { BrowserCredentialMetadata } from '@sim/desktop-bridge' import { createLogger } from '@sim/logger' +import { generateId } from '@sim/utils/id' import type { BrowserWindow, WebContents } from 'electron' -import { Menu } from 'electron' import { normalizeOrigin } from '@/main/browser-credentials/origin' +import { CredentialPicker } from '@/main/browser-credentials/picker' import type { CredentialVault } from '@/main/browser-credentials/vault' +import type { + CredentialFieldBounds, + CredentialFillResult, + CredentialFillStatus, + CredentialFormReport, +} from '@/shared/browser-credentials' const logger = createLogger('BrowserCredentialFill') +const FILL_TIMEOUT_MS = 2_000 -/** - * Decides when a credential may be filled, and does the filling. - * - * The page can navigate at any point between "this page has a login form", - * "the user opened the chooser", and "the user picked an account" — and a fill - * aimed at the wrong document means a password handed to the wrong site. So - * every authorization is bound to a specific tab and a specific navigation - * generation, and that binding is revalidated immediately before plaintext - * leaves the vault, not just when the chooser opened. - * - * Renderer-owned chrome receives only password-free metadata. The main - * process keeps the corresponding one-shot authorization, and the password - * travels only from the vault to the browser page after every binding is - * revalidated. - */ - -interface FormState { - origin: string - hasLoginForm: boolean - /** - * Whether the page currently has somewhere to put a password. - * - * False on the first step of an identifier-first sign-in, which asks for an - * email and only reveals the password field after it is submitted. Those - * steps are still worth filling — the username is what they want — so the - * password simply is not sent to a page that has nowhere to put it. - */ - hasPasswordField: boolean - /** Bumped on every navigation, so a stale authorization cannot be replayed. */ +interface FormState extends CredentialFormReport { generation: number } +interface PickerHost { + window: BrowserWindow + anchor: CredentialFieldBounds +} + export interface FillCoordinatorDeps { vault: CredentialVault - /** The tab the user is actually looking at, optionally constrained to one chat scope. */ getActiveContents: (scopeId?: string) => WebContents | null - /** Whether a live tab belongs to the renderer-requested chat scope. */ scopeOwnsContents: (scopeId: string, contents: WebContents) => boolean - /** Push the fill affordance's visibility to the Sim renderer. */ onAvailabilityChanged: (available: boolean, contents: WebContents | null) => void + /** Screen coordinates for the focused field, only while its native page is visible. */ + pickerHost?: (contents: WebContents, bounds: CredentialFieldBounds) => PickerHost | null +} + +interface SelectionAuthorization { + pickerVersion?: number + credentialIds: ReadonlySet + generation: number + targetId: string } -export interface FormStateReport { - origin: string - hasLoginForm: boolean - hasPasswordField: boolean +interface PendingFill { + pickerVersion?: number + requestId: string + targetId: string + generation: number + complete: (status: CredentialFillStatus) => void } +/** Keeps authorization in main and binds every user selection to a live document and form. */ export class FillCoordinator { private readonly states = new WeakMap() private readonly generations = new WeakMap() - private readonly selectionAuthorizations = new WeakMap< - WebContents, - { credentialIds: ReadonlySet; generation: number } - >() + private readonly selectionAuthorizations = new WeakMap() + private readonly pendingFills = new Map() private readonly availabilityRefreshes = new WeakMap() private availabilityRefreshWithoutContents = 0 private readonly lastAvailability = new WeakMap() private lastAvailabilityWithoutContents = false + private picker: { view: CredentialPicker; contents: WebContents; targetId: string } | null = null + private pickerVersion = 0 + private disposed = false constructor(private readonly deps: FillCoordinatorDeps) {} @@ -73,230 +68,314 @@ export class FillCoordinator { return this.generations.get(contents) ?? 0 } - /** - * Records what the browser preload observed. The report is trusted only as - * far as it goes: it can claim a form exists, but the origin it names is - * checked against the live URL before any fill. - */ - noteFormState(contents: WebContents, report: FormStateReport): void { - this.selectionAuthorizations.delete(contents) + noteFormState(contents: WebContents, report: CredentialFormReport): void { + const previous = this.states.get(contents) const origin = normalizeOrigin(report.origin) - if (origin === null) { + if (previous?.targetId !== report.targetId || previous?.origin !== origin) { + this.selectionAuthorizations.delete(contents) + this.pendingFills.get(contents)?.complete('stale-target') + if (this.picker?.contents === contents) this.dismissPicker() + } + if (origin === null || !report.targetId || !report.hasLoginForm) { this.states.delete(contents) } else { - this.states.set(contents, { - origin, - hasLoginForm: report.hasLoginForm, - hasPasswordField: report.hasPasswordField, - generation: this.generationFor(contents), - }) + this.states.set(contents, { ...report, origin, generation: this.generationFor(contents) }) + } + if (this.picker?.contents === contents) { + const host = report.bounds ? this.deps.pickerHost?.(contents, report.bounds) : null + if (host) this.picker.view.position(host.anchor) + else this.dismissPicker() } void this.refreshAvailability() } - /** - * Invalidates everything known about a tab's page. Called on every - * navigation, including in-page ones — a single-page app can swap a login - * form for a different site's UI without a document load. - */ noteNavigation(contents: WebContents, sameDocument = false): void { this.generations.set(contents, this.generationFor(contents) + 1) this.states.delete(contents) this.selectionAuthorizations.delete(contents) - // A same-document navigation keeps the preload and its report fingerprint - // alive. Ask it to report again after clearing main-process state, or an - // unchanged login form remains invisible until a full page reload. - // Literal channel name: the IPC contract audit resolves constants only on - // the preload side, so main-process sends must inline the channel. + this.pendingFills.get(contents)?.complete('stale-target') + if (this.picker?.contents === contents) this.dismissPicker() if (sameDocument && !contents.isDestroyed()) contents.send('browser-credentials:rescan') void this.refreshAvailability() } forget(contents: WebContents): void { - this.states.delete(contents) + this.noteNavigation(contents) this.generations.delete(contents) - this.selectionAuthorizations.delete(contents) - void this.refreshAvailability() } - /** Whether one tab has a login form with at least one saved match. */ + dispose(): void { + this.disposed = true + this.dismissPicker() + for (const pending of this.pendingFills.values()) pending.complete('stale-target') + } + private async isFillAvailableFor(contents: WebContents | null): Promise { - if (!contents || contents.isDestroyed()) return false + if (this.disposed || !contents || contents.isDestroyed()) return false const state = this.states.get(contents) - if (!state?.hasLoginForm) return false - if (!this.deps.vault.isAvailable()) return false + if (!state?.hasLoginForm || !this.deps.vault.isAvailable()) return false return (await this.deps.vault.listForOrigin(state.origin)).length > 0 } - /** Whether the active tab can currently be filled. */ isFillAvailable(): Promise { return this.isFillAvailableFor(this.deps.getActiveContents()) } - /** - * Lists only password-safe metadata matching the active scoped login form. - * - * The result also establishes a one-shot selection authorization bound to - * the current tab and navigation generation. A renderer-owned menu can then - * ask to fill one of these ids without letting a stale menu follow a - * navigation into a replacement document. - */ - async listFillOptions(scopeId?: string): Promise { + /** Kept for hosted renderers that still use the previous toolbar chooser. */ + async listFillOptions( + scopeId?: string, + pickerVersion?: number + ): Promise { const contents = this.deps.getActiveContents(scopeId) if (!contents || contents.isDestroyed()) return [] - if (scopeId && !this.deps.scopeOwnsContents(scopeId, contents)) return [] - const state = this.states.get(contents) - if (!state?.hasLoginForm || !this.deps.vault.isAvailable()) return [] - const authorizedGeneration = state.generation - if (!this.isStillAuthorized(contents, authorizedGeneration, scopeId)) return [] - if (normalizeOrigin(contents.getURL()) !== state.origin) return [] - + if (!state?.targetId || !this.deps.vault.isAvailable()) return [] + const authorization = { + pickerVersion, + generation: state.generation, + targetId: state.targetId, + credentialIds: new Set(), + } + if (!this.isStillAuthorized(contents, authorization, scopeId)) return [] const matches = await this.deps.vault.listForOrigin(state.origin) - if (!this.isStillAuthorized(contents, authorizedGeneration, scopeId)) return [] - if (normalizeOrigin(contents.getURL()) !== state.origin) return [] - - this.selectionAuthorizations.set(contents, { - credentialIds: new Set(matches.map((credential) => credential.id)), - generation: authorizedGeneration, - }) + if (!this.isStillAuthorized(contents, authorization, scopeId)) return [] + authorization.credentialIds = new Set(matches.map((credential) => credential.id)) + this.selectionAuthorizations.set(contents, authorization) return matches } - /** Fills one option from the latest scoped list without returning its password. */ async fillCredential(credentialId: string, scopeId?: string): Promise { - const contents = this.deps.getActiveContents(scopeId) - if (!contents || contents.isDestroyed()) return false - if (scopeId && !this.deps.scopeOwnsContents(scopeId, contents)) return false + return (await this.fillSelected(credentialId, scopeId)) === 'filled' + } + private async fillSelected( + credentialId: string, + scopeId?: string, + expected?: SelectionAuthorization + ): Promise { + const contents = this.deps.getActiveContents(scopeId) + if (!contents || contents.isDestroyed()) return 'stale-target' const authorization = this.selectionAuthorizations.get(contents) this.selectionAuthorizations.delete(contents) - if (!authorization?.credentialIds.has(credentialId)) return false + if (expected && authorization !== expected) return 'stale-target' + if (!authorization?.credentialIds.has(credentialId)) return 'stale-target' + if (!this.isStillAuthorized(contents, authorization, scopeId)) return 'stale-target' + const state = this.states.get(contents)! + const credential = await this.deps.vault.readForFill(credentialId, state.origin) + if (!credential) return 'failed' + if (!this.isStillAuthorized(contents, authorization, scopeId)) return 'stale-target' + this.pendingFills.get(contents)?.complete('stale-target') + const requestId = generateId() + return new Promise((resolve) => { + const complete = (status: CredentialFillStatus) => { + if (this.pendingFills.get(contents)?.requestId !== requestId) return + clearTimeout(timeout) + this.pendingFills.delete(contents) + if (status === 'filled') logger.info('Filled a user-selected saved credential') + resolve(status) + } + const timeout = setTimeout(() => complete('failed'), FILL_TIMEOUT_MS) + this.pendingFills.set(contents, { requestId, ...authorization, complete }) + try { + contents.send('browser-credentials:fill', { + requestId, + targetId: authorization.targetId, + origin: state.origin, + username: credential.username, + password: state.hasPasswordField ? credential.password : undefined, + }) + } catch { + complete('failed') + } + }) + } - return this.fill(contents, credentialId, authorization.generation, scopeId) + noteFillResult(contents: WebContents, result: CredentialFillResult): void { + const pending = this.pendingFills.get(contents) + if (!pending || pending.requestId !== result.requestId) return + const authorized = this.isStillAuthorized(contents, pending) + pending.complete(authorized ? result.status : 'stale-target') } - /** - * Publishes the active tab's current state. Activation forces a replay so a - * newly mounted chat receives its value even when the previous chat happened - * to have the same boolean availability. - */ async refreshAvailability(force = false): Promise { + if (this.disposed) return const contents = this.deps.getActiveContents() + if (this.picker && this.picker.contents !== contents) this.dismissPicker() + for (const [owner, pending] of this.pendingFills) { + if (owner !== contents) pending.complete('stale-target') + } const refresh = contents ? (this.availabilityRefreshes.get(contents) ?? 0) + 1 : this.availabilityRefreshWithoutContents + 1 - if (contents) { - this.availabilityRefreshes.set(contents, refresh) - } else { - this.availabilityRefreshWithoutContents = refresh - } + if (contents) this.availabilityRefreshes.set(contents, refresh) + else this.availabilityRefreshWithoutContents = refresh const available = await this.isFillAvailableFor(contents) - if (this.deps.getActiveContents() !== contents) return + if (this.disposed || this.deps.getActiveContents() !== contents) return if ( - contents - ? this.availabilityRefreshes.get(contents) !== refresh - : this.availabilityRefreshWithoutContents !== refresh - ) { + (contents + ? this.availabilityRefreshes.get(contents) + : this.availabilityRefreshWithoutContents) !== refresh + ) return - } const previous = contents ? this.lastAvailability.get(contents) : this.lastAvailabilityWithoutContents + if (!available && this.picker?.contents === contents) this.dismissPicker() if (!force && available === previous) return - if (contents) { - this.lastAvailability.set(contents, available) - } else { - this.lastAvailabilityWithoutContents = available - } + if (contents) this.lastAvailability.set(contents, available) + else this.lastAvailabilityWithoutContents = available this.deps.onAvailabilityChanged(available, contents) } - /** - * Shows the native account chooser near a point in the window. - * - * Only usernames are listed; no password is read until the user picks one. - * The navigation generation is captured here and carried into the fill, so a - * page that moves while the menu is open invalidates the choice. - */ + /** Real input in the active page opens or focuses the trusted account picker. */ + async requestPicker(contents: WebContents, action: 'open' | 'focus' | 'dismiss'): Promise { + if (this.deps.getActiveContents() !== contents) return + if (action === 'dismiss') { + this.dismissPicker() + return + } + const state = this.states.get(contents) + if (!state || !this.fieldPickerHost(contents)) return + if (!this.picker || this.picker.targetId !== state.targetId) { + if (!(await this.openPicker(contents, () => this.fieldPickerHost(contents)))) return + } + if ( + action === 'focus' && + this.picker?.contents === contents && + this.picker.targetId === state.targetId + ) + this.picker.view.focus() + } + + /** Older hosted clients retain a functional chooser during independent desktop/web rollouts. */ async showChooser( window: BrowserWindow, anchor: { x: number; y: number }, scopeId?: string ): Promise { const contents = this.deps.getActiveContents(scopeId) - if (!contents || contents.isDestroyed()) return false - if (scopeId && !this.deps.scopeOwnsContents(scopeId, contents)) return false - const state = this.states.get(contents) - if (!state?.hasLoginForm) return false - - const matches = await this.deps.vault.listForOrigin(state.origin) - if (matches.length === 0) return false - - const authorizedGeneration = state.generation - const menu = Menu.buildFromTemplate( - matches.map((credential) => ({ - label: credential.username || '(no username)', - click: () => { - void this.fill(contents, credential.id, authorizedGeneration, scopeId).catch(() => {}) - }, - })) + if (!contents) return false + const useFieldHost = Boolean(this.fieldPickerHost(contents)) + const opened = await this.openPicker( + contents, + () => { + if (useFieldHost) return this.fieldPickerHost(contents) + if (window.isDestroyed()) return null + const bounds = window.getContentBounds() + return { + window, + anchor: { x: bounds.x + anchor.x, y: bounds.y + anchor.y, width: 1, height: 1 }, + } + }, + scopeId ) - menu.popup({ window, x: Math.round(anchor.x), y: Math.round(anchor.y) }) - return true + if (opened) this.picker?.view.focus() + return opened } - /** - * Performs one authorized fill. - * - * Every precondition is checked again here rather than trusted from when the - * chooser opened, and once more after the vault read, because that read is - * asynchronous and the page can navigate inside it. - */ - private async fill( + private async openPicker( contents: WebContents, - credentialId: string, - authorizedGeneration: number, + resolveHost: () => PickerHost | null, scopeId?: string ): Promise { - if (!this.isStillAuthorized(contents, authorizedGeneration, scopeId)) return false + this.dismissPicker() + const version = this.pickerVersion + const matches = await this.listFillOptions(scopeId, version) + if ( + version !== this.pickerVersion || + this.deps.getActiveContents(scopeId) !== contents || + contents.isDestroyed() + ) + return false const state = this.states.get(contents) - if (!state) return false - - // The origin the preload reported must still be the document's real - // origin. This is the check that stops a fill following a page that - // navigated to another site. - if (normalizeOrigin(contents.getURL()) !== state.origin) return false - - const credential = await this.deps.vault.readForFill(credentialId, state.origin) - if (credential === null) return false - - if (!this.isStillAuthorized(contents, authorizedGeneration, scopeId)) return false - if (normalizeOrigin(contents.getURL()) !== state.origin) return false - - contents.send('browser-credentials:fill', { - origin: state.origin, - username: credential.username, - // Withheld on an identifier-first step: the page has no password field, - // so sending it would put plaintext in a document that cannot use it. - password: state.hasPasswordField ? credential.password : undefined, + const authorization = this.selectionAuthorizations.get(contents) + if ( + !state?.targetId || + !authorization || + !this.isStillAuthorized(contents, authorization, scopeId) + ) + return false + if (!matches.length) return false + const host = resolveHost() + if ( + !host || + host.window.isDestroyed() || + !host.window.isVisible() || + host.window.isMinimized() || + !host.window.isFocused() + ) { + this.dismissPicker() + return false + } + const { window, anchor } = host + const view = new CredentialPicker({ + parent: window, + anchor, + configuration: { + origin: state.origin, + accounts: matches.map(({ id, username }) => ({ id, username })), + }, + select: (id) => this.fillSelected(id, scopeId, authorization), + restoreFocus: () => { + if ( + !window.isDestroyed() && + window.isVisible() && + !window.isMinimized() && + this.isStillAuthorized( + contents, + { generation: authorization.generation, targetId: authorization.targetId }, + scopeId + ) + ) { + window.focus() + contents.focus() + } + }, + closed: () => { + if (this.picker?.view === view) this.dismissPicker() + if (this.selectionAuthorizations.get(contents) === authorization) + this.selectionAuthorizations.delete(contents) + }, }) - // Counts and outcomes only — never the origin, username, or password. - logger.info('Filled a saved credential at the user\u2019s request') + this.picker = { view, contents, targetId: state.targetId } return true } + private fieldPickerHost(contents: WebContents): PickerHost | null { + const bounds = this.states.get(contents)?.bounds + return bounds ? (this.deps.pickerHost?.(contents, bounds) ?? null) : null + } + + dismissPicker(): void { + this.pickerVersion++ + for (const pending of this.pendingFills.values()) { + if (pending.pickerVersion !== undefined) pending.complete('stale-target') + } + const picker = this.picker + this.picker = null + picker?.view.close() + } + private isStillAuthorized( contents: WebContents, - authorizedGeneration: number, + authorization: { generation: number; targetId: string; pickerVersion?: number }, scopeId?: string ): boolean { - if (contents.isDestroyed()) return false - // A fill must land in the tab the user is looking at. Switching tabs - // between choosing and filling cancels it. - if (this.deps.getActiveContents(scopeId) !== contents) return false + if ( + this.disposed || + (authorization.pickerVersion !== undefined && + authorization.pickerVersion !== this.pickerVersion) + ) + return false + if (contents.isDestroyed() || this.deps.getActiveContents(scopeId) !== contents) return false if (scopeId && !this.deps.scopeOwnsContents(scopeId, contents)) return false - if (this.generationFor(contents) !== authorizedGeneration) return false - return this.states.get(contents)?.generation === authorizedGeneration + const state = this.states.get(contents) + return Boolean( + state?.hasLoginForm && + state.targetId === authorization.targetId && + this.generationFor(contents) === authorization.generation && + state.generation === authorization.generation && + normalizeOrigin(contents.getURL()) === state.origin + ) } } diff --git a/apps/desktop/src/main/browser-credentials/index.ts b/apps/desktop/src/main/browser-credentials/index.ts index de21a6b7d68..e78086e4d2d 100644 --- a/apps/desktop/src/main/browser-credentials/index.ts +++ b/apps/desktop/src/main/browser-credentials/index.ts @@ -30,6 +30,7 @@ export function credentialVault(): CredentialVault { /** Creates the coordinator once the browser session can report its active tab. */ export function initFillCoordinator(deps: Omit): FillCoordinator { + coordinatorInstance?.dispose() coordinatorInstance = new FillCoordinator({ ...deps, vault: credentialVault() }) return coordinatorInstance } diff --git a/apps/desktop/src/main/browser-credentials/picker.ts b/apps/desktop/src/main/browser-credentials/picker.ts new file mode 100644 index 00000000000..e00d062a82a --- /dev/null +++ b/apps/desktop/src/main/browser-credentials/picker.ts @@ -0,0 +1,181 @@ +import { join } from 'node:path' +import { createLogger } from '@sim/logger' +import { getErrorMessage } from '@sim/utils/errors' +import { app, BrowserWindow, type InputEvent, screen, session } from 'electron' +import { hasRecentDiscreteInput, trackInputActivity } from '@/main/input-activity' +import { attachLocalPageProtocol, localPageUrl } from '@/main/local-pages' +import { attachShellTheme } from '@/main/shell-theme' +import { isShellWindowSender } from '@/main/shell-window' +import { createSecureWebPreferences } from '@/main/window-preferences' +import type { + CredentialFieldBounds, + CredentialFillStatus, + CredentialPickerConfiguration, +} from '@/shared/browser-credentials' + +const logger = createLogger('CredentialPicker') +const WIDTH = 320 +const PARTITION = 'credential-picker' + +interface CredentialPickerOptions { + parent: BrowserWindow + anchor: CredentialFieldBounds + configuration: CredentialPickerConfiguration + select: (id: string) => Promise + closed: () => void + restoreFocus: () => void +} + +/** A trusted, nonmodal emcn menu above the native page; website scripts cannot cover its rows. */ +export class CredentialPicker { + private readonly window: BrowserWindow + private anchor: CredentialFieldBounds + private height = 420 + private wantsFocus = false + private selecting = false + + constructor(options: CredentialPickerOptions) { + this.anchor = options.anchor + const ses = session.fromPartition(PARTITION) + ses.setPermissionRequestHandler((_contents, _permission, callback) => callback(false)) + ses.setPermissionCheckHandler(() => false) + attachLocalPageProtocol(ses) + const win = new BrowserWindow({ + parent: options.parent, + width: WIDTH, + height: this.height, + useContentSize: true, + frame: false, + transparent: true, + hasShadow: true, + resizable: false, + minimizable: false, + maximizable: false, + fullscreenable: false, + skipTaskbar: true, + show: false, + title: 'Saved passwords', + webPreferences: createSecureWebPreferences( + PARTITION, + join(__dirname, 'credential-picker-preload.cjs'), + app.isPackaged + ), + }) + this.window = win + const pageUrl = localPageUrl('credential-picker.html') + attachShellTheme(win) + trackInputActivity(win.webContents) + win.webContents.setWindowOpenHandler(() => ({ action: 'deny' })) + win.webContents.on('will-navigate', (event) => event.preventDefault()) + win.webContents.ipc.handle('credential-picker:configuration', (event) => { + if (!isShellWindowSender(win, pageUrl, event)) throw new Error('Untrusted picker sender') + return options.configuration + }) + win.webContents.ipc.handle('credential-picker:select', async (event, id: unknown) => { + if ( + !isShellWindowSender(win, pageUrl, event) || + !hasRecentDiscreteInput(win.webContents) || + this.selecting + ) + return 'failed' + if ( + typeof id !== 'string' || + !options.configuration.accounts.some((account) => account.id === id) + ) + return 'failed' + this.selecting = true + try { + return await options.select(id) + } catch (error) { + logger.warn('Saved password fill failed', { error: getErrorMessage(error) }) + return 'failed' + } finally { + this.selecting = false + } + }) + win.webContents.ipc.on('credential-picker:dismiss', (event) => { + if (!isShellWindowSender(win, pageUrl, event)) return + const restoreFocus = win.isFocused() + this.close() + if (restoreFocus) options.restoreFocus() + }) + win.webContents.ipc.on('credential-picker:resize', (event, height: unknown) => { + if ( + !isShellWindowSender(win, pageUrl, event) || + typeof height !== 'number' || + !Number.isFinite(height) + ) + return + clearTimeout(timeout) + this.height = Math.min(420, Math.max(40, Math.ceil(height))) + this.position(this.anchor) + if (!win.isVisible()) { + if (this.wantsFocus) win.show() + else win.showInactive() + } + if (this.wantsFocus) win.focus() + }) + const close = () => this.close() + options.parent.on('move', close) + options.parent.on('resize', close) + options.parent.on('hide', close) + options.parent.on('minimize', close) + options.parent.on('closed', close) + const parentInput = (_event: Electron.Event, input: InputEvent) => { + if (['mouseDown', 'rawKeyDown', 'keyDown', 'touchStart'].includes(input.type)) this.close() + } + const windowFocused = (_event: Electron.Event, focused: BrowserWindow) => { + if (focused !== options.parent && focused !== win) this.close() + } + options.parent.webContents.on('input-event', parentInput) + app.on('browser-window-focus', windowFocused) + app.on('did-resign-active', close) + win.on('blur', close) + const timeout = setTimeout(close, 5_000) + win.on('closed', () => { + clearTimeout(timeout) + options.parent.removeListener('move', close) + options.parent.removeListener('resize', close) + options.parent.removeListener('hide', close) + options.parent.removeListener('minimize', close) + options.parent.removeListener('closed', close) + options.parent.webContents.removeListener('input-event', parentInput) + app.removeListener('browser-window-focus', windowFocused) + app.removeListener('did-resign-active', close) + options.closed() + }) + void win.loadURL(pageUrl).catch((error) => { + logger.warn('Could not open saved password picker', { error: getErrorMessage(error) }) + close() + }) + } + + position(anchor: CredentialFieldBounds): void { + if (this.window.isDestroyed()) return + this.anchor = anchor + const area = screen.getDisplayMatching({ + ...anchor, + x: Math.round(anchor.x), + y: Math.round(anchor.y), + width: Math.ceil(anchor.width), + height: Math.ceil(anchor.height), + }).workArea + const below = anchor.y + anchor.height + 4 + const y = below + this.height <= area.y + area.height ? below : anchor.y - this.height - 4 + this.window.setBounds({ + x: Math.round(Math.max(area.x, Math.min(anchor.x, area.x + area.width - WIDTH))), + y: Math.round(Math.max(area.y, Math.min(y, area.y + area.height - this.height))), + width: WIDTH, + height: this.height, + }) + } + + focus(): void { + this.wantsFocus = true + if (!this.window.isDestroyed() && this.window.isVisible()) this.window.focus() + } + + close(): void { + if (!this.window.isDestroyed()) this.window.destroy() + } +} diff --git a/apps/desktop/src/main/downloads.ts b/apps/desktop/src/main/downloads.ts index 1267eeac3ea..a26582f379b 100644 --- a/apps/desktop/src/main/downloads.ts +++ b/apps/desktop/src/main/downloads.ts @@ -4,6 +4,7 @@ import { createLogger } from '@sim/logger' import { generateShortId } from '@sim/utils/id' import type { Session } from 'electron' import { app } from 'electron' +import { isAgentWebContents } from '@/main/browser-agent/registry' import type { EventRecorder } from '@/main/observability' const logger = createLogger('DesktopDownloads') @@ -136,7 +137,8 @@ export async function uniqueDownloadPath( * downloads bounce the Dock Downloads stack. */ export function attachDownloadHandling(session: Session, events: EventRecorder): void { - session.on('will-download', (_event, item) => { + session.on('will-download', (_event, item, contents) => { + if (contents && isAgentWebContents(contents)) return const filename = suggestedFilename(item.getFilename(), item.getMimeType()) item.setSaveDialogOptions({ defaultPath: join(app.getPath('downloads'), filename), diff --git a/apps/desktop/src/main/index.ts b/apps/desktop/src/main/index.ts index 18996aedcce..56888943323 100644 --- a/apps/desktop/src/main/index.ts +++ b/apps/desktop/src/main/index.ts @@ -744,7 +744,9 @@ function main(): void { }, { getDirectory: () => desktopSettings.getPreferences().browserDownloadDirectory, - } + }, + { origin: processOrigin, session: ensureAppSession() }, + localFilesystem ) if (accountDataAvailable()) { await localFilesystem.initialize() diff --git a/apps/desktop/src/main/ipc.test.ts b/apps/desktop/src/main/ipc.test.ts index 9ebb7d6f0d9..b74b899fb45 100644 --- a/apps/desktop/src/main/ipc.test.ts +++ b/apps/desktop/src/main/ipc.test.ts @@ -63,6 +63,8 @@ vi.mock('@/main/terminal-themes', () => ({ const { mockCoordinator } = vi.hoisted(() => ({ mockCoordinator: { noteFormState: vi.fn(), + noteFillResult: vi.fn(), + requestPicker: vi.fn(async () => {}), noteNavigation: vi.fn(), forget: vi.fn(), refreshAvailability: vi.fn(), @@ -505,6 +507,71 @@ describe('registerIpcHandlers', () => { }) }) + it('reads a native file through canonical IPC arguments without folder grants or user activation', async () => { + const { invoke } = collectHandlers() + const handler = invoke.get('desktop:local-files') + const path = fileURLToPath(import.meta.url) + const fetchAuthorization = vi.fn(async () => + Response.json({ chatId: 'chat-1', toolName: 'read_local_file', args: { path, limit: 64 } }) + ) + const authorizedEvent = { + senderFrame: { url: `${APP}/o/org/home` }, + sender: { session: { fetch: fetchAuthorization } }, + } + const mounts = vi.spyOn(deps.localFilesystem, 'handle') + expect( + await handler?.(authorizedEvent, { + operation: 'read', + toolCallId: 'tool-native', + path: '/not/the/canonical/path', + }) + ).toMatchObject({ + ok: true, + data: { kind: 'read', path, text: readFileSync(path, 'utf8').slice(0, 64) }, + }) + expect(mounts).not.toHaveBeenCalled() + expect(fetchAuthorization).toHaveBeenCalledWith( + `${APP}/api/desktop/tool/authorize`, + expect.objectContaining({ body: JSON.stringify({ toolCallId: 'tool-native' }) }) + ) + expect( + await handler?.(evilEvent, { operation: 'read', toolCallId: 'tool-native' }) + ).toMatchObject({ ok: false }) + }) + + it('claims native imports at IPC before traversal and rejects a replay', async () => { + const { invoke } = collectHandlers() + const handler = invoke.get('desktop:local-files') + const fetchAuthorization = vi + .fn() + .mockResolvedValueOnce( + Response.json({ + chatId: 'chat-1', + toolName: 'import_local_files', + args: { path: fileURLToPath(import.meta.url), targetWorkspaceId: 'workspace' }, + }) + ) + .mockResolvedValueOnce(Response.json({ error: 'already started' }, { status: 409 })) + const event = { + senderFrame: { url: `${APP}/o/org/home` }, + sender: { session: { fetch: fetchAuthorization } }, + } + const request = { operation: 'manifest', toolCallId: 'tool-import' } + expect(await handler?.(event, request)).toMatchObject({ + ok: true, + data: { + kind: 'manifest', + targetWorkspaceId: 'workspace', + entries: [{ relativePath: '', kind: 'file' }], + }, + }) + expect(fetchAuthorization).toHaveBeenCalledWith( + `${APP}/api/desktop/tool/authorize`, + expect.objectContaining({ body: JSON.stringify({ toolCallId: 'tool-import', claim: true }) }) + ) + expect(await handler?.(event, request)).toMatchObject({ ok: false, code: 'ALREADY_STARTED' }) + }) + it('requires server authorization for every privileged filesystem tool request', async () => { const { invoke } = collectHandlers() const handler = invoke.get('desktop:local-filesystem') @@ -1664,6 +1731,7 @@ describe('registerIpcHandlers', () => { expect(credentialChannels.sort()).toEqual([ 'browser-credentials:available', 'browser-credentials:copy', + 'browser-credentials:fill-result', 'browser-credentials:fill-selected', 'browser-credentials:forget', 'browser-credentials:forget-all', @@ -1671,6 +1739,7 @@ describe('registerIpcHandlers', () => { 'browser-credentials:import', 'browser-credentials:list', 'browser-credentials:list-fill-options', + 'browser-credentials:picker', 'browser-credentials:reveal', 'browser-credentials:show-chooser', ]) @@ -1730,6 +1799,8 @@ describe('registerIpcHandlers', () => { origin: 'https://example.com', hasLoginForm: true, hasPasswordField: false, + targetId: 'target-1', + bounds: null, } const browserPageEvent = { senderFrame: { url: 'https://example.com/login' }, @@ -1766,6 +1837,56 @@ describe('registerIpcHandlers', () => { expect(mockCoordinator.noteFormState).not.toHaveBeenCalled() }) + it('requires native browser input before opening the field picker', () => { + const { on } = collectHandlers() + const handler = on.get('browser-credentials:picker') + const tracked = trackedSender() + Object.assign(tracked.sender, { isBrowserTab: true }) + const event = { sender: tracked.sender, senderFrame: { url: 'https://example.com/login' } } + handler?.(activeAppEvent, 'open') + handler?.(event, 'open') + expect(mockCoordinator.requestPicker).not.toHaveBeenCalled() + tracked.press() + handler?.(event, 'unknown') + expect(mockCoordinator.requestPicker).not.toHaveBeenCalled() + handler?.(event, 'open') + expect(mockCoordinator.requestPicker).toHaveBeenCalledWith(tracked.sender, 'open') + }) + + it('accepts fill acknowledgements only from browser pages with a valid status', () => { + const { on } = collectHandlers() + const handler = on.get('browser-credentials:fill-result') + const event = { + sender: { isBrowserTab: true }, + senderFrame: { url: 'https://example.com/login' }, + } + const result = { requestId: 'request-1', status: 'filled' } + handler?.(activeAppEvent, result) + handler?.(evilEvent, result) + handler?.(event, { ...result, status: 'unknown' }) + expect(mockCoordinator.noteFillResult).not.toHaveBeenCalled() + handler?.(event, result) + expect(mockCoordinator.noteFillResult).toHaveBeenCalledWith(event.sender, result) + }) + + it.each([0, -1, Number.NaN, Number.POSITIVE_INFINITY])( + 'rejects invalid focused field dimensions: %s', + (width) => { + const { on } = collectHandlers() + on.get('browser-credentials:form-state')?.( + { sender: { isBrowserTab: true }, senderFrame: { url: 'https://example.com/login' } }, + { + origin: 'https://example.com', + targetId: 'target-1', + hasLoginForm: true, + hasPasswordField: true, + bounds: { x: 0, y: 0, width, height: 30 }, + } + ) + expect(mockCoordinator.noteFormState).not.toHaveBeenCalled() + } + ) + it('requires a live user gesture before opening the credential chooser', async () => { const { invoke } = collectHandlers() const handler = invoke.get('browser-credentials:show-chooser') diff --git a/apps/desktop/src/main/ipc.ts b/apps/desktop/src/main/ipc.ts index 50c3d64efd2..840334ce29c 100644 --- a/apps/desktop/src/main/ipc.ts +++ b/apps/desktop/src/main/ipc.ts @@ -81,6 +81,7 @@ import { isSafeInternalPath } from '@/main/config' import type { DesktopSettingsService } from '@/main/desktop-settings' import { isDesktopPreferenceKey } from '@/main/desktop-settings' import { hasRecentDeliberateInput, hasRecentDiscreteInput } from '@/main/input-activity' +import { executeLocalFileRequest } from '@/main/local-files' import type { LocalFilesystemService } from '@/main/local-filesystem' import { isAppOrigin, openExternalSafe } from '@/main/navigation' import type { ScopedEventRouter } from '@/main/scoped-event-router' @@ -507,7 +508,9 @@ interface DesktopToolAuthorization { async function fetchDesktopToolAuthorization( event: IpcMainInvokeEvent, deps: IpcDeps, - toolCallId: unknown + toolCallId: unknown, + claim = false, + onFailureStatus?: (status: number) => void ): Promise { if (!isDesktopToolCallId(toolCallId)) return null const startedAt = Date.now() @@ -518,11 +521,12 @@ async function fetchDesktopToolAuthorization( method: 'POST', credentials: 'include', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ toolCallId }), + body: JSON.stringify({ toolCallId, ...(claim ? { claim: true } : {}) }), signal: AbortSignal.timeout(BROWSER_TOOL_AUTHORIZATION_TIMEOUT_MS), } ) if (!response.ok) { + onFailureStatus?.(response.status) logger.warn('Desktop tool authorization was rejected', { toolCallId, status: response.status, @@ -712,6 +716,15 @@ export function registerIpcHandlers(deps: IpcDeps): void { return deps.beginOAuthConnect(providerId, parsedScope) }, }, + 'desktop:local-files': { + kind: 'invoke', + gate: 'app-origin', + requiresAccountData: true, + passSender: true, + denied: { ok: false, error: 'Local file tools are unavailable from this page.' }, + handler: (_sender, request, authorization) => + executeLocalFileRequest(request, authorization as DesktopToolAuthorization), + }, 'desktop:local-filesystem': { kind: 'invoke', gate: 'app-origin', @@ -1342,20 +1355,33 @@ export function registerIpcHandlers(deps: IpcDeps): void { kind: 'send', gate: 'browser-page', deviationReason: - "the only sender in this family that is a browser PAGE rather than the Sim app, so browser-page is the correct gate and requires:'browser' follows — with the browser off no such page exists", + 'the isolated browser preload reports its own form; app renderers cannot claim browser targets', requires: 'browser', passSender: true, handler: (sender, report) => { if (!isRecordLike(report)) return - const { origin, hasLoginForm, hasPasswordField } = report as { + const { origin, hasLoginForm, hasPasswordField, targetId, bounds } = report as { origin?: unknown hasLoginForm?: unknown hasPasswordField?: unknown + targetId?: unknown + bounds?: unknown } if ( typeof origin !== 'string' || typeof hasLoginForm !== 'boolean' || - typeof hasPasswordField !== 'boolean' + typeof hasPasswordField !== 'boolean' || + (targetId !== null && (typeof targetId !== 'string' || !ID_PATTERN.test(targetId))) || + (bounds !== null && + (!isRecordLike(bounds) || + !['x', 'y', 'width', 'height'].every( + (key) => + typeof bounds[key] === 'number' && + Number.isFinite(bounds[key]) && + Math.abs(bounds[key]) <= 100_000 + ) || + Number(bounds.width) <= 0 || + Number(bounds.height) <= 0)) ) { return } @@ -1363,12 +1389,66 @@ export function registerIpcHandlers(deps: IpcDeps): void { origin, hasLoginForm, hasPasswordField, + targetId: typeof targetId === 'string' ? targetId : null, + bounds: isRecordLike(bounds) + ? { + x: Number(bounds.x), + y: Number(bounds.y), + width: Number(bounds.width), + height: Number(bounds.height), + } + : null, }) }, }, + 'browser-credentials:fill-result': { + kind: 'send', + gate: 'browser-page', + deviationReason: + 'only the isolated browser preload acknowledges a fill; no secret values are returned', + requires: 'browser', + passSender: true, + handler: (sender, result) => { + if ( + !isRecordLike(result) || + typeof result.requestId !== 'string' || + !ID_PATTERN.test(result.requestId) + ) + return + if ( + result.status !== 'filled' && + result.status !== 'stale-target' && + result.status !== 'failed' + ) + return + fillCoordinator()?.noteFillResult(sender as WebContents, { + requestId: result.requestId, + status: result.status, + }) + }, + }, + 'browser-credentials:picker': { + kind: 'send', + gate: 'browser-page', + deviationReason: + 'real input in the browser page opens the trusted picker; selection is authorized separately in its bundled window', + requires: 'browser', + needsUserActivation: true, + passSender: true, + handler: (sender, action) => { + if (action !== 'open' && action !== 'focus' && action !== 'dismiss') return + void fillCoordinator() + ?.requestPicker(sender as WebContents, action) + .catch((error) => { + logger.warn('Could not open saved password picker', { error: getErrorMessage(error) }) + }) + }, + }, 'browser-credentials:available': { kind: 'invoke', gate: 'app-origin', + deviationReason: + 'vault availability is account data and remains readable while the browser surface is disabled', requiresAccountData: true, denied: false, handler: () => credentialsAvailable(), @@ -1376,6 +1456,8 @@ export function registerIpcHandlers(deps: IpcDeps): void { 'browser-credentials:list': { kind: 'invoke', gate: 'app-origin', + deviationReason: + 'password management remains available while the browser surface is disabled', requiresAccountData: true, denied: [], handler: () => listCredentials(), @@ -1413,6 +1495,8 @@ export function registerIpcHandlers(deps: IpcDeps): void { 'browser-credentials:reveal': { kind: 'invoke', gate: 'app-origin', + deviationReason: + 'OS-authenticated password management does not require an active browser surface', requiresAccountData: true, needsUserActivation: true, denied: null, @@ -1421,6 +1505,8 @@ export function registerIpcHandlers(deps: IpcDeps): void { 'browser-credentials:copy': { kind: 'invoke', gate: 'app-origin', + deviationReason: + 'OS-authenticated password copying does not require an active browser surface', requiresAccountData: true, needsUserActivation: true, denied: false, @@ -1429,6 +1515,8 @@ export function registerIpcHandlers(deps: IpcDeps): void { 'browser-credentials:forget': { kind: 'invoke', gate: 'app-origin', + deviationReason: + 'users must be able to remove saved credentials while the browser surface is disabled', requiresAccountData: true, needsUserActivation: true, denied: [], @@ -1437,6 +1525,8 @@ export function registerIpcHandlers(deps: IpcDeps): void { 'browser-credentials:forget-all': { kind: 'invoke', gate: 'app-origin', + deviationReason: + 'users must be able to clear saved credentials while the browser surface is disabled', requiresAccountData: true, needsUserActivation: true, denied: [], @@ -1465,9 +1555,7 @@ export function registerIpcHandlers(deps: IpcDeps): void { ) }, }, - // Opens the native account chooser. The renderer only says "the user - // clicked the key icon, here"; it never learns which accounts exist, never - // names one, and never receives a password. The shell performs the fill. + /** The app requests a trusted chooser; only its selected account reaches the live page. */ 'browser-credentials:show-chooser': { kind: 'invoke', gate: 'app-origin', @@ -1984,6 +2072,32 @@ export function registerIpcHandlers(deps: IpcDeps): void { error: 'This local filesystem request is not an authorized pending Copilot tool call.', } } + if (channel === 'desktop:local-files') { + const request = args[0] + if (!isRecordLike(request)) return { ok: false, error: 'Invalid local file request.' } + let failureStatus: number | undefined + const authorization = await fetchDesktopToolAuthorization( + event, + deps, + request.toolCallId, + request.operation === 'manifest', + (status) => { + failureStatus = status + } + ) + if (failureStatus === 409) + return { + ok: false, + code: 'ALREADY_STARTED', + error: 'This import is already running or was already started.', + } + if ( + !authorization || + !['read_local_file', 'import_local_files'].includes(authorization.toolName) + ) + return { ok: false, error: 'This is not an authorized pending local file tool call.' } + handlerArgs = [request, authorization] + } if (spec.passSender) { handlerArgs = [event.sender, ...handlerArgs] } diff --git a/apps/desktop/src/main/local-files.test.ts b/apps/desktop/src/main/local-files.test.ts new file mode 100644 index 00000000000..1661834ca2b --- /dev/null +++ b/apps/desktop/src/main/local-files.test.ts @@ -0,0 +1,303 @@ +import { mkdir, mkdtemp, rm, symlink, truncate, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { PDFDocument } from 'pdf-lib' +import { afterEach, beforeEach, expect, it } from 'vitest' +import { executeLocalFileRequest } from '@/main/local-files' + +let root: string +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), 'sim-native-files-')) +}) +afterEach(async () => { + await rm(root, { recursive: true, force: true }) +}) + +it('reads an ordinary OS path without mounting or granting a folder, with bounded text pagination', async () => { + const path = join(root, 'notes.txt') + await writeFile(path, 'hello world') + const result = await executeLocalFileRequest( + { operation: 'read', toolCallId: 'read', path: '/untrusted' }, + { toolName: 'read_local_file', args: { path, offset: 6, limit: 5 } } + ) + expect(result).toEqual({ + ok: true, + data: { + kind: 'read', + path, + representation: 'text', + text: 'world', + offset: 6, + nextOffset: 11, + truncated: false, + }, + }) + const listing = await executeLocalFileRequest( + { operation: 'read' }, + { toolName: 'read_local_file', args: { path: root } } + ) + expect(listing).toMatchObject({ + ok: true, + data: { representation: 'directory', entries: [{ name: 'notes.txt', kind: 'file' }] }, + }) +}) + +it('returns actual image and bounded PDF observations instead of base64 text', async () => { + const png = Buffer.from( + 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+j2ioAAAAASUVORK5CYII=', + 'base64' + ) + const imagePath = join(root, 'image.png') + await writeFile(imagePath, png) + expect( + await executeLocalFileRequest( + { operation: 'read' }, + { toolName: 'read_local_file', args: { path: imagePath } } + ) + ).toMatchObject({ + ok: true, + data: { + representation: 'visual', + observations: [{ mediaType: 'image/png', data: png.toString('base64') }], + }, + }) + const pdf = await PDFDocument.create() + for (let i = 0; i < 22; i++) pdf.addPage() + const path = join(root, 'pages.pdf') + await writeFile(path, await pdf.save()) + const response = await executeLocalFileRequest( + { operation: 'read' }, + { toolName: 'read_local_file', args: { path } } + ) + expect(response).toMatchObject({ + ok: true, + data: { + representation: 'visual', + truncated: true, + observations: [{ mediaType: 'application/pdf', pageCount: 20 }], + }, + }) + if (!response.ok || response.data.kind !== 'read' || !response.data.observations?.[0]) + throw new Error('Missing PDF observation') + expect( + ( + await PDFDocument.load(Buffer.from(response.data.observations[0].data, 'base64')) + ).getPageCount() + ).toBe(20) +}) + +it('preserves nested and empty directories, imports binary bytes and detects changed files', async () => { + await mkdir(join(root, 'empty')) + await mkdir(join(root, 'nested')) + const path = join(root, 'nested', 'bytes.bin') + const bytes = Buffer.from([0, 1, 2, 255]) + await writeFile(path, bytes) + const authorization = { + toolName: 'import_local_files', + args: { path: root, targetWorkspaceId: 'target' }, + } + const result = await executeLocalFileRequest({ operation: 'manifest' }, authorization) + if (!result.ok || result.data.kind !== 'manifest') throw new Error('Expected manifest') + expect(result.data.entries.map((entry) => entry.relativePath)).toEqual([ + '', + 'empty', + 'nested', + 'nested/bytes.bin', + ]) + const entry = result.data.entries.at(-1)! + expect( + await executeLocalFileRequest( + { operation: 'chunk', relativePath: entry.relativePath, revision: entry.revision, offset: 0 }, + authorization + ) + ).toEqual({ ok: true, data: { kind: 'chunk', bytes: new Uint8Array(bytes), eof: true } }) + await writeFile(path, 'changed') + expect( + await executeLocalFileRequest( + { operation: 'chunk', relativePath: entry.relativePath, revision: entry.revision, offset: 0 }, + authorization + ) + ).toMatchObject({ ok: false, error: expect.stringContaining('changed') }) + expect( + await executeLocalFileRequest( + { operation: 'chunk', relativePath: '../other', revision: entry.revision, offset: 0 }, + authorization + ) + ).toMatchObject({ ok: false, error: expect.stringContaining('outside this import') }) +}) + +it('rejects missing paths and directory cycles with explicit errors before uploading', async () => { + expect( + await executeLocalFileRequest( + { operation: 'read' }, + { toolName: 'read_local_file', args: { path: join(root, 'missing') } } + ) + ).toMatchObject({ ok: false }) + await symlink(root, join(root, 'cycle')) + expect( + await executeLocalFileRequest( + { operation: 'manifest' }, + { toolName: 'import_local_files', args: { path: root, targetWorkspaceId: 'target' } } + ) + ).toMatchObject({ ok: false, error: expect.stringContaining('cycle') }) +}) + +it('refuses oversized import files before any workspace mutation or bulk allocation', async () => { + const path = join(root, 'large.bin') + await writeFile(path, '') + await truncate(path, 64 * 1024 * 1024 + 1) + expect( + await executeLocalFileRequest( + { operation: 'manifest' }, + { toolName: 'import_local_files', args: { path, targetWorkspaceId: 'workspace' } } + ) + ).toMatchObject({ ok: false, error: expect.stringContaining('64 MB') }) +}) + +it.each(['file', 'directory'] as const)( + 'rejects %s symlinks outside the import source during manifest and chunk reads', + async (kind) => { + const source = join(root, 'selected') + const outside = join(root, 'selected-other') + await mkdir(source) + await mkdir(outside) + await writeFile(join(outside, 'private.txt'), 'outside content') + const target = kind === 'file' ? join(outside, 'private.txt') : outside + await symlink(target, join(source, 'link')) + const authorization = { + toolName: 'import_local_files', + args: { path: source, targetWorkspaceId: 'target' }, + } + expect(await executeLocalFileRequest({ operation: 'manifest' }, authorization)).toMatchObject({ + ok: false, + error: expect.stringContaining('outside this import source'), + }) + const outsideManifest = await executeLocalFileRequest( + { operation: 'manifest' }, + { toolName: 'import_local_files', args: { path: target, targetWorkspaceId: 'target' } } + ) + if (!outsideManifest.ok || outsideManifest.data.kind !== 'manifest') + throw new Error('Expected manifest') + const entry = outsideManifest.data.entries.find((item) => item.kind === 'file')! + expect( + await executeLocalFileRequest( + { + operation: 'chunk', + relativePath: kind === 'file' ? 'link' : 'link/private.txt', + revision: entry.revision, + offset: 0, + }, + authorization + ) + ).toMatchObject({ ok: false, error: expect.stringContaining('outside this import source') }) + } +) + +it('supports internal symlinks and an explicitly selected symlink root, but rejects a retargeted child', async () => { + const source = join(root, 'selected') + await mkdir(source) + await writeFile(join(source, 'notes.txt'), 'inside') + await symlink(join(source, 'notes.txt'), join(source, 'alias.txt')) + const selectedAlias = join(root, 'selected-alias') + await symlink(source, selectedAlias) + const authorization = { + toolName: 'import_local_files', + args: { path: selectedAlias, targetWorkspaceId: 'target' }, + } + const manifest = await executeLocalFileRequest({ operation: 'manifest' }, authorization) + if (!manifest.ok || manifest.data.kind !== 'manifest') throw new Error('Expected manifest') + const entry = manifest.data.entries.find((item) => item.relativePath === 'alias.txt')! + const request = { + operation: 'chunk', + relativePath: 'alias.txt', + revision: entry.revision, + offset: 0, + } + expect(await executeLocalFileRequest(request, authorization)).toEqual({ + ok: true, + data: { kind: 'chunk', bytes: new Uint8Array(Buffer.from('inside')), eof: true }, + }) + await writeFile(join(root, 'outside.txt'), 'outside') + await rm(join(source, 'alias.txt')) + await symlink(join(root, 'outside.txt'), join(source, 'alias.txt')) + expect(await executeLocalFileRequest(request, authorization)).toMatchObject({ + ok: false, + error: expect.stringContaining('outside this import source'), + }) +}) + +it.each(['é', '界', '😀', '\uFEFF'])( + 'preserves %s across the default text-page boundary', + async (character) => { + const path = join(root, 'unicode.txt') + const prefix = 'a'.repeat(63_999) + const content = `${prefix}${character}end` + await writeFile(path, content) + const first = await executeLocalFileRequest( + { operation: 'read' }, + { + toolName: 'read_local_file', + args: { path }, + } + ) + expect(first).toMatchObject({ + ok: true, + data: { + representation: 'text', + text: prefix, + offset: 0, + nextOffset: 63_999, + truncated: true, + }, + }) + if (!first.ok || first.data.kind !== 'read') throw new Error('Expected text') + const second = await executeLocalFileRequest( + { operation: 'read' }, + { + toolName: 'read_local_file', + args: { path, offset: first.data.nextOffset }, + } + ) + expect(second).toMatchObject({ + ok: true, + data: { + representation: 'text', + text: `${character}end`, + nextOffset: Buffer.byteLength(content), + truncated: false, + }, + }) + } +) + +it('preserves a UTF-8 BOM and rejects split offsets and limits that cannot fit a character', async () => { + const path = join(root, 'unicode.txt') + await writeFile(path, '\uFEFF😀end') + expect( + await executeLocalFileRequest( + { operation: 'read' }, + { + toolName: 'read_local_file', + args: { path, limit: 4 }, + } + ) + ).toMatchObject({ ok: true, data: { text: '\uFEFF', nextOffset: 3, truncated: true } }) + expect( + await executeLocalFileRequest( + { operation: 'read' }, + { + toolName: 'read_local_file', + args: { path, offset: 4 }, + } + ) + ).toMatchObject({ ok: false, error: expect.stringContaining('UTF-8') }) + expect( + await executeLocalFileRequest( + { operation: 'read' }, + { + toolName: 'read_local_file', + args: { path, offset: 3, limit: 1 }, + } + ) + ).toMatchObject({ ok: false, error: expect.stringContaining('limit') }) +}) diff --git a/apps/desktop/src/main/local-files.ts b/apps/desktop/src/main/local-files.ts new file mode 100644 index 00000000000..a4b9cd9a42c --- /dev/null +++ b/apps/desktop/src/main/local-files.ts @@ -0,0 +1,265 @@ +import { open, readdir, realpath, stat } from 'node:fs/promises' +import { homedir } from 'node:os' +import { basename, isAbsolute, join, relative, resolve, sep } from 'node:path' +import type { + DesktopLocalFileEntry, + DesktopLocalFileManifest, + DesktopLocalFileRead, + DesktopLocalFileResponse, +} from '@sim/desktop-bridge' +import { MAX_DESKTOP_IMPORT_FILE_BYTES } from '@sim/desktop-bridge' +import { getErrorMessage } from '@sim/utils/errors' +import { isRecordLike } from '@sim/utils/object' +import { PDFDocument } from 'pdf-lib' + +const CHUNK_BYTES = 8 * 1024 * 1024 +const MAX_ENTRIES = 1000 +const MAX_READ_BYTES = 64_000 + +export interface LocalFileAuthorization { + toolName: string + args: Record +} + +/** Resolve normal native paths; macOS, not Sim folder grants, owns filesystem access. */ +function nativePath(value: unknown): string { + if (typeof value !== 'string' || value.length > 4096 || value.includes('\0')) + throw new Error('A native absolute path or ~/ path is required.') + const path = + value === '~' ? homedir() : value.startsWith('~/') ? join(homedir(), value.slice(2)) : value + if (!isAbsolute(path)) throw new Error('Use an absolute path or ~/ path.') + return resolve(path) +} + +function revision(info: Awaited>): string { + return `${info.dev}:${info.ino}:${info.size}:${info.mtimeMs}` +} + +function boundedInteger(value: unknown, fallback: number, max: number): number { + if (value === undefined) return fallback + if (typeof value !== 'number' || !Number.isSafeInteger(value) || value < 0 || value > max) + throw new Error(`Expected an integer between 0 and ${max}.`) + return value +} + +function assertImportPath(root: string, candidate: string): void { + const rel = relative(root, candidate) + if (isAbsolute(rel) || rel === '..' || rel.startsWith(`..${sep}`)) + throw new Error('The file is outside this import source.') +} + +async function inspect(path: string, args: Record): Promise { + const info = await stat(path) + if (info.isDirectory()) { + const entries = await readdir(path, { withFileTypes: true }) + return { + kind: 'read', + path, + representation: 'directory', + truncated: entries.length > MAX_ENTRIES, + entries: entries + .sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)) + .slice(0, MAX_ENTRIES) + .map((entry) => ({ + name: entry.name, + kind: entry.isFile() + ? 'file' + : entry.isDirectory() + ? 'directory' + : entry.isSymbolicLink() + ? 'symlink' + : 'other', + })), + } + } + if (!info.isFile()) throw new Error('The path is not a regular file or directory.') + const file = await open(path, 'r') + try { + const header = Buffer.alloc(16) + await file.read(header, 0, header.length, 0) + const mediaType = header.subarray(0, 8).equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10])) + ? 'image/png' + : header[0] === 255 && header[1] === 216 + ? 'image/jpeg' + : header.toString('ascii', 0, 3) === 'GIF' + ? 'image/gif' + : header.toString('ascii', 0, 4) === 'RIFF' && header.toString('ascii', 8, 12) === 'WEBP' + ? 'image/webp' + : header.toString('ascii', 0, 5) === '%PDF-' + ? 'application/pdf' + : null + if (mediaType) { + if (info.size > CHUNK_BYTES) + throw new Error( + 'Visual files larger than 8 MB must be imported into Workspace Files first.' + ) + let bytes = Buffer.alloc(info.size) + const { bytesRead } = await file.read(bytes, 0, bytes.length, 0) + bytes = bytes.subarray(0, bytesRead) + let pageCount: number | undefined + let truncated = false + if (mediaType === 'application/pdf') { + const source = await PDFDocument.load(bytes) + pageCount = Math.min(source.getPageCount(), 20) + if (!pageCount) throw new Error('The PDF has no pages.') + truncated = source.getPageCount() > pageCount + const subset = await PDFDocument.create() + for (const page of await subset.copyPages( + source, + Array.from({ length: pageCount }, (_, i) => i) + )) + subset.addPage(page) + bytes = Buffer.from(await subset.save()) + } + if (bytes.length > CHUNK_BYTES) + throw new Error('The rendered file exceeds the 8 MB visual limit; import it first.') + return { + kind: 'read', + path, + representation: 'visual', + truncated, + observations: [ + { + name: basename(path), + mediaType, + data: bytes.toString('base64'), + ...(pageCount ? { pageCount } : {}), + }, + ], + } + } + const offset = boundedInteger(args.offset, 0, Number.MAX_SAFE_INTEGER) + const limit = boundedInteger(args.limit, MAX_READ_BYTES, MAX_READ_BYTES) + if (!limit) throw new Error('Read limit must be positive.') + const buffer = Buffer.alloc(Math.min(limit, Math.max(0, info.size - offset))) + const { bytesRead } = await file.read(buffer, 0, buffer.length, offset) + const bytes = buffer.subarray(0, bytesRead) + if (bytes.includes(0)) + return { + kind: 'read', + path, + representation: 'binary', + note: 'Binary content cannot be decoded as text. Import this file into Workspace Files for document extraction.', + } + /** Retain partial characters for the next page; preserving BOM keeps byte offsets exact. */ + let text: string + try { + text = new TextDecoder('utf-8', { fatal: true, ignoreBOM: true }).decode(bytes, { + stream: offset + bytesRead < info.size, + }) + } catch { + throw new Error('Text reads require valid UTF-8 and an offset at a character boundary.') + } + const textBytes = Buffer.byteLength(text) + if (bytesRead > 0 && textBytes === 0) + throw new Error('The read limit must fit at least one complete UTF-8 character.') + return { + kind: 'read', + path, + representation: 'text', + text, + offset, + nextOffset: offset + textBytes, + truncated: offset + textBytes < info.size, + } + } finally { + await file.close() + } +} + +async function manifest( + path: string, + args: Record +): Promise { + if (typeof args.targetWorkspaceId !== 'string') throw new Error('A target workspace is required.') + const root = await realpath(path) + const entries: DesktopLocalFileEntry[] = [] + async function walk(current: string, ancestors: ReadonlySet): Promise { + if (entries.length >= MAX_ENTRIES) + throw new Error( + 'The directory exceeds 1,000 entries. Import smaller subdirectories separately.' + ) + const canonical = await realpath(current) + assertImportPath(root, canonical) + const info = await stat(canonical) + if (!info.isDirectory() && !info.isFile()) + throw new Error(`Cannot import special filesystem entry: ${current}`) + if (ancestors.has(canonical)) throw new Error(`Directory link cycle: ${current}`) + if (info.isFile() && info.size > MAX_DESKTOP_IMPORT_FILE_BYTES) + throw new Error( + 'Desktop imports support files up to 64 MB. Use the file uploader for larger files.' + ) + entries.push({ + relativePath: relative(path, current).split(sep).join('/'), + kind: info.isDirectory() ? 'directory' : 'file', + size: info.size, + revision: revision(info), + }) + if (info.isDirectory()) { + const next = new Set([...ancestors, canonical]) + for (const name of (await readdir(current)).sort()) await walk(join(current, name), next) + } + } + await walk(path, new Set()) + return { + kind: 'manifest', + name: basename(path), + targetWorkspaceId: args.targetWorkspaceId, + ...(typeof args.folderId === 'string' ? { folderId: args.folderId } : {}), + entries, + } +} + +/** Calls have already been authorized against the pending server record by the IPC boundary. */ +export async function executeLocalFileRequest( + request: unknown, + authorization: LocalFileAuthorization +): Promise { + try { + if (!isRecordLike(request)) throw new Error('Invalid local file request.') + const path = nativePath(authorization.args.path) + if (request.operation === 'read' && authorization.toolName === 'read_local_file') + return { ok: true, data: await inspect(path, authorization.args) } + if (authorization.toolName !== 'import_local_files') + throw new Error('The operation does not match the pending tool call.') + if (request.operation === 'manifest') + return { ok: true, data: await manifest(path, authorization.args) } + if ( + request.operation !== 'chunk' || + typeof request.relativePath !== 'string' || + typeof request.revision !== 'string' + ) + throw new Error('Invalid file chunk request.') + const child = resolve(path, request.relativePath) + assertImportPath(path, child) + const root = await realpath(path) + const canonical = await realpath(child) + assertImportPath(root, canonical) + const offset = boundedInteger(request.offset, 0, Number.MAX_SAFE_INTEGER) + const file = await open(canonical, 'r') + try { + const info = await file.stat() + if (!info.isFile() || revision(info) !== request.revision) + throw new Error( + 'The source file changed during import. Inspect the partial result before retrying.' + ) + if (offset > info.size) throw new Error('The requested offset is outside the file.') + const buffer = Buffer.alloc(Math.min(CHUNK_BYTES, info.size - offset)) + const { bytesRead } = await file.read(buffer, 0, buffer.length, offset) + if (revision(await file.stat()) !== request.revision) + throw new Error('The source file changed during import.') + return { + ok: true, + data: { + kind: 'chunk', + bytes: new Uint8Array(buffer.subarray(0, bytesRead)), + eof: offset + bytesRead >= info.size, + }, + } + } finally { + await file.close() + } + } catch (error) { + return { ok: false, error: getErrorMessage(error) } + } +} diff --git a/apps/desktop/src/main/local-filesystem.test.ts b/apps/desktop/src/main/local-filesystem.test.ts index a04e532bf93..ffad9b8b62e 100644 --- a/apps/desktop/src/main/local-filesystem.test.ts +++ b/apps/desktop/src/main/local-filesystem.test.ts @@ -1,9 +1,23 @@ -import { mkdir, mkdtemp, realpath, symlink, writeFile } from 'node:fs/promises' +import { + type FileHandle, + mkdir, + mkdtemp, + open, + realpath, + rename, + rm, + symlink, + writeFile, +} from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' import { beforeEach, describe, expect, it, vi } from 'vitest' vi.mock('electron', () => import('@/test/electron-mock')) +vi.mock('node:fs/promises', async (importOriginal) => { + const actual = await importOriginal() + return { ...actual, open: vi.fn(actual.open) } +}) import type { LocalFilesystemMount, LocalFilesystemResponse } from '@sim/desktop-bridge' import { @@ -492,6 +506,110 @@ describe('LocalFilesystemService', () => { expect(escaped).toMatchObject({ ok: false, code: 'ACCESS_DENIED' }) }) + it('resolves a granted file for upload and refuses escapes, directories, and oversize files', async () => { + const granted = await mount(service) + const vfsRoot = `user-local/${encodeURIComponent(granted.name)}--${granted.id}` + const outside = await mkdtemp(join(tmpdir(), 'sim-localfs-outside-')) + await writeFile(join(outside, 'secret.txt'), 'secret') + await symlink(join(outside, 'secret.txt'), join(root, 'secret-link.txt')) + + const file = await service.resolveGrantedFile(`${vfsRoot}/README.md`, 1024) + try { + expect(file).toMatchObject({ name: 'README.md', size: 24 }) + await expect(file.handle.readFile('utf8')).resolves.toBe('hello world\nsecond line\n') + } finally { + await file.handle.close() + } + await expect( + service.resolveGrantedFile(`${vfsRoot}/secret-link.txt`, 1024) + ).rejects.toMatchObject({ code: 'ACCESS_DENIED' }) + await expect(service.resolveGrantedFile(`${vfsRoot}/src`, 1024)).rejects.toMatchObject({ + code: 'NOT_A_FILE', + }) + await expect(service.resolveGrantedFile(`${vfsRoot}/README.md`, 4)).rejects.toMatchObject({ + code: 'FILE_TOO_LARGE', + }) + await expect( + service.resolveGrantedFile('user-local/Other--missing/README.md', 1024) + ).rejects.toMatchObject({ code: 'MOUNT_NOT_FOUND' }) + }) + + it.runIf(process.platform !== 'win32')( + 'resolves a listed POSIX file with a literal backslash for upload', + async () => { + const name = 'report\\draft.txt' + await writeFile(join(root, name), 'draft') + const granted = await mount(service) + const listing = dataOf(await service.handle({ operation: 'list', uri: granted.uri })) + const entry = 'entries' in listing && listing.entries.find((item) => item.name === name) + expect(entry).toMatchObject({ name, uri: `${granted.uri}report%5Cdraft.txt` }) + if (!entry) throw new Error('Expected the backslash file in the directory listing') + const vfsPath = `user-local/${encodeURIComponent(granted.name)}--${granted.id}/${new URL(entry.uri).pathname.slice(1)}` + + const file = await service.resolveGrantedFile(vfsPath, 1024) + try { + expect(file).toMatchObject({ name, size: 5 }) + await expect(file.handle.readFile('utf8')).resolves.toBe('draft') + } finally { + await file.handle.close() + } + } + ) + + it.each(['..', '%2e%2e', 'src%2F..%2FREADME.md', 'README.md%00'])( + 'rejects unsafe upload path segment %s', + async (segment) => { + const granted = await mount(service) + const vfsRoot = `user-local/${encodeURIComponent(granted.name)}--${granted.id}` + + await expect(service.resolveGrantedFile(`${vfsRoot}/${segment}`, 1024)).rejects.toMatchObject( + { + code: expect.stringMatching(/^(ACCESS_DENIED|INVALID_URI)$/), + } + ) + } + ) + + it.each([false, true])( + 'closes a file opened through a swapped ancestor (ancestor restored: %s)', + async (restoreAncestor) => { + const granted = await mount(service) + const vfsRoot = `user-local/${encodeURIComponent(granted.name)}--${granted.id}` + const outside = await mkdtemp(join(tmpdir(), 'sim-localfs-outside-')) + await writeFile(join(outside, 'index.ts'), 'outside secret') + const sourceDirectory = join(root, 'src') + const originalDirectory = join(root, 'original-src') + const openFile = vi.mocked(open).getMockImplementation() + if (!openFile) throw new Error('Expected the original file-open implementation') + let opened: FileHandle | undefined + const openSpy = vi + .mocked(open) + .mockClear() + .mockImplementationOnce(async (...args) => { + await rename(sourceDirectory, originalDirectory) + await symlink(outside, sourceDirectory) + opened = await openFile(...args) + if (restoreAncestor) { + await rm(sourceDirectory) + await rename(originalDirectory, sourceDirectory) + } + return opened + }) + + try { + await expect( + service.resolveGrantedFile(`${vfsRoot}/src/index.ts`, 1024) + ).rejects.toMatchObject({ code: 'ACCESS_DENIED' }) + expect(openSpy).toHaveBeenCalledTimes(1) + expect(opened?.fd).toBe(-1) + } finally { + openSpy.mockReset().mockImplementation(openFile) + await opened?.close() + await rm(outside, { recursive: true, force: true }) + } + } + ) + it('rejects lexical traversal before URL normalization can reinterpret it', async () => { const granted = await mount(service) const traversal = await service.handle({ diff --git a/apps/desktop/src/main/local-filesystem.ts b/apps/desktop/src/main/local-filesystem.ts index edf226d4592..46a6c30b9fb 100644 --- a/apps/desktop/src/main/local-filesystem.ts +++ b/apps/desktop/src/main/local-filesystem.ts @@ -1,5 +1,5 @@ -import type { Dirent } from 'node:fs' -import { lstat, opendir, readFile, realpath, stat } from 'node:fs/promises' +import { constants, type Dirent } from 'node:fs' +import { type FileHandle, lstat, open, opendir, readFile, realpath, stat } from 'node:fs/promises' import { basename, isAbsolute, relative, resolve, sep } from 'node:path' import type { LocalFilesystemData, @@ -495,18 +495,7 @@ export class LocalFilesystemService { return false } const args = authorization.args - - const expectedUriForPath = (path: unknown): string | null => { - if (typeof path !== 'string') return null - for (const mount of this.mounts.values()) { - const root = mountVfsRoot(mount) - if (path === root) return mount.uri - if (path.startsWith(`${root}/`)) { - return `${mount.uri}${path.slice(root.length + 1)}` - } - } - return null - } + const expectedUriForPath = (path: unknown): string | null => this.uriForVfsPath(path) switch (authorization.toolName) { case 'read': { @@ -795,6 +784,72 @@ export class LocalFilesystemService { return { revealed: true } } + /** + * Resolves a granted `user-local/…` file the browser agent attaches to a page. It applies the + * same VFS mapping and realpath containment as reads and returns an open handle so later path + * replacements cannot redirect the upload. The caller owns and must close the handle, and must + * enforce the byte limit while reading because the file can grow after its initial size check. + * Rechecks containment and identity after opening; Node's path-based lookups cannot make + * ancestor resolution atomic. + */ + async resolveGrantedFile( + vfsPath: string, + maxBytes: number + ): Promise<{ handle: FileHandle; name: string; size: number }> { + const uri = this.uriForVfsPath(vfsPath) + if (!uri) { + throw new LocalFilesystemError( + 'MOUNT_NOT_FOUND', + 'That local folder is not shared with Sim. Ask the user to select it again.' + ) + } + const resolved = await this.resolveUri(uri) + const handle = await open( + resolved.realPath, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ) + try { + const metadata = await handle.stat() + if (!metadata.isFile()) { + throw new LocalFilesystemError('NOT_A_FILE', 'The local path is not a file.') + } + const verified = await this.resolveUri(uri) + const currentMetadata = await lstat(verified.realPath) + if ( + verified.realPath !== resolved.realPath || + !currentMetadata.isFile() || + metadata.dev !== currentMetadata.dev || + metadata.ino !== currentMetadata.ino + ) { + throw new LocalFilesystemError( + 'ACCESS_DENIED', + 'The local file changed while it was being opened. Try again.' + ) + } + if (metadata.size > maxBytes) { + throw new LocalFilesystemError( + 'FILE_TOO_LARGE', + `The local file exceeds the ${Math.floor(maxBytes / 1024 / 1024)} MB upload limit.` + ) + } + return { handle, name: basename(resolved.realPath), size: metadata.size } + } catch (error) { + await handle.close() + throw error + } + } + + /** Maps a `user-local/--/…` VFS path onto its granted mount's localfs URI. */ + private uriForVfsPath(path: unknown): string | null { + if (typeof path !== 'string') return null + for (const mount of this.mounts.values()) { + const root = mountVfsRoot(mount) + if (path === root) return mount.uri + if (path.startsWith(`${root}/`)) return `${mount.uri}${path.slice(root.length + 1)}` + } + return null + } + private parseUri(uri: string): { mount: GrantedMount; relativePath: string } { if (!uri.startsWith('localfs://')) { throw new LocalFilesystemError('INVALID_URI', 'The localfs URI is invalid.') @@ -854,7 +909,7 @@ export class LocalFilesystemService { decoded === '.' || decoded === '..' || decoded.includes('/') || - decoded.includes('\\') || + (sep === '\\' && decoded.includes('\\')) || decoded.includes('\0') ) { throw new LocalFilesystemError('INVALID_URI', 'The localfs URI is invalid.') diff --git a/apps/desktop/src/main/local-pages.ts b/apps/desktop/src/main/local-pages.ts index 2675fe3fe49..42e80f6b2ad 100644 --- a/apps/desktop/src/main/local-pages.ts +++ b/apps/desktop/src/main/local-pages.ts @@ -25,7 +25,7 @@ export const LOCAL_PAGE_SCHEME = 'sim-shell' const LOCAL_PAGE_HOST = 'pages' export const LOCAL_PAGE_ORIGIN = `${LOCAL_PAGE_SCHEME}://${LOCAL_PAGE_HOST}` -export type LocalPage = 'offline.html' | 'server.html' | 'dialog.html' +export type LocalPage = 'offline.html' | 'server.html' | 'dialog.html' | 'credential-picker.html' const LOCAL_PAGES: ReadonlySet = new Set(['offline.html', 'server.html']) @@ -44,6 +44,9 @@ const SERVABLE_FILES: ReadonlySet = new Set([ 'dialog.html', 'dialog.js', 'dialog.css', + 'credential-picker.html', + 'credential-picker.js', + 'credential-picker.css', ]) const CONTENT_TYPES: Readonly> = { diff --git a/apps/desktop/src/main/shell-theme.ts b/apps/desktop/src/main/shell-theme.ts index a1eb4afc723..ede9f6745b0 100644 --- a/apps/desktop/src/main/shell-theme.ts +++ b/apps/desktop/src/main/shell-theme.ts @@ -10,7 +10,11 @@ export function getShellTheme(): ShellTheme | undefined { } function isShellPage(url: string): boolean { - return isLocalPageUrl(url) || url === localPageUrl('dialog.html') + return ( + isLocalPageUrl(url) || + url === localPageUrl('dialog.html') || + url === localPageUrl('credential-picker.html') + ) } /** Retains Sim's last resolved theme so recovery works even after its renderer stops. */ diff --git a/apps/desktop/src/main/telemetry-policy.test.ts b/apps/desktop/src/main/telemetry-policy.test.ts index c4d7ab876b0..857f0e7310c 100644 --- a/apps/desktop/src/main/telemetry-policy.test.ts +++ b/apps/desktop/src/main/telemetry-policy.test.ts @@ -3,7 +3,15 @@ import { describe, expect, it, vi } from 'vitest' // telemetry-policy pulls in @/main/navigation, which imports electron. vi.mock('electron', () => import('@/test/electron-mock')) -import { shouldBlockRequest } from '@/main/telemetry-policy' +const requestPolicy = vi.hoisted(() => ({ + handleBrowserRequest: vi.fn(), +})) +vi.mock('@/main/browser-agent/request-policy', () => requestPolicy) + +import type { OnBeforeRequestListenerDetails } from 'electron' +import { WebContentsView } from 'electron' +import { registerAgentWebContents } from '@/main/browser-agent/registry' +import { attachTelemetryPolicy, shouldBlockRequest } from '@/main/telemetry-policy' describe('shouldBlockRequest', () => { it('blocks third-party analytics hosts and their subdomains', () => { @@ -25,3 +33,77 @@ describe('shouldBlockRequest', () => { expect(shouldBlockRequest('not a url')).toBe(false) }) }) + +describe('attachTelemetryPolicy', () => { + it.each([true, false])('retains browser isolation when analytics blocking is %s', (enabled) => { + const contents = new WebContentsView().webContents + const onBeforeRequest = vi.mocked(contents.session.webRequest.onBeforeRequest) + onBeforeRequest.mockClear() + requestPolicy.handleBrowserRequest.mockClear() + attachTelemetryPolicy(contents.session, enabled) + const listener = onBeforeRequest.mock.calls[0][0] + if (typeof listener !== 'function') throw new Error('Missing request policy') + const request: OnBeforeRequestListenerDetails = { + id: 1, + url: 'https://sim.ai/home', + method: 'GET', + resourceType: 'mainFrame', + webContents: contents, + referrer: '', + timestamp: 0, + uploadData: [], + } + const callback = vi.fn() + listener(request, callback) + expect(callback).toHaveBeenCalledExactlyOnceWith({ cancel: false }) + expect(requestPolicy.handleBrowserRequest).not.toHaveBeenCalled() + callback.mockClear() + registerAgentWebContents(contents, 'https://sim.ai') + listener(request, callback) + expect(requestPolicy.handleBrowserRequest).toHaveBeenCalledExactlyOnceWith(request, callback) + expect(callback).not.toHaveBeenCalled() + requestPolicy.handleBrowserRequest.mockClear() + const workerRequest = { + ...request, + url: 'http://169.254.169.254/metadata', + webContents: undefined, + resourceType: 'other' as const, + } + listener(workerRequest, callback) + expect(requestPolicy.handleBrowserRequest).toHaveBeenCalledExactlyOnceWith( + workerRequest, + callback + ) + expect(callback).not.toHaveBeenCalled() + }) + + it('preserves ordinary workers on the exact configured LAN app origin', () => { + const contents = new WebContentsView().webContents + const onBeforeRequest = vi.mocked(contents.session.webRequest.onBeforeRequest) + onBeforeRequest.mockClear() + requestPolicy.handleBrowserRequest.mockClear() + registerAgentWebContents(contents, 'http://192.168.1.10:3000') + attachTelemetryPolicy(contents.session, false) + const listener = onBeforeRequest.mock.calls[0][0] + if (typeof listener !== 'function') throw new Error('Missing request policy') + const request: OnBeforeRequestListenerDetails = { + id: 1, + url: 'http://192.168.1.10:3000/editor.worker.js', + method: 'GET', + resourceType: 'script', + referrer: '', + timestamp: 0, + uploadData: [], + } + const callback = vi.fn() + listener(request, callback) + expect(callback).toHaveBeenCalledExactlyOnceWith({ cancel: false }) + expect(requestPolicy.handleBrowserRequest).not.toHaveBeenCalled() + + for (const url of ['http://192.168.1.11/data', 'http://192.168.1.10:4000/data']) { + const otherRequest = { ...request, url } + listener(otherRequest, callback) + expect(requestPolicy.handleBrowserRequest).toHaveBeenCalledWith(otherRequest, callback) + } + }) +}) diff --git a/apps/desktop/src/main/telemetry-policy.ts b/apps/desktop/src/main/telemetry-policy.ts index b79411f61fd..49d52a43917 100644 --- a/apps/desktop/src/main/telemetry-policy.ts +++ b/apps/desktop/src/main/telemetry-policy.ts @@ -1,5 +1,7 @@ import { createLogger } from '@sim/logger' import type { Session } from 'electron' +import { isAgentWebContents, shouldGuardUnownedAgentRequest } from '@/main/browser-agent/registry' +import { handleBrowserRequest } from '@/main/browser-agent/request-policy' import { matchesHostList } from '@/main/navigation' const logger = createLogger('DesktopTelemetryPolicy') @@ -17,11 +19,6 @@ export const BLOCKED_ANALYTICS_HOSTS: readonly string[] = [ 'stats.g.doubleclick.net', ] -const BLOCK_URL_PATTERNS = BLOCKED_ANALYTICS_HOSTS.flatMap((host) => [ - `*://${host}/*`, - `*://*.${host}/*`, -]) - /** * Suffix-matches a URL's hostname against the blocked analytics hosts. */ @@ -40,11 +37,18 @@ export function shouldBlockRequest(rawUrl: string): boolean { * onBeforeRequest consumer — Electron allows a single listener per session. */ export function attachTelemetryPolicy(session: Session, enabled: boolean): void { - if (!enabled) { - return - } - session.webRequest.onBeforeRequest({ urls: BLOCK_URL_PATTERNS }, (details, callback) => { - callback({ cancel: shouldBlockRequest(details.url) }) + session.webRequest.onBeforeRequest((details, callback) => { + if (enabled && shouldBlockRequest(details.url)) { + callback({ cancel: true }) + } else if ( + details.webContents + ? isAgentWebContents(details.webContents) + : shouldGuardUnownedAgentRequest(session, details.url) + ) { + handleBrowserRequest(details, callback) + } else { + callback({ cancel: false }) + } }) - logger.info('Third-party analytics blocking enabled') + if (enabled) logger.info('Third-party analytics blocking enabled') } diff --git a/apps/desktop/src/main/window.test.ts b/apps/desktop/src/main/window.test.ts index 69a02eea19d..58f44ada89d 100644 --- a/apps/desktop/src/main/window.test.ts +++ b/apps/desktop/src/main/window.test.ts @@ -4,7 +4,8 @@ import { createSecureWebPreferences } from '@/main/window-preferences' vi.mock('electron', () => import('@/test/electron-mock')) -import { BrowserWindow, dialog, screen, systemPreferences } from 'electron' +import { BrowserWindow, dialog, screen, systemPreferences, WebContentsView } from 'electron' +import { registerAgentWebContents } from '@/main/browser-agent/registry' import type { ConfigStore } from '@/main/config' import type { EventRecorder } from '@/main/observability' import { @@ -184,6 +185,22 @@ describe('setupPermissionHandlers', () => { expect(check(null, 'media', APP, {})).toBe(false) expect(check(null, 'media', 'https://evil.example', { mediaType: 'audio' })).toBe(false) }) + + it('keeps browser permission prompts when a Sim page shares the app session', () => { + const { request, check } = createSession() + const contents = new WebContentsView().webContents + const browserRequest = vi.fn((_contents, _permission, callback) => callback(false)) + const browserCheck = vi.fn(() => false) + registerAgentWebContents(contents, APP, { request: browserRequest, check: browserCheck }) + const callback = vi.fn() + request(contents, 'media', callback, { requestingUrl: `${APP}/chat`, mediaTypes: ['audio'] }) + expect(browserRequest).toHaveBeenCalledOnce() + expect(callback).toHaveBeenCalledWith(false) + expect(check(contents, 'media', APP, { mediaType: 'audio' })).toBe(false) + expect(browserCheck).toHaveBeenCalledOnce() + expect(systemPreferences.getMediaAccessStatus).not.toHaveBeenCalled() + expect(check(null, 'media', APP, { mediaType: 'audio' })).toBe(true) + }) }) describe('backgroundColorFor', () => { diff --git a/apps/desktop/src/main/window.ts b/apps/desktop/src/main/window.ts index 19481b4ff89..095385970f8 100644 --- a/apps/desktop/src/main/window.ts +++ b/apps/desktop/src/main/window.ts @@ -2,6 +2,7 @@ import { createLogger } from '@sim/logger' import { getErrorMessage } from '@sim/utils/errors' import type { Event, Rectangle, Session } from 'electron' import { app, BrowserWindow, dialog, nativeTheme, screen, systemPreferences } from 'electron' +import { agentPermissionHandlers } from '@/main/browser-agent/registry' import { type ConfigStore, isSafeInternalPath, type WindowBounds } from '@/main/config' import { showShellDialog } from '@/main/dialogs' import { isAppOrigin, isAuthSurfacePath } from '@/main/navigation' @@ -99,6 +100,11 @@ function originOf(raw: string): string { */ export function setupPermissionHandlers(session: Session, getAppOrigin: () => string): void { session.setPermissionRequestHandler((webContents, permission, callback, details) => { + const browserPermissions = agentPermissionHandlers(webContents) + if (browserPermissions) { + browserPermissions.request(webContents, permission, callback, details) + return + } const requestingUrl = details.requestingUrl || webContents?.getURL() || '' const mediaTypes = 'mediaTypes' in details ? details.mediaTypes : undefined if (!resolvePermission(permission, originOf(requestingUrl), getAppOrigin(), mediaTypes)) { @@ -112,7 +118,10 @@ export function setupPermissionHandlers(session: Session, getAppOrigin: () => st callback(true) }) - session.setPermissionCheckHandler((_webContents, permission, requestingOrigin, details) => { + session.setPermissionCheckHandler((webContents, permission, requestingOrigin, details) => { + const browserPermissions = agentPermissionHandlers(webContents) + if (browserPermissions) + return browserPermissions.check(webContents, permission, requestingOrigin, details) const mediaTypes = details.mediaType ? [details.mediaType] : undefined return resolvePermission(permission, originOf(requestingOrigin), getAppOrigin(), mediaTypes) }) diff --git a/apps/desktop/src/preload/browser/forms.test.ts b/apps/desktop/src/preload/browser/forms.test.ts new file mode 100644 index 00000000000..1a45c9e8b24 --- /dev/null +++ b/apps/desktop/src/preload/browser/forms.test.ts @@ -0,0 +1,179 @@ +/** @vitest-environment jsdom */ +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { + collectLoginFields, + findLoginTarget, + isFillable, + targetSignature, +} from '@/preload/browser/forms' + +beforeEach(() => { + document.body.innerHTML = '' + vi.spyOn(HTMLInputElement.prototype, 'getBoundingClientRect').mockReturnValue({ + x: 20, + y: 20, + width: 200, + height: 32, + top: 20, + left: 20, + right: 220, + bottom: 52, + toJSON: () => ({}), + }) +}) + +function target(markup: string) { + document.body.innerHTML = markup + return findLoginTarget(collectLoginFields().fields) +} + +function field(id: string): HTMLInputElement { + return document.getElementById(id) as HTMLInputElement +} + +describe('saved credential target detection', () => { + it('pairs a standard login within its form', () => { + const result = target( + '
' + ) + expect(result).toEqual({ username: field('user'), password: field('pass') }) + }) + it('supports password-only challenges', () => { + const result = target('
') + expect(result).toEqual({ username: null, password: field('pass') }) + }) + it('supports semantic identifier steps', () => { + const result = target( + '
' + ) + expect(result).toEqual({ username: field('user'), password: null }) + }) + it('recognizes an email-only sign-in but not a newsletter', () => { + expect(target('
')).not.toBeNull() + expect( + target('
') + ).toBeNull() + }) + it.each(['new-password', 'section-signup new-password', 'one-time-code'])( + 'never selects %s as a saved password', + (autocomplete) => { + expect( + target(`
`) + ).toBeNull() + } + ) + it('fills the focused second form and leaves the first form alone', () => { + target( + '
' + ) + field('u2').focus() + expect(findLoginTarget(collectLoginFields().fields)).toEqual({ + username: field('u2'), + password: field('p2'), + }) + field('p2').focus() + expect(findLoginTarget(collectLoginFields().fields)?.password).toBe(field('p2')) + }) + it('prefers semantic usernames over the closest unrelated field', () => { + const result = target( + '
' + ) + expect(result?.username).toBe(field('user')) + }) + it('excludes one-time codes from username candidates', () => { + const result = target( + '
' + ) + expect(result?.username).toBe(field('user')) + }) + it.each([ + 'hidden', + 'inert', + 'aria-hidden="true"', + 'style="visibility:hidden"', + 'style="display:none"', + 'style="opacity:0"', + ])('rejects a field inside %s', (attribute) => { + expect(target(`
`)).toBeNull() + }) + it.each(['disabled', 'readonly'])('rejects a %s password', (attribute) => { + expect(target(``)).toBeNull() + }) + it('rechecks detachment and disabling immediately', () => { + target('') + const password = field('pass') + expect(isFillable(password)).toBe(true) + password.disabled = true + expect(isFillable(password)).toBe(false) + password.disabled = false + password.remove() + expect(isFillable(password)).toBe(false) + }) + it('traverses open shadow roots and respects focus within them', () => { + document.body.innerHTML = '
' + const root = document.getElementById('host')!.attachShadow({ mode: 'open' }) + root.innerHTML = + '
' + const password = root.querySelector('#pass')! + password.focus() + const collected = collectLoginFields() + expect(collected.roots).toContain(root) + expect(findLoginTarget(collected.fields)).toEqual({ + username: root.querySelector('#user'), + password, + }) + root.host.setAttribute('inert', '') + expect(findLoginTarget(collected.fields)).toBeNull() + }) + it('invalidates the signature when field purpose changes', () => { + const result = target('') + const before = targetSignature(result) + field('pass').autocomplete = 'new-password' + expect(targetSignature(result)).not.toBe(before) + expect(findLoginTarget(collectLoginFields().fields)).toBeNull() + }) + it('respects disabled fieldsets and their first-legend exception', () => { + target( + '
' + ) + expect(isFillable(field('blocked'))).toBe(false) + expect(isFillable(field('allowed'))).toBe(true) + }) + it('keeps separate formless login groups together', () => { + target( + '
' + ) + field('p2').focus() + expect(findLoginTarget(collectLoginFields().fields)).toEqual({ + username: field('u2'), + password: field('p2'), + }) + }) + it('caps collection on large input-heavy pages', () => { + document.body.innerHTML = ''.repeat(500) + expect(collectLoginFields().fields).toHaveLength(256) + }) + it('recognizes revealed password boundaries between formless groups', () => { + target( + '' + ) + field('u2').focus() + expect(findLoginTarget(collectLoginFields().fields)).toEqual({ + username: field('u2'), + password: field('p2'), + }) + }) + it('prefers a focused identifier-only form over another login form', () => { + target( + '
' + ) + field('u2').focus() + expect(findLoginTarget(collectLoginFields().fields)).toEqual({ + username: field('u2'), + password: null, + }) + }) + it('does not accept an unsupported current-password input type', () => { + expect(target('')).toBeNull() + }) +}) diff --git a/apps/desktop/src/preload/browser/forms.ts b/apps/desktop/src/preload/browser/forms.ts new file mode 100644 index 00000000000..7d67224d09e --- /dev/null +++ b/apps/desktop/src/preload/browser/forms.ts @@ -0,0 +1,165 @@ +const MAX_NODES = 10_000 +const MAX_FIELDS = 256 + +export interface LoginTarget { + username: HTMLInputElement | null + password: HTMLInputElement | null +} + +function autocompleteTokens(field: HTMLInputElement): string[] { + return (field.getAttribute('autocomplete') ?? '').toLowerCase().split(/\s+/) +} + +function parentElementAcrossRoots(element: Element): Element | null { + const root = element.getRootNode() + return element.parentElement ?? (root instanceof ShadowRoot ? root.host : null) +} + +export function isFillable(field: HTMLInputElement): boolean { + if (!field.isConnected || field.matches(':disabled') || field.readOnly) return false + const rect = field.getBoundingClientRect() + if (rect.width <= 0 || rect.height <= 0) return false + for (let node: Element | null = field; node; node = parentElementAcrossRoots(node)) { + if (node.hasAttribute('inert') || node.hasAttribute('hidden')) return false + if (node.getAttribute('aria-hidden') === 'true') return false + const style = getComputedStyle(node) + if (style.visibility === 'hidden' || style.visibility === 'collapse') return false + if (style.display === 'none' || style.opacity === '0') return false + } + return true +} + +/** Bounds work on large pages and visits open shadow roots without modifying the page. */ +export function collectLoginFields(): { fields: HTMLInputElement[]; roots: ParentNode[] } { + const fields: HTMLInputElement[] = [] + const roots: ParentNode[] = [document] + const pending: Element[] = document.documentElement ? [document.documentElement] : [] + let visited = 0 + while (pending.length && visited < MAX_NODES && fields.length < MAX_FIELDS) { + const node = pending.pop()! + visited++ + if (node instanceof HTMLInputElement) fields.push(node) + const appendChildren = (parent: ParentNode) => { + const remaining = MAX_NODES - visited - pending.length + for (let index = Math.min(parent.children.length, remaining) - 1; index >= 0; index--) { + pending.push(parent.children[index]) + } + } + appendChildren(node) + if (node.shadowRoot) { + roots.push(node.shadowRoot) + appendChildren(node.shadowRoot) + } + } + return { fields, roots } +} + +export function focusedInput(): HTMLInputElement | null { + let active = document.activeElement + while (active?.shadowRoot?.activeElement) active = active.shadowRoot.activeElement + return active instanceof HTMLInputElement ? active : null +} + +function isCurrentPassword(field: HTMLInputElement): boolean { + const tokens = autocompleteTokens(field) + if (tokens.includes('new-password') || tokens.includes('one-time-code')) return false + return field.type === 'password' || (field.type === 'text' && tokens.includes('current-password')) +} + +function isUsernameCandidate(field: HTMLInputElement): boolean { + const tokens = autocompleteTokens(field) + return ( + ['text', 'email', 'tel'].includes(field.type) && + !tokens.some((token) => + ['new-password', 'current-password', 'one-time-code'].includes(token) + ) && + !/(?:one.?time|otp|verification.?code)/i.test(`${field.name} ${field.id}`) + ) +} + +function sameForm(left: HTMLInputElement, right: HTMLInputElement): boolean { + return left.form === right.form && left.getRootNode() === right.getRootNode() +} + +function usernameFor( + password: HTMLInputElement, + fields: HTMLInputElement[] +): HTMLInputElement | null { + const passwordIndex = fields.indexOf(password) + let previousPassword = -1 + for (let index = passwordIndex - 1; index >= 0; index--) { + if ( + fields[index].type === 'password' || + autocompleteTokens(fields[index]).some( + (token) => token === 'current-password' || token === 'new-password' + ) + ) { + previousPassword = index + break + } + } + const candidates = fields.filter( + (field, index) => + sameForm(field, password) && + isUsernameCandidate(field) && + (password.form !== null || (index > previousPassword && index < passwordIndex)) + ) + const explicit = candidates.find((field) => autocompleteTokens(field).includes('username')) + if (explicit) return explicit + const named = candidates.find((field) => + /(?:user.?name|email|login)/i.test(`${field.name} ${field.id}`) + ) + if (named) return named + const preceding = candidates.filter( + (field) => (password.compareDocumentPosition(field) & Node.DOCUMENT_POSITION_PRECEDING) !== 0 + ) + return preceding.at(-1) ?? candidates[0] ?? null +} + +function isIdentifierStep(field: HTMLInputElement): boolean { + if (!isUsernameCandidate(field)) return false + const tokens = autocompleteTokens(field) + if (tokens.includes('username')) return true + if (field.type !== 'email' && !tokens.includes('email')) return false + const scope = field.form ?? field.parentElement + const text = scope?.textContent ?? '' + return ( + /\b(sign\s*in|log\s*in|continue|next)\b/i.test(text) && + !/\b(subscribe|newsletter)\b/i.test(text) + ) +} + +/** Selects the focused login first and keeps account creation and OTP inputs out of saved fills. */ +export function findLoginTarget(fields: HTMLInputElement[]): LoginTarget | null { + const fillable = fields.filter(isFillable) + const focused = focusedInput() + const passwords = fillable.filter(isCurrentPassword) + const focusedPassword = + passwords.find((field) => field === focused) ?? + (focused + ? passwords.find( + (field) => + sameForm(field, focused) && + (field.form !== null || usernameFor(field, fillable) === focused) + ) + : undefined) + if (focused && !focusedPassword && fillable.includes(focused) && isIdentifierStep(focused)) { + return { username: focused, password: null } + } + const password = focusedPassword ?? passwords[0] + if (password) return { username: usernameFor(password, fillable), password } + const identifiers = fillable.filter(isIdentifierStep) + const username = identifiers.find((field) => field === focused) ?? identifiers[0] + return username ? { username, password: null } : null +} + +/** Includes field purpose so changing an existing input invalidates its old authorization. */ +export function targetSignature(target: LoginTarget | null): string { + return [target?.username, target?.password] + .map((field) => (field ? `${field.type}|${field.autocomplete}|${field.name}|${field.id}` : '')) + .join('\n') +} + +export function sameTarget(left: LoginTarget | null, right: LoginTarget | null): boolean { + return left?.username === right?.username && left?.password === right?.password +} diff --git a/apps/desktop/src/preload/browser/index.test.ts b/apps/desktop/src/preload/browser/index.test.ts index 03f766595bf..f68828868e1 100644 --- a/apps/desktop/src/preload/browser/index.test.ts +++ b/apps/desktop/src/preload/browser/index.test.ts @@ -58,6 +58,8 @@ describe('browser credential preload', () => { origin: window.location.origin, hasLoginForm: false, hasPasswordField: false, + targetId: null, + bounds: null, }) // No DOM mutation here: this models stylesheet/layout completion after @@ -69,6 +71,8 @@ describe('browser credential preload', () => { origin: window.location.origin, hasLoginForm: true, hasPasswordField: true, + targetId: expect.any(String), + bounds: null, }) await vi.advanceTimersByTimeAsync(3_000) @@ -82,6 +86,8 @@ describe('browser credential preload', () => { origin: window.location.origin, hasLoginForm: false, hasPasswordField: false, + targetId: null, + bounds: null, }) }) }) diff --git a/apps/desktop/src/preload/browser/index.ts b/apps/desktop/src/preload/browser/index.ts index 6001d4c28ba..465a1a4869b 100644 --- a/apps/desktop/src/preload/browser/index.ts +++ b/apps/desktop/src/preload/browser/index.ts @@ -1,238 +1,194 @@ +import { generateId } from '@sim/utils/id' import { ipcRenderer } from 'electron' - -/** - * The preload for pages inside the built-in browser. - * - * It exists for exactly one job: notice that the page has a login form, tell - * the main process only that fact, and — when the main process says the user - * chose an account — put the credential into the two fields it already found. - * - * It deliberately exposes nothing. There is no `contextBridge` call here, so - * the page cannot see or call any of this, and the fill can only be initiated - * by the main process. What travels out is a page origin and two booleans; - * field names, values, and page content never do. - * - * Runs in the top-level frame only (subframe preloads are not enabled), so - * cross-origin iframe login flows are out of scope for now — filling those - * needs its own threat review. - */ - -const FORM_STATE_CHANNEL = 'browser-credentials:form-state' -const FILL_CHANNEL = 'browser-credentials:fill' -const RESCAN_CHANNEL = 'browser-credentials:rescan' -const RESCAN_DEBOUNCE_MS = 250 +import { + collectLoginFields, + findLoginTarget, + focusedInput, + isFillable, + type LoginTarget, + sameTarget, + targetSignature, +} from '@/preload/browser/forms' +import type { + CredentialFillRequest, + CredentialFillStatus, + CredentialFormReport, +} from '@/shared/browser-credentials' + +const RESCAN_DELAY_MS = 250 const INITIAL_RESCAN_DELAYS_MS = [250, 750, 1_500, 3_000] as const - -interface DetectedForm { - username: HTMLInputElement | null - /** Absent on an identifier-first step, which asks for the email alone. */ - password: HTMLInputElement | null -} - -/** Held only in this isolated world; never serialized to main or the page. */ -let detected: DetectedForm | null = null +let target: LoginTarget | null = null +let signature = '' +let targetId: string | null = null let lastReported = '' -let rescanTimer: ReturnType | null = null -let formObserver: MutationObserver | null = null -let initialRescansScheduled = false - -function isFillable(field: HTMLInputElement): boolean { - if (field.disabled || field.readOnly) return false - const rect = field.getBoundingClientRect() - return rect.width > 0 && rect.height > 0 -} - -/** - * The field's `autocomplete` tokens. - * - * Token membership, not whole-string equality: the spec allows space-separated - * detail tokens and WebAuthn recommends `current-password webauthn`. Equality - * here while the agent guards split tokens would leave fill blind to exactly - * the fields they protect. - */ -function autocompleteTokens(field: HTMLInputElement): string[] { - return String(field.getAttribute('autocomplete') || '') - .toLowerCase() - .split(/\s+/) -} - -/** - * Matches the same definition the agent guards use: a reveal toggle flips a - * password field to `type="text"` without making it any less secret, and the - * autocomplete token is the page's own declaration either way. - */ -function isPasswordField(field: HTMLInputElement): boolean { - if (String(field.type || '').toLowerCase() === 'password') return true - const tokens = autocompleteTokens(field) - return tokens.includes('current-password') || tokens.includes('new-password') -} - -function findPasswordField(): HTMLInputElement | null { - for (const field of document.querySelectorAll('input')) { - if (isPasswordField(field) && isFillable(field)) return field +let timer: ReturnType | null = null +let initialScansScheduled = false +let filling = false +const observer = new MutationObserver(scheduleRescan) + +function scan(): void { + const { fields, roots } = collectLoginFields() + const next = findLoginTarget(fields) + const nextSignature = targetSignature(next) + if (!sameTarget(target, next) || signature !== nextSignature) { + targetId = next ? generateId() : null } - return null -} - -/** - * The username field for a password field: the nearest preceding text-like - * input in the same form, which is how essentially every login form is built. - */ -function findUsernameField(password: HTMLInputElement): HTMLInputElement | null { - const scope: ParentNode = password.form ?? document - const candidates: HTMLInputElement[] = [] - for (const field of scope.querySelectorAll('input')) { - const type = String(field.type || 'text').toLowerCase() - if (['text', 'email', 'tel', 'username'].includes(type) && isFillable(field)) { - candidates.push(field) - } + target = next + signature = nextSignature + observer.disconnect() + for (const root of roots) { + if (root === document && !document.documentElement) continue + observer.observe(root, { + childList: true, + subtree: true, + attributes: true, + attributeFilter: [ + 'type', + 'autocomplete', + 'name', + 'id', + 'disabled', + 'readonly', + 'class', + 'style', + 'hidden', + 'inert', + 'aria-hidden', + ], + }) } - const preceding = candidates.filter( - (field) => (password.compareDocumentPosition(field) & Node.DOCUMENT_POSITION_PRECEDING) !== 0 - ) - return preceding.at(-1) ?? candidates[0] ?? null } -/** - * The identifier field of a sign-in step that has no password field yet. - * - * Two-step sign-in — email, Continue, then the password on the next screen — - * is now the norm at Google, Okta, and most workplace tools. Requiring a - * password field would mean the key icon never appears on the step where the - * user actually needs it. Only the page's own declaration counts here: an - * `autocomplete` token naming a username or email, or an email input. That is - * narrow enough to leave newsletter boxes and search fields alone. - */ -function findIdentifierField(): HTMLInputElement | null { - for (const field of document.querySelectorAll('input')) { - if (!isFillable(field)) continue - const tokens = autocompleteTokens(field) - if (tokens.includes('username') || tokens.includes('email')) return field - if (String(field.type || '').toLowerCase() === 'email') return field +function report(): void { + scan() + const focused = focusedInput() + const rect = + focused && (focused === target?.username || focused === target?.password) + ? focused.getBoundingClientRect() + : null + const visible = + rect && rect.bottom > 0 && rect.top < innerHeight && rect.right > 0 && rect.left < innerWidth + const state: CredentialFormReport = { + origin: location.origin, + targetId, + hasLoginForm: target !== null, + hasPasswordField: target?.password != null, + bounds: visible ? { x: rect.x, y: rect.y, width: rect.width, height: rect.height } : null, } - return null -} - -function detectForm(): DetectedForm | null { - const password = findPasswordField() - if (password) return { password, username: findUsernameField(password) } - const username = findIdentifierField() - return username ? { password: null, username } : null -} - -function reportFormState(): void { - detected = detectForm() - - const origin = window.location.origin - const hasPasswordField = detected?.password != null - const fingerprint = `${origin}|${detected !== null}|${hasPasswordField}` + const fingerprint = JSON.stringify(state) if (fingerprint === lastReported) return lastReported = fingerprint - ipcRenderer.send(FORM_STATE_CHANNEL, { - origin, - hasLoginForm: detected !== null, - hasPasswordField, - }) + ipcRenderer.send('browser-credentials:form-state', state) } function scheduleRescan(): void { - // Coalesce a mutation burst without letting an animated page postpone the - // scan forever by continually resetting a trailing-edge debounce. - if (rescanTimer !== null) return - rescanTimer = setTimeout(() => { - rescanTimer = null - reportFormState() - }, RESCAN_DEBOUNCE_MS) + if (timer !== null || filling) return + timer = setTimeout(() => { + timer = null + report() + }, RESCAN_DELAY_MS) } -function observeDocument(): void { - if (formObserver || !document.documentElement) return - formObserver = new MutationObserver(scheduleRescan) - formObserver.observe(document.documentElement, { - childList: true, - subtree: true, - attributes: true, - // Login steps often exist at first paint and become usable only after a - // framework flips an ancestor's visibility class/style. - attributeFilter: [ - 'type', - 'autocomplete', - 'disabled', - 'readonly', - 'class', - 'style', - 'hidden', - 'aria-hidden', - ], - }) +function initialize(): void { + report() + if (initialScansScheduled) return + initialScansScheduled = true + for (const delay of INITIAL_RESCAN_DELAYS_MS) setTimeout(report, delay) +} + +/** Native setters plus composed events reach framework listeners across shadow boundaries. */ +function setFieldValue(field: HTMLInputElement, value: string): void { + const setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value')?.set + if (!setter) throw new Error('Input value setter unavailable') + setter.call(field, value) + field.dispatchEvent(new Event('input', { bubbles: true, composed: true })) + field.dispatchEvent(new Event('change', { bubbles: true, composed: true })) } -function reportInitialFormState(): void { - observeDocument() - reportFormState() - if (initialRescansScheduled) return - initialRescansScheduled = true - // Stylesheets, hydration, and password-step transitions can settle without - // a useful child mutation. A short bounded tail covers that first-load - // window without leaving a permanent poller behind. - for (const delay of INITIAL_RESCAN_DELAYS_MS) { - setTimeout(reportFormState, delay) +async function fill(payload: CredentialFillRequest): Promise { + if (payload.origin !== location.origin || payload.targetId !== targetId || filling) + return 'stale-target' + const intended = target + scan() + if (!intended || payload.targetId !== targetId || !sameTarget(intended, target)) + return 'stale-target' + const intendedSignature = signature + const writes: Array<[HTMLInputElement, string]> = [] + if (intended.username && payload.username) writes.push([intended.username, payload.username]) + if (intended.password && payload.password) writes.push([intended.password, payload.password]) + if (!writes.length) return 'failed' + filling = true + try { + for (const [field, value] of writes) { + if ( + !isFillable(field) || + targetSignature(intended) !== intendedSignature || + !sameTarget(intended, findLoginTarget(collectLoginFields().fields)) + ) + return 'stale-target' + setFieldValue(field, value) + } + await new Promise((resolve) => requestAnimationFrame(() => resolve())) + return writes.every(([field, value]) => isFillable(field) && field.value === value) + ? 'filled' + : 'failed' + } catch { + return 'failed' + } finally { + filling = false + scheduleRescan() } } -ipcRenderer.on(RESCAN_CHANNEL, () => { - // Same-document navigation does not recreate this preload. Main invalidates - // its old form state for safety, so bypass the unchanged fingerprint once - // and republish the live form after the SPA route has committed. +ipcRenderer.on('browser-credentials:fill', (_event, payload: CredentialFillRequest) => { + void fill(payload).then((status) => { + ipcRenderer.send('browser-credentials:fill-result', { requestId: payload.requestId, status }) + }) +}) +ipcRenderer.on('browser-credentials:rescan', () => { + target = null + targetId = null lastReported = '' scheduleRescan() }) -/** - * Writes through the native value setter so frameworks that track their own - * input state (React and friends) see the change instead of reverting it on - * the next render. - */ -function setFieldValue(field: HTMLInputElement, value: string): void { - const setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value')?.set - if (setter) { - setter.call(field, value) - } else { - field.value = value - } - field.dispatchEvent(new Event('input', { bubbles: true })) - field.dispatchEvent(new Event('change', { bubbles: true })) -} - -ipcRenderer.on( - FILL_CHANNEL, - (_event, payload: { origin: string; username: string; password?: string }) => { - // Last line of defence against a navigation between the user's choice and - // this message arriving: the main process binds the fill to an origin, and - // the live document has to still agree. - if (!payload || payload.origin !== window.location.origin || !detected) return - if (detected.username && payload.username) { - setFieldValue(detected.username, payload.username) - } - if (detected.password && payload.password) { - setFieldValue(detected.password, payload.password) +document.addEventListener('focusin', () => { + report() + ipcRenderer.send('browser-credentials:picker', 'open') +}) +document.addEventListener( + 'pointerdown', + (event) => { + const input = event.composedPath().find((node) => node instanceof HTMLInputElement) + if (input) { + setTimeout(() => { + report() + ipcRenderer.send('browser-credentials:picker', 'open') + }, 0) + } else ipcRenderer.send('browser-credentials:picker', 'dismiss') + }, + true +) +document.addEventListener( + 'keydown', + (event) => { + if (!event.isTrusted) return + if (event.key === 'Escape') ipcRenderer.send('browser-credentials:picker', 'dismiss') + const focused = focusedInput() + if ( + event.key === 'ArrowDown' && + focused && + (focused === target?.username || focused === target?.password) + ) { + ipcRenderer.send('browser-credentials:picker', 'focus') } - // Never submitted. Autofill and submission stay separate so the user can - // confirm the site and the account before anything is sent. Focus lands on - // whichever field the user still has to deal with. - ;(detected.password ?? detected.username)?.focus() - } + }, + true ) - -// Electron preloads can run before `` exists. Never pass that null root -// to MutationObserver: the exception would permanently disable dynamic-form -// detection for this document. -observeDocument() -document.addEventListener('readystatechange', observeDocument) -document.addEventListener('DOMContentLoaded', reportInitialFormState) -window.addEventListener('load', reportInitialFormState) -window.addEventListener('pageshow', reportInitialFormState) - -// Login forms are routinely rendered after first paint, behind a "Sign in" -// toggle, or swapped in by a single-page router. The observer is installed by -// `observeDocument` as soon as a real document root exists. +document.addEventListener('scroll', scheduleRescan, true) +window.addEventListener('resize', scheduleRescan) +document.addEventListener('readystatechange', scheduleRescan) +document.addEventListener('DOMContentLoaded', initialize) +window.addEventListener('load', initialize) +window.addEventListener('pageshow', initialize) +scan() diff --git a/apps/desktop/src/preload/credential-picker.ts b/apps/desktop/src/preload/credential-picker.ts new file mode 100644 index 00000000000..e2c5d24aed0 --- /dev/null +++ b/apps/desktop/src/preload/credential-picker.ts @@ -0,0 +1,13 @@ +import { contextBridge, ipcRenderer } from 'electron' +import { exposeShellTheme } from '@/preload/shell-theme' +import type { CredentialPickerApi } from '@/shared/browser-credentials' + +const api: CredentialPickerApi = { + configuration: () => ipcRenderer.invoke('credential-picker:configuration'), + select: (id) => ipcRenderer.invoke('credential-picker:select', id), + dismiss: () => ipcRenderer.send('credential-picker:dismiss'), + resize: (height) => ipcRenderer.send('credential-picker:resize', height), +} + +contextBridge.exposeInMainWorld('simCredentialPicker', api) +exposeShellTheme() diff --git a/apps/desktop/src/preload/index.ts b/apps/desktop/src/preload/index.ts index 6e6ed65a51c..50fe5d3ff25 100644 --- a/apps/desktop/src/preload/index.ts +++ b/apps/desktop/src/preload/index.ts @@ -28,6 +28,8 @@ import type { BrowserToolbarCommand, DesktopAppearanceTheme, DesktopCommand, + DesktopLocalFileRequest, + DesktopLocalFileResponse, DesktopNotificationPayload, DesktopOAuthConnectResult, DesktopOAuthConnectScope, @@ -146,6 +148,8 @@ const api: SimDesktopApi = { }, localFilesystem: (request: LocalFilesystemRequest): Promise => ipcRenderer.invoke('desktop:local-filesystem', request), + localFiles: (request: DesktopLocalFileRequest): Promise => + ipcRenderer.invoke('desktop:local-files', request), onCommand: (callback: (command: DesktopCommand) => void): (() => void) => { const listener = (_event: unknown, command: DesktopCommand) => callback(command) ipcRenderer.on('desktop:command', listener) diff --git a/apps/desktop/src/renderer/credential-picker/index.tsx b/apps/desktop/src/renderer/credential-picker/index.tsx new file mode 100644 index 00000000000..622daf6f053 --- /dev/null +++ b/apps/desktop/src/renderer/credential-picker/index.tsx @@ -0,0 +1,107 @@ +import { useRef, useState } from 'react' +import { + DropdownMenu, + DropdownMenuContent, + DropdownMenuItem, + DropdownMenuLabel, + DropdownMenuTrigger, + OverflowText, +} from '@sim/emcn' +import { createRoot } from 'react-dom/client' +import { initializeShellPage } from '@/renderer/shell' +import type { + CredentialPickerApi, + CredentialPickerConfiguration, +} from '@/shared/browser-credentials' +import '@/renderer/shell.css' + +const api = (window as Window & { simCredentialPicker?: CredentialPickerApi }).simCredentialPicker + +interface CredentialPickerProps { + configuration: CredentialPickerConfiguration + api: CredentialPickerApi +} + +function CredentialPicker({ configuration, api }: CredentialPickerProps) { + const firstItem = useRef(null) + const [pending, setPending] = useState(false) + const [error, setError] = useState(null) + const select = async (id: string) => { + setPending(true) + setError(null) + try { + const result = await api.select(id) + if (result === 'filled') api.dismiss() + else + setError( + result === 'stale-target' + ? 'The form changed. Select the field again.' + : 'Could not fill this form. Select the field again.' + ) + } catch { + setError('Could not fill this form. Select the field again.') + } finally { + setPending(false) + } + } + return ( + { + if (!open) api.dismiss() + }} + > + + { + event.preventDefault() + firstItem.current?.focus() + }} + onCloseAutoFocus={(event) => event.preventDefault()} + ref={(element) => { + if (!element) return + const resize = () => + api.resize(element.offsetHeight + element.scrollHeight - element.clientHeight) + const observer = new ResizeObserver(resize) + observer.observe(element) + resize() + return () => observer.disconnect() + }} + > + + + + {configuration.accounts.map((account, index) => ( + { + event.preventDefault() + void select(account.id) + }} + > + {account.username || 'No username'} + + ))} + {error && ( +

+ {error} +

+ )} + + + ) +} + +const container = document.getElementById('root') +if (!container || !api) throw new Error('Credential picker unavailable') +void Promise.all([initializeShellPage(), api.configuration()]).then(([, configuration]) => { + createRoot(container).render() +}) diff --git a/apps/desktop/src/shared/browser-credentials.ts b/apps/desktop/src/shared/browser-credentials.ts new file mode 100644 index 00000000000..6f48573d293 --- /dev/null +++ b/apps/desktop/src/shared/browser-credentials.ts @@ -0,0 +1,43 @@ +export interface CredentialFieldBounds { + x: number + y: number + width: number + height: number +} + +/** No input values leave the isolated page world when discovering a login. */ +export interface CredentialFormReport { + origin: string + targetId: string | null + hasLoginForm: boolean + hasPasswordField: boolean + bounds: CredentialFieldBounds | null +} + +export type CredentialFillStatus = 'filled' | 'stale-target' | 'failed' + +export interface CredentialFillRequest { + requestId: string + targetId: string + origin: string + username: string + password?: string +} + +export interface CredentialFillResult { + requestId: string + status: CredentialFillStatus +} + +export interface CredentialPickerConfiguration { + origin: string + accounts: Array<{ id: string; username: string }> +} + +/** Only the bundled account picker receives this bridge; it never receives a password. */ +export interface CredentialPickerApi { + configuration(): Promise + select(id: string): Promise + dismiss(): void + resize(height: number): void +} diff --git a/apps/desktop/static/credential-picker.html b/apps/desktop/static/credential-picker.html new file mode 100644 index 00000000000..0ea123b0c9e --- /dev/null +++ b/apps/desktop/static/credential-picker.html @@ -0,0 +1,12 @@ + + + + + + + Saved passwords + + + +
+ diff --git a/apps/docs/content/docs/agents/index.mdx b/apps/docs/content/docs/agents/index.mdx index f1300676781..7ab7f7afc70 100644 --- a/apps/docs/content/docs/agents/index.mdx +++ b/apps/docs/content/docs/agents/index.mdx @@ -17,7 +17,7 @@ The example throughout is an agent that scores inbound sales leads. ## The Agent block -You set up the reasoning step by giving the Agent block a **model** and a **prompt**. The model is the LLM that powers it; you pick one from the available providers, and the default is `claude-sonnet-4-6`. The prompt is a system message that defines who the agent is and how it should behave, plus a user message that carries the input, usually a reference like ``. +You set up the reasoning step by giving the Agent block a **model** and a **prompt**. The model is the LLM that powers it; you pick one from the available providers, and the default is `claude-sonnet-5`. The prompt is a system message that defines who the agent is and how it should behave, plus a user message that carries the input, usually a reference like ``. When it runs, the Agent block reasons, calls any tools it needs, and stores its result under its own name. By default that result is free text in `content`, read by a later block as ``, alongside run details like the model used, token counts, tool calls, and cost. Every setting and output field is in the [Agent block reference](/workflows/blocks/agent). diff --git a/apps/docs/content/docs/chat/files.mdx b/apps/docs/content/docs/chat/files.mdx index 42ba0101f90..db82437d9d9 100644 --- a/apps/docs/content/docs/chat/files.mdx +++ b/apps/docs/content/docs/chat/files.mdx @@ -21,7 +21,7 @@ Use this to: - Drop in a PDF and ask Sim to turn it into a knowledge base document - Attach a design mockup and ask Sim to describe it or generate code from it -Uploaded files appear in the Files panel in the sidebar and are accessible to all workflows in the workspace. Sim can also fetch a file directly from a URL and save it for you: "Download the JSON at [URL] and save it to the workspace." +An attached file is stored under `uploads/` and readable by Sim for the rest of the conversation, but it does not appear in the Files panel and is not listed for workflows. Ask Sim to save a copy into workspace Files if you need it there. Sim can also fetch a file directly from a URL and save it for you: "Download the JSON at [URL] and save it to the workspace." ## Creating Documents @@ -114,4 +114,5 @@ When a file opens in the resource panel, you can switch between three views: diff --git a/apps/docs/content/docs/chat/index.mdx b/apps/docs/content/docs/chat/index.mdx index a15623c9536..c8ab0137ba5 100644 --- a/apps/docs/content/docs/chat/index.mdx +++ b/apps/docs/content/docs/chat/index.mdx @@ -65,6 +65,7 @@ Chat has two panes. On the left: the chat thread, where your messages and Sim's diff --git a/apps/docs/content/docs/chat/knowledge.mdx b/apps/docs/content/docs/chat/knowledge.mdx index 384ccd013d1..2cafd69e439 100644 --- a/apps/docs/content/docs/chat/knowledge.mdx +++ b/apps/docs/content/docs/chat/knowledge.mdx @@ -46,7 +46,7 @@ Ask Sim a question and it searches the specified knowledge base to answer: For knowledge bases that should stay current automatically, connectors sync content from external services on a schedule — no manual uploads needed. New content is added, changed content is re-processed, and deleted content is removed on every run. -Connectors are configured through the knowledge base settings, not through Chat. Once connected, all synced content is immediately searchable by Sim and by any Agent block with the knowledge base attached. +Connectors are usually configured in the knowledge base settings. Sim can also create, update, and sync them for you. Content becomes searchable after syncing and indexing, subject to the caller's access permissions. Use a [connector](/knowledgebase/connectors) to sync sources such as Notion, Google Drive, Slack, GitHub, or Confluence. diff --git a/apps/docs/content/docs/chat/research.mdx b/apps/docs/content/docs/chat/research.mdx index 532e665b205..2bb0bab87b3 100644 --- a/apps/docs/content/docs/chat/research.mdx +++ b/apps/docs/content/docs/chat/research.mdx @@ -39,4 +39,5 @@ When you need a structured, saved document rather than a chat answer, ask Sim to diff --git a/apps/docs/content/docs/chat/tasks.mdx b/apps/docs/content/docs/chat/tasks.mdx index a1b8bbeab7d..324731b9d2a 100644 --- a/apps/docs/content/docs/chat/tasks.mdx +++ b/apps/docs/content/docs/chat/tasks.mdx @@ -124,4 +124,6 @@ Sim can build custom tools from a description: { question: "What's the difference between a scheduled job and a deployed workflow?", answer: "A scheduled job runs a Chat prompt on a cron schedule — Sim decides what to do each time based on current workspace state. A deployed workflow runs a saved graph of blocks, which can include Agent reasoning. Use jobs when you want Sim to reason and adapt; use workflows when you want predictable, auditable execution." }, { question: "Can a scheduled job trigger a workflow?", answer: "Yes. Include it in the job prompt: 'Run the invoice sync workflow and then post the results to Slack.'" }, { question: "Can direct actions be undone?", answer: "It depends on the service and action. Some records can be edited or deleted afterward; sending an email cannot be undone by deleting the Chat task." }, + { question: "How do I know what integrations are connected?", answer: "Ask Sim: 'What integrations are connected to this workspace?' or check the Integrations page." }, + { question: "How do workflows reference environment variables?", answer: "Use {{ENV_VAR}} syntax. Resolution depends on the variable's scope and the execution context." }, ]} /> diff --git a/apps/docs/content/docs/chat/workflows.mdx b/apps/docs/content/docs/chat/workflows.mdx index c2d3e5dc2b3..cdfa33e0764 100644 --- a/apps/docs/content/docs/chat/workflows.mdx +++ b/apps/docs/content/docs/chat/workflows.mdx @@ -27,7 +27,7 @@ Describe what the workflow should do — what triggers it, what it should do, wh Open an existing workflow with `@workflow-name` or the **+** menu, then describe the change. Sim reads the current structure before modifying it — you don't need to explain what already exists. - "Add a condition that routes to a different branch if the confidence score is below 0.7" -- "Replace the GPT-4o model with Claude Opus 4.6 on the summarizer block" +- "Replace the GPT-4o model with Claude Opus 5 on the summarizer block" - "Add a Slack notification at the end that includes the output" ## Running Workflows @@ -112,4 +112,5 @@ Variables set this way are available via `` syntax insid diff --git a/apps/docs/content/docs/cli/authentication.mdx b/apps/docs/content/docs/cli/authentication.mdx index cf70ed37fa5..e9c8afb91f1 100644 --- a/apps/docs/content/docs/cli/authentication.mdx +++ b/apps/docs/content/docs/cli/authentication.mdx @@ -5,6 +5,12 @@ description: Sign in from the terminal, authenticate CI with an API key, and kee import { Callout } from 'fumadocs-ui/components/callout' + +These settings apply to the CLI you install locally. When Sim runs a `sim` +command for you inside Chat, it uses your session identity — there is no profile, +no `sim login`, and no config file. + + `sim login` signs you in through your browser. It prefers OAuth, which stores a short-lived login that renews itself, and selects API-key pairing for remote terminals or servers without OAuth support. In CI you supply an existing API @@ -196,7 +202,7 @@ export SIM_WORKSPACE="2f6d0b1c-8a34-4d92-b7e5-31c8a0f45d67" sim workflows run 3a9e21d8-5f47-4c0b-b2ea-91d7c6034ef8 --input '{"source":"nightly"}' --output json ``` -Create the key in Sim under **Settings → API keys**. Store it as a secret in your +Create the key in Sim under **Settings → Sim API keys**. Store it as a secret in your CI provider — never commit it. `SIM_CONFIG_DIR` relocates both files if you need them somewhere other than diff --git a/apps/docs/content/docs/cli/blocks.mdx b/apps/docs/content/docs/cli/blocks.mdx index 96166c9fee2..93e6d3ce480 100644 --- a/apps/docs/content/docs/cli/blocks.mdx +++ b/apps/docs/content/docs/cli/blocks.mdx @@ -38,7 +38,9 @@ sim blocks list [options] | `--search ` | No | Case-insensitive substring match against the block id, name, and description. | | `--category ` | No | Restrict to one toolbar category. Accepted values: `blocks`, `tools`, `triggers`. | | `--capability ` | No | Restrict to blocks that can start a workflow — the `triggers` category, blocks declaring `triggerAllowed`, and blocks with trigger-mode fields. Accepted values: `trigger`. | -| `--source ` | No | Restrict to built-in blocks or this workspace's deployed custom blocks. Accepted values: `builtin`, `custom`. | +| `--source ` | No | Restrict to shipped blocks or to this workspace’s deployed custom blocks. Accepted values: `builtin`, `custom`. | +| `--include-sunset` | No | Include `legacy` and `deprecated` blocks. Off by default: a sunset block keeps executing where it is already placed, but it is not offered for new authoring. Each returned entry carries `sunset.replacedBy`, the block to build with instead. | +| `--no-include-sunset` | No | Send --include-sunset as false. | | `--sort-by ` | No | Field used to sort the result. Sorting by `name` is case-sensitive and follows the storage collation, so do not rely on a case-insensitive order. Accepted values: `id`, `name`, `category`. | | `--sort-order ` | No | Sort direction. Accepted values: `asc`, `desc`. | | `--limit ` | No | Maximum items to return (0 for everything). Defaults to `0`. | diff --git a/apps/docs/content/docs/cli/configuration.mdx b/apps/docs/content/docs/cli/configuration.mdx index 3a1f6ac6814..7963852f478 100644 --- a/apps/docs/content/docs/cli/configuration.mdx +++ b/apps/docs/content/docs/cli/configuration.mdx @@ -3,6 +3,10 @@ title: Configuration description: Profiles, config files, environment variables, and how each setting is resolved --- +These settings apply to the CLI you install locally. When Sim runs a `sim` +command for you inside Chat, it uses your session identity — there is no profile, +no `sim login`, and no config file. + The CLI resolves an **endpoint**, **credential**, **workspace**, and **output format**. The credential can be a stored OAuth login or an API key. Each resolves independently, so a saved default can still be overridden for a single command. diff --git a/apps/docs/content/docs/cli/connector-types.mdx b/apps/docs/content/docs/cli/connector-types.mdx index 2823ed60e0c..88dd05a0cde 100644 --- a/apps/docs/content/docs/cli/connector-types.mdx +++ b/apps/docs/content/docs/cli/connector-types.mdx @@ -20,5 +20,8 @@ sim connector-types list [options] | Option | Required | Description | | --- | --- | --- | | `--search ` | No | Case-insensitive substring match against the connector name. | +| `--detail ` | No | Projection of each item. `summary` (the default) carries the identifier, name, description, and auth mode; `full` adds the version, the complete auth settings, the `sourceConfig` field schema, incremental-sync support, and tag definitions. Accepted values: `summary`, `full`. | +| `--limit ` | No | Maximum items to return (0 for everything). Defaults to `100`. | +| `--cursor ` | No | Continue from nextCursor returned by a previous result. | diff --git a/apps/docs/content/docs/cli/files.mdx b/apps/docs/content/docs/cli/files.mdx index db168e20bfe..e8934f98bb6 100644 --- a/apps/docs/content/docs/cli/files.mdx +++ b/apps/docs/content/docs/cli/files.mdx @@ -21,7 +21,7 @@ sim files batch-delete [options] | Option | Required | Description | | --- | --- | --- | -| `--file-ids ` | Yes | File identifiers to update. (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | +| `--file-ids ` | Yes | File identifiers to delete. (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `-y, --yes` | Yes | Confirm this operation. | @@ -486,7 +486,7 @@ sim files read [options] | Argument | Required | Description | | --- | --- | --- | -| `fileId` | Yes | File identifier. | +| `fileId` | Yes | File identifier, or the file’s VFS path: `files/<folder>/<name>`, or `uploads/<name>` for a Chat upload. | diff --git a/apps/docs/content/docs/cli/logs.mdx b/apps/docs/content/docs/cli/logs.mdx index f29733d9b76..4b35e0db28b 100644 --- a/apps/docs/content/docs/cli/logs.mdx +++ b/apps/docs/content/docs/cli/logs.mdx @@ -51,9 +51,12 @@ sim logs stats [options] | `--folder ` | No | Folder path as shown in the app; the leading / is optional (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `--trigger ` | No | Comma-separated trigger types to include. An empty entry is rejected. The vocabulary is open, so an unrecognized member selects no runs; the literal `all` disables this filter. (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `--level ` | No | Severity level to include. Accepted values: `info`, `error`. | +| `--include-handled-errors` | No | Whether runs with a handled block error are counted as `handledErrorRuns`, and whether `level=error` also selects them. Off by default: counting them scans each run’s stored trace. | +| `--no-include-handled-errors` | No | Send --include-handled-errors as false. | | `--start-date ` | No | Only include runs started at or after this UTC ISO 8601 timestamp, e.g. `2026-08-06T00:00:00Z`. A date without a time, or a timestamp carrying a UTC offset instead of `Z`, is rejected, as is year `0000`, which names no storable instant. | | `--end-date ` | No | Only include runs started at or before this UTC ISO 8601 timestamp, e.g. `2026-08-06T00:00:00Z`. A date without a time, or a timestamp carrying a UTC offset instead of `Z`, is rejected, as is year `0000`, which names no storable instant. | -| `--segment-count ` | No | Number of time buckets, up to 500. Exactly this many are returned, each at least one minute wide. Short windows extend past the requested end and include empty trailing buckets. | +| `--segment-count ` | No | Number of equal time buckets to divide the window into, from 1 to 500. It is the ceiling on how many buckets a series carries: with `includeEmpty=true` exactly this many are returned, otherwise only the buckets holding at least one run. Buckets are never narrower than one minute, so on a short window the series extends past the end of the window rather than being compressed, and the trailing buckets are empty. | +| `--include-empty ` | No | Whether buckets with no runs are included in every series. Off by default, so each series carries only the buckets that hold at least one run; set it to publish exactly `segmentCount` buckets per series, empty ones included. The listed spellings are the whole accepted vocabulary and are case-sensitive; any other value is rejected. Accepted values: `true`, `1`, `yes`, `on`, `y`, `enabled`, `false`, `0`, `no`, `off`, `n`, `disabled`. | @@ -84,6 +87,8 @@ sim logs list [options] | `--include-final-output` | No | Include final output in JSON or YAML output (implies full detail). | | `--limit ` | No | Maximum items to return (0 for everything). Defaults to `100`. | | `--cursor ` | No | Continue from nextCursor returned by a previous result. | +| `--include-handled-errors` | No | Whether `level=error` also selects runs that finished at `info` after a block error was recovered by an error path. Off by default: such a run succeeded, so it is an error only to a caller auditing error handling. Every row reports `hasHandledErrors` whether or not this is set. Job runs carry no block trace, so the flag never widens that branch. | +| `--no-include-handled-errors` | No | Send --include-handled-errors as false. | | `--status ` | No | Comma-separated execution statuses to include, from `pending` \| `running` \| `paused` \| `redacting` \| `completed` \| `failed` \| `cancelled`. An empty entry is rejected. ANDed with `level`, which reports severity rather than lifecycle. | | `--workflow-name ` | No | Case-insensitive substring match against the run's workflow name. Runs whose workflow has been deleted match nothing, because the name is no longer joinable. | | `--include-job-runs` | No | Include Chat and Sim-agent jobs alongside workflow runs. Jobs use `kind: "job"` and have no workflow or cost ledger. Workflow, folder, model, or status filters exclude jobs. This option is valid only when sorting by `startedAt`. | diff --git a/apps/docs/content/docs/cli/output.mdx b/apps/docs/content/docs/cli/output.mdx index 85ca9d95c97..234149a1a5d 100644 --- a/apps/docs/content/docs/cli/output.mdx +++ b/apps/docs/content/docs/cli/output.mdx @@ -68,7 +68,7 @@ sim logs get 9c4f0b7e-2d81-4a35-b6e9-70f1c8a2d543 --output json | jq '.traceSpan ## Exceptions -`sim profiles` and `sim configure` print local configuration for humans. +`sim configure` prints local configuration for humans. `sim profiles` honors `--output`. `sim files get` writes the file’s raw content to stdout or `--output-file`. `sim chat` streams reply text in `table` and `text` modes. In `json` or `yaml` diff --git a/apps/docs/content/docs/cli/reference.mdx b/apps/docs/content/docs/cli/reference.mdx index 5a0af9c37e0..cec31d1d71b 100644 --- a/apps/docs/content/docs/cli/reference.mdx +++ b/apps/docs/content/docs/cli/reference.mdx @@ -357,7 +357,9 @@ sim blocks list [options] | `--search ` | No | Case-insensitive substring match against the block id, name, and description. | | `--category ` | No | Restrict to one toolbar category. Accepted values: `blocks`, `tools`, `triggers`. | | `--capability ` | No | Restrict to blocks that can start a workflow — the `triggers` category, blocks declaring `triggerAllowed`, and blocks with trigger-mode fields. Accepted values: `trigger`. | -| `--source ` | No | Restrict to built-in blocks or this workspace's deployed custom blocks. Accepted values: `builtin`, `custom`. | +| `--source ` | No | Restrict to shipped blocks or to this workspace’s deployed custom blocks. Accepted values: `builtin`, `custom`. | +| `--include-sunset` | No | Include `legacy` and `deprecated` blocks. Off by default: a sunset block keeps executing where it is already placed, but it is not offered for new authoring. Each returned entry carries `sunset.replacedBy`, the block to build with instead. | +| `--no-include-sunset` | No | Send --include-sunset as false. | | `--sort-by ` | No | Field used to sort the result. Sorting by `name` is case-sensitive and follows the storage collation, so do not rely on a case-insensitive order. Accepted values: `id`, `name`, `category`. | | `--sort-order ` | No | Sort direction. Accepted values: `asc`, `desc`. | | `--limit ` | No | Maximum items to return (0 for everything). Defaults to `0`. | @@ -406,6 +408,9 @@ sim connector-types list [options] | Option | Required | Description | | --- | --- | --- | | `--search ` | No | Case-insensitive substring match against the connector name. | +| `--detail ` | No | Projection of each item. `summary` (the default) carries the identifier, name, description, and auth mode; `full` adds the version, the complete auth settings, the `sourceConfig` field schema, incremental-sync support, and tag definitions. Accepted values: `summary`, `full`. | +| `--limit ` | No | Maximum items to return (0 for everything). Defaults to `100`. | +| `--cursor ` | No | Continue from nextCursor returned by a previous result. | @@ -742,7 +747,7 @@ sim files batch-delete [options] | Option | Required | Description | | --- | --- | --- | -| `--file-ids ` | Yes | File identifiers to update. (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | +| `--file-ids ` | Yes | File identifiers to delete. (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `-y, --yes` | Yes | Confirm this operation. | @@ -1245,7 +1250,7 @@ sim files read [options] | Argument | Required | Description | | --- | --- | --- | -| `fileId` | Yes | File identifier. | +| `fileId` | Yes | File identifier, or the file’s VFS path: `files/<folder>/<name>`, or `uploads/<name>` for a Chat upload. | @@ -2748,9 +2753,12 @@ sim logs stats [options] | `--folder ` | No | Folder path as shown in the app; the leading / is optional (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `--trigger ` | No | Comma-separated trigger types to include. An empty entry is rejected. The vocabulary is open, so an unrecognized member selects no runs; the literal `all` disables this filter. (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `--level ` | No | Severity level to include. Accepted values: `info`, `error`. | +| `--include-handled-errors` | No | Whether runs with a handled block error are counted as `handledErrorRuns`, and whether `level=error` also selects them. Off by default: counting them scans each run’s stored trace. | +| `--no-include-handled-errors` | No | Send --include-handled-errors as false. | | `--start-date ` | No | Only include runs started at or after this UTC ISO 8601 timestamp, e.g. `2026-08-06T00:00:00Z`. A date without a time, or a timestamp carrying a UTC offset instead of `Z`, is rejected, as is year `0000`, which names no storable instant. | | `--end-date ` | No | Only include runs started at or before this UTC ISO 8601 timestamp, e.g. `2026-08-06T00:00:00Z`. A date without a time, or a timestamp carrying a UTC offset instead of `Z`, is rejected, as is year `0000`, which names no storable instant. | -| `--segment-count ` | No | Number of time buckets, up to 500. Exactly this many are returned, each at least one minute wide. Short windows extend past the requested end and include empty trailing buckets. | +| `--segment-count ` | No | Number of equal time buckets to divide the window into, from 1 to 500. It is the ceiling on how many buckets a series carries: with `includeEmpty=true` exactly this many are returned, otherwise only the buckets holding at least one run. Buckets are never narrower than one minute, so on a short window the series extends past the end of the window rather than being compressed, and the trailing buckets are empty. | +| `--include-empty ` | No | Whether buckets with no runs are included in every series. Off by default, so each series carries only the buckets that hold at least one run; set it to publish exactly `segmentCount` buckets per series, empty ones included. The listed spellings are the whole accepted vocabulary and are case-sensitive; any other value is rejected. Accepted values: `true`, `1`, `yes`, `on`, `y`, `enabled`, `false`, `0`, `no`, `off`, `n`, `disabled`. | @@ -2783,6 +2791,8 @@ sim logs list [options] | `--include-final-output` | No | Include final output in JSON or YAML output (implies full detail). | | `--limit ` | No | Maximum items to return (0 for everything). Defaults to `100`. | | `--cursor ` | No | Continue from nextCursor returned by a previous result. | +| `--include-handled-errors` | No | Whether `level=error` also selects runs that finished at `info` after a block error was recovered by an error path. Off by default: such a run succeeded, so it is an error only to a caller auditing error handling. Every row reports `hasHandledErrors` whether or not this is set. Job runs carry no block trace, so the flag never widens that branch. | +| `--no-include-handled-errors` | No | Send --include-handled-errors as false. | | `--status ` | No | Comma-separated execution statuses to include, from `pending` \| `running` \| `paused` \| `redacting` \| `completed` \| `failed` \| `cancelled`. An empty entry is rejected. ANDed with `level`, which reports severity rather than lifecycle. | | `--workflow-name ` | No | Case-insensitive substring match against the run's workflow name. Runs whose workflow has been deleted match nothing, because the name is no longer joinable. | | `--include-job-runs` | No | Include Chat and Sim-agent jobs alongside workflow runs. Jobs use `kind: "job"` and have no workflow or cost ledger. Workflow, folder, model, or status filters exclude jobs. This option is valid only when sorting by `startedAt`. | @@ -4461,7 +4471,7 @@ sim tables groups create [options] | Option | Required | Description | | --- | --- | --- | | `--group ` | Yes | Workflow or enrichment producer definition. (JSON, or @path / @- to read a file or stdin). | -| `--output-columns ` | Yes | Columns created for producer outputs. (JSON, or @path / @- to read a file or stdin). | +| `--output-columns ` | No | Columns to create for producer outputs. An entry naming a column the table already has attaches that column to the group instead of creating it (its `type` must match), and an output whose column already exists may omit its entry entirely — so `[]` attaches existing columns only. (JSON, or @path / @- to read a file or stdin). | | `--auto-run` | No | Whether to schedule existing rows after group creation. | | `--no-auto-run` | No | Send --auto-run as false. | @@ -5005,7 +5015,7 @@ sim tables dispatches get ### sim tables dispatches list -List Active Run Dispatches +List Run Dispatches ```bash sim tables dispatches list @@ -5472,7 +5482,7 @@ sim tables delete [options] ### sim tables enrichment get -Get Enrichment Run Detail +Get Row Group Run ```bash sim tables enrichment get @@ -6060,7 +6070,7 @@ sim workflows operations apply [options] | Option | Required | Description | | --- | --- | --- | -| `--dry-run` | No | Validate and lint without persisting. The response is identical to the committed write of the same body, so a caller can inspect `lint` and then re-send the request for real. Nothing is written, no audit entry is recorded, and collaborators are not notified. | +| `--dry-run` | No | Validate and lint without writing, auditing, or notifying collaborators. Returns the same validation, preparation warnings, lint findings, and ID-ownership conflicts (`409`) as a committed write. `needsRedeployment` describes the pre-write state. For semantic operations, `mintedBlockIds` is empty; `previewBlockIds` contains provisional IDs with a warning, since committing mints new IDs. | | `--no-dry-run` | No | Send --dry-run as false. | | `--operations ` | Yes | Edits to apply, in a single batch, keyed by operation_type: [{"operation_type":"add","block_id":"my-fn","params":{"type":"function","name":"My Fn","inputs":{"code":"return {ok:true}"}}},{"operation_type":"edit","block_id":"<uuid>","params":{"name":"Renamed","connections":{"success":"my-fn"}}},{"operation_type":"delete","block_id":"<uuid>"}]. Also extract_from_subflow, whose params carry {"subflowId":"<loop-id>"}, and insert_into_subflow, which creates a block and so takes an add’s params plus that subflowId (JSON, or @path / @- to read a file or stdin). | | `--atomic` | No | Fail the whole batch when any operation is declined or any block input would be dropped. The default applies what it can and reports the rest in `skipped` and `inputValidationErrors`; `true` writes nothing and answers `409` instead. | @@ -6154,7 +6164,7 @@ sim workflows runs get [options] | --- | --- | --- | | `--workflow ` | Yes | Workflow ID. | | `--include-output` | No | Include the final output in JSON or YAML output. | -| `--select-output ` | No | Include blockId or blockId.path values in JSON or YAML output; block names are not resolved on a finished run (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | +| `--select-output ` | No | Include blockName.path or blockId.path values (e.g. agent_1.content) in JSON or YAML output; names resolve against the workflow’s current blocks, and missing paths are omitted (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `--include-file-base64` | No | Inline each produced file's bytes as base64. Requires `includeOutput`. A file above the inline ceiling answers `413` naming its download path; fetch large files from `downloadPath` instead. | | `--no-include-file-base64` | No | Send --include-file-base64 as false. | | `--base64-max-bytes ` | No | Per-file inline ceiling, lowering but never raising the server limit of 16 MiB. | @@ -6555,14 +6565,14 @@ sim workflows run [options] | `--input ` | No | Trigger input as JSON (JSON, or @path / @- to read a file or stdin). | | `--async` | No | Queue the run and return immediately. | | `--execution-timeout-seconds ` | No | Maximum duration of an asynchronous run, in seconds, capped by the plan's execution timeout. Requires `async: true`; otherwise returns `400`. | -| `--select-output ` | No | Return streamed outputs as blockName.path or childWorkflowId.blockName.path; selecting a child workflow applies to every invocation, requires --follow (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | +| `--select-output ` | No | Return blockName.path values (e.g. agent_1.content), or childWorkflowId.blockName.path for a child workflow (applies to every invocation) — in blockOutputs on a sync run, or from the streamed result with --follow; missing paths are omitted. Not available with --async (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `--include-file-base64` | No | Inline eligible output files as base64 content. Rejected when `async` is true. | | `--no-include-file-base64` | No | Send --include-file-base64 as false. | | `--base64-max-bytes ` | No | Maximum total bytes of file content to inline as base64, lowering but never raising the server limit of 16 MiB. Rejected when `async` is true. | | `--run-id ` | No | One-shot identifier for this run; NOT an idempotency key — reusing a claimed value fails with RUN_ID_CONFLICT instead of replaying the first result, and a fresh value starts another run. | | `--manual` | No | Run the current saved workflow state instead of the active deployment. | -| `--trigger ` | No | Enter a manual run through this runnable trigger (requires --manual). | -| `--mock-payload` | No | Use the selected trigger's server-derived mock payload (requires --manual). | +| `--trigger ` | No | Enter the run through this runnable trigger; runs the current saved workflow state (implies --manual). | +| `--mock-payload` | No | Use the selected trigger's server-derived mock payload; runs the current saved workflow state (implies --manual). | | `--from-block ` | No | Run manually from this saved workflow block. | | `--source-run ` | No | Prior run whose persisted state supplies upstream outputs (requires --from-block). | | `--follow` | No | Stream the run as it happens; progress on stderr, result on stdout. The stream reports only success and output, so the result omits the run id and timings a non-streaming run returns. | @@ -6596,6 +6606,8 @@ sim workflows export [options] | Option | Required | Description | | --- | --- | --- | | `--include-references` | No | Include non-secret resource identities for mapped import. | +| `--include-workspace-bindings` | No | Whether to keep workspace-scoped bindings — table, knowledge base, document, folder, channel, and other resource selectors — in the exported state. Defaults to false, the sharing-safe export in which those ids are cleared because they resolve nowhere else. Send true for a same-workspace round trip so the re-imported workflow can run without re-selecting them. Credentials, passwords, and table sub-block values are cleared either way. | +| `--no-include-workspace-bindings` | No | Send --include-workspace-bindings as false. | @@ -6705,7 +6717,7 @@ sim workflows state replace [options] | Option | Required | Description | | --- | --- | --- | -| `--dry-run` | No | Validate and lint without persisting. The response is identical to the committed write of the same body, so a caller can inspect `lint` and then re-send the request for real. Nothing is written, no audit entry is recorded, and collaborators are not notified. | +| `--dry-run` | No | Validate and lint without writing, auditing, or notifying collaborators. Returns the same validation, preparation warnings, lint findings, and ID-ownership conflicts (`409`) as a committed write. `needsRedeployment` describes the pre-write state. For semantic operations, `mintedBlockIds` is empty; `previewBlockIds` contains provisional IDs with a warning, since committing mints new IDs. | | `--no-dry-run` | No | Send --dry-run as false. | | `--blocks ` | Yes | Blocks keyed by block id. (JSON, or @path / @- to read a file or stdin). | | `--edges ` | Yes | Directed connections between blocks. (JSON, or @path / @- to read a file or stdin). | diff --git a/apps/docs/content/docs/cli/scripting.mdx b/apps/docs/content/docs/cli/scripting.mdx index e0d111736cf..7c494033d0f 100644 --- a/apps/docs/content/docs/cli/scripting.mdx +++ b/apps/docs/content/docs/cli/scripting.mdx @@ -12,6 +12,9 @@ Use these patterns to pass inputs, page through results, and handle command outc Any flag that takes JSON or a list also accepts `@path` to read a file, or `@-` to read stdin. +When Sim runs a command for you, `@path` reads from the chat's sandbox, not from +workspace Files. Use `sim files read` for a workspace path. + ```bash sim workflows import --workflow @wf.json sim tables rows query tbl_9f3c1a05d4b7426e8c2f0917ab35de64 --filter @filter.json @@ -221,19 +224,18 @@ esac ## Selecting workflow output -`--select-output` shapes a streamed result, so it requires `--follow`. It takes -`blockName.field` selectors; fields that a run did not produce are simply -omitted: +`--select-output` returns the named values in `blockOutputs` on a sync run, or +from the streamed result with `--follow`. It cannot be combined with `--async`. +It takes `blockName.field` selectors; fields that a run did not produce are +simply omitted: ```bash sim workflows run 3a9e21d8-5f47-4c0b-b2ea-91d7c6034ef8 --follow --select-output agent_1.content --output json ``` -Without `--follow` the CLI refuses the pair rather than spending a request on a -response that carries no outputs, and `--async` cannot be combined with it -either — there is no stream to shape. To narrow a run that has already finished, -read it back with `workflows runs get`, which matches block **ids** rather than -the block names `workflows run` takes: +To narrow a run that has already finished, read it back with +`workflows runs get`, which takes the same `blockName.path` or `blockId.path` +selectors; names resolve against the workflow's current blocks: ```bash sim workflows runs get "$run_id" --workflow 3a9e21d8-5f47-4c0b-b2ea-91d7c6034ef8 \ diff --git a/apps/docs/content/docs/cli/tables.mdx b/apps/docs/content/docs/cli/tables.mdx index 7cf0b265b3f..3f92acd7992 100644 --- a/apps/docs/content/docs/cli/tables.mdx +++ b/apps/docs/content/docs/cli/tables.mdx @@ -112,7 +112,7 @@ sim tables groups create [options] | Option | Required | Description | | --- | --- | --- | | `--group ` | Yes | Workflow or enrichment producer definition. (JSON, or @path / @- to read a file or stdin). | -| `--output-columns ` | Yes | Columns created for producer outputs. (JSON, or @path / @- to read a file or stdin). | +| `--output-columns ` | No | Columns to create for producer outputs. An entry naming a column the table already has attaches that column to the group instead of creating it (its `type` must match), and an output whose column already exists may omit its entry entirely — so `[]` attaches existing columns only. (JSON, or @path / @- to read a file or stdin). | | `--auto-run` | No | Whether to schedule existing rows after group creation. | | `--no-auto-run` | No | Send --auto-run as false. | @@ -616,7 +616,7 @@ sim tables dispatches get -## List active run dispatches +## List run dispatches ```bash sim tables dispatches list @@ -1043,7 +1043,7 @@ sim tables delete [options] -## Get enrichment run detail +## Get row group run ```bash sim tables enrichment get diff --git a/apps/docs/content/docs/cli/troubleshooting.mdx b/apps/docs/content/docs/cli/troubleshooting.mdx index f5bd7da2281..4ad1a9b2f22 100644 --- a/apps/docs/content/docs/cli/troubleshooting.mdx +++ b/apps/docs/content/docs/cli/troubleshooting.mdx @@ -70,7 +70,7 @@ switch to a machine format to see it in full: sim logs get 9c4f0b7e-2d81-4a35-b6e9-70f1c8a2d543 --output json ``` -## `sim files get` refuses to print to the terminal +## In an interactive terminal, `sim files get` refuses to print to the terminal Writing arbitrary binary to an interactive terminal can corrupt it, so non-text content has to go to a file or a pipe: diff --git a/apps/docs/content/docs/cli/workflows.mdx b/apps/docs/content/docs/cli/workflows.mdx index 7dbfc39efc2..786e93d54db 100644 --- a/apps/docs/content/docs/cli/workflows.mdx +++ b/apps/docs/content/docs/cli/workflows.mdx @@ -62,7 +62,7 @@ Apply Workflow Operations (OAuth login or personal API key required) | Option | Required | Description | | --- | --- | --- | -| `--dry-run` | No | Validate and lint without persisting. The response is identical to the committed write of the same body, so a caller can inspect `lint` and then re-send the request for real. Nothing is written, no audit entry is recorded, and collaborators are not notified. | +| `--dry-run` | No | Validate and lint without writing, auditing, or notifying collaborators. Returns the same validation, preparation warnings, lint findings, and ID-ownership conflicts (`409`) as a committed write. `needsRedeployment` describes the pre-write state. For semantic operations, `mintedBlockIds` is empty; `previewBlockIds` contains provisional IDs with a warning, since committing mints new IDs. | | `--no-dry-run` | No | Send --dry-run as false. | | `--operations ` | Yes | Edits to apply, in a single batch, keyed by operation_type: [{"operation_type":"add","block_id":"my-fn","params":{"type":"function","name":"My Fn","inputs":{"code":"return {ok:true}"}}},{"operation_type":"edit","block_id":"<uuid>","params":{"name":"Renamed","connections":{"success":"my-fn"}}},{"operation_type":"delete","block_id":"<uuid>"}]. Also extract_from_subflow, whose params carry {"subflowId":"<loop-id>"}, and insert_into_subflow, which creates a block and so takes an add’s params plus that subflowId (JSON, or @path / @- to read a file or stdin). | | `--atomic` | No | Fail the whole batch when any operation is declined or any block input would be dropped. The default applies what it can and reports the rest in `skipped` and `inputValidationErrors`; `true` writes nothing and answers `409` instead. | @@ -152,7 +152,7 @@ Show run status (requested outputs are included in JSON or YAML output) | --- | --- | --- | | `--workflow ` | Yes | Workflow ID. | | `--include-output` | No | Include the final output in JSON or YAML output. | -| `--select-output ` | No | Include blockId or blockId.path values in JSON or YAML output; block names are not resolved on a finished run (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | +| `--select-output ` | No | Include blockName.path or blockId.path values (e.g. agent_1.content) in JSON or YAML output; names resolve against the workflow’s current blocks, and missing paths are omitted (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `--include-file-base64` | No | Inline each produced file's bytes as base64. Requires `includeOutput`. A file above the inline ceiling answers `413` naming its download path; fetch large files from `downloadPath` instead. | | `--no-include-file-base64` | No | Send --include-file-base64 as false. | | `--base64-max-bytes ` | No | Per-file inline ceiling, lowering but never raising the server limit of 16 MiB. | @@ -533,14 +533,14 @@ sim workflows run [options] | `--input ` | No | Trigger input as JSON (JSON, or @path / @- to read a file or stdin). | | `--async` | No | Queue the run and return immediately. | | `--execution-timeout-seconds ` | No | Maximum duration of an asynchronous run, in seconds, capped by the plan's execution timeout. Requires `async: true`; otherwise returns `400`. | -| `--select-output ` | No | Return streamed outputs as blockName.path or childWorkflowId.blockName.path; selecting a child workflow applies to every invocation, requires --follow (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | +| `--select-output ` | No | Return blockName.path values (e.g. agent_1.content), or childWorkflowId.blockName.path for a child workflow (applies to every invocation) — in blockOutputs on a sync run, or from the streamed result with --follow; missing paths are omitted. Not available with --async (space-separated, or @path / @- with one value per line; @@value for a literal leading @). | | `--include-file-base64` | No | Inline eligible output files as base64 content. Rejected when `async` is true. | | `--no-include-file-base64` | No | Send --include-file-base64 as false. | | `--base64-max-bytes ` | No | Maximum total bytes of file content to inline as base64, lowering but never raising the server limit of 16 MiB. Rejected when `async` is true. | | `--run-id ` | No | One-shot identifier for this run; NOT an idempotency key — reusing a claimed value fails with RUN_ID_CONFLICT instead of replaying the first result, and a fresh value starts another run. | | `--manual` | No | Run the current saved workflow state instead of the active deployment. | -| `--trigger ` | No | Enter a manual run through this runnable trigger (requires --manual). | -| `--mock-payload` | No | Use the selected trigger's server-derived mock payload (requires --manual). | +| `--trigger ` | No | Enter the run through this runnable trigger; runs the current saved workflow state (implies --manual). | +| `--mock-payload` | No | Use the selected trigger's server-derived mock payload; runs the current saved workflow state (implies --manual). | | `--from-block ` | No | Run manually from this saved workflow block. | | `--source-run ` | No | Prior run whose persisted state supplies upstream outputs (requires --from-block). | | `--follow` | No | Stream the run as it happens; progress on stderr, result on stdout. The stream reports only success and output, so the result omits the run id and timings a non-streaming run returns. | @@ -572,6 +572,8 @@ sim workflows export [options] | Option | Required | Description | | --- | --- | --- | | `--include-references` | No | Include non-secret resource identities for mapped import. | +| `--include-workspace-bindings` | No | Whether to keep workspace-scoped bindings — table, knowledge base, document, folder, channel, and other resource selectors — in the exported state. Defaults to false, the sharing-safe export in which those ids are cleared because they resolve nowhere else. Send true for a same-workspace round trip so the re-imported workflow can run without re-selecting them. Credentials, passwords, and table sub-block values are cleared either way. | +| `--no-include-workspace-bindings` | No | Send --include-workspace-bindings as false. | @@ -675,7 +677,7 @@ Replace Workflow State (OAuth login or personal API key required) | Option | Required | Description | | --- | --- | --- | -| `--dry-run` | No | Validate and lint without persisting. The response is identical to the committed write of the same body, so a caller can inspect `lint` and then re-send the request for real. Nothing is written, no audit entry is recorded, and collaborators are not notified. | +| `--dry-run` | No | Validate and lint without writing, auditing, or notifying collaborators. Returns the same validation, preparation warnings, lint findings, and ID-ownership conflicts (`409`) as a committed write. `needsRedeployment` describes the pre-write state. For semantic operations, `mintedBlockIds` is empty; `previewBlockIds` contains provisional IDs with a warning, since committing mints new IDs. | | `--no-dry-run` | No | Send --dry-run as false. | | `--blocks ` | Yes | Blocks keyed by block id. (JSON, or @path / @- to read a file or stdin). | | `--edges ` | Yes | Directed connections between blocks. (JSON, or @path / @- to read a file or stdin). | diff --git a/apps/docs/content/docs/files/editor.mdx b/apps/docs/content/docs/files/editor.mdx index 046ed565cac..b40166a2c80 100644 --- a/apps/docs/content/docs/files/editor.mdx +++ b/apps/docs/content/docs/files/editor.mdx @@ -46,7 +46,7 @@ Type `/` anywhere to insert any block — heading, list, table, code block, imag The editor round-trips your markdown exactly — it saves what you wrote, with no reformatting churn. -A few constructs can't be represented visually without losing information on save — footnotes, raw HTML, and HTML comments. When a file contains one of these, it opens **read-only** so the original source is preserved untouched. Everything is still rendered faithfully; you just can't edit that file inline. +A few constructs still can't round-trip: a `
` inside a table cell, a hard line break inside a heading, and non-canonical HTML entities such as `©`. Files over 256 KB also open read-only. Footnotes, HTML comments, and raw HTML are preserved byte-for-byte and stay editable. diff --git a/apps/docs/content/docs/files/generating.mdx b/apps/docs/content/docs/files/generating.mdx index 6c2f1a4db77..7ad2759531d 100644 --- a/apps/docs/content/docs/files/generating.mdx +++ b/apps/docs/content/docs/files/generating.mdx @@ -4,6 +4,7 @@ description: How a workflow produces a document, report, or media file and saves --- import { Callout } from 'fumadocs-ui/components/callout' +import { Card, Cards } from 'fumadocs-ui/components/card' A generated file is an artifact a workflow run creates: a report, a CSV, a rendered audio clip. It starts as a value a block produces and becomes a workspace file when a [File](/integrations/file) block writes it to the [Files](/files) store. Once saved, it has a name, a size, and a URL, and any later run can read it back. @@ -54,3 +55,12 @@ During a run, the file also appears in the output panel as the File block's outp ## Returning a generated file from a deployment When a workflow is deployed as an [API](/workflows/deployment/api), a generated file can be part of the response. Reference the file in a [Response](/workflows/blocks/response) block, or include its ID in the object you return. The caller uses the `url` or `id` to fetch the file from the workspace store. The file itself stays in the Files store, and the response carries a pointer to it, not the bytes. + +## Next + + + + + + + diff --git a/apps/docs/content/docs/getting-started/index.mdx b/apps/docs/content/docs/getting-started/index.mdx index 18abe1ab3dd..99737fa5f73 100644 --- a/apps/docs/content/docs/getting-started/index.mdx +++ b/apps/docs/content/docs/getting-started/index.mdx @@ -28,7 +28,7 @@ Build a people research agent in 10 minutes. It takes a name through a chat inte - **System**: "You are a people research agent. When given a person's name, use your search tools to find their location, profession, educational background, and other relevant details." - **User**: insert `` so the agent reads whatever the chat receives. - Leave the **Model** on the default (`claude-sonnet-4-6`), or pick any other. + Leave the **Model** on the default (`claude-sonnet-5`), or pick any other.