diff --git a/.claude/skills/playwright-roll/SKILL.md b/.claude/skills/playwright-roll/SKILL.md index 3295bafda..30a078c08 100644 --- a/.claude/skills/playwright-roll/SKILL.md +++ b/.claude/skills/playwright-roll/SKILL.md @@ -162,7 +162,7 @@ Implement the change in `playwright/_impl/.py`. Use the upstream JS impl | `(await this._channel.foo()).value` | `await self._channel.send("foo", None)` (Python's `send()` auto-unwraps single-key responses; only call `send_return_as_dict` when the protocol returns multiple keys.) | | `(await this._channel.foo()).artifact` (multi-key, may be empty) | `result = await self._channel.send_return_as_dict("foo", None); (result or {}).get("artifact")` — `send_return_as_dict` returns **`None`** (not `{}`) when the protocol response carries no fields. | | `try { ... } catch (e) { if (isTargetClosedError(e)) return; throw e; }` | `try: ...; except Exception as e: if is_target_closed_error(e): return; raise` (import from `playwright._impl._errors`) | -| Inline `[Object]` return like `{endpoint: string}` | A `TypedDict` in `playwright/_impl/_api_structures.py`. The generator names the struct from its docs `* alias:` (or a `structRenames` / `structNamesByPlace` entry in `python_rules.js`) and imports that name from `_api_structures.py`. See `RemoteAddr`, `BrowserBindResult`, `DebuggerPausedDetails`. | +| Inline `[Object]` return like `{endpoint: string}` | A `TypedDict` in `playwright/_impl/_api_structures.py`, which the generator writes. It names the struct from its docs `* alias:` (or a `structRenames` / `structNamesByPlace` entry in `python_rules.js`). See `RemoteAddr`, `BrowserBindResult`, `DebuggerPausedDetails`. | | `binary` event/return field | The Python channel layer hands you a base64 string. Decode with `base64.b64decode(value)` before exposing as `bytes`. See `Screencast._dispatch_frame`. | When implementing a new ChannelOwner subclass (one constructed by the protocol with `(parent, type, guid, initializer)`): @@ -172,8 +172,9 @@ When implementing a new ChannelOwner subclass (one constructed by the protocol w When implementing a non-ChannelOwner wrapper class (a plain class that holds a Page/Context reference, like `Screencast`, `Clock`): - Set `self._loop = parent._loop` and `self._dispatcher_fiber = parent._dispatcher_fiber` in `__init__`. The generated `AsyncBase`/`SyncBase` wrappers read these; missing them gives `AttributeError: 'X' object has no attribute '_loop'` at first use. -When adding a new TypedDict in `_api_structures.py`: -- Its name must match what the generator emits: the struct's `* alias:` in the docs, or a `structRenames` / `structNamesByPlace` entry in `python_rules.js`. The generator fails with "No Python name for the struct at …" until one exists. +When the docs add a new struct (an inline `[Object]` with named fields): +- The generator writes its TypedDict into `_api_structures.py`. Never edit that file by hand; internal types go in `_internal_structures.py`. +- It needs a Python name: the struct's `* alias:` in the docs, or a `structRenames` / `structNamesByPlace` entry in `python_rules.js`. The generator fails with "No Python name for the struct at …" until one exists. Nested structs need names too. - Re-export it from both `playwright/async_api/__init__.py` and `playwright/sync_api/__init__.py`: assignment line plus an entry in `__all__`. Same pattern as `ViewportSize`, `RemoteAddr`. If the new API was previously recorded in `python_rules.js` (for example in `skipArgs`), **remove that entry** when implementing it. The generator fails on entries that no longer match anything, so stale entries cannot pile up. @@ -203,7 +204,7 @@ The generator fails with "the entry … no longer matches anything; remove it" w - "Internal tooling" classes/methods whose `langs` field is empty (`Screencast.*`, `Debugger.*`, `Page.pick_locator`, `BrowserContext.debugger`, `Browser.bind/unbind`, `Page.{clear,console}_*`). The 1.59 roll suppressed all of these initially, then had to walk every one back. Verify `langs` first. - A renamed positional parameter (mypy: unexpected keyword argument `newName` for the impl method). Just rename in `_impl/` and update the channel-send dict key. -- A struct return the generator cannot name ("No Python name for the struct at …"). Add a TypedDict and name it — don't type it `Dict`. +- A struct return the generator cannot name ("No Python name for the struct at …"). Name it, so that the generator writes a TypedDict — don't type it `Dict`. #### N/A @@ -247,11 +248,12 @@ Failure modes and fixes: | mypy: `"XImpl" has no attribute "y"` in `_generated.py` | The docs have a new member `X.y`; `_impl/` does not. | PORT it. | | mypy: `Unexpected keyword argument "z" for "y" of "XImpl"` | New parameter or option on an existing method. | Add the kwarg (docs camelCase name) in `_impl/`. If Python must not have it, `skipArgs`. | | mypy: incompatible argument type in `_generated.py` | The docs type changed, or `_impl/` declares a narrower type. | Match the type in `_impl/`, or record a `typeOverrides` entry for a justified divergence. | -| generator: `No Python name for the struct at X.y.z` | A new inline `[Object]` with no `* alias:` in the docs. | Add the TypedDict and name it: `alias-python` upstream, or `structNamesByPlace` / `structRenames`. | +| generator: `No Python name for the struct at X.y.z` | A new inline `[Object]` with no `* alias:` in the docs. | Name it: `alias-python` upstream, or `structNamesByPlace` / `structRenames`. | +| generator: `The struct X has different fields at A and B` | Two places in the docs share one struct name, but their fields differ. | Give one place its own name in `structNamesByPlace`, or fix the docs. | | generator: `Unknown docs type "T" at X.y` | A docs type the type table in `generate_api.js` (`toPy`) does not know. | Add it to `toPy`. | | generator: `required parameter z follows an optional one` | The docs order params so that Python would need a default before a non-default. | Add a `positional` entry with the Python order. | | generator: `the entry "…" no longer matches anything; remove it` | An upstream change made a rules entry unnecessary (or renamed what it points at). | Delete it, or re-key it to the new docs name. | -| generator: `legacy_order.json: … has no …; remove it` | A member or parameter in the temporary legacy order was removed or renamed upstream. | Delete that name from `scripts/codegen/legacy_order.json`. New members need no entry; they go last. | +| generator: `legacy_order.json: … has no …; remove it` | A member, parameter or TypedDict in the temporary legacy order was removed or renamed upstream. | Delete that name from `scripts/codegen/legacy_order.json`. New members need no entry; they go last. | | `pyright … reportInconsistentOverload` (single-event class) | A class gained its first event, so there is one `Literal[…]` overload + impl, which pyright wants ≥2 of. | The generator already handles this — `emitEvents` adds a second `@typing.overload` with `event: str` when a class has exactly one event. | | `pre-commit` keeps reformatting `_generated.py` on each run | First run after regen always reformats once; rerun until idle. | `pre-commit run --all-files` to settle. | | A kwarg in `_impl/` that the docs don't mention (e.g. legacy name from a doc rename) | Nothing fails: the generator simply never passes it. | If the param is positional with no default, just rename it in `_impl/`. Check `packages/protocol/spec/*.yml` and the server dispatcher — if the wire field renamed too, also update the channel-send dict key. Only use `extraOptions` for genuinely hidden internal kwargs (`default_browser_type`). | @@ -303,6 +305,7 @@ All keys use docs names: `Class.member` or `Class.member.arg`, with the upstream | `positional` | Methods whose positional parameters differ from "params positional, options keyword-only". | | `positionalOptions` | Option names that are positional everywhere (`predicate`). | | `structRenames`, `structNamesByPlace` | TypedDict names. | +| `structTotal` | `total=` of a TypedDict, where it differs from "total=False if any field is optional". | | `implModules` | `_impl/` module of classes not in `_.py`. | ## Tips & gotchas @@ -313,7 +316,7 @@ All keys use docs names: `Class.member` or `Class.member.arg`, with the upstream - **Watch for revert pairs in the same range.** 1.59 added and reverted `Browser.isRemote` (#39613 / #39620) inside the same release. Walking chronologically lets you skip the add when you see the revert later. - **Watch for rename-revert pairs.** 1.59 had `Locator.normalize` → `Locator.toCode` (#39648) → `Locator.normalize` (#39754). Final state wins; only port the last. - **Doc renames almost always include a wire-protocol rename.** Whenever you see `### param: X.y.oldName` → `### param: X.y.newName` in a doc commit, also `git -C "$PW_SRC_DIR" show -- packages/protocol/spec/` and the corresponding `*Dispatcher.ts` file. If the wire field changed too, the channel-send dict key in `_impl/` must change. Suppressing the doc-side mismatch is hiding a real bug — the previous Python code is silently sending an unknown field that the new server ignores. -- **TypedDicts beat `Dict[str, X]` for any structured return.** When the docs describe a return as `[Object]` with named fields, define a `TypedDict` in `_api_structures.py`, name it for the generator, and re-export it from both public `__init__.py` files. Zero runtime cost (it's still a `dict`), and the docstrings expand its fields from the docs. +- **TypedDicts beat `Dict[str, X]` for any structured return.** When the docs describe a return as `[Object]` with named fields, name the struct for the generator, which writes its `TypedDict` into `_api_structures.py`, and re-export it from both public `__init__.py` files. Zero runtime cost (it's still a `dict`), and the docstrings expand its fields from the docs. - **Positional renames are free.** A param with no default before any `*` separator is positional-or-keyword in Python, but realistic call sites pass it positionally. Renaming such a param doesn't break callers. - **The "Backport changes" GitHub issue can be misleading.** In the 1.59 roll its checkboxes were all marked `[x]` with annotations like "✅ IMPLEMENTED", but several of those features had not actually been merged into the Python port. Trust the `docs/src/api/` walk over the issue. - **The docs parser has quirks.** In a type expression, everything after a function's `:` is its return type, so `[function]([A]):[any]|[B]` is one function returning `any|B`. Put a function with a return type last in a union. The parser also drops the second argument of a function type when the first is a union (see the `WebSocketRoute.onClose.handler` override, which builds the type by hand). And when a `### param:` header name differs from the backticked name on its type line, the header name is the docs name and the type-line name becomes the alias (see `argNames`). diff --git a/CLAUDE.md b/CLAUDE.md index 0ae206b2b..bbbc9269e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,11 +8,11 @@ Python bindings for [Playwright](https://playwright.dev). The Python client talk ## Layout -- `playwright/_impl/` — hand-written client implementation (one module per object: `_browser.py`, `_page.py`, `_locator.py`, `_network.py`, etc.). Edit these to add or change behavior. -- `playwright/async_api/_generated.py`, `playwright/sync_api/_generated.py` — **auto-generated** from the upstream API docs. Never edit by hand; rerun `./scripts/update_api.sh` after a roll or after changing `scripts/codegen/`. -- `scripts/codegen/generate_api.js` — the generator. A Node.js script that loads the upstream doc parser (`utils/doclint/api_parser.js`) from `$PW_SRC_DIR`, filters the docs for Python, and writes the whole public API: signatures, types, docstrings, event overloads, and the calls into `_impl/`. It does not read `_impl/`. -- `scripts/codegen/python_rules.js` — the Python facts that the docs do not contain (positional parameters, struct names, type overrides, hidden members). Policy: the shipped Python API wins; each entry is a candidate for an upstream docs fix, and the generator fails on entries that no longer match anything. Every entry needs a one-line rationale comment above it. -- `scripts/codegen/legacy_order.json` — **temporary**. The old generator's order of classes, members, keyword-only parameters and union members. The generator applies it, and more old habits marked `TEMPORARY` in the generator, so that the switch to the docs generator has a small diff. New members go last. If the generator reports an entry that no longer matches, remove the entry. A follow-up change deletes the file and the `TEMPORARY` code. +- `playwright/_impl/` — hand-written client implementation (one module per object: `_browser.py`, `_page.py`, `_locator.py`, `_network.py`, etc.). Edit these to add or change behavior. Internal types are in `_internal_structures.py`. +- `playwright/async_api/_generated.py`, `playwright/sync_api/_generated.py`, and the public TypedDicts in `playwright/_impl/_api_structures.py` — **auto-generated** from the upstream API docs. Never edit by hand; rerun `./scripts/update_api.sh` after a roll or after changing `scripts/codegen/`. +- `scripts/codegen/generate_api.js` — the generator. A Node.js script that loads the upstream doc parser (`utils/doclint/api_parser.js`) from `$PW_SRC_DIR`, filters the docs for Python, and writes the whole public API: signatures, types, docstrings, event overloads, the calls into `_impl/`, and the TypedDicts. It does not read `_impl/`. +- `scripts/codegen/python_rules.js` — the Python facts that the docs do not contain (positional parameters, struct names and `total=`, type overrides, hidden members). Policy: the shipped Python API wins; each entry is a candidate for an upstream docs fix, and the generator fails on entries that no longer match anything. Every entry needs a one-line rationale comment above it. +- `scripts/codegen/legacy_order.json` — **temporary**. The old generator's order of classes, members, keyword-only parameters and union members, and the hand-written order of the TypedDicts. The generator applies it, and more old habits marked `TEMPORARY` in the generator, so that the switch to the docs generator has a small diff. New members go last. If the generator reports an entry that no longer matches, remove the entry. A follow-up change deletes the file and the `TEMPORARY` code. - `scripts/codegen/compare_api.py` — compares two generated files (signatures, annotations, call bodies, docstrings) and lists every difference. Use it to check that a generator change does not change the API. - `tests/async/`, `tests/sync/` — pytest suites. Most new tests are added to the async file with a sync mirror. - `DRIVER_VERSION` — the single source of truth for which Playwright release the driver is assembled from (one line, the `playwright-core` npm version, e.g. `1.61.0`, no `v` prefix). Read by `setup.py`, `scripts/build_driver.py`, and CI. The wheel build downloads `playwright-core` at this version from npm plus the matching Node.js binary and assembles the per-platform bundles — no source build. The version is baked into the staged bundle filenames (`driver/playwright--.zip`), so it doubles as the build cache key. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dda1526f9..49cf6f1b7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -75,9 +75,10 @@ For more details look at the [CI configuration](./.github/workflows/ci.yml). ### Regenerating APIs -`update_api.sh` generates `playwright/{async,sync}_api/_generated.py` from the -API docs in a nearby `microsoft/playwright` checkout (at the release branch -matching `DRIVER_VERSION`); point `PW_SRC_DIR` at it. The generator is +`update_api.sh` generates `playwright/{async,sync}_api/_generated.py` and the +public TypedDicts in `playwright/_impl/_api_structures.py` from the API docs in +a nearby `microsoft/playwright` checkout (at the release branch matching +`DRIVER_VERSION`); point `PW_SRC_DIR` at it. The generator is `scripts/codegen/generate_api.js`; Python facts that the docs do not contain live in `scripts/codegen/python_rules.js`. diff --git a/playwright/_impl/_api_structures.py b/playwright/_impl/_api_structures.py index 0de5527ae..de8488049 100644 --- a/playwright/_impl/_api_structures.py +++ b/playwright/_impl/_api_structures.py @@ -12,16 +12,11 @@ # See the License for the specific language governing permissions and # limitations under the License. +# Generated from the docs by scripts/codegen/generate_api.js. Do not edit. + from pathlib import Path from typing import Any, Dict, List, Literal, Optional, Sequence, TypedDict, Union -# These are the structures that we like keeping in a JSON form for their potential -# reuse between SDKs / services. They are public and are a part of the -# stable API. - -# Explicitly mark optional params as such for the documentation -# If there is at least one optional param, set total=False for better mypy handling. - class Cookie(TypedDict, total=False): name: str @@ -46,7 +41,6 @@ class StorageStateCookie(TypedDict, total=False): sameSite: Literal["Lax", "None", "Strict"] -# TODO: We are waiting for PEP705 so SetCookieParam can be readonly and matches Cookie. class SetCookieParam(TypedDict, total=False): name: str value: str @@ -171,7 +165,7 @@ class FilePayload(TypedDict): class DropPayload(TypedDict, total=False): files: Optional[ - Union[str, Path, FilePayload, Sequence[Union[str, Path]], Sequence[FilePayload]] + Union[str, Path, Sequence[Union[str, Path]], FilePayload, Sequence[FilePayload]] ] data: Optional[Dict[str, str]] @@ -198,136 +192,6 @@ class NameValue(TypedDict): value: str -HeadersArray = List[NameValue] -Headers = Dict[str, str] - - -class ServerFilePayload(TypedDict): - name: str - mimeType: str - buffer: str - - -class FormField(TypedDict, total=False): - name: str - value: Optional[str] - file: Optional[ServerFilePayload] - - -class ExpectedTextValue(TypedDict, total=False): - string: str - regexSource: str - regexFlags: str - matchSubstring: bool - normalizeWhiteSpace: bool - ignoreCase: Optional[bool] - - -class FrameExpectOptions(TypedDict, total=False): - expressionArg: Any - expectedText: Optional[Sequence[ExpectedTextValue]] - expectedNumber: Optional[float] - expectedValue: Optional[Any] - useInnerText: Optional[bool] - isNot: bool - timeout: Optional[float] - pseudo: Optional[str] - - -class FrameExpectResult(TypedDict, total=False): - matches: bool - received: Any - log: Optional[List[str]] - errorMessage: Optional[str] - timedOut: Optional[bool] - - -AriaRole = Literal[ - "alert", - "alertdialog", - "application", - "article", - "banner", - "blockquote", - "button", - "caption", - "cell", - "checkbox", - "code", - "columnheader", - "combobox", - "complementary", - "contentinfo", - "definition", - "deletion", - "dialog", - "directory", - "document", - "emphasis", - "feed", - "figure", - "form", - "generic", - "grid", - "gridcell", - "group", - "heading", - "img", - "insertion", - "link", - "list", - "listbox", - "listitem", - "log", - "main", - "marquee", - "math", - "menu", - "menubar", - "menuitem", - "menuitemcheckbox", - "menuitemradio", - "meter", - "navigation", - "none", - "note", - "option", - "paragraph", - "presentation", - "progressbar", - "radio", - "radiogroup", - "region", - "row", - "rowgroup", - "rowheader", - "scrollbar", - "search", - "searchbox", - "separator", - "slider", - "spinbutton", - "status", - "strong", - "subscript", - "superscript", - "switch", - "tab", - "table", - "tablist", - "tabpanel", - "term", - "textbox", - "time", - "timer", - "toolbar", - "tooltip", - "tree", - "treegrid", - "treeitem", -] - - class TracingGroupLocation(TypedDict): file: str line: Optional[int] diff --git a/playwright/_impl/_assertions.py b/playwright/_impl/_assertions.py index 706fd6c26..381d4f6fd 100644 --- a/playwright/_impl/_assertions.py +++ b/playwright/_impl/_assertions.py @@ -17,16 +17,16 @@ from typing import Any, Iterator, List, Literal, Optional, Pattern, Sequence, Union from urllib.parse import urljoin -from playwright._impl._api_structures import ( +from playwright._impl._connection import format_call_log +from playwright._impl._errors import Error +from playwright._impl._fetch import APIResponse +from playwright._impl._helper import is_textual_mime_type +from playwright._impl._internal_structures import ( AriaRole, ExpectedTextValue, FrameExpectOptions, FrameExpectResult, ) -from playwright._impl._connection import format_call_log -from playwright._impl._errors import Error -from playwright._impl._fetch import APIResponse -from playwright._impl._helper import is_textual_mime_type from playwright._impl._js_handle import parse_value from playwright._impl._locator import Locator from playwright._impl._page import Page diff --git a/playwright/_impl/_fetch.py b/playwright/_impl/_fetch.py index be8f783df..faa247003 100644 --- a/playwright/_impl/_fetch.py +++ b/playwright/_impl/_fetch.py @@ -25,14 +25,11 @@ ClientCertificate, Cookie, FilePayload, - FormField, - Headers, HttpCredentials, ProxySettings, RemoteAddr, ResourceTiming, SecurityDetails, - ServerFilePayload, SetCookieParam, StorageState, ) @@ -51,6 +48,7 @@ object_to_array, to_impl, ) +from playwright._impl._internal_structures import FormField, Headers, ServerFilePayload from playwright._impl._network import ( serialize_headers, to_clear_cookies_params, diff --git a/playwright/_impl/_frame.py b/playwright/_impl/_frame.py index 501bb6bd7..c0e781558 100644 --- a/playwright/_impl/_frame.py +++ b/playwright/_impl/_frame.py @@ -30,14 +30,7 @@ from pyee import EventEmitter -from playwright._impl._api_structures import ( - AriaRole, - DropPayload, - FilePayload, - FrameExpectOptions, - FrameExpectResult, - Position, -) +from playwright._impl._api_structures import DropPayload, FilePayload, Position from playwright._impl._connection import ( ChannelOwner, from_channel, @@ -58,6 +51,11 @@ monotonic_time, url_matches, ) +from playwright._impl._internal_structures import ( + AriaRole, + FrameExpectOptions, + FrameExpectResult, +) from playwright._impl._js_handle import ( JSHandle, Serializable, diff --git a/playwright/_impl/_har_router.py b/playwright/_impl/_har_router.py index a9729bbb2..9b92f706c 100644 --- a/playwright/_impl/_har_router.py +++ b/playwright/_impl/_har_router.py @@ -15,12 +15,12 @@ import base64 from typing import TYPE_CHECKING, Optional, cast -from playwright._impl._api_structures import HeadersArray from playwright._impl._helper import ( HarLookupResult, RouteFromHarNotFoundPolicy, URLMatch, ) +from playwright._impl._internal_structures import HeadersArray from playwright._impl._local_utils import LocalUtils if TYPE_CHECKING: # pragma: no cover diff --git a/playwright/_impl/_helper.py b/playwright/_impl/_helper.py index b3e6820a6..4f979378f 100644 --- a/playwright/_impl/_helper.py +++ b/playwright/_impl/_helper.py @@ -53,7 +53,7 @@ from playwright._impl._str_utils import escape_regex_flags if TYPE_CHECKING: # pragma: no cover - from playwright._impl._api_structures import HeadersArray + from playwright._impl._internal_structures import HeadersArray from playwright._impl._network import Request, Response, Route, WebSocketRoute URLMatch = Union[str, Pattern[str], Callable[[str], bool]] diff --git a/playwright/_impl/_internal_structures.py b/playwright/_impl/_internal_structures.py new file mode 100644 index 000000000..1b34754f8 --- /dev/null +++ b/playwright/_impl/_internal_structures.py @@ -0,0 +1,149 @@ +# Copyright (c) Microsoft Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Types that the implementation uses internally. The public TypedDicts are +# generated into _api_structures.py. + +from typing import Any, Dict, List, Literal, Optional, Sequence, TypedDict + +from playwright._impl._api_structures import NameValue + +HeadersArray = List[NameValue] +Headers = Dict[str, str] + + +class ServerFilePayload(TypedDict): + name: str + mimeType: str + buffer: str + + +class FormField(TypedDict, total=False): + name: str + value: Optional[str] + file: Optional[ServerFilePayload] + + +class ExpectedTextValue(TypedDict, total=False): + string: str + regexSource: str + regexFlags: str + matchSubstring: bool + normalizeWhiteSpace: bool + ignoreCase: Optional[bool] + + +class FrameExpectOptions(TypedDict, total=False): + expressionArg: Any + expectedText: Optional[Sequence[ExpectedTextValue]] + expectedNumber: Optional[float] + expectedValue: Optional[Any] + useInnerText: Optional[bool] + isNot: bool + timeout: Optional[float] + pseudo: Optional[str] + + +class FrameExpectResult(TypedDict, total=False): + matches: bool + received: Any + log: Optional[List[str]] + errorMessage: Optional[str] + timedOut: Optional[bool] + + +AriaRole = Literal[ + "alert", + "alertdialog", + "application", + "article", + "banner", + "blockquote", + "button", + "caption", + "cell", + "checkbox", + "code", + "columnheader", + "combobox", + "complementary", + "contentinfo", + "definition", + "deletion", + "dialog", + "directory", + "document", + "emphasis", + "feed", + "figure", + "form", + "generic", + "grid", + "gridcell", + "group", + "heading", + "img", + "insertion", + "link", + "list", + "listbox", + "listitem", + "log", + "main", + "marquee", + "math", + "menu", + "menubar", + "menuitem", + "menuitemcheckbox", + "menuitemradio", + "meter", + "navigation", + "none", + "note", + "option", + "paragraph", + "presentation", + "progressbar", + "radio", + "radiogroup", + "region", + "row", + "rowgroup", + "rowheader", + "scrollbar", + "search", + "searchbox", + "separator", + "slider", + "spinbutton", + "status", + "strong", + "subscript", + "superscript", + "switch", + "tab", + "table", + "tablist", + "tabpanel", + "term", + "textbox", + "time", + "timer", + "toolbar", + "tooltip", + "tree", + "treegrid", + "treeitem", +] diff --git a/playwright/_impl/_local_utils.py b/playwright/_impl/_local_utils.py index da4b50103..7813c5279 100644 --- a/playwright/_impl/_local_utils.py +++ b/playwright/_impl/_local_utils.py @@ -15,9 +15,9 @@ import base64 from typing import Dict, List, Optional, cast -from playwright._impl._api_structures import HeadersArray from playwright._impl._connection import ChannelOwner, StackFrame from playwright._impl._helper import HarLookupResult, locals_to_params +from playwright._impl._internal_structures import HeadersArray class LocalUtils(ChannelOwner): diff --git a/playwright/_impl/_locator.py b/playwright/_impl/_locator.py index fb4ed49cb..5e3df1bcc 100644 --- a/playwright/_impl/_locator.py +++ b/playwright/_impl/_locator.py @@ -32,12 +32,9 @@ ) from playwright._impl._api_structures import ( - AriaRole, DropPayload, FilePayload, FloatRect, - FrameExpectOptions, - FrameExpectResult, Position, ) from playwright._impl._element_handle import ElementHandle @@ -49,6 +46,11 @@ monotonic_time, to_impl, ) +from playwright._impl._internal_structures import ( + AriaRole, + FrameExpectOptions, + FrameExpectResult, +) from playwright._impl._js_handle import Serializable, serialize_argument from playwright._impl._str_utils import ( escape_for_attribute_selector, diff --git a/playwright/_impl/_network.py b/playwright/_impl/_network.py index 3e5b8fe89..da470e3af 100644 --- a/playwright/_impl/_network.py +++ b/playwright/_impl/_network.py @@ -39,8 +39,6 @@ from playwright._impl._api_structures import ( ClientCertificate, - Headers, - HeadersArray, HttpCredentials, RemoteAddr, RequestSizes, @@ -62,6 +60,7 @@ locals_to_params, url_matches, ) +from playwright._impl._internal_structures import Headers, HeadersArray from playwright._impl._str_utils import escape_regex_flags from playwright._impl._waiter import Waiter diff --git a/playwright/_impl/_page.py b/playwright/_impl/_page.py index ce7e9ff21..589bc3539 100644 --- a/playwright/_impl/_page.py +++ b/playwright/_impl/_page.py @@ -35,7 +35,6 @@ ) from playwright._impl._api_structures import ( - AriaRole, FilePayload, FloatRect, PdfMargins, @@ -85,6 +84,7 @@ url_matches, ) from playwright._impl._input import Keyboard, Mouse, Touchscreen +from playwright._impl._internal_structures import AriaRole from playwright._impl._js_handle import ( JSHandle, Serializable, diff --git a/scripts/codegen/compare_api.py b/scripts/codegen/compare_api.py index ea818bd71..71d2a9216 100644 --- a/scripts/codegen/compare_api.py +++ b/scripts/codegen/compare_api.py @@ -27,7 +27,10 @@ (unions are flattened and sorted, Optional[X] is Union[X, None], quotes are removed, Literal values are sorted and merged); - the body: the call into the implementation, with keyword arguments sorted; -- docstrings (reported separately; they are allowed to differ). +- docstrings (reported separately; they are allowed to differ); +- for TypedDicts (`playwright/_impl/_api_structures.py`): `total=`, the field + names and the field annotations (the field order is reported separately, + because it is allowed to differ). Exit code is 1 if any difference outside the allowed kinds is found. """ @@ -40,7 +43,7 @@ from dataclasses import dataclass, field from typing import Dict, List, Optional, Set, Tuple -ALLOWED_KINDS = {"kwonly-order", "docstring", "member-inherited"} +ALLOWED_KINDS = {"kwonly-order", "docstring", "member-inherited", "field-order"} @dataclass @@ -66,7 +69,9 @@ class Func: class Klass: name: str bases: List[str] + keywords: List[str] = field(default_factory=list) members: Dict[str, Func] = field(default_factory=dict) + fields: Dict[str, str] = field(default_factory=dict) def _strip_typing(name: str) -> str: @@ -233,8 +238,15 @@ def load(path: str) -> Dict[str, Klass]: for node in module.body: if not isinstance(node, ast.ClassDef): continue - klass = Klass(node.name, [ast.unparse(b) for b in node.bases]) + klass = Klass( + node.name, + [ast.unparse(b) for b in node.bases], + [ast.unparse(k) for k in node.keywords], + ) for item in node.body: + if isinstance(item, ast.AnnAssign) and isinstance(item.target, ast.Name): + klass.fields[item.target.id] = canon(item.annotation) + continue if not isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)): continue decorators = [ast.unparse(d) for d in item.decorator_list] @@ -364,6 +376,29 @@ def _inherited(classes: Dict[str, Klass], klass: Klass, member: str) -> Optional return None +def compare_fields( + where: str, old: Dict[str, str], new: Dict[str, str], out: List[Difference] +) -> None: + for name in old: + if name not in new: + out.append(Difference("field-missing", f"{where}.{name}", old[name], "-")) + elif old[name] != new[name]: + out.append( + Difference("field-type", f"{where}.{name}", old[name], new[name]) + ) + for name in new: + if name not in old: + out.append(Difference("field-extra", f"{where}.{name}", "-", new[name])) + common_old = [n for n in old if n in new] + common_new = [n for n in new if n in old] + if common_old != common_new: + out.append( + Difference( + "field-order", where, ", ".join(common_old), ", ".join(common_new) + ) + ) + + def compare(old: Dict[str, Klass], new: Dict[str, Klass]) -> List[Difference]: out: List[Difference] = [] for name in sorted(set(old) - set(new)): @@ -374,6 +409,9 @@ def compare(old: Dict[str, Klass], new: Dict[str, Klass]) -> List[Difference]: o, n = old[name], new[name] if o.bases != n.bases: out.append(Difference("bases", name, str(o.bases), str(n.bases))) + if o.keywords != n.keywords: + out.append(Difference("keywords", name, str(o.keywords), str(n.keywords))) + compare_fields(name, o.fields, n.fields, out) for member in sorted(set(o.members) - set(n.members)): inherited = _inherited(new, n, member) if inherited: diff --git a/scripts/codegen/generate_api.js b/scripts/codegen/generate_api.js index e6216b8cb..d1b67a833 100644 --- a/scripts/codegen/generate_api.js +++ b/scripts/codegen/generate_api.js @@ -18,10 +18,11 @@ // @ts-check // Generates the public Python API (playwright/{async,sync}_api/_generated.py) -// from the upstream Playwright docs (docs/src/api in microsoft/playwright). +// and its TypedDicts (playwright/_impl/_api_structures.py) from the upstream +// Playwright docs (docs/src/api in microsoft/playwright). // // Usage: -// PW_SRC_DIR= node scripts/codegen/generate_api.js --flavor async|sync [--out ] +// PW_SRC_DIR= node scripts/codegen/generate_api.js --flavor async|sync|structs [--out ] // // The docs are parsed with the upstream parser (utils/doclint/api_parser.js) and // filtered for Python. Facts that the docs do not contain come from @@ -102,8 +103,8 @@ function parseArgs(argv) { else fail(`Unknown argument: ${argv[i]}`); } - if (!['async', 'sync'].includes(result.flavor)) - fail(`--flavor must be "async" or "sync", got "${result.flavor}"`); + if (!['async', 'sync', 'structs'].includes(result.flavor)) + fail(`--flavor must be "async", "sync" or "structs", got "${result.flavor}"`); return result; } @@ -133,6 +134,8 @@ const errors = []; const usedRules = new Set(); /** @type {Set} */ const usedStructs = new Set(); +/** @type {Map} every use of each struct name */ +const structUses = new Map(); // --------------------------------------------------------------------------- // Names. @@ -477,6 +480,10 @@ function structName(t) { return 'typing.Dict'; } usedStructs.add(name); + const uses = structUses.get(name) || []; + if (!uses.some(use => use.place === t.place && use.dir === t.dir)) + uses.push(t); + structUses.set(name, uses); return name; } @@ -844,6 +851,7 @@ function orderedClasses() { // parameters in the _generated.py files that the old generator wrote. That // generator read playwright/_impl, so its order followed the implementation. // The file lists only the places where that order differs from the docs order. +// It also holds the class order of the hand-written _api_structures.py. // It keeps the diff of the switch to this generator small enough to review. // // The section also reproduces two more habits of the old generator: @@ -1192,8 +1200,96 @@ function emitClass(clazz, out) { out.push(`mapping.register(${clazz.name}Impl, ${clazz.name})`); } +// --------------------------------------------------------------------------- +// The public TypedDicts (playwright/_impl/_api_structures.py). + +/** + * The fields of one use of a struct, as Python annotation text. + * @param {PyType} t + */ +function structFields(t) { + return t.docType.properties.map(p => { + const place = `${t.place}.${p.name}`; + const type = toPy(typeAt(place, p.type), { dir: t.dir || 'out', place }); + if (someType(type, c => c.kind === 'class' && !!c.wrapper)) + errors.push(`The struct field ${place} refers to a class; _api_structures.py cannot import it`); + const text = annotation(type); + const optional = !p.required; + return { name: p.name, optional, text: optional && !text.startsWith('typing.Optional[') ? `typing.Optional[${text}]` : text }; + }); +} + +/** + * Every struct that the API uses, with its fields. Nested structs are found + * while the fields are rendered. + */ +function buildStructs() { + /** @type {Map} */ + const structs = new Map(); + for (let pending = [...structUses.keys()]; pending.length; pending = [...structUses.keys()].filter(name => !structs.has(name))) { + for (const name of pending) { + const uses = /** @type {PyType[]} */ (structUses.get(name)); + // A struct used for input and output takes the output types: + // `List` rather than `Sequence`, `pathlib.Path` rather than `str | Path`. + const dir = uses.some(use => use.dir === 'out') ? 'out' : 'in'; + const fields = structFields({ ...uses[0], dir }); + const key = JSON.stringify(fields); + for (const use of uses.slice(1)) { + if (JSON.stringify(structFields({ ...use, dir })) !== key) + errors.push(`The struct ${name} has different fields at ${uses[0].place} and ${use.place}`); + } + structs.set(name, { fields }); + } + } + return structs; +} + +/** + * Struct names in alphabetical order, each after the structs that its fields + * use: the annotations are evaluated when the class is created. + * @param {Map} structs + */ +function orderedStructNames(structs) { + const result = []; + const done = new Set(); + const visit = name => { + if (done.has(name)) + return; + done.add(name); + const text = /** @type {any} */ (structs.get(name)).fields.map(f => f.text).join(' '); + for (const other of [...structs.keys()].sort()) { + if (other !== name && new RegExp(`\\b${other}\\b`).test(text)) + visit(other); + } + result.push(name); + }; + [...structs.keys()].sort().forEach(visit); + return result; +} + +function emitStructs() { + const structs = buildStructs(); + const body = []; + // TEMPORARY: sortByLegacyOrder() is the legacy order. + for (const name of sortByLegacyOrder(orderedStructNames(structs), legacyOrder?.structs, name => name, '_api_structures.py')) { + const { fields } = /** @type {any} */ (structs.get(name)); + const totalRule = lookupRule('structTotal', name); + const total = totalRule !== undefined ? totalRule : !fields.some(f => f.optional); + body.push('', '', `class ${name}(TypedDict${total ? '' : ', total=False'}):`); + for (const field of fields) + body.push(` ${field.name}: ${field.text.replace(/\btyping\./g, '').replace(/\bpathlib\.Path\b/g, 'Path')}`); + } + const text = body.join('\n'); + const typingNames = ['Any', 'Dict', 'List', 'Literal', 'Optional', 'Pattern', 'Sequence', 'TypedDict', 'Union'].filter(n => new RegExp(`\\b${n}\\b`).test(text)); + const header = [LICENSE, '# Generated from the docs by scripts/codegen/generate_api.js. Do not edit.', '']; + if (/\bPath\b/.test(text)) + header.push('from pathlib import Path'); + header.push(`from typing import ${typingNames.join(', ')}`); + return [...header, text, ''].join('\n'); +} + function checkRules() { - for (const table of ['extraMembers', 'extraOptions', 'skipArgs', 'argNames', 'typeOverrides', 'positional', 'structRenames', 'structNamesByPlace', 'implModules']) { + for (const table of ['extraMembers', 'extraOptions', 'skipArgs', 'argNames', 'typeOverrides', 'positional', 'structRenames', 'structNamesByPlace', 'structTotal', 'implModules']) { for (const key of Object.keys(/** @type {any} */ (rules)[table])) { if (!usedRules.has(`${table}:${key}`)) errors.push(`${table}: the entry "${key}" no longer matches anything; remove it`); @@ -1227,6 +1323,10 @@ function main() { else header.push('from playwright._impl._sync_base import EventContextManager, SyncBase, SyncContextManager, mapping'); + // Every flavor builds the structs, so that every run checks them. + const structsText = emitStructs(); + const text = options.flavor === 'structs' ? structsText : [...header, ...body, ''].join('\n'); + checkRules(); checkLegacyUnions(); // TEMPORARY: the legacy order. if (errors.length) { @@ -1235,7 +1335,6 @@ function main() { process.exit(1); } - const text = [...header, ...body, ''].join('\n'); if (options.out) fs.writeFileSync(options.out, text); else diff --git a/scripts/codegen/legacy_order.json b/scripts/codegen/legacy_order.json index 8dc9f1861..1a908c387 100644 --- a/scripts/codegen/legacy_order.json +++ b/scripts/codegen/legacy_order.json @@ -289,5 +289,40 @@ "LocatorAssertions.not_to_have_values.values": ["typing.Sequence[str]", "typing.Sequence[typing.Pattern[str]]", "typing.Sequence[typing.Union[typing.Pattern[str], str]]", "typing.Pattern[str]", "str"], "LocatorAssertions.to_have_text.expected": ["typing.Sequence[str]", "typing.Sequence[typing.Pattern[str]]", "typing.Sequence[typing.Union[typing.Pattern[str], str]]", "typing.Pattern[str]", "str"], "LocatorAssertions.not_to_have_text.expected": ["typing.Sequence[str]", "typing.Sequence[typing.Pattern[str]]", "typing.Sequence[typing.Union[typing.Pattern[str], str]]", "typing.Pattern[str]", "str"] - } + }, + "structs": [ + "Cookie", + "StorageStateCookie", + "SetCookieParam", + "FloatRect", + "Geolocation", + "HttpCredentials", + "LocalStorageEntry", + "OriginState", + "PdfMargins", + "Position", + "ProxySettings", + "StorageState", + "ClientCertificate", + "ResourceTiming", + "RequestSizes", + "ViewportSize", + "SourceLocation", + "WebErrorLocation", + "FilePayload", + "DropPayload", + "RemoteAddr", + "BrowserBindResult", + "SecurityDetails", + "NameValue", + "TracingGroupLocation", + "DebuggerLocation", + "DebuggerPausedDetails", + "ScreencastSize", + "ScreencastActionStyle", + "VirtualCredential", + "WebMCPToolAnnotations", + "WebMCPTool", + "ScreencastFrame" + ] } diff --git a/scripts/codegen/python_rules.js b/scripts/codegen/python_rules.js index d0c5fd54d..cefa9a0b4 100644 --- a/scripts/codegen/python_rules.js +++ b/scripts/codegen/python_rules.js @@ -191,6 +191,18 @@ module.exports = { // No docs alias. '*.*.storageState': 'StorageState', // No docs alias. + '*.storageState.cookies': 'StorageStateCookie', + // No docs alias. + '*.*.storageState.cookies': 'StorageStateCookie', + // No docs alias. + '*.storageState.origins': 'OriginState', + // No docs alias. + '*.*.storageState.origins': 'OriginState', + // The docs alias is `WebStorageItem`, which maps to NameValue elsewhere. + '*.storageState.origins.localStorage': 'LocalStorageEntry', + // The docs alias is `WebStorageItem`, which maps to NameValue elsewhere. + '*.*.storageState.origins.localStorage': 'LocalStorageEntry', + // No docs alias. '*.addCookies.cookies': 'SetCookieParam', // No docs alias. '*.timing': 'ResourceTiming', @@ -218,6 +230,21 @@ module.exports = { 'Tracing.group.location': 'TracingGroupLocation', }, + // `total=` of a TypedDict, where it differs from the rule "total=False if + // any field is optional". Keyed by struct name. + structTotal: { + // Shipped as total=True, although `line` and `column` are optional. + DebuggerLocation: true, + // Shipped as total=True, although every field is optional. + SecurityDetails: true, + // Shipped as total=False, although every field is required. + StorageState: false, + // Shipped as total=False, although every field is required. + StorageStateCookie: false, + // Shipped as total=True, although `line` and `column` are optional. + TracingGroupLocation: true, + }, + // Old implementation: the module in playwright/_impl/ that holds each class. // Classes not listed are in `_.py`. implModules: { diff --git a/scripts/update_api.sh b/scripts/update_api.sh index 3451c05c0..fe0bf975f 100755 --- a/scripts/update_api.sh +++ b/scripts/update_api.sh @@ -1,10 +1,11 @@ #!/bin/bash -# Generates the public API (playwright/{async,sync}_api/_generated.py) from the -# Playwright API docs (docs/src/api) in a microsoft/playwright checkout, pointed -# to by PW_SRC_DIR. Use the release branch for the version in DRIVER_VERSION -# (for example release-1.64). The generator is scripts/codegen/generate_api.js; -# Python facts that the docs do not contain are in scripts/codegen/python_rules.js. +# Generates the public API (playwright/{async,sync}_api/_generated.py) and its +# TypedDicts (playwright/_impl/_api_structures.py) from the Playwright API docs +# (docs/src/api) in a microsoft/playwright checkout, pointed to by PW_SRC_DIR. +# Use the release branch for the version in DRIVER_VERSION (for example +# release-1.64). The generator is scripts/codegen/generate_api.js; Python facts +# that the docs do not contain are in scripts/codegen/python_rules.js. driver_version="$(cat DRIVER_VERSION)" if [[ -z "$PW_SRC_DIR" ]]; then echo "Set PW_SRC_DIR to a microsoft/playwright checkout at the release branch for v${driver_version}, e.g.:" >&2 @@ -15,7 +16,7 @@ export PW_SRC_DIR function update_api { flavor="$1" - file_name="playwright/${flavor}_api/_generated.py" + file_name="${2:-playwright/${flavor}_api/_generated.py}" echo "Generating $file_name" if node scripts/codegen/generate_api.js --flavor "$flavor" --out "$file_name"; then pre-commit run --files "$file_name" @@ -26,6 +27,7 @@ function update_api { fi } +update_api structs playwright/_impl/_api_structures.py update_api sync update_api async