Skip to content

Phase 7.2: the JS lexer, proven total and span-covering - #187

Open
atiernan wants to merge 7 commits into
claude/js-lexer-specfrom
claude/js-lexer-impl
Open

atiernan wants to merge 7 commits into
claude/js-lexer-specfrom
claude/js-lexer-impl

Conversation

@atiernan

Copy link
Copy Markdown
Member

The second half of Phase 7 item 7.2: the JS lexer's implementation, against the spec and contracts in #178. Totality and span coverage are proven, but full agreement with the spec is not, so the crate stays at tier 0 and doesn't implement api::JsLexer yet. spec.rs, contract.rs and api.rs are unchanged.

Stacked on #178. Merge that PR first, or retarget this one afterwards.

What's here

  • src/exec.rs: the lexer, written as exec Verus code.
  • Unicode tables: xtask/src/js_idents.rs generates src/unicode_data.rs from the vendored DerivedCoreProperties.txt (Unicode 18.0.0, in clazel-js-regexp's ucd-regexp suite), checking its SHA-256 and version header. It holds 693 ID_Start ranges and 810 ID_Continue ranges above U+007F. It runs under cargo xtask tables, and cargo xtask check fails if the table is stale. src/unicode.rs looks it up by bisection.
  • Tests: tests/lexer.rs, tests/test262.rs and tests/unicode_tables.rs.

Proven

  • next_input_element:
    • it always returns, with no panic and no overflow;
    • it returns Eof exactly at the end of the input, and Err only before it;
    • each element starts at pos and is non-empty, so repeated calls cover the input with no gaps;
    • the value's variant matches the element kind;
    • white space, line terminators, the hashbang, / and } match the contract's element_matches.
  • next_token:
    • the token lies inside the input and is not trivia, with the same kind agreement;
    • if newline_before is set, a line terminator lies between pos and the token.
  • lemma_contract_implies_proven: the full contract implies what is proven here, so the proven claim is a weaker form of the contract, not a different one.
  • The character-class helpers equal their spec counterparts, as does push_utf16.
  • Every scan_* function is total and stays in bounds.
  • No escape hatches. The only verifier attribute is #[verifier::opaque] on the generated table consts, as clazel-idna does.

Not proven

  • That comments, identifiers, punctuators, numbers, strings, templates and regexps match the grammar. Proving the longest-match rule needs negative facts: that no longer element of any kind exists. That is the large remaining piece of work.
  • That errors are exactly lex_error.
  • That newline_before is set whenever a line terminator lies in between (only the other direction is proven).
  • Every token value: identifier code points, SV, TV, TRV, bigint and legacy.
  • The link between the Unicode table and the spec's uninterpreted unicode_id_start_non_ascii / unicode_id_continue_non_ascii. Nothing about it is claimed.
  • Also missing: the independent transcription and mutations.json.

For the reviewer

  • Unterminated /*: read literally, the longest-element rule lexes it as / then * in a goal that allows a division punctuator, rather than as an error. That is faithful to the grammar, and the parser will reject it.

Tests

  • Unit tests: 13 in tests/lexer.rs, positive and negative cases for every kind. They include a totality sweep over every input of up to 3 code points from a 38-code-point alphabet, in every goal.
  • Test262: over all 5,212 vendored files, the elements cover each file with no gaps in the Div and RegExp goals. All 829 must-parse tests lex to the end. 246 of the 4,383 negative tests are rejected already by the lexer; that is for information only.
  • Miri (pinned nightly, in WSL): passes. The Test262 test is skipped there because it reads files. After that run the stride was raised from 97 to 331, to keep the run near a minute, and Miri was not re-run after the change.

Verification

  • Verus, on a fresh target dir in WSL: verification results:: 118 verified, 0 errors.
  • cargo xtask check: STATIC CHECKS PASSED. cargo xtask tables is up to date. cargo test -p clazel-js-lexer and cargo test -p xtask pass, and clippy -D warnings and rustfmt are clean.
  • cargo xtask verify --changed was not run, because the branch changes xtask/. CI's full verification is the check, and its proof jobs need approval.

🤖 Generated with Claude Code

atiernan and others added 5 commits November 17, 2026 11:52
…0 (7.2)

`cargo xtask tables` now writes crates/clazel-js-lexer/src/unicode_data.rs
from DerivedCoreProperties.txt (Unicode 18.0.0), the copy vendored for
clazel-js-regexp, after checking its SHA-256 against that SOURCE.md; `check`
fails if the table is stale. `unicode.rs` looks code points up by bisection.

Nothing ties the tables to the spec's uninterpreted
unicode_id_start_non_ascii / unicode_id_continue_non_ascii: Verus proves only
that a lookup is total and false outside U+0080..U+10FFFF. Tests check the
tables are sorted and disjoint, ID_Start is within ID_Continue, and known
code points.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
src/exec.rs: next_input_element and next_token as Verus exec code, one
scan_ function per production of spec.rs, no recursion. Verus proves (no
escape hatch) that every input gives a result without panic or overflow,
that Eof comes exactly at the end, that an element is a non-empty range at
pos inside the source text (so elements tile the input), that the value's
variant matches the kind, and that next_token's token is after pos, not
trivia, with newline_before set only by a line terminator in between.

Not proven: agreement with spec.rs (the kind derives the range, the length
is longest_len, the errors are lex_error's, the values are SV/TV/TRV/
IdentifierCodePoints). So the functions do not implement api::JsLexer and
the crate stays at tier 0. lemma_contract_implies_proven shows the proven
part is a weakening of the contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/lexer.rs: positive and negative cases for every element kind, the
Annex B HTML-like comments, the numeric-literal follow rule and escape
early errors, SV/TV/TRV values, and a totality sweep over every input of up
to three code points from a branch-covering alphabet (a stride under Miri).

tests/test262.rs (ignored under Miri: it reads files): over all 5,212
vendored tests, the elements tile each file in two goals, and every test that
must parse (829) lexes to the end with goals guessed from the previous
token. This is not the conformance run, which needs the parser (7.3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
next_input_element and next_token now also ensure kind_agrees_proven: a
WhiteSpace, LineTerminator, HashbangComment, DivPunctuator or
RightBracePunctuator element is derived by its kind in the goal
(element_matches, the contract's own conjunct). The other kinds' derivation,
the longest-element rule, the errors and the values remain unproven, as the
module header lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch is being deployed

1 queued deployment
full-verification — 9f4fd040 Deployed Nov 20, 2026 by atiernan via Approve the proofs #613
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant