gh-153569: consolidate tokenizer source and lexer state

[pablogsal]

Consolidate tokenizer source and lexer state after #156482.

Replace the generic lexer mode array with active formatted-string frames that hold expression and comment spans. Carry raw-string context with emitted and cached parser tokens, so interpreting a token does not depend on the current frame. Keep field entry, punctuation, closing, and debug-expression transitions together in the string scanner, with explicit dispatch over its three modes.

Group indentation and logical-line state in a dedicated layout module, with operations for leading whitespace, continuations, pending indent/dedent tokens, and newline classification. Keep normal and alternate indentation columns together in each stack entry.

Give the reader ownership of input-specific state and source-retention decisions. Derive locations and failure status from existing scanner state, and keep buffer relocation inside the reader instead of a separate lexer buffer layer.

#156654 builds on this to replace the remaining scanner pointers and complete the opaque consumer API.

• Issue: Base the tokenizer API on source offsets #153569

[pablogsal]

Do not review until #156482 lands.