Skip to content

Keep a surrogate pair in one output chunk [patch] - #135

Merged
matt-edmondson merged 1 commit into
mainfrom
fix/109-surrogate-pair-chunks
Nov 4, 2026
Merged

matt-edmondson merged 1 commit into
mainfrom
fix/109-surrogate-pair-chunks

Conversation

@matt-edmondson

Copy link
Copy Markdown
Contributor

AsyncProcessStreamReader.ReadToEnd passed each 4096-char read straight to the handler. When a read stopped between the two halves of a surrogate pair, a raw OutputHandler callback got one chunk ending in a lone high surrogate and the next starting with a lone low surrogate. A callback that encodes each chunk on its own then wrote two U+FFFD characters in place of the emoji.

The read loop now lives in an internal static ReadChunks(TextReader, char[], Action<string>). It holds back a trailing high surrogate, puts it in front of the next read, and hands it over on its own if the stream ends without the low half. BOM stripping and the delivery lock work as before, in the callback ReadToEnd passes in.

Tests (AsyncProcessStreamReaderTests, which drive ReadChunks over a StringReader so the chunk boundaries are deterministic):

  • the issue's case (4095 chars + 😀 + 5000 chars, 4096-char buffer): no chunk splits the pair, and the concatenated chunks equal the input
  • a 1-char buffer delivers each emoji whole
  • a lone high surrogate at end of stream is still delivered
  • output without surrogates is delivered as read

With the hold-back removed, the first two tests fail. Full suite on Linux/net10.0: 68 passed, 2 skipped (the existing elevation skips). The full multi-target build passes.

Fixes #109


Generated by Claude Code

A read of the 4096-char buffer could stop between the two halves of a
surrogate pair, so a raw OutputHandler callback got one chunk ending in a
lone high surrogate and the next starting with a lone low surrogate. A
trailing high surrogate is now held back and put in front of the next
read, and flushed on its own at end of stream.

Fixes #109

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vfJW5g4znzysmAf6XdCd2
@sonarqubecloud

sonarqubecloud Bot commented Nov 2, 2026

Copy link
Copy Markdown

@matt-edmondson
matt-edmondson merged commit 6c3bbd8 into main Nov 4, 2026
15 checks passed
@matt-edmondson
matt-edmondson deleted the fix/109-surrogate-pair-chunks branch November 4, 2026 00:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OutputHandler chunks can split a surrogate pair, delivering unpaired surrogates to the callback

2 participants