Repository navigation
Keep a surrogate pair in one output chunk [patch] - #135
Merged
Merged
Conversation
A read of the 4096-char buffer could stop between the two halves of a surrogate pair, so a raw OutputHandler callback got one chunk ending in a lone high surrogate and the next starting with a lone low surrogate. A trailing high surrogate is now held back and put in front of the next read, and flushed on its own at end of stream. Fixes #109 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vfJW5g4znzysmAf6XdCd2
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



AsyncProcessStreamReader.ReadToEndpassed each 4096-char read straight to the handler. When a read stopped between the two halves of a surrogate pair, a rawOutputHandlercallback got one chunk ending in a lone high surrogate and the next starting with a lone low surrogate. A callback that encodes each chunk on its own then wrote two U+FFFD characters in place of the emoji.The read loop now lives in an internal static
ReadChunks(TextReader, char[], Action<string>). It holds back a trailing high surrogate, puts it in front of the next read, and hands it over on its own if the stream ends without the low half. BOM stripping and the delivery lock work as before, in the callbackReadToEndpasses in.Tests (
AsyncProcessStreamReaderTests, which driveReadChunksover aStringReaderso the chunk boundaries are deterministic):With the hold-back removed, the first two tests fail. Full suite on Linux/net10.0: 68 passed, 2 skipped (the existing elevation skips). The full multi-target build passes.
Fixes #109
Generated by Claude Code