Skip to content

fix(fs): detect binary files that contain no NUL byte - #115

Open
Aikiooo wants to merge 1 commit into
shellular-org:mainfrom
Aikiooo:fix/binary-file-detection
Open

fix(fs): detect binary files that contain no NUL byte#115
Aikiooo wants to merge 1 commit into
shellular-org:mainfrom
Aikiooo:fix/binary-file-detection

Conversation

@Aikiooo

@Aikiooo Aikiooo commented Sep 10, 2026

Copy link
Copy Markdown

What

FS_READ decides whether a file is binary with a NUL-byte check:

const buffer = fs.readFileSync(filePath);
const isBinary = buffer.includes(0);

Plenty of binary formats do not contain a NUL byte in the bytes read — JPEG, PNG, PDF and ZIP all have contents where one is not guaranteed. When the check misses, the buffer is sent as buffer.toString("utf-8") with encoding: "utf-8".

toString("utf-8") is not fatal: invalid sequences are replaced with U+FFFD rather than throwing. The client sees encoding: "utf-8" and re-encodes that string, so the replaced bytes are gone for good. This affects previews and downloads, which trust the encoding the host reports rather than sniffing the payload themselves.

Change

Reuse the heuristic the app already applies in isLikelyBinaryBytes: a NUL byte, a fatal UTF-8 decode, then a control-character ratio above 0.3. Both ends now agree on what counts as binary.

No protocol change — the same FS_READ / FS_READ_RESULT shapes and the same encoding values, just more files classified as base64.

Risk

Text in a legacy single-byte encoding (Windows-1252, Latin-1) with a high proportion of control characters would now be classified as binary. That is the trade-off the app already makes with the same heuristic on its own reads. UTF-8, UTF-8 with a BOM, and empty files stay text.

Tests

The cli package has no test setup, so I did not add a test rather than introduce a framework as part of this change. tsc and biome check are clean.

Notes

GIT_READ has the same weakness, but it reads with encoding: "utf-8" up front, so a fix there also means switching that call to encoding: "buffer". I left it out to keep this to a single behavior change.

`FS_READ` decided binary vs text with `buffer.includes(0)`. JPEG, PNG, PDF
and ZIP files do not have to contain a NUL byte, and Node's
`toString("utf-8")` is not fatal — invalid sequences become U+FFFD instead
of being reported. Those files were sent with `encoding: "utf-8"`, so the
client re-encoded the replaced characters and the bytes were lost.

Use the heuristic the app already applies in `isLikelyBinaryBytes` — a NUL
byte, a fatal UTF-8 decode, then a control-character ratio — so both ends
agree on what counts as binary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant