Skip to content

[core] Add compression extension to text format data file names - #9853

Open
jackylee-ch wants to merge 1 commit into
apache:masterfrom
jackylee-ch:text-compression-suffix
Open

jackylee-ch wants to merge 1 commit into
apache:masterfrom
jackylee-ch:text-compression-suffix

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Purpose

A text file is always written through the configured codec (AbstractTextFileWriter:44), while the reader recovers the codec from the file name alone; file.compression is not consulted on read. DataFilePathFactory.isTextFormat listed only json and csv, so file.format=text wrote zstd bytes into data-<uuid>-0.text and read them back as UTF-8 lines — garbage rows, no exception.

Format tables are unaffected (formatTableFileCompression() returns none for text); managed tables hit it with no configuration at all, since file.compression defaults to zstd.

Files already written compressed under the old name stay unreadable; this changes new writes only.

Tests

DataFilePathFactoryTest#testTextFormatCarriesTheCompressionExtension. Reverting the fix:

Expecting actual: ...-0.text
to end with: .text.zst

org.apache.paimon.io.*Test: 113 run, 0 failures.

Written with Claude Code; verification is mine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant