Repository navigation
fix(flexible-csv): Skip the CSV header once per file instead of once per split - #417
Merged
Merged
Conversation
…per split The filesystem connector decoded flexible-csv files with a line deserializer that drops the first record it sees, cloned for every file split. A file read in several splits therefore lost the first data row of every split after the first. A new BulkReaderFormatFactory reads each split the same way, but skips the header only in the split that starts at offset 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
12 of 16 tasks
mateczagany
approved these changes
Oct 16, 2026
9 of 14 tasks
ferenc-csaky
added a commit
that referenced
this pull request
Oct 21, 2026
…per split (#417) Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
The filesystem connector now skips a
flexible-csvfile's header once per file, instead of once per split. Before, a file read in several splits lost the first data row of every later split. Follow-up to DataSQRL/sqrl#2476.FlexibleCsvFileFormat: aBulkReaderFormatFactorywith theflexible-csvidentifier. The filesystem connector prefers it overFlexibleCsv, and Kafka keeps usingFlexibleCsvunchanged.FlexibleCsvBulkFormat: reads each split through aDeserializationSchemaAdapter, as before, so parsing and options are unchanged. Only the split at offset 0 skips the header, and restored splits stay consistent.README.md: documents theskip-headerdefault, that it applies per file, the Kafka caveat, and theflexible-csv.option prefix.Only files read in several splits were affected: uncompressed files on a filesystem that reports several blocks, such as HDFS. Local files, compressed files and S3 objects are always read in one split.
skip-headerstill defaults totrue, so Kafka still drops the first record of each reader. Changing that is a separate decision.Area
flink-sql-runner)connectors)formats)stdlib)env-utils)testing)README.md,stdlib-docs)How did you test it?
FlexibleCsvFileFormatTestcovers factory lookup, a file read in two splits at every split point,skip-header = false, and restoring the first and a later split. 20 of its 28 tests fail with the old behavior.FlexibleCsvFileFormatITruns SQL on a MiniCluster: plain and gzipped files with headers, a single-column query, andignore-parse-errors.mvn verifyon the module,mvn testonflink-sql-runner, andmvn spotless:checkall pass.Checklist
feat:,fix:ordocs:, optionally with a scope (e.g.fix(flexible-json):)mvn clean installpasses locally and the code is formatted withmvn spotless:apply(formatted; ran only the builds above, with-Dlicense.skip=truebecause the license plugin can't read a git worktree)given_when_thennaming, AssertJ assertions)pom.xmlgit commit -s -S)README.mdandstdlib-docsare updated if user-facing behavior or functions changed🤖 Generated with Claude Code