Skip to content

refactor!: Remove the flexible-csv format - #421

Merged
ferenc-csaky merged 1 commit into
mainfrom
refactor/flexible-csv-remove-skip-header
Oct 20, 2026
Merged

ferenc-csaky merged 1 commit into
mainfrom
refactor/flexible-csv-remove-skip-header

Conversation

@ferenc-csaky

@ferenc-csaky ferenc-csaky commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

What does this change?

Removes the flexible-csv format. Its only addition over Flink's csv format was the skip-header option. DataSQRL no longer infers the schema of CSV files, and it copies data files unchanged (DataSQRL/sqrl#2476), so CSV sources declare their columns and their files have no header row. Without skip-header, flexible-csv is the same as csv.

  • Removed: the formats/flexible-csv-format module, including FlexibleCsv and the per-file header skipping from fix(flexible-csv): Skip the CSV header once per file instead of once per split #417 (FlexibleCsvFileFormat, FlexibleCsvBulkFormat) with their tests. Also its dependency in flink-sql-runner/pom.xml and its entry in FactoryAutoServiceTest.
  • README.md: removes the CSV Format section and the "Additional configuration options for CSV format" feature bullet. CLAUDE.md no longer lists the module.

Breaking change

Tables with 'format' = 'flexible-csv' fail with Flink's Could not find any factory for identifier 'flexible-csv' that implements '...DeserializationFormatFactory' in the classpath.. To migrate:

  1. Switch to 'format' = 'csv'.
  2. Rename flexible-csv.* options to csv.*.
  3. Drop skip-header, and remove header rows from the data.

This also removes the old Kafka behaviour, where skip-header = true by default dropped the first record each reader decoded.

Area

  • Runner CLI and Docker image (flink-sql-runner)
  • Connectors (connectors)
  • Formats (formats)
  • Function libraries (stdlib)
  • Environment utilities (env-utils)
  • Tests and samples (testing)
  • Build / CI / dependencies
  • Docs (README.md, stdlib-docs)

How did you test it?

  • mvn compile -pl flink-sql-runner -am passes, so nothing else referenced the module.
  • FactoryAutoServiceTest passes.
  • mvn spotless:check passes.
  • git grep -i flexible-csv finds nothing.

Checklist

  • The PR title starts with a type such as feat:, fix: or docs:, optionally with a scope (e.g. fix(flexible-json):)
  • mvn clean install passes locally and the code is formatted with mvn spotless:apply (formatted; ran only the builds above, with -Dlicense.skip=true because the license plugin can't read a git worktree)
  • New or changed behavior is covered by tests (given_when_then naming, AssertJ assertions) (removal only)
  • New dependency and plugin versions are defined as properties in the root pom.xml (no new dependencies)
  • Commits are signed off and signed (git commit -s -S)
  • README.md and stdlib-docs are updated if user-facing behavior or functions changed

🤖 Generated with Claude Code

Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
@ferenc-csaky
ferenc-csaky force-pushed the refactor/flexible-csv-remove-skip-header branch from efddb9b to 4af5a67 Compare October 19, 2026 20:05
@ferenc-csaky ferenc-csaky changed the title refactor(flexible-csv)!: Remove the skip-header option and decode like the csv format refactor!: Remove the flexible-csv format Oct 19, 2026
@ferenc-csaky ferenc-csaky added this to the 0.12.1 milestone Oct 20, 2026
@ferenc-csaky
ferenc-csaky merged commit 294d002 into main Oct 20, 2026
13 checks passed
@ferenc-csaky
ferenc-csaky deleted the refactor/flexible-csv-remove-skip-header branch October 20, 2026 08:39
ferenc-csaky added a commit that referenced this pull request Oct 21, 2026
Signed-off-by: Ferenc Csaky <ferenc@datasqrl.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant