[suxiaogang223]

Purpose

Linked issue: Closes #1326.

Related to #911 (partial scope).

Repeated strings can retain several MiB of Arrow input while
producing a small Parquet dictionary. Checking only encoded size can exhaust
the input budget even when flushing the current file would allow writing to
continue. This addresses one part of the resource-governance issue.

Prepared with AI assistance. No throughput measurements are available.

Brief change log

Expose the current format writer's input reservation internally and include it
in the data-file flush threshold. Reuse existing reservation release accounting;
other files and readers sharing the context cannot trigger this file's flush.
Writers without a resource context keep the encoded-size policy.

Tests

Validated with the existing Rust 1.97.0 toolchain on macOS arm64:

• cargo +1.97.0 test --locked -p paimon --lib: 4804 passed, 6 ignored.
• cargo +1.97.0 test --locked -p paimon --test writer_resources_test: 10 passed.
• New regressions cover four distinct approximately 4 MiB dictionary-friendly
  batches with a 6 MiB budget and a 1 MiB threshold, value readback and row-group
  boundaries, shared-context isolation, and reservation lifetime after failed
  flush/close. The existing auto-flush-with-tail test also checks per-file bytes.
• cargo +1.97.0 fmt --all -- --check and git diff --check: passed.
• cargo +1.97.0 clippy --locked -p paimon --all-targets -- -D warnings: passed.

Not run locally: CI-pinned Rust 1.98.0 checks, the full workspace/Vortex
suite, or the Linux/MSRV matrix. The results above cover the listed commands.

API and Format

Internal change; no public Rust or C API and no storage format change.
Row-group boundaries can change with resource accounting. Formats whose flush
is a no-op do not gain memory reclamation. Single-batch admission failures,
background close overlap and allocations outside reservations remain unchanged.
No spill, automatic retry or peak-memory guarantee is introduced.

Documentation

Document the per-file retained-input threshold in the C writer resource guide.

[JingsongLi]

[P2] Document the buffer option that controls this flush

The public table writer gets the threshold from CoreOptions::write_parquet_buffer_size(), which reads write.parquet-buffer-size and defaults to 256 MiB. The documented write-buffer-size key is read by the sorted global-index builder and leaves this file-writer threshold at its default.

I reproduced the difference through Table::new_write_builder().with_resources(...) with a 6 MiB budget and distinct 4 MiB dictionary-friendly batches. With the documented key set to 1 MiB, the second write fails with ResourceExhausted. With write.parquet-buffer-size=1 MB, all four batches write, commit, and scan back as 4,096 rows with the expected payload.

Please use write.parquet-buffer-size here and cover the option through the public table writer so the test also exercises configuration wiring.

[JeonDaehong]

Append tables pass all four writes here (main fails on the second), but postpone-bucket tables (bucket=-2) still hit this: postpone_file_writer.rs:226 checks only in_progress_size. Same 6 MiB budget, write.parquet-buffer-size=1 mb and 4 MiB dictionary batches: the first write keeps 4113 KiB reserved and the second fails with ResourceExhausted, on this branch and on main. kv_data_file_writer.rs:314 has the same check too, though a bucket=1 PK table already fails on the first write here because the sort buffer and the file writer each reserve the batch, so that one may be separate.