Skip to content

[AMORO-1604][Iceberg] Preserve string event-time metrics in mixed-table writers - #4445

Open
pallavithegod wants to merge 1 commit into
apache:masterfrom
pallavithegod:fix/amoro-1604-event-time-metrics
Open

pallavithegod wants to merge 1 commit into
apache:masterfrom
pallavithegod:fix/amoro-1604-event-time-metrics

Conversation

@pallavithegod

Copy link
Copy Markdown

Why are the changes needed?

Close #1604.

A string event-time field can inherit Iceberg's default truncate(16) metrics mode. Timestamp strings such as 2026-10-16 12:34:56 need more than 16 characters, so a truncated upper bound cannot reliably drive watermark calculation.

Default the configured string event-time column to full metrics in BasicUnkeyedTable.properties(), after catalog properties are merged. Mixed-table writers consume these effective properties for both newly created and existing tables. This covers the table-loading path rather than only internal-catalog creation. Explicit per-column metrics settings remain authoritative, and unrelated columns keep their existing behavior.

Brief change log

  • Supply write.metadata.metrics.column.<event-time-field>=full when the event-time column is a string and has no explicit column override.
  • Leave timestamp-typed, absent, and unknown fields unchanged.
  • Avoid changing the source property map or persisted metadata.
  • Document the effective-writer scope: existing file bounds are not repaired, and writers accessing the underlying Iceberg table directly must configure metrics separately.
  • Add real Parquet/ORC bounds checks, Avro writer configuration coverage, catalog defaults, and explicit override cases.

How was this patch tested?

  • Added new unit test cases that verify the changes (TestEventTimeMetrics, 12 cases).
  • Verified existing unit tests pass locally (TestWatermarkGenerator and TestTableWatermark); 41 selected tests passed in total.
  • Tested manually in local environment.

Using Java 11 and Maven 3.9.11:

  • ./mvnw spotless:apply checkstyle:check passed.
  • ./mvnw test -pl amoro-format-iceberg -am -Dtest=TestEventTimeMetrics,TestWatermarkGenerator,TestTableWatermark -Dsurefire.failIfNoSpecifiedTests=false passed in Linux.
  • The 12 new cases also passed on Windows.
  • Against unmodified master, the new suite had four failures and no errors.

@github-actions github-actions Bot added the type:docs Improvements or additions to documentation label Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: When manually setting table.event-time-field to a field, it should widen its data file mertic length

1 participant