Skip to content

[AMORO-1121][Spark] Protect internal mixed-table properties in ALTER TABLE - #4446

Open
pallavithegod wants to merge 1 commit into
apache:masterfrom
pallavithegod:fix/amoro-1121-protect-mixed-table-metadata
Open

pallavithegod wants to merge 1 commit into
apache:masterfrom
pallavithegod:fix/amoro-1121-protect-mixed-table-metadata

Conversation

@pallavithegod

Copy link
Copy Markdown

Why are the changes needed?

Addresses #1121 for Spark's mixed-table catalog entry points.

Spark ALTER TABLE currently forwards user property changes to mixed-table commits without restricting internal transaction, partition, and watermark bookkeeping. Changing or removing these values can invalidate table metadata or interfere with consistency tracking.

Validate the complete property-change list before loading or mutating a table in each supported mixed-catalog implementation. Reject requests containing an internal bookkeeping property before any part can be committed. This covers both direct mixed-catalog registration and unified-catalog delegation across Spark 3.3, 3.4, and 3.5.

Brief change log

  • Add shared validation for table.partition-properties, base.table.max-transaction-id, max-txId, watermark.table, and watermark.base.
  • Apply the gate to both SET and UNSET operations in all three Spark catalog implementations.
  • Preserve supported location, format-version, name-mapping, ordinary configuration, and schema changes.
  • Keep low-level table commit APIs available to runtime maintenance.
  • Document the Spark boundary and add shared regression cases exercised through each version's actual catalog entry point.

How was this patch tested?

  • Added new unit test cases that verify the changes (TestMixedCatalogPropertyProtection).
  • Verified local builds, test compilation, and the selected tests with Spark 3.3, 3.4, and 3.5; 16 cases passed per profile.
  • Tested manually in local environment.

Using Java 11:

  • ./mvnw spotless:apply checkstyle:check passed.
  • For each version, ran ./mvnw -Pspark-<version> test -pl amoro-format-mixed/amoro-mixed-spark/v<version>/amoro-mixed-spark-<version> -am -Dtest=TestMixedCatalogPropertyProtection -Dsurefire.failIfNoSpecifiedTests=false.
  • The 11 public-entry rejection cases failed against unmodified master: the calls reached catalog access instead of rejecting the internal property first. The five validation-control cases exercise the new shared gate and are not part of that baseline run.

This PR addresses the Spark portion of #1121; it does not close the umbrella issue for other interfaces.

@github-actions github-actions Bot added type:docs Improvements or additions to documentation module:mixed-spark Spark module for Mixed Format labels Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:mixed-spark Spark module for Mixed Format type:docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant