Skip to content

[AMORO-4292][Iceberg] Guard orphan cleanup against shared table locations - #4443

Open
pallavithegod wants to merge 1 commit into
apache:masterfrom
pallavithegod:fix/amoro-4292-shared-location-cleanup
Open

pallavithegod wants to merge 1 commit into
apache:masterfrom
pallavithegod:fix/amoro-4292-shared-location-cleanup

Conversation

@pallavithegod

Copy link
Copy Markdown

Why are the changes needed?

Close #4292.

IcebergTableMaintainer.cleanOrphanFiles computes reachability for one table but scans its location when deleting content and metadata. Metadata for a second table in that location is not part of the first table's reachable set, so cleanup can delete another table's files.

Check metadata ownership before either deletion phase. A foreign or unknown UUID, unreadable metadata, or failed inspection skips cleanup with a warning. Old metadata with the same UUID remains eligible for normal cleanup. The check detects metadata present at inspection time; it does not coordinate concurrent table creation or detect shared files outside the inspected location.

Brief change log

  • Inspect ordinary and compressed metadata files before orphan deletion.
  • Preserve normal cleanup for a single table and same-UUID metadata history.
  • Add the default-false clean-orphan-file.ignore-location-conflict escape hatch and document its data-loss risk and the guard's scope.
  • Add real-file regression coverage for foreign UUIDs, unreadable metadata, same-UUID history, and explicit opt-out.

How was this patch tested?

  • Added new unit test cases that verify the changes (TestSharedLocationOrphanCleanup).
  • Verified existing unit tests pass locally: TestIcebergTableMaintainer and TestIcebergV3MaintenanceGuard; 25 selected tests passed in total.
  • Tested manually in local environment.

Validation used Maven 3.9.11 and Java 11. The file-backed tests ran in Linux using maven:3.9.11-eclipse-temurin-11 because Windows Hadoop requires native filesystem support.

  • ./mvnw spotless:apply checkstyle:check passed.
  • ./mvnw test -pl amoro-format-iceberg -am -Dtest=TestSharedLocationOrphanCleanup,TestIcebergTableMaintainer,TestIcebergV3MaintenanceGuard -Dsurefire.failIfNoSpecifiedTests=false passed.
  • Against unmodified master, the six new cases had four failures and no errors.

@github-actions github-actions Bot added the type:docs Improvements or additions to documentation label Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: IcebergTableMaintainer orphan file cleanup can corrupt other tables sharing the same location

1 participant