Repository navigation
[AMORO-4292][Iceberg] Guard orphan cleanup against shared table locations - #4443
Open
pallavithegod wants to merge 1 commit into
Open
pallavithegod wants to merge 1 commit into
pallavithegod wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Close #4292.
IcebergTableMaintainer.cleanOrphanFilescomputes reachability for one table but scans its location when deleting content and metadata. Metadata for a second table in that location is not part of the first table's reachable set, so cleanup can delete another table's files.Check metadata ownership before either deletion phase. A foreign or unknown UUID, unreadable metadata, or failed inspection skips cleanup with a warning. Old metadata with the same UUID remains eligible for normal cleanup. The check detects metadata present at inspection time; it does not coordinate concurrent table creation or detect shared files outside the inspected location.
Brief change log
clean-orphan-file.ignore-location-conflictescape hatch and document its data-loss risk and the guard's scope.How was this patch tested?
TestSharedLocationOrphanCleanup).TestIcebergTableMaintainerandTestIcebergV3MaintenanceGuard; 25 selected tests passed in total.Validation used Maven 3.9.11 and Java 11. The file-backed tests ran in Linux using
maven:3.9.11-eclipse-temurin-11because Windows Hadoop requires native filesystem support../mvnw spotless:apply checkstyle:checkpassed../mvnw test -pl amoro-format-iceberg -am -Dtest=TestSharedLocationOrphanCleanup,TestIcebergTableMaintainer,TestIcebergV3MaintenanceGuard -Dsurefire.failIfNoSpecifiedTests=falsepassed.