Implement Iceberg Side-Input Table Cache Integration with Fallback - #40080
jrmccluskey merged 3 commits into
Conversation
|
Assigning reviewers: R: @Abacn for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
R: @ahmedabu98 |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
ahmedabu98
left a comment
ahmedabu98
left a comment
There was a problem hiding this comment.
Overall looks good, just a couple comments
| String canonicalTableId; | ||
| try { | ||
| canonicalTableId = | ||
| IcebergUtils.tableIdentifierToString( | ||
| IcebergUtils.parseTableIdentifier(tableIdentifier)); | ||
| } catch (Exception e) { | ||
| canonicalTableId = tableIdentifier.trim(); | ||
| } |
There was a problem hiding this comment.
I don't think this is necessary?
There was a problem hiding this comment.
Sorry I mean the roundtrip itself feels unnecessary (i.e. we can just use tableIdentifier directly without doing the tableIdentifierToString(parseTableIdentifier(...)) roundtrip)
There was a problem hiding this comment.
Okay yeah this is unnecessary. It only makes changes if, for some reason, the underlying table identifiers being passed aren't formatted correctly.
| } else if (tableSpec != null) { | ||
| spec = tableSpec.getPartitionSpec(); | ||
| if (data.getSchema().getFieldCount() == tableSpec.getSchema().columns().size()) { | ||
| schema = tableSpec.getSchema(); | ||
| } | ||
| checkStateNotNull(cachedSpecIds).put(tableIdentifier, tableSpec.getSpecId()); |
There was a problem hiding this comment.
Can we move this to be the first check? So we can make sure cachedSpecIds is populated even when the user gives a createConfig
There was a problem hiding this comment.
Gotcha, flipped the logic here to have this check execute first
2 participants
Wires through the Iceberg side-input table cache transform into the IO but does not expose the API to users yet. Largely handled by introducing overloads of constuctors and methods to determine if a side input is being used.
Part of #39723
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.