normalize Spanner CDC - #40040
normalize Spanner CDC#40040
Conversation
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
087221f to
1deb624
Compare
| private static final HashMap<String, SpannerSchema> TABLE_SCHEMAS = new HashMap<>(); | ||
|
|
||
| private static Schema getTableSchema(SpannerChangestreamsReadConfiguration config) { | ||
| Pipeline miniPipeline = Pipeline.create(); |
There was a problem hiding this comment.
Either way, this requires pipeline submission time access to the Spanner instance. As a follow up, to allow pass a schema so that it won't need IO resource access at pipeline submission time, or create a Issue to track
There was a problem hiding this comment.
ok and yes both still need Spanner access at expansion and submission time
I'll create a follow up issue to allow an optional schema config on the Spanner CDC SchemaTransform so expansion can skip live information_schema access when the schema is provided
| if (allowedTableNames.isEmpty()) { | ||
| return true; | ||
| } | ||
| for (String allowed : allowedTableNames) { |
There was a problem hiding this comment.
Two things unusual
- allowedTableNames is a Set and we're doing a linear search
- equalsIgnoreCase. Is table name supposed to be case inSensitive?
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.