feat(resharding): call post-data-validation during replication

[meskill]

Integrate the FK key validation into the resharding process. This is done by extending the COPY_DATA task to run the replication until all the tables are synchronized (so the fact that we copy it from different snapshots won't cause issues). After the tables are synchronized we can run the sync_schema with PostDataValidation to run the actual validation and mark the table as valid.

There are different settings to when the run the actual validation, because in theory we can fail at any step during resharding so the constraints will be false because we copy and replicate data with replication role that disables the checks for it. That means even if we ran the validation after COPY_DATA and the pg marked the table as valid during replication we can break the rules and pg won't know about it (although, this should not happen the way we do the replication, and I can think only of the case where the related rows in tables hit different shards so they appeared on different instances).

So, the flow is following:

• COPY_DATA copies the tables like it was
• after the copy the internal replication starts that waits the moment when all the tables from the source will reach the same LSN on destination tables.
• then the REPLICATION task is started
• depending on the setting we can start the validation check - by default, it's running in parallel with the replication itself, since it should not block the writes and doesn't hold other tables.
• the process fails if validation failed

[codecov]

Codecov Report

❌ Patch coverage is 94.96644% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pgdog/src/api/replication.rs	84.16%	19 Missing ⚠️
pgdog/src/api/resharding.rs	80.00%	8 Missing ⚠️
pgdog/src/api/copy_data.rs	97.10%	2 Missing ⚠️
...nd/replication/logical/publisher/cutover_policy.rs	99.43%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!