Skip to content

[Feature] Add Pipe logical backup sink and restore tooling - #18623

Open
Caideyipi wants to merge 2 commits into
masterfrom
feat/pipe-logical-backup-sink
Open

Caideyipi wants to merge 2 commits into
masterfrom
feat/pipe-logical-backup-sink

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Summary

  • add built-in logical-backup-sink / logical-backup-connector plugins that persist Pipe metadata and data events as ordered WAL-like .pwal segments
  • add CRC32C/SHA-256 integrity checks, atomic manifests, segment recovery/rotation, directory locking, append identity validation, and committed event groups
  • support tablet, TsFile, delete, schema plan/snapshot, heartbeat, and terminate records for tree and table models
  • add archive inspect/verify/stats/export/import/restore commands with safe ZIP handling, dry-run, checkpoints, schema-first replay, and replay request allowlisting
  • package Linux and Windows import/export wrappers in the CLI distribution and provide English/Chinese messages

Validation

  • node-commons targeted tests: 32 passed (LogicalBackupWriterTest, LogicalBackupArchiveReaderTest, BuiltinPipePluginTest)
  • CLI targeted tests: 4 passed (PipeLogicalBackupToolTest)
  • node-commons and CLI install/package validation passed
  • Spotless, Checkstyle, RAT, and forbidden-apis checks passed
  • English and Chinese locale compilation passed for node-commons and CLI
  • distribution packaging passed; the CLI ZIP contains all six wrappers plus CLI, node-commons, and Gson JARs
  • verified the CLI JAR contains PipeLogicalBackupTool and PipeTransferDataNodeHandshakeV2Req

Known baseline issue

DataNode compile was checked in both English and Chinese locales. Both reach the same four existing generated-interface errors in MPPDataExchangeManager and DeviceEntryRpcSegmentFetcher; neither file is modified by this PR, and no logical-backup compilation errors are reported.

@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.31784% with 714 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.87%. Comparing base (31621ef) to head (ce4b30d).

Files with missing lines Patch % Lines
...sink/protocol/logicalbackup/LogicalBackupSink.java 0.00% 274 Missing ⚠️
.../apache/iotdb/tool/pipe/PipeLogicalBackupTool.java 48.11% 234 Missing ⚠️
...s/pipe/sink/logicalbackup/LogicalBackupWriter.java 87.93% 86 Missing ⚠️
...sink/logicalbackup/LogicalBackupSegmentReader.java 76.27% 56 Missing ⚠️
...sink/logicalbackup/LogicalBackupArchiveReader.java 83.33% 39 Missing ⚠️
...s/pipe/sink/logicalbackup/LogicalBackupRecord.java 57.14% 18 Missing ⚠️
...ugin/dataregion/PipeDataRegionSinkConstructor.java 66.66% 2 Missing ⚠️
.../schemaregion/PipeSchemaRegionSinkConstructor.java 66.66% 2 Missing ⚠️
...s/pipe/sink/logicalbackup/LogicalBackupFormat.java 71.42% 2 Missing ⚠️
.../builtin/sink/logicalbackup/LogicalBackupSink.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18623      +/-   ##
============================================
+ Coverage     42.77%   42.87%   +0.10%     
  Complexity      442      442              
============================================
  Files          5443     5453      +10     
  Lines        394537   396538    +2001     
  Branches      51723    52040     +317     
============================================
+ Hits         168747   170032    +1285     
- Misses       225790   226506     +716     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant