Skip to content

Antalya 26.8: auto-grp-pr-2141: combined port of 13 PRs - #2399

Open
zvonand wants to merge 20 commits into
antalya-26.8from
feature/antalya-26.8/auto-grp-pr-2141
Open

Antalya 26.8: auto-grp-pr-2141: combined port of 13 PRs#2399
zvonand wants to merge 20 commits into
antalya-26.8from
feature/antalya-26.8/auto-grp-pr-2141

Conversation

@zvonand

@zvonand zvonand commented Sep 17, 2026

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add support for exporting MergeTree parts and partitions to object storage, including experimental Apache Iceberg exports, with a command to schedule every partition at once, automatic type casting as in INSERT ... SELECT, retries with backoff, flexible column and partition-expression matching between source and destination, and a persistent export history in system.replicated_partition_exports. Also add alternative cluster table function syntax with distributed JOIN execution on swarm nodes, expose Iceberg partition and sorting keys in system.tables, allow cluster discovery settings to be reloaded from remote_servers without a restart, and fix Iceberg and Parquet read problems, JOIN filter pushdown, and incomplete data when a multi-file partition export resumes after a restart.
Add support for exporting MergeTree parts and partitions to object storage, including experimental Apache Iceberg exports, with a command to schedule every partition at once, automatic type casting as in INSERT ... SELECT, retries with backoff, flexible column and partition-expression matching between source and destination, and a persistent export history in system.replicated_partition_exports. Also add alternative cluster table function syntax with distributed JOIN execution on swarm nodes, expose Iceberg partition and sorting keys in system.tables, allow cluster discovery settings to be reloaded from remote_servers without a restart, and fix Iceberg and Parquet read problems, JOIN filter pushdown, and incomplete data when a multi-file partition export resumes after a restart (#2141 by @zvonand, #2145 by @zvonand, #2146 by @zvonand, #2200 by @zvonand, #2201 by @zvonand, #2209 by @zvonand, #2210 by @zvonand, #2218 by @k-morozov, #2229 by @k-morozov, #2253 by @arthurpassos, #2249 by @ianton-ru, #2197 by @ianton-ru, #2294 by @arthurpassos).

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Combined port of 13 PR(s) (group auto-grp-pr-2141). Cherry-picked from #2141, #2145, #2146, #2200, #2201, #2209, #2210, #2218, #2229, #2253, #2249, #2197, #2294.

zvonand and others added 18 commits September 15, 2026 16:55
@zvonand
Cherry-pick of #2141 with unresolved conflict markers (resolution in …
6d7cf47 
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2141 from Altinity/feature/antalya-26.6/auto-grp-pr-1662

Antalya 26.6: Expose IcebergS3 partition_key / sorting_key in system.tables
# Conflicts:
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h
@zvonand
Resolve conflicts in cherry-pick of #2141
852645a 
Source-PR: #2141 (#2141)
@mkmkme @zvonand
Cherry-pick of #2145 with unresolved conflict markers (resolution in …
c104023 
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2145 from Altinity/feature/antalya-26.6/auto-grp-pr-1687

Antalya 26.6: Cluster request improvements + Iceberg/Parquet read fixes
# Conflicts:
#	programs/server/Server.cpp
#	src/Common/ProfileEvents.cpp
#	src/Core/Settings.cpp
#	src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h
#	src/Interpreters/ClusterFunctionReadTask.cpp
#	src/QueryPipeline/RemoteQueryExecutor.cpp
#	src/QueryPipeline/RemoteQueryExecutorReadContext.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp
#	src/Storages/ObjectStorage/StorageObjectStorageSource.cpp
#	src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp
#	tests/integration/test_storage_s3/test.py
#	tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql
@zvonand
Resolve conflicts in cherry-pick of #2145
b6b8545 
Kept both sides where antalya-26.8 and the port add independent lines
(ProfileEvents rows, Settings declarations, obsolete settings, test
configs / experimental-settings exception list, `got_data_from_replica`
plus `replica_has_processed_data`, `read_source_index` plus
`file_meta_info`).

Dropped the cherry-pick's stale duplicates of code that antalya-26.8
already has elsewhere: the second copy of the shutdown `SCOPE_EXIT_SAFE`
block in Server.cpp (only `global_context->stopSwarmMode()` was added
into the existing block) and the duplicated `namespace ErrorCodes` block
in StorageObjectStorageSource.cpp.

Adapted: moved `deserializeFieldFromBinaryRepr` out of
ManifestFileIterator.cpp into Iceberg::IcebergFieldParseHelpers as the port
does, but carrying antalya-26.8's body (Decimal128/256 support added by
acb870c) instead of the port's older implementation.
Adapted: used antalya-26.8's `getSchedulingIdentifier()` helper (added by
cbd81e0) for the `unprocessed_files` keys and log message in
StorageObjectStorageStableTaskDistributor instead of the port's raw
path expressions.
Adapted: kept the `AsyncTaskExecutor` name argument required on
antalya-26.8 in RemoteQueryExecutorReadContext's constructor.

Source-PR: #2145 (#2145)
@zvonand
Cherry-pick of #2146 with unresolved conflict markers (resolution in …
80ee409 
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2146 from Altinity/feature/antalya-26.6/auto-grp-pr-1718

Antalya 26.6: Partition export + cluster functions
# Conflicts:
#	ci/jobs/scripts/integration_tests_configs.py
#	docs/reference/engines/database-engines/datalake.mdx
#	src/CMakeLists.txt
#	src/Common/ErrorCodes.cpp
#	src/Common/ProfileEvents.cpp
#	src/Common/setThreadName.h
#	src/Core/ServerSettings.cpp
#	src/Core/Settings.cpp
#	src/Core/Settings.h
#	src/Databases/DataLake/DatabaseDataLake.cpp
#	src/Databases/DataLake/DatabaseDataLake.h
#	src/Databases/DataLake/DatabaseDataLakeSettings.cpp
#	src/Databases/DataLake/GlueCatalog.cpp
#	src/Databases/DataLake/ICatalog.h
#	src/Databases/DataLake/RestCatalog.cpp
#	src/Databases/DataLake/RestCatalog.h
#	src/Databases/DataLake/S3TablesCatalog.cpp
#	src/Databases/DataLake/UnityCatalog.h
#	src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.cpp
#	src/Interpreters/IcebergMetadataLog.h
#	src/Interpreters/InterpreterSystemQuery.cpp
#	src/Interpreters/PreparedSets.cpp
#	src/Interpreters/PreparedSets.h
#	src/Parsers/ASTSetQuery.cpp
#	src/Parsers/ASTSystemQuery.h
#	src/Parsers/FunctionSecretArgumentsFinder.cpp
#	src/Planner/Planner.cpp
#	src/Processors/QueryPlan/ReadFromRemote.cpp
#	src/Storages/IStorage.h
#	src/Storages/IStorageCluster.cpp
#	src/Storages/MergeTree/MergeTreeData.cpp
#	src/Storages/MergeTree/MergeTreeData.h
#	src/Storages/ObjectStorage/Azure/Configuration.cpp
#	src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/PersistentTableComponents.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/StatelessMetadataFileGetter.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp
#	src/Storages/ObjectStorage/ReadBufferIterator.cpp
#	src/Storages/ObjectStorage/StorageObjectStorage.cpp
#	src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp
#	src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h
#	src/Storages/ObjectStorage/StorageObjectStorageSource.cpp
#	src/Storages/ObjectStorage/registerStorageObjectStorage.cpp
#	src/Storages/ObjectStorageQueue/StorageObjectStorageQueue.cpp
#	src/Storages/ObjectStorageQueue/registerQueueStorage.cpp
#	src/Storages/StorageDistributed.cpp
#	src/Storages/StorageMergeTree.cpp
#	src/Storages/System/StorageSystemIcebergHistory.cpp
#	src/Storages/System/attachSystemTables.cpp
#	src/Storages/buildQueryTreeForShard.cpp
#	src/Storages/buildQueryTreeForShard.h
#	src/TableFunctions/TableFunctionObjectStorage.cpp
#	src/TableFunctions/TableFunctionObjectStorageCluster.cpp
#	src/TableFunctions/TableFunctionRemote.h
#	tests/integration/test_database_iceberg/test.py
#	tests/queries/0_stateless/01625_constraints_index_append.reference
#	tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql
#	tests/queries/0_stateless/03745_system_background_schedule_pool.reference
@zvonand
Resolve conflicts in cherry-pick of #2146
c7553f2 
Kept antalya-26.8 behaviour everywhere the cherry-pick only carried
pre-existing antalya-26.6 code, and translated the PR's real changes to
the current shape of antalya-26.8:

- Append-only registries (ErrorCodes, ProfileEvents, setThreadName,
  Settings/ServerSettings, obsolete settings): kept only the rows the PR
  adds, renumbering the new error codes after the ones antalya-26.8
  already uses.
- `IcebergSchemaProcessor`: took the PR's context-carrying signatures
  (`addIcebergTableSchema(ptr, context)`,
  `getSchemaTransformationDagByIds(context, ...)`, `getSimpleType(name,
  context, ...)`) and kept antalya-26.8's `getClickHouseTableSchemaById`
  spelling, adapting the call sites accordingly.
- `generateManifestFile`: appended the PR's `per_file_stats` after the
  extra defaulted parameters antalya-26.8 added, and named the
  intermediate defaults at the PR's call site.
- `StorageObjectStorageConfiguration::initialize` became an instance
  method, so the remaining static call sites (StorageURL,
  TableFunctionURL, registerQueueStorage) were converted.
- `RestCatalog`/`DeltaSharingCatalog`/`HorizonCatalog` (antalya-26.8-only
  subclasses) gained the `namespaces` constructor argument the PR adds to
  the base catalog.
- `MergeTreeData::changeSettings`: dropped the now-unused
  `has_storage_policy_changed` flag, since the PR replaced the
  conditional `startBackgroundMovesIfNeeded()` with an unconditional
  `startBackgroundMoves()` (exports run on the moves assignee).
- `registerTableFunctionObjectStorage` / `registerTableFunctionIceberg` /
  `registerTableFunctionDeltaLake` / `registerTableFunctionHudi`: dropped
  the registrations the PR moved into the new
  `TableFunctionObjectStorageClusterFallback`, keeping the antalya-26.8
  documentation for the functions that stay.
- Dropped the parts of "theirs" that antalya-26.8 already implements in
  another shape (lazy `insertRowToLogTable`, catalog-aware
  `getLatestMetadataFileAndVersionWithCatalog`, the parallel-replicas
  branch in `DatabaseDataLake::tryGetTableImpl`, the
  `RewriteInToGlobalInVisitor` copy in StorageDistributed) and the
  antalya-26.6-only test bodies that do not apply here
  (`test_partitioning_by_time`, the second result block of
  01625_constraints_index_append).

Source-PR: #2146 (#2146)
@zvonand
Merge pull request #2200 from Altinity/feature/antalya-26.6/pr-1747
fefaff6 
Antalya 26.6: Fix rescheduleTasksFromReplica

Source-PR: #2200 (#2200)
@zvonand
Cherry-pick of #2201 with unresolved conflict markers (resolution in …
d72fb88 
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2201 from Altinity/feature/antalya-26.6/pr-1748

Antalya 26.6: Fix file identifier in rescheduleTasksFromReplica
# Conflicts:
#	src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp
@zvonand
Resolve conflicts in cherry-pick of #2201
6ccd444 
antalya-26.8 already carries the file-identifier unification the source PR
introduces as StorageObjectStorageStableTaskDistributor::getFileIdentifier:
the free function getSchedulingIdentifier(object_info, send_over_whole_archive)
is already used at all call sites, including rescheduleTasksFromReplica, and
uses the newer archive identifier scheme from "Preserve URL shard identity in
object tasks" (cbd81e0). Kept "ours" at every conflict and routed the
source PR's new code through the existing helper.

Adapted: source PR's getFileIdentifier(file) call in getNextTask routed to the
equivalent getSchedulingIdentifier(file, send_over_whole_archive) already on
antalya-26.8; the duplicate helper and its header declaration were removed.

Source-PR: #2201 (#2201)
@zvonand
Merge pull request #2209 from Altinity/feature/antalya-26.6/pr-1843
df8ff97 
Antalya 26.6: Make the export task timeout 24 hours instead of 1 hour

Source-PR: #2209 (#2209)
@zvonand
Merge pull request #2210 from Altinity/feature/antalya-26.6/pr-1832
42619eb 
Antalya 26.6: Add commit info to partition exports table

Source-PR: #2210 (#2210)
@mkmkme @zvonand
Merge pull request #2218 from Altinity/bugfix/antalya/26.6/export-inc…
555167c 
…orect-order-partition-key

Antalya 26.6 Export partition - check name and pos of columns in partition key

Source-PR: #2218 (#2218)
@mkmkme @zvonand
Merge pull request #2229 from Altinity/feature/antalya/26.6/allow_non…
229cd78 
…_matching_schema_export_partition_by_position

Atnalya 26.6 EXPORT PARTITION with position matching + extra columns in source

Source-PR: #2229 (#2229)
@arthurpassos @zvonand
Merge pull request #2253 from Altinity/feature/antalya-26.6/export-pa…
f4c1fa5 
…rtition-monotonic-compat

Export partition (Antalya 26.6) - allow non matching partition expressions in case we can prove the destination expression does not split the data

Source-PR: #2253 (#2253)
@zvonand
Cherry-pick of #2249 with unresolved conflict markers (resolution in …
b970bee 
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2249 from Altinity/fix/join-filter-pushdown-through-rename

Fix join filter pushdown
# Conflicts:
#	src/Analyzer/Utils.cpp
#	src/Processors/QueryPlan/Optimizations/filterPushDown.cpp
@zvonand
Resolve conflicts in cherry-pick of #2249
20cf1dc 
Kept antalya-26.8's ANY INNER JOIN pushdown guard (used later in the function) after the new canPrefilterJoinSide assignments; dropped the logical_join Left/Right lines the PR replaces. Used std::deque<QueryTreeNodePtr> instead of the QueryTreeNodesDeque alias, which does not exist on antalya-26.8, and dropped the Core/Streaming/CursorTree_fwd.h include (context from the source branch, not part of the PR diff; the header does not exist here).

Source-PR: #2249 (#2249)
@zvonand
Merge pull request #2197 from Altinity/feature/antalya-26.6/cluster_d…
19a8472 
…iscovery_update

Reload cluster discovery settings from remote_servers without restart.

Source-PR: #2197 (#2197)
@zvonand
Merge pull request #2294 from Altinity/fix/antalya-26.6/export-skip-p…
3754151 
…olicy-commit-marker

Partition export - fix incomplete data on multi file export after restart

Source-PR: #2294 (#2294)
@zvonand zvonand added releasy Created/managed by RelEasy antalya-26.8 Session label (releasy session config) forwardport This is a frontport of code that existed in previous Antalya versions ai-resolved Port conflict auto-resolved by Claude labels Sep 17, 2026
@zvonand zvonand mentioned this pull request Sep 17, 2026
Open
27 tasks
@zvonand
Merge origin/antalya-26.8 into feature/antalya-26.8/auto-grp-pr-2141
1187302 
# Conflicts:
#	src/Databases/DataLake/DatabaseDataLake.cpp
#	src/Interpreters/InterpreterOptimizeQuery.cpp
#	src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp
#	src/Storages/ObjectStorage/StorageObjectStorage.cpp
#	src/Storages/ObjectStorage/StorageObjectStorage.h
#	src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp
#	src/Storages/ObjectStorage/StorageObjectStorageCluster.h
#	src/Storages/StorageFileCluster.cpp
#	src/Storages/StorageFileCluster.h
#	src/Storages/System/StorageSystemIcebergFiles.cpp
#	src/Storages/buildQueryTreeForShard.cpp
#	tests/integration/test_database_iceberg/test.py
#	tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql
@github-actions

github-actions Bot commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown

Workflow [PR], commit [23dc27f]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ai-resolved Port conflict auto-resolved by Claude antalya-26.8 Session label (releasy session config) forwardport This is a frontport of code that existed in previous Antalya versions releasy Created/managed by RelEasy

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zvonand @mkmkme @arthurpassos