Conversation
…next commit) --- Original cherry-pick message follows: Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2 Iceberg deletion vector support (attempt #2) # Conflicts: # docs/en/sql-reference/table-functions/icebergCluster.md # docs/reference/statements/system.mdx # src/Access/Common/AccessType.h # src/AggregateFunctions/AggregateFunctionGroupBitmapData.h # src/Common/ProfileEvents.cpp # src/Core/Settings.cpp # src/Interpreters/InterpreterSystemQuery.cpp # src/Parsers/ASTSystemQuery.cpp # src/Parsers/ASTSystemQuery.h # src/Parsers/ParserSystemQuery.cpp # src/Processors/Formats/Impl/Parquet/Reader.cpp # src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h # src/Storages/ObjectStorage/IObjectIterator.h # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp # tests/queries/0_stateless/01271_show_privileges.reference # tests/queries/0_stateless/04117_parser_system_query_variants.reference # tests/queries/0_stateless/04117_parser_system_query_variants.sql
Kept the antalya-26.8 side for everything outside the source PR's scope (point-in-polygon / paimon cache entries, lazy-materialization `rows_to_read`, identity partition columns, `_headers` count-cache guard, 26.8 path resolution without secondary storages, existing manifest-count and snapshot-summary comments) and applied the PR's deletion-vector changes on top of it. Also moved the PR's `use_puffin_files_cache` settings-history row into the branch's own (empty) `26.8.1.20001.altinityantalya` block: the three-way merge had spliced the PR's `26.6.2.20001.altinityantalya` block into the middle of the `26.8` row list, breaking the file. Applied the PR's 6-line `icebergCluster` deletion-vector docs section to `docs/reference/functions/table-functions/icebergCluster.mdx`, the current location of that page on antalya-26.8, instead of re-adding the whole pre-restructure `docs/en/sql-reference/table-functions/icebergCluster.md` file that the cherry-pick resurrected. Adapted: Parquet `Reader::prefilterAndInitRowGroups` lazy-materialization slice and the trailing `rows_to_read` bound now read `global_offsets[...]` / `global_offsets.back()`, because the PR replaces the running `total_rows` counter with `buildRowGroupGlobalOffsets()` Adapted: `InterpreterSystemQuery::getRequiredAccessForDDLOnCluster` emplaces `SYSTEM_DROP_PUFFIN_FILES_CACHE` in its own case, because antalya-26.8 replaced the grouped fall-through with one `required_access.emplace_back` per cache type Adapted: the PR's delete-file classification (deletion vectors vs parquet position deletes) is applied in `IcebergIterator::decodeDeleteManifests()`, because antalya-26.8 moved delete-manifest decoding out of the constructor into that lazy method Adapted: deletion-vector loading uses `object_storage` + `persistent_components.path_resolver.resolve(...)` instead of `resolveObjectStorageForPath(..., *secondary_storages, ...)`, which does not exist on antalya-26.8 Adapted: `ManifestFileEntriesHandle::getRowsCountInAllFilesExcludingDeleted` keeps antalya-26.8's `std::optional<UInt64>` signature while delegating to the PR's shared `getRecordCountInAllFilesExcludingDeleted` helper Adapted: `IcebergMetadata::totalRows` keeps antalya-26.8's delete-file bail-out, comment and mismatch warning, and only adds the PR's `common::addOverflow` guard on the cross-manifest total Adapted: `canUseCountFromFilesCache` replaces antalya-26.8's `hasAttachedDeletes` gate (same purpose, both call sites), which is what the PR's `effective_need_only_count` refinement supersedes; antalya-26.8's `rows_to_read` and `_headers` guards are preserved Adapted: added `ObjectInfo::clone()` / `IcebergDataObjectInfo::clone()` (two lines, verbatim from the source branch) as the minimum scaffolding for the PR's new `gtest_iceberg_data_object_info_clone` test; that virtual came to antalya-26.6 via #2154 (upstream ClickHouse#90740) and is not on antalya-26.8 Adapted: `#include <Poco/String.h>` in AvroForIcebergDeserializer.cpp, which antalya-26.8 does not include but the PR's `Poco::toLower(file_format)` needs Source-PR: #2183 (#2183)
…next commit) --- Original cherry-pick message follows: Merge pull request #2271 from Altinity/iceberg-delta-bin-deletion-vectors Support for Databricks deletion vectors # Conflicts: # docs/en/sql-reference/table-functions/icebergCluster.md
The legacy Docusaurus doc docs/en/sql-reference/table-functions/icebergCluster.md was removed on antalya-26.8 (commit 9440ca7, new layout docs/reference/...mdx), so the cherry-pick re-added it as a modify/delete conflict. Applied the PR's one-line deletion-vector wording change to the current file docs/reference/functions/table-functions/icebergCluster.mdx instead and dropped the re-added legacy copy. Source-PR: #2271 (#2271)
zvonand
added
releasy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add read support for Iceberg deletion vectors, applying Puffin
deletion-vector-v1bitmaps when querying Iceberg tables over local, object storage, and cluster reads, together with newPuffinandPuffinMetadatainput formats and a configurable process-global Puffin files cache that can be cleared withSYSTEM DROP PUFFIN FILES CACHE. Deletion vectors stored by Databricks in standalone.binfiles are also recognised and applied.Add read support for Iceberg deletion vectors, applying Puffin
deletion-vector-v1bitmaps when querying Iceberg tables over local, object storage, and cluster reads, together with newPuffinandPuffinMetadatainput formats and a configurable process-global Puffin files cache that can be cleared withSYSTEM DROP PUFFIN FILES CACHE. Deletion vectors stored by Databricks in standalone.binfiles are also recognised and applied (#2183 by @ianton-ru, #2271 by @ianton-ru).CI/CD Options
Exclude tests:
Regression jobs to run:
Combined port of 2 PR(s) (group
auto-grp-pr-2183). Cherry-picked from #2183, #2271.