Skip to content

[python] Keep local search and lookup on one snapshot - #9843

Open
TheR1sing3un wants to merge 3 commits into
apache:masterfrom
TheR1sing3un:codex/search-snapshot-consistency
Open

TheR1sing3un wants to merge 3 commits into
apache:masterfrom
TheR1sing3un:codex/search-snapshot-consistency

Conversation

@TheR1sing3un

@TheR1sing3un TheR1sing3un commented Sep 15, 2026

Copy link
Copy Markdown
Member

Purpose

Keep each local multimodal search execution on one snapshot. A commit between candidate search and result lookup can currently drop a selected row or return its updated values even when they violate the original pre-filter. Hybrid routes can also plan against different commits, and full-text reads lose the planning snapshot.

Capture an execution-local read view and carry it through vector, batch, full-text, hybrid, and result lookup paths. Reusing a query resolves a fresh view; explicit snapshot, tag, timestamp, and scan.version selectors remain effective. Retain resolved snapshot metadata for tags and preserve an empty view across the first concurrent commit. Share the read-view helper with existing snapshot-aware readers without mutating reusable queries or builders.

Tests

The latest full CI run passed, including Rust main 66f655b30a38: 5,204 tests passed, 121 skipped, and 2,250 native plans exercised.

  • Regression coverage includes concurrent delete/update before lookup, query reuse, hybrid route planning, empty-table append, snapshot/tag/timestamp/scan.version selection, retained tag metadata, read-view copies, branch switching, full-text live-row/raw reads, and native indexed-vector lookup.
  • Updated the existing PK snapshot scan test to exercise the shared table-view helper and assert that source scanning and index planning retain the same snapshot.
  • After syncing with master, the snapshot, PK definitions, expanded native-planner, and incremental-read suites passed locally: 74 passed, 35 skipped (native runtime unavailable in that Python 3.9 environment). The integrated explain/profile branch also passed the broader related regression run: 399 passed, 47 skipped.
  • Changed Python files pass repository-configured flake8. License-header checks and git diff --check pass.
  • Full Python matrix and Rust-main validation: https://github.com/apache/paimon/actions/runs/34991660805

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant