Skip to content

feat(index): add SeekDB derived-index backend - #449

Closed
knqiufan wants to merge 5 commits into
EverMind-AI:mainfrom
knqiufan:codex/feat-seekdb-index-backend
Closed

knqiufan wants to merge 5 commits into
EverMind-AI:mainfrom
knqiufan:codex/feat-seekdb-index-backend

Conversation

@knqiufan

@knqiufan knqiufan commented Sep 15, 2026

Copy link
Copy Markdown

Summary

Adds SeekDB as a third rebuildable derived-index backend while preserving
LanceDB as the default and keeping Markdown as the source of truth.

  • route all seven typed index repositories and lifecycle operations through a
    new SeekDB adapter;
  • support embedded SeekDB on Linux/macOS and remote seekdb Server or OceanBase
    connections, with optional dependency extras and validated configuration;
  • map portable schemas to typed scalar, JSON, whitespace-tokenized full-text,
    and cosine HNSW vector columns/indexes;
  • provide safe SQL rendering, exact UTC epoch-millisecond conversion, batched
    writes, native pagination, BM25 search, vector search, and strict column/index
    drift diagnostics;
  • guard embedded directory ownership, retain connection locks until cancelled
    driver calls finish, restore physical-session settings after reconnects, and
    suppress raw driver exception chains in application logs;
  • integrate cascade rebuild, configuration examples, architecture/storage
    docs, changelog, unit/contract/integration/e2e fixtures, and both embedded and
    latest remote SeekDB CI jobs.

Closes #448.

Area

  • Architecture method
  • Benchmark
  • Use case
  • Documentation
  • Developer experience
  • CI, build, or release

Verification

  • Local Windows validation: 81 passed for SeekDB unit tests,
    and configuration tests.
    A process-local fcntl import shim was used only to collect portable tests;
    actual flock calls still raise. Real-engine tests require Linux/macOS or a configured remote endpoint and
    were not executed locally.
  • Added cancellation/query/shutdown ordering, repeated cancellation, failed ping,
    physical connection replacement, and caller-side exception-log privacy tests.
  • Real-engine validation also covers SeekDB internal full-text/vector catalog
    metadata without treating engine-generated backing indexes as schema drift.
  • Added a real remote regression that replaces the physical connection with
    NO_BACKSLASH_ESCAPES enabled and verifies text round trips after recovery.
  • Passed repository-wide Ruff lint/format, all four import-linter contracts,
    asset/file-size gates, deprecated-name/contributor-doc gates, datetime discipline,
    and documentation checks.
  • Real-engine checks passed on the latest commit in the
    fork CI run:
    • Embedded SeekDB 1.4: 25 passed, 1 remote-only test skipped for repository/
      port contracts; 27 passed for the tiered API suites.
    • Remote SeekDB latest: 26 passed for repository/port contracts, including
      the connection replacement regression; 27 passed for tiered API suites.
    • Lint, package build, and the complete Python 3.12/3.13 unit and general
      integration suites passed.
    • The existing Milvus job fails before tests because the upstream compose
      references an unavailable minio/minio image; this run is not fully green.
  • The CI workflow supports manual dispatch for fork validation. Upstream PR
    workflows require maintainer approval; pending approval is not a passing check.

Checklist

  • I kept the change scoped to the relevant area.
  • I am opening this from a separate branch, not pushing directly to main.
  • I updated docs, examples, or setup notes when behavior changed.
  • I added or updated tests when the change affects behavior.
  • I did not commit secrets, .env files, dependency folders, or generated output.
  • Active relative links in Markdown files resolve.

Notes for Reviewers

The adapter isolates the SQL-oriented pyseekdb 1.4.x client surface in one
connection manager and pins the optional dependency below 1.5 so future client
surface changes cannot silently alter the contract. Schema drift is fail-fast
because every physical table is disposable and rebuilt from Markdown with
everos cascade rebuild.

The default install and default [index] backend = "lancedb" behavior are
unchanged.

By submitting this pull request, I agree that my contribution is licensed under
the Apache License 2.0.

@knqiufan knqiufan closed this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Add SeekDB as an optional derived-index backend

1 participant