Skip to content

feat(rag): prepare local E5 corpus and retrieval for staging - #95

Draft
SHcommit wants to merge 11 commits into
developfrom
feature/local-e5-embeddings
Draft

SHcommit wants to merge 11 commits into
developfrom
feature/local-e5-embeddings

Conversation

@SHcommit

@SHcommit SHcommit commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Refs #94 (does not close).

Status: held for decomposition and staging evidence

This branch currently contains local E5 corpus preparation, retrieval changes, and the pg_search migration. Per review, these changes need to be split into reviewable exporter/corpus, search-runtime, and BM25 migration units before merge. Keep this PR draft until that split is ready. CI status alone is not rollout approval.

Verified data state (read-only, 2026-11-20)

  • Current kto-korean-tour source pointer ffda4e5f-3315-4f25-8a8c-d8825ba69ae0 contains 23,573 places. Production AI active revision rev-20261115025249 has 23,573 documents/chunks/embeddings, and its public place ID set exactly matches the current pointer export and the local 23,573-row spool.
  • The raw status='ACTIVE' union count is 23,600 because 27 places are only present in obsolete ACTIVE revision e3362d30-f668-41da-8496-ff84e58afa30. The 23,600 union spool includes those stale rows and must not be staged. The current-pointer 23,573 spool is the reconciled candidate, still local and un-staged.
  • Production has pg_search 0.27.0 but no chunks_search_bm25_idx; no migration or ANALYZE was applied.
  • No staging DSN, production-data snapshot/backup, or compatible staging image is available in this session. No production writes were performed.

Embedding profile decision

Keep ONNX INT8 as a separate profile and corpus revision: multilingual-e5-small-384-onnx-int8@614241f622f5. Do not mix it with the existing DB profile label multilingual-e5-small-384@614241f622f5. The deployed AI Engine image revision c51530249b5098f9a6aeae7198633f29cfd06c81 uses Gemini query embeddings, while the active DB is tagged E5 FP32. On 120 exact stored chunk texts, local E5 ONNX passage vectors had median cosine -0.0012 against stored vectors (p90 0.0412), so the DB profile label does not establish vector provenance. Exact historical provenance remains unknown; regenerating FP32 to match the label would be unsafe without resolving it.

Search behavior and tests

  • Hybrid currently invokes BM25 and then vector retrieval sequentially. Architecture, request lifecycle, observability, and telemetry docs now describe that behavior. Parallel execution is deferred until 2GB staging measurements compare p95, CPU, and DB connections.
  • Added gated PostgreSQL integration tests for Korean lexical search, dense search, hybrid/router modes, filters, top-K order, empty results, and query dimension mismatch. They require ONMARU_TEST_POSTGRES_DSN pointing to a disposable PostgreSQL 17 + pgvector + pg_search 0.27.0 database; the shared fixture drops the ai schema. Local environment lacks that image/DSN, so all 7 integration cases skipped. Synthetic tests are not search-quality evidence.
  • Local verification: 153 search/router tests passed; 7 DB integration cases skipped; Ruff, mypy, and git diff --check passed. Prior full CI passed before the integration-test/doc follow-up; the current head check is pending.

Required before merge / staging

  1. Split exporter/corpus prep, search-runtime, and BM25 migration into separate review units; update this PR and link each PR to [Embedding] 로컬 multilingual-e5-small ONNX 임베딩 및 재개 가능한 corpus staging #94 as appropriate.
  2. Restore an operational snapshot into isolated staging. Apply 004_pg_search_bm25.sql there only, recording index build RSS, disk, and ANALYZE results on the 2GB target.
  3. Run real-corpus Korean query/filter/top-K checks and compare BM25, dense, and hybrid quality on a fixed labeled set. Record Recall@10/nDCG@10; test vectors and unit tests do not substitute for this.
  4. Keep production unchanged and Plan/Execute disabled until staging review, reviewed-fact approval, healthy revocation heartbeat, version sync, and E2E gates pass. Production writes require the operator's separate approval.

@SHcommit SHcommit changed the title feat(rag): stage local E5 ONNX corpus and BM25 feat(rag): prepare local E5 corpus and retrieval for staging Nov 20, 2026
@SHcommit
SHcommit marked this pull request as draft November 20, 2026 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant