Repository navigation
Conversation
SHcommit
marked this pull request as draft
November 20, 2026 07:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #94 (does not close).
Status: held for decomposition and staging evidence
This branch currently contains local E5 corpus preparation, retrieval changes, and the pg_search migration. Per review, these changes need to be split into reviewable exporter/corpus, search-runtime, and BM25 migration units before merge. Keep this PR draft until that split is ready. CI status alone is not rollout approval.
Verified data state (read-only, 2026-11-20)
kto-korean-toursource pointerffda4e5f-3315-4f25-8a8c-d8825ba69ae0contains 23,573 places. Production AI active revisionrev-20261115025249has 23,573 documents/chunks/embeddings, and its public place ID set exactly matches the current pointer export and the local 23,573-row spool.status='ACTIVE'union count is 23,600 because 27 places are only present in obsolete ACTIVE revisione3362d30-f668-41da-8496-ff84e58afa30. The 23,600 union spool includes those stale rows and must not be staged. The current-pointer 23,573 spool is the reconciled candidate, still local and un-staged.pg_search0.27.0 but nochunks_search_bm25_idx; no migration or ANALYZE was applied.Embedding profile decision
Keep ONNX INT8 as a separate profile and corpus revision:
multilingual-e5-small-384-onnx-int8@614241f622f5. Do not mix it with the existing DB profile labelmultilingual-e5-small-384@614241f622f5. The deployed AI Engine image revisionc51530249b5098f9a6aeae7198633f29cfd06c81uses Gemini query embeddings, while the active DB is tagged E5 FP32. On 120 exact stored chunk texts, local E5 ONNX passage vectors had median cosine-0.0012against stored vectors (p900.0412), so the DB profile label does not establish vector provenance. Exact historical provenance remains unknown; regenerating FP32 to match the label would be unsafe without resolving it.Search behavior and tests
ONMARU_TEST_POSTGRES_DSNpointing to a disposable PostgreSQL 17 + pgvector + pg_search 0.27.0 database; the shared fixture drops theaischema. Local environment lacks that image/DSN, so all 7 integration cases skipped. Synthetic tests are not search-quality evidence.git diff --checkpassed. Prior full CI passed before the integration-test/doc follow-up; the current head check is pending.Required before merge / staging
004_pg_search_bm25.sqlthere only, recording index build RSS, disk, andANALYZEresults on the 2GB target.