Repository navigation
Conversation
Register zapx v18 as the default segment plugin, with v11-v17 still registered so existing indexes stay readable. go.mod points zapx/v18, scorch_segment_api, vellum and freeway at the local checkouts of the branches this is built against. zapx v18 doesn't implement the optimizable postings interfaces, so the "disjunction:unadorned" optimization finds nothing to work with: * search_multi_term.go: when it can't optimize a batch of term searchers, a multi term query over more terms than DisjunctionMaxClauseCount fell over with "unable to optimize". It now falls back to a plain disjunction over the batch (optimizeOrDisjunct). * search_term_range_test.go: the test counts searchers started, which depends on whether the optimization applies; it accepts both counts. * index_test.go: byte counts of the v18 format differ from v17's, and the bytes read are not bit-for-bit deterministic across runs (the order docs are numbered in within a batch is not), so the read checks use a tolerance (bytesReadSame) and the baselines are v18's. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ueries
A second way to run a search, alongside the existing one, for the requests it
serves; anything else is run the old way, untouched. It works with the segments
of the index rather than hiding them, hands scalars (doc number, term frequency,
norm, score) from the segment's blocks up to the collector 128 at a time, and
scores them with SIMD kernels.
Which requests: a term query, a match of terms (no fuzziness), or conjunctions
and disjunctions of those, sorted by descending score, without explanations,
locations, highlights, facets, KNN, search after/before, pre search data,
synonyms, nested documents, score fusion or a custom match handler. Whether the
index can do it is up to the index: it needs segments that hand out block
cursors (zapx v18); otherwise the regular searchers are built.
Structure
* index/scorch: PerSegmentIndexReader and the per segment term field reader, a
concrete type on top of the segment's block cursor. Readers, and the
dictionaries they use, are pooled (see the notes on the pool and on the
MB-64604 hazard in snapshot_index_tfr_perseg.go).
* search/perseg_collect.go: the contract between searcher and collector, with
scalars only. The collector is generic: it calls the searcher's NextBlock until
it is exhausted, puts the matches in the heap, and merges. A searcher that can
do better offers CollectOptimized, handed a sink (the heap shared by all
segments, so a later segment starts from the threshold of the earlier ones).
* search/scorer: PerSegmentTermScorer, tf-idf and bm25 on freeway's float32
kernels, with upper bounds for pruning.
* search/searcher: PerSegmentTermSearcher (top-k with block skipping; no score
with an exact total from the postings count), and the conjunction and
disjunction searchers, with generic cursors for any nesting and:
- disjunction: block-max WAND, with the coord factor in the bound; a bitset
union for no score;
- conjunction: tantivy's block windows around the rarest term, and its
dense/sparse count.
Pruning makes the total a lower bound (TotalRelation gte); a search for no
hits is a count and is exact. Scores are float32, within float32 precision of
the regular path's.
* search/collector: PerSegmentTopNCollector.
* index_impl.go: the request opts in (search.PerSegmentSearchKey) when it's
eligible, the TopN collector is only built when the regular path is taken.
* earlystop_test.go: an exact total may say "eq"; what's not allowed is calling a
lower bound exact.
Tests: differential tests against the regular path through real requests; an
oracle in plain code for composites, a stress test and a direct test of the
soundness of the pruning windows; concurrency under -race for the pools;
benchmarks of latency and of allocations (go test -bench SearchAllocs).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Two profiles of the per segment path (CPU, BM25) put most of the cost of the AND and OR searchers in the term cursor: * Seek: almost all of its self time was the binary search inside the decoded block, a coin toss for the branch predictor, repeated for every candidate of every other term. It now looks at the next posting first (seeks ascend and are usually a few apart), and otherwise does a search with no branch on the comparison. * ShallowSeek: WAND calls it for each pivot, and every call recomputed an upper bound (a scalar BM25 score: a sqrt and two divisions) and searched the skip data again, though the pivots stay in one block for a long time. The bound of a block is now kept while the targets go up inside it. On the 200k doc benchmark (BM25): AND of 3 very common terms 1.86 -> 1.64ms, OR of 5 terms 1.28 -> 1.01ms, wandSegment's share of the search 11.6% -> 4.4%. Also BenchmarkSearchCPU (PERSEG_SHAPE=term|and|or PERSEG_MODEL=bm25), for CPU profiles of one shape at a time. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…to the searchers More queries take the per segment path, and the decision of whether to is no longer made by each query. Which queries * BooleanQuery: must-only and should-only are the conjunction and disjunction themselves; the mixed shapes (must + should, anything with a must not) are a new PerSegmentBooleanSearcher with BooleanSearcher's semantics -- a should whose own minimum is above 0 is required -- and no pruning, as in tantivy, which only prunes plain unions and intersections of terms. * BoolFieldQuery is a term; QueryStringQuery is its parsed query. SupportsPerSegment (the shape check done before opting in) knows both, and a boolean with a filter, or only a must not, is out. * A one clause disjunction or conjunction is its clause, to the bit (a match of a single token is one): unwrapped, so that an OR of matches is an OR of terms and gets WAND, an AND the block windows. * Without scores every composite stops once it has the first matches, as the regular path does, with the total a lower bound; a search for no hits counts. Where the decision is made Not in the Query.Searcher methods, which are back as they were, but in the constructors of the searchers: NewTermSearcherBytes, which all terms go through, and those of the conjunction, disjunction and boolean searchers (the last as NewBooleanSearcherOrPerSegment, as NewBooleanSearcher has a concrete type). They build the per segment kind when the request has opted in and what they're given is of that kind (perseg_dispatch.go). A tree that is a mix of the two kinds is an explicit error, ErrPerSegmentMixed, not reachable for the shapes that opt in. Regular searchers that can't match anything -- a match of a text with no tokens is one -- go with either kind, and are made into searchers of no matches; whether a should clause is required is read from the clause before that. Found on the way: the regular path leaves the score of an optional should clause out of some of the docs it matches when a boolean is nested in a conjunction (its own explanation of such a doc has no component for the clause; which docs depends on how they are numbered). The per segment path scores them right, and the test for that shape asks for the same docs and no lower score. Tests: old vs new through real requests for boolean, bool field and query string shapes, with and without scores; clauses that match nothing; the clause limit; the guard against mixing the kinds. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… for ANDs A lone scored term skips the blocks that can't make the heap: once it is full, the skip data bounds the next block and a block whose bound is at or below the worst hit is passed over undecoded. A segment without deletions still counts exactly (the term's count is its matches); with deletions a segment that skipped reports what it visited, and the search is marked as pruned (total is a lower bound). A plain OR of terms uses block-max MAXSCORE, after tantivy's block_maxscore.rs (as ParadeDB's fork has it), adapted to bleve's scoring: bounds carry the coord factor, scores are summed in the order of the query so a doc scores the same whatever the algorithm, and the split between terms that can drive a match and those that can't is redone for each batch, with small batches until the heap has a threshold. Working memory is pooled. WAND stays for queries whose terms have too few postings for batches to pay. An AND works a narrow window on bitmaps when enough of the leader's postings could make the heap: the candidates and each term's postings are put in bitmaps, ANDed, and the matches scored from lanes of block scores. Everything else takes the seek path as before. The AND's working memory has a pool of its own: its lanes aren't cleaned, so they can't be shared with MAXSCORE's. Tests compare each algorithm's top k with the regular path's ranking of every match on random queries, with and without deletions, for both scoring models; the bound of a window over several blocks has a unit test of its own. The benchmarks take PERSEG_DISJ, PERSEG_CONJ_MINCAND, PERSEG_SIZE and PERSEG_CPUPROF. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A search with Explain set is no longer kept off the per segment path. Nothing is explained while matches are found and scored: the collector asks the searcher (search.PerSegmentExplainer) to explain each hit it returns, once the searcher has been read, and no others. A term looks its doc up in its postings (PerSegmentIndexSnapshotTermFieldReader. Probe, with a cursor of its own) and has its scorer build the explanation TermQueryScorer would, with the same messages and structure. A conjunction, disjunction or boolean explains its clauses and puts them together with the same float32 sums, in the same order, as it scores with, so the value at the root of an explanation is the hit's score to the bit, whichever algorithm found the hit. The clauses are listed in the order the regular searchers list them (by count, the sort they use), and the wrappers around one clause composites, which are unwrapped for speed, are put back in the explanation. Two small differences from the regular path's explanations: the bm25 saturation message has the term in it (the regular one leaves it empty), and the clauses of a disjunction of more than DisjunctionHeapTakeover are listed in query order, where the regular heap lists them in an order of its own. The tests compare, for every kind of query, each algorithm and both scoring models, with and without deletions, the explanation of each hit with the regular path's, and its root with the hit's score; and check that a collector explains nothing until the searcher is exhausted, and only the page. Asking for explanations costs about 1us a hit for a term, 5us for an OR of five. TestBM25GlobalScoring compared the scores of a search that is now served by the per segment path (float32) with one that isn't (float64): it now allows for float32 precision. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The generic loop (NextBlock, and the collection that has to visit every match) moved a cursor from match to match: a union looked for the lowest doc of all its terms at every step, through calls, and scored each match with the scalar scorer. That cost about 40-50ns a match for an OR or an AND, against 3ns for a lone term. A plain OR of terms is now read a window of 4096 docs at a time, as tantivy's BufferedUnionScorer does: each term puts its postings in the window in a bitmap and adds their block scores to the slot of their doc, in the order of the query, and the matches are the bits, taken lowest first. A block of matches is filled without a call for each. An AND of terms none of which has more than 8 times the postings of the one with the fewest is done the same way, the bitmaps ANDed (tantivy's intersection has no such thing: it seeks, as our other path does, which stays for terms that differ more). Scores are the sums the other paths make, in the same order, so a doc scores the same whichever way it was found. With every match visited (a search for no hits, ORs of 5 and 10 terms, ANDs of 3 and 4): the OR of 5 goes from 6.8ms to 2.0ms, of 10 from 15.5ms to 3.2ms, the AND of 3 from 2.7ms to 1.4ms, of 4 from 3.0ms to 1.6ms. The tests that compare the algorithms with the regular path's ranking of every match now run on segments of 9000 docs, more than a window (4096) and the outer window of MAXSCORE (8192): they had only one window each, so the edges between windows were never crossed. New tests read ORs and ANDs through the generic loop and compare every hit and the exact total. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Tuning knobs and shared parts - The knobs that force an algorithm (disjunction algorithm, the AND's bitmap threshold) are atomics in perseg_tuning.go, safe to change while searching. - Window constants, the four kinds of scratch memory and the invariants of their pools are in perseg_scratch.go, with one table saying what each must leave behind. Duplicated pieces are gone: the skew rule, scatter / scatterAdd (MAXSCORE adds into a zeroed lane), the term cursor builders. Buffered cursors - bufferedUnionCursor and bufferedIntersectionCursor have a real Seek instead of advancing one match at a time, and release(), which cleans and returns the scratch. Whoever owns a cursor releases it (the block loop, drainCursor, the unscored loop, Close), so cursors given up on halfway no longer cost a new scratch each. - A Seek just past the window starts a new window; one of more than a whole window hands over to the classic cursor over the same terms, which stays until the cursor has been advanced 32 times in a row. The next window after a seek is small and doubles. Nested and boolean clauses - A clause is given its cursor by segCursorSeeked, told how often its consumer will seek it (about the driver's cost), and reads a window at a time only if it has no more than 4 times that many matches. Booleans read the clause that drives them through, and seek the others. Sought sparsely, a nested OR stays on the classic cursor: a sparse leader over a dense OR was twice as slow otherwise. Boolean 'must + should' 3.3 -> 2.2ms, AND of an OR 5.4 -> 3.2ms, OR of an AND 5.1 -> 4.1ms, the sparse case unchanged. Safety - A runtime panic in an algorithm (an index out of range in a window) fails the search that hit it with an error saying where, and not the process. - The MAXSCORE scratch is given back only when it isn't being worked in: not after an error, and not after a panic, where the check on the error value alone would have returned it dirty. Tests - The oracle tests of the searcher package, which had segments of 2500-3000 docs, now include segments of 9000, more than a window (4096) and than MAXSCORE's outer window (8192). New: the buffered cursors move like the classic ones under random Advance and Seek (far, near, in the window, beyond it, runs of Advance), a released union goes back to its pool all zeros, nested and boolean queries match the regular path on big segments, a runtime panic becomes an error. Comments that still described the old behaviour are updated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cores The explain test now covers booleans with only musts, only shoulds, shoulds and a must not, required shoulds (a minimum), composite clauses and a boolean inside an AND; ORs with a minimum of two; boosted ORs and ANDs; and clauses that are a match of nothing -- 23 shapes in all, each against the regular path's explanation and the hit's score. A search with Explain and without scores (Score: "none") stays on the regular path: there the explanation is a tree of zeros, and the per segment path would have shown the scores the hits do not have. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The fallback path (a search whose searcher has no algorithm of its own for the kind of search it is -- scored booleans, ORs and ANDs with clauses that aren't terms, a minimum above one) took its matches a block at a time. Only the lone term and the windowed cursors had a block to give; the other cursors filled one by calling Advance and Score 128 times, for the collector to unpack again. Read through a union of terms the two cost the same (11.0 ns a match), so the layer was complexity that bought nothing there. PerSegmentSearcher is now NextMatch, which returns the next match -- its segment, its doc across the index and its score -- and false when there are none; the searchers comply with it. DrainPerSegmentSearcher loops over matches. The block type, the block filling of the windowed cursors, and the interface the generic loop used to ask for it are gone. The lone term still reads and SIMD-scores its postings a block at a time, and hands them out one by one. The scored boolean and nested queries, which only have this path, are 0-6% slower for the call per match (e.g. 2.68 -> 2.78ms, 3.13 -> 3.28ms), the sparse leader over an OR and the OR with only a should clause are unchanged. Every other path is untouched. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cher The per segment searchers are built by query.PerSegmentQuery and consumed by the PerSegmentTopNCollector through NextMatch. They no longer implement Searcher (Next/Advance/DocumentMatchPoolSize stubs gone), and the dispatch hooks, the ctx flag and ErrPerSegmentMixed are removed. Searches the path can't serve return search.ErrPerSegmentUnsupported and use the regular path. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ront The term searcher embedded a 512 byte score array that only the NextMatch fallback uses, so every term query (count included) paid for it, and the per segment totals grew one append at a time. Allocate the scores on the first NextMatch that needs them and keep room for 16 segments' totals in the collection state. Count: 1905 -> 1401 B/op, 30 -> 26 allocs; top10: 7256 -> 6751 B/op. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ExplainMatch is a method of PerSegmentSearcher, no longer an optional interface the collector discovers after the search has run. The coord arithmetic of a disjunction lives in one place (disjunctionScore), used by the cursors, the windows, WAND and MaxScore and by the explanation. The collector checks that the root of an explanation is the score the hit was ranked by and fails the search if not. A clause that excludes docs is only asked whether the doc matches (explain with build=false): nothing is built. Tests: explanations agree with the search for every match of every shape, and asking only for a match agrees with asking for the explanation; the collector refuses an explanation of another score. Mutating a scoring site away from the shared arithmetic fails them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A sorted collector for any sort order but descending score alone: PerSegmentSortedCollector keeps the top N the way tantivy's TopNComputer does (a buffer of 2N, a strict threshold, a quickselect when it is full), reads sort values from the doc values as bytes (new optional search.SortValueBytes, which SortField implements) and compares them to the threshold before keeping a copy, and makes DocumentMatches only for the hits returned. It visits every match, so totals are exact, and never asks a searcher for its optimized collection, which prunes by score. Facets, search after/before, locations and sorts of an application's own stay with the regular path. A score as a sort is compared as a float32, and a hit's Score is float32 precision, as on the score ordered path (one test that compared it for equality with a regular path score now allows for that). Measured on 200k docs: 1.07-1.21x for numeric sorts, 1.23-1.59x for string sorts, 20-100x fewer allocations. 97% of the collector's time is reading doc values, 68% of that snappy decoding their chunks, which the regular path pays too. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The sorted collector no longer limits sort orders to the four sorts that come with bleve. A sort that requires scoring is ordered by the hit's score (as TopNCollector does) and the DocumentMatch that sorts are asked the value of now carries the score; any other sort gives its value. The tests use sorts of an application's own (by score, by id, one that reads the score) and fail when either is taken away. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… falling back A search with Explain on and Score "none" is now served by the per segment path like any other: it has no scores to explain, so its hits come without explanations. (It was left to the regular path, whose explanation of scores that were never computed is a tree of zeros.) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The sorted collector counts facets (the classic FacetsBuilder, so results are the regular collector's) for every match, in the same doc values visit as the sort values, and keeps only the hits that rank strictly after the search after values (a tie of them on every sort is not after), skipping nothing by offset. Search before is the regular logic over a reversed sort order, which the path now serves like any other. A request with facets or a search after goes to the sorted collector even when ordered by score, as every match counts and so nothing can be pruned. Also: no sort at all is not score order (hits come in the order found, as in the regular path), and the doc values bytes read are reported as the regular collector reports them (the reader's running total, after each visit), which TestBytesRead depends on. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…32 in the versus test TestPerSegmentSearchMatchesRegularSearchOnLargeSegments compares the two paths on 100000 documents in segments past the window sizes (every kind of request the path serves: scores, explain, sorts, facets, search after/before, unscored), TestPerSegmentSearchThroughAnAlias does it for a search of two indexes, and compareScoreTiedResults is for hits ordered by score and then something else, where float32 scores that are alike are one tie. test.TestDisjunctionSearchScoreIndexWithCompositeFields compared the scores of scorch and upsidedown with !=, which has failed since explanations moved to the per segment path (7c05865): they are the same to float32 precision. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Drop the unused unwrapSingle (its doc moves to unwrapSingleKinds), the unused compareExplanations test helper and a few placeholder declarations; simplify a nil check; and correct comments that predate later changes: the top N collector shares one heap across segments (it was described as merging per segment heaps) and no longer "supports nothing beyond" score order, the file header called the searcher a Searcher. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 4, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Follows the rename of segment.BlockCursorProvider's method in scorch_segment_api and zapx. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
segment.BlockPostingsList, BlockPostingsIterator, BlockMaxPostingsIterator and the BlockIterator method (scorch_segment_api, zapx). The reader's field is blockIter. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
search.PerSegmentBlockLen and scorer.PerSegmentBlockLen were two more copies of the block length: use the one the segment API defines. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… descendants desc, the DocumentMatch that stands in for each descendant, was allocated for every hit, whether it had descendants or not, and most don't. A document without descendants has nothing more to load: return before allocating anything. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A termCursor allocated a 512 byte buffer of its own for the scores of a block, and a query has a cursor per term and segment: for an AND or an OR that was about half of what the search allocated. The buffers now come from a pool and go back when the cursor is released (the releaser contract of the cursors is unchanged: whoever is done with a cursor releases it, and one that isn't is only garbage collected). The loops over segments release a segment's cursors when it is done, the composite cursors pass the release on to their clauses, and the buffered cursors, which share their terms with the fallback they switch to, release the terms only when they themselves are released, not when they give their scratch back. Releasing is harmless at any time and any number of times: a cursor that is asked for scores again takes a buffer again and works the block's scores out again. The tests read the scores of the whole block (not Score, which doesn't use the buffer), releasing in the middle and overwriting the pool's buffers meanwhile, and run cursors of several goroutines against the pool (with -race). AND: 24738 -> 10322 B/op, 151 -> 99 allocs/op; OR of 5: 26181 -> 14187 B/op, 166 -> 119 (200k docs, 8 segments). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Each hit returned cost two allocations, its DocumentMatch and its 8 byte internal id. They are now made together for all the hits (a slab, not a pool: nothing is asked of whoever gets the hits, and any one of them keeps the slab alive). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…or on It runs the regular path's searches over 100000 documents, which take minutes with the race detector; the algorithms are raced by the searcher package's tests and the smaller tests of this package. Also: PERSEG_FIELDS=1 makes BenchmarkSearchCPU ask for the stored fields of the hits, as a service does. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… same bounds ShallowSeek computed the upper bound of the block it landed on every time it moved to another block, though the bounds are two bytes (the highest frequency the entry holds and the shortest field) that neighboring blocks mostly share. It now keeps the last pair and what it came to. UpperBound was 3.6% of an AND's top 10 on 5M documents. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s much rarer
windowSegment cuts a window wherever a block of any term ends, so its work is in
proportion to the blocks of all the terms. A real query is a rare word and some
common ones ("tallest trees in the world", "battle of the bulge"), where the
common terms' blocks span a couple of hundred docs: tens of thousands of windows,
almost none with a posting of the rare term in them, each with its bookkeeping.
On 5M documents, for 163 of 300 AOL AND queries a scored top 10 took 2.4 times
what counting every match did.
candidateSegment goes through the postings of the leader a block at a time with
bounds that cost the least first: the leader's block with the best score of each
other term anywhere (passing over its 128 postings by the skip data alone), the
posting's own score with the same, the blocks the others have at the doc one term
after the other, and only then the lookups. It is chosen, per segment, when the
leader has 16 times fewer postings than all the terms (candidateDriven); the
setting forces either algorithm.
On 5M documents, AOL ANDs, whole workload: top 10 average 1.01 -> 0.78 ms, p95
3.56 -> 2.39 ms, p99 4.75 -> 3.94 ms; top 100 1.14 -> 0.91 ms; top 1000 1.40 ->
1.18 ms. For ANDs whose leader is under 1% of the postings, 5.3 times faster.
Tests: the oracle, stress and planted outlier tests of ANDs and the root level top-k
test run with each algorithm forced; the 100000 document comparison with the
regular path runs by what fits best and with candidates; the choice has a test.
Mutations of each bound, over-pruning, are caught.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes to existing code (review these first)
Everything else in the diff is new files; these are the edits to files that already existed:
index/scorch/segment_plugin.gobitpack-simd) becomes the default segment plugin, so new and merged segments are written in the v18 format; v17 stays registered and readable. This is an on-disk format default change, independent of the search path.search/searcher/search_multi_term.gooptimizeOrDisjunct. A multi-term query (prefix, wildcard, regexp, fuzzy, ...) used to returnunable to optimizewhen the index can't apply thedisjunction:unadornedoptimization to its searchers (zapx v18 has no implementation of it); it now falls back to a plain disjunction. This changes the classic path too.index_impl.goSearchInContextbuilds a per segment searcher and collector when the request is eligible, otherwise unchanged. Facets, results and the memory estimate go through small interfaces so either collector fits. Also, in classic code used by both paths:LoadAndHighlightAllFieldsreturns before allocating the stand-inDocumentMatchfor descendants when a hit has none (it allocated one per hit regardless): about 10 allocations and 2 KB less per 10-hit search.search/sort.goSortValueBytes;SortField.Valueis nowstring(ValueBytes())(same result).search/query/match.goanalyze, shared by the classic and per segment paths.index/scorch/snapshot_index.goperSegmentDicts: dictionaries kept for reuse by the per segment term readers.go.mod/go.sumreplacedirectives noted above.What
A per segment search path: a second search pipeline next to the classic searcher/collector one, built on zapx v18 block iterators and the freeway SIMD scoring kernels. A request it can serve skips
DocumentMatchconstruction for everything but the hits returned; anything it can't serve falls back to the classic path untouched.Structure
search.PerSegmentSearcher: its own interface, not aSearcher(NextMatch,ExplainMatch,Count,Min,Weight, ...). Built byquery.PerSegmentQuery(term, bool-field, match, conjunction, disjunction, boolean, query string). A build the path can't serve returnssearch.ErrPerSegmentUnsupportedandSearchInContextuses the classic searcher.PerSegmentTopNCollector(descending score only): searchers may offer an optimized collection. Lone-term block skipping with block-max bounds; plain ORs by batched block-max MAXSCORE (WAND for sparse queries); ANDs by block windows with bitmap windows when dense, or, when the leader (the term with the fewest postings) has 16 times fewer postings than all the terms together, by going through the leader's candidates with bounds that cost the least first (candidateSegment: the windows of a query with a rare word and common ones are almost all empty of the rare word). Pruned searches report a lower-bound total (TotalRelationGte).PerSegmentSortedCollector(every other sort order, facets, search after / before): visits every match, never prunes, so totals are exact. Keeps the top N the way tantivy'sTopNComputerdoes (buffer of 2N, strict threshold, quickselect), reads sort values from doc values as bytes (new optionalsearch.SortValueBytes, implemented bySortField) and buildsDocumentMatches only for the hits returned. Facets reuse the classicFacetsBuilder, in the same doc values visit as the sort keys.NextMatchcalls over windowed unions and intersections (tantivy'sBufferedUnionScoreras the model).disjunctionScore) is shared by the cursors, the windows, WAND, MaxScore and the explanation, and the collector fails the search if an explanation's root is not the score the hit was ranked by.Served by this path
Term, match (without fuzziness), conjunction, disjunction, boolean (without a filter) and query-string queries, in any sort order, with explain, facets, search after and search before.
Falls back to the classic path
KNN, pre search data, score fusion, nested documents, locations / highlighting, synonyms, an application's custom match handler, fuzzy / prefix / phrase / other query types, and indexes with segments that can't hand out block iterators.
Behavior differences from the classic path (please review)
PerSegmentSortedCollector.SetSearchAfter.TotalRelationGtewith a lower bound; sorted / faceted searches are exact. WithScore: "none"the total is exact where the classic path stops early aftersize+frommatches.Score: "none"returns no explanation (the classic path returns a tree of zeros).TestBM25GlobalScoring,TestFilteredBooleanQuery, andtest.TestDisjunctionSearchScoreIndexWithCompositeFields.SortField.Valueis now built onValueBytes(same result).Testing
go test . ./search/... ./index/scorch/...and the rest of the repo's packages pass; race detector clean onsearch/searcher,search/collector,searchand the per segment tests in the root package (that race run predates the large-segment and alias tests, which were run without it).runBothPaths): small multi-segment index with updates and deletes (all sort kinds, facet kinds, search after / before walks, edge cases, explain), plusTestPerSegmentSearchMatchesRegularSearchOnLargeSegments(100,000 docs, segments past the 4096 / 8192 window sizes, 234 comparisons) and a two-index alias test.Measured (200k docs, 8 segments, BM25)
Searches sorted by a field: 1.07-1.21x for numeric sorts and 1.23-1.59x for string sorts against the classic path, with 20-100x fewer allocations; facets 1.03-1.24x; search after about 1.07-1.09x (
BenchmarkSortedSearch,BenchmarkFacetedSearch). The gain is modest because about 97% of the sorted collector's time is reading doc values (68% of that snappy decoding their chunks), which the classic path pays too; a bigger win would need a change to how doc values are stored. Score-ordered searches use the pruning algorithms above; seeBenchmarkSearchCPU.Measured on 5M documents (AOL AND queries, the workload of bleve-bench)
300 real AND queries over the Wikipedia corpus, one worker. Cost from 1M to 5M documents (5.03x) was 3.6-4.2x on every mode, and a scored top 10 was slower than counting every match (1.01 ms against 0.65 ms) because windows follow the blocks of the common terms. With
candidateSegmentfor ANDs of a rare leader: top 10 average 1.01 -> 0.78 ms (p95 3.56 -> 2.39, p99 4.75 -> 3.94), top 100 1.14 -> 0.91 ms, top 1000 1.40 -> 1.18 ms; at 1M documents the gains are 1.1-1.2x. Counts are unchanged. The cost still grows about 3.4-3.8x for 5x the documents: what is left is mostly the lookups in the dense terms (block decodes and the search in them).Dependencies
Builds against, via local
replacedirectives: blevesearch/zapxperseg/block-cursor(based onbitpack-simd), blevesearch/scorch_segment_apiperseg/block-cursor, blevesearch/freewayperseg/score32, blevesearch/vellumperf/tantivy-inspired.Stack (review order)
bitpack.BlockLenis the block length everything else uses)BlockPostingsIteratorinterfaces;PostingsBlockLenfrom freewayBlockPostingsIteratorover bit-packed postings (basebitpack-simd)Only the last one is the design under review. freeway, scorch_segment_api and zapx are the pieces it reads postings and scores through (the block length is defined once, in freeway, and handed on by scorch_segment_api), and vellum is independent FST work that bleve builds against.
🤖 Generated with Claude Code