Repository navigation
MB-65016: More Like This query - #2411
Open
maneuvertomars wants to merge 1 commit into
Open
maneuvertomars wants to merge 1 commit into
maneuvertomars wants to merge 1 commit into
Conversation
|
coverage: 52.122% (+0.5%) from 51.659% — MB-65016 into master |
maneuvertomars
requested review from
Likith101,
Samsonnyyeet and
steveyen
and
a balanced review from Copilot
November 4, 2026 09:45
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved query-rewrite issues can admit synonym-only matches, silently discard nested-field sources, and make alias term selection depend on index grouping or response order.
5 open issues
What changed in this PR
Adds more_like_this similarity queries to Bleve, supporting source document IDs or raw text across individual indexes and index aliases.
Changes:
- Collects term statistics, ranks candidates by TF-IDF or BM25, and rewrites similarity queries into standard queries.
- Integrates distributed statistics collection and merging into alias preSearch.
- Adds rewrite-only responses, explanatory notes, and unit and integration tests.
| File | Description |
|---|---|
| search/util.go | Defines similarity statistics and the preSearch marker. |
| search/query/term.go | Adds optional notes to term queries. |
| search/query/query.go | Parses similarity queries and extracts their fields. |
| search/query/more_like_this.go | Implements validation, statistics collection, ranking, and rewriting. |
| search/query/more_like_this_test.go | Tests parsing, validation, rewriting, and query traversal. |
| search/query/disjunction.go | Preserves optional disjunction notes. |
| search/query/boolean.go | Preserves optional boolean query notes. |
| search.go | Adds statistics and rewritten queries to search results. |
| search_no_knn.go | Adds rewrite-only and notes support to non-vector requests. |
| search_knn.go | Adds request options and rejects similarity queries in vector filters. |
| pre_search.go | Merges similarity statistics and builds rewritten responses. |
| more_like_this_test.go | Tests single-index and alias similarity searches. |
| index_impl.go | Integrates local statistics collection and query rewriting. |
| index_alias_impl.go | Coordinates distributed rewriting and forwards rewritten queries. |
Review effort: Balanced
Comment on lines
+1090
to
+1092
| if m := in.Mapping(); m != nil { | ||
| return m | ||
| } |
| key := mltTermKey(st) | ||
| cur, ok := n.terms[key] | ||
| if !ok { | ||
| cur = &search.MLTTermStat{Field: st.Field, Term: st.Term, TF: st.TF} |
Comment on lines
+387
to
+390
| dvReader, err := r.DocValueReader(fields) | ||
| if err != nil { | ||
| return nil, err | ||
| } |
| type fieldTerm struct{ field, term string } | ||
| var docTerms []fieldTerm | ||
| seen := make(map[fieldTerm]struct{}) | ||
| err = dvReader.VisitDocValues(internalID, func(field string, term []byte) { |
Comment on lines
+560
to
+561
| tq := NewTermQuery(w.stat.Term) | ||
| tq.SetField(w.stat.Field) |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved MLT defects can produce response-order-dependent alias results, unintended synonym matches, and missing matches for nested fields.
8 open issues
Review effort: Max
Comment on lines
+375
to
379
| // an explained result also shows the query that was actually run | ||
| if req.Explain && searchReq != req { | ||
| sr.RewrittenQuery = &RewrittenQuery{Query: searchReq.Query} | ||
| } | ||
| } |
| if knnFiltersContainMoreLikeThis(req) { | ||
| return nil, fmt.Errorf("knn filter query cannot contain a more_like_this query") | ||
| } | ||
| mlt := len(mltNodes) > 0 |
Comment on lines
+300
to
+308
| case fm.Type == "": | ||
| // not explicitly mapped: indexed dynamically, if at all | ||
| if !im.DocValuesDynamic { | ||
| return fmt.Errorf("more_like_this: field %q has no doc values", field) | ||
| } | ||
| case fm.Type != "text": | ||
| return fmt.Errorf("more_like_this: field %q is not a text field", field) | ||
| case !fm.DocValues: | ||
| return fmt.Errorf("more_like_this: field %q has no doc values", field) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
more_like_thisqueryA new query type that finds documents similar to given documents (
like_ids) or to raw text (like_text), in the spirit of Lucene'sMoreLikeThis/ Elasticsearch'smore_like_this.How it works
fieldsare read from doc values, each term's tf comes from its posting entry for the document, df from the term dictionary, N from the reader.min_term_freq/min_doc_freq, and the topmax_query_termsare kept.minapplied), wrapped in a boolean with amust_notonlike_ids, so the source documents are excluded.On a single index this happens in
SearchInContext(and inSearcher()). On an index alias it rides on the existing preSearch round: a fourthpreSearchFlagsentry asks every member for the term stats of each node (SearchResult.MLTStats, keyed by the node's position in the query tree), the alias merges them order-independently (a source document replicated in two members is counted once; df and N are summed per contributing member), substitutes the nodes, andMultiSearchruns the plain query. The caller's request is never mutated: the rewritten query travels throughmultiSearchParams, so score fusion and therequestecho keep working.API
{ "query": { "more_like_this": { "fields": ["description"], "like_ids": ["hotel_10025"], "min_term_freq": 1, "min_doc_freq": 1, "max_query_terms": 25, "min": "30%", "boost_terms": 0 } } }