semantic_search: truncation warning misses SMW's server-side result cap #162

Activity

1. LukasGold commented on Sep 7, 2026

   

   LukasGold

   on Sep 7, 2026

   Last edited by LukasGold

   ContributorAuthor

   More actions

   The same gap exists in the service layer added by PR #133

   #133 adds osw.service.ops.search, whose search operations return a truncated field to CLI and MCP callers. Commit 5c26a76 makes that field fire when the result count reaches the limit that reached the wiki:

   osw-python/src/osw/service/ops/search.py

   Lines 17 to 26 in bca9d1f

   	def _hit_limit(total: int, limit: Optional[int]) -> bool:
   	"""Whether a result set is as large as the limit that produced it.
   	The wiki applies the limit itself, so ``cap_list`` never has to cut these
   	results and its own flag stays False. A full result set is then the only
   	signal left that the wiki may hold further matches. ``limit=0`` asks for
   	no results, so meeting it says nothing about truncation.
   	"""
   	return bool(limit) and total >= limit

   That mirrors the if limit and n >= limit condition in semantic_search, so it inherits exactly the blind spot this issue describes. A server-side cap below the requested limit returns fewer rows than asked for, total >= limit is false, and truncated reads False for a result set that was cut.

   The service layer widens the gap to two more APIs:

   Operation	API	Server cap
   search_entities, list_instances_of_category	SMW ask	$smwgQMaxLimit
   search_titles	MediaWiki prefixsearch	pslimit maximum
   search_content	MediaWiki search	srlimit maximum

   At the default OSW_MAX_RESULTS of 100 none of these caps is reached. A caller passing a larger limit can reach all three.

   What this means for the proposed fix

   Reading query.meta.hasFurtherResults, the third option above, would fix the service layer too, but only if semantic_search returns the signal instead of only warning about it. search_entities calls it with the default return_json=False and receives a flat list of titles:

   osw-python/src/osw/service/ops/search.py

   Lines 64 to 79 in bca9d1f

   	lim = ctx.limit(limit)
   	titles = ctx.osw.site.semantic_search(
   	WtSite.SearchParam(query=ask_query, limit=lim)
   	)
   	# semantic_search lets a 'limit=' written into the query win over `lim`,
   	# so the flag has to compare against the limit that reached the wiki.
   	# `titles` excludes hits whose page does not exist, so a result set
   	# thinned that way reads as not truncated.
   	query_limit = get_query_limit(ask_query)
   	effective_limit = lim if query_limit is None else query_limit
   	capped, total, truncated = cap_list(titles, lim)
   	return {
   	"titles": capped,
   	"count": total,
   	"truncated": truncated or _hit_limit(total, effective_limit),
   	}

   So the signal is not reachable from there today. Whichever option is chosen, returning the truncation state from semantic_search rather than only emitting a warning would let callers report it instead of re-deriving it from the result count.
2. LukasGold commented on Sep 7, 2026

   

   LukasGold

   on Sep 7, 2026

   ContributorAuthor

   More actions

   hasFurtherResults does not exist, query-continue-offset does

   Measured with mwclient, the client this library uses, against https://www.semantic-mediawiki.org.

   A truncated ask response carries a top-level query-continue-offset holding the offset the remainder starts at. A complete one omits the key.

   query "[[Modification date::+]]|limit=3"
     top-level keys : ['query', 'query-continue-offset']
     query-continue-offset: 3
     query keys     : ['meta', 'printrequests', 'results', 'serializer', 'version']
   
   query "[[Category:NoSuchCat12345]]|limit=10"
     top-level keys : ['query']
     query-continue-offset: <absent>
   

   hasFurtherResults appears nowhere in the response, neither under query nor under query.meta, whose keys are hash, count, offset, source and time. So option 3 as written is not implementable, but its idea is: the wiki does report completeness, under a different key and one level higher. query.meta in the issue text is the right path for count and offset.

   The count comparison also invents truncation

   Checking the case the issue does not mention, a complete result set whose size equals the limit. [[Has type::Date]] has 18 results in total:

   query	count returned	query-continue-offset
   [[Has type::Date]]|limit=18	18	absent
   [[Has type::Date]]|limit=17	17	17

   18 >= 18 is true, so the old check warns about a result set that is complete. That is not a rare shape: at the OSW_MAX_RESULTS default of 100, any category with exactly 100 instances would set truncated wrongly in osw.service.ops.search.

   Since the offset is present exactly when the set was cut short, it replaces the count comparison rather than supplementing it. Keeping the count as a fallback would only re-add the false positive.

   A limit=0 count query returns no offset even when the query matches pages, so it stays silent as before.

   Fix

   #171 changes semantic_search in wiki_tools on main:

   • truncation is read from query-continue-offset
   • SearchParam.return_meta=True returns one SemanticSearchResult per query, with query, titles, count, truncated and next_offset, so a caller can report the state instead of re-deriving it
   • the return_json=True path is unchanged, since the raw response it returns already carries the offset

   Wiring osw.service.ops.search to return_meta stays with #133, since src/osw/service/ does not exist on main yet.

   search_titles and search_content are not covered. They use prefixsearch and search, which report continuation differently, so they need separate handling in #133.
3. LukasGold commented on Sep 18, 2026

   

   LukasGold

   on Sep 18, 2026

   ContributorAuthor

   More actions

   Another confirmation of the false positive, measured on a second wiki
   (arkeve.isc.fraunhofer.de, 2026-09-17).

   The query [[Category:OSWc5d4829ed2744a219ba027171c75fa1d]] matches 408 tasks.

   limit	rows returned	top-level query-continue-offset	wiki_tools warns
   3	3	present, value 3	yes, correctly
   408	408	absent	yes, wrongly
   500	408	absent	no

   The limit=408 row is the false positive. The result set is complete and the
   server sends no continuation offset, but the count check at
   https://github.com/OpenSemanticLab/osw-python/blob/9aa6322/src/osw/wiki_tools.py#L394
   still emits the UserWarning:

   if limit and n >= limit:

   This repeats what was already measured against semanticmediawiki.org and shows
   the behaviour is not instance specific.

   A working implementation of the check now ships in this repository, in
   _ask_rows in src/osw/service/ops/tasks.py on the feat/mcp-server branch:

   response = raw[0] if raw else {}
   ...
   return rows, "query-continue-offset" in response

   It reads the top-level key and nothing else. semantic_search can adopt the
   same test.
4. LukasGold

   closed this as completedin #171on Sep 21, 2026