Repository navigation
Query on an index returns a LastEvaluatedKey on the page that holds the last item; the service does so for base tables only #434
Description
Summary
For a Query whose Limit lands exactly on the last matching item, the service answers differently depending on the target. On a base table it returns the full page with a LastEvaluatedKey, and the next request returns an empty page without one. On a global secondary index it returns the full page without a LastEvaluatedKey. ExtendDB applies the base-table rule to both, so an index Query paged to exhaustion makes one request more than against the service and ends on an empty page.
Measured
2026-10-22, us-east-1. Table with HASH pk and RANGE sk, a GSI gsi with HASH g and RANGE gs, five items in one partition of each. Each walk follows LastEvaluatedKey until it is absent; the pairs are (items on the page, LastEvaluatedKey present).
| Query | service | ExtendDB (SQLite and PostgreSQL, main) |
|---|---|---|
base table, Limit 5 over 5 items |
(5, yes), (0, no) |
(5, yes), (0, no) |
base table, Limit 1 walk |
(1, yes) x5, then (0, no) |
same |
base table, Limit 2 walk |
(2, yes), (2, yes), (1, no) |
same |
GSI, Limit 5 over 5 items |
(5, no) |
(5, yes), (0, no) |
GSI, Limit 1 walk |
(1, yes) x4, then (1, no) |
(1, yes) x5, then (0, no) |
GSI, Limit 2 walk |
(2, yes), (2, yes), (1, no) |
same |
The same shape was measured on 2026-10-21 on a GSI with four RANGE attributes and seven items: h = :h forward and reverse, h = :h AND a = :a, h = :h AND a = :a AND b = :b both directions, h = :h AND a = :a AND b > :hn reverse. In each walk the service's last page holds the last item and no key; ExtendDB adds one empty page. Item order is identical on both sides in every walk.
Expected
On an index Query, no LastEvaluatedKey when the page holds the last matching item. On a base table Query the current behaviour matches the service and must stay.
Where
crates/engine/src/read_helpers.rs, read_page: reaching Limit ends the page with a key whether or not storage reported more rows, with the comment citing the 2026-09-16 base-table measurement (Limit 5 over exactly 5 items returns a key). That rule is right for a base table and wrong for an index. The storage backends already fetch Limit + 1 rows and report whether a row follows, so the index path can end the page without a key when storage reports none.
Activity
LeeroyHannigan commentedon Oct 23, 2026
Closing: the premise does not hold. Measured again today on a fresh table with five items in one partition,
Limit 5, ten calls per series:LastEvaluatedKeyConsistentRead: trueThe same request returns a key on one call and none on the next. The 22 October rows in the issue (base table always a key, GSI never) were one sample each and happened to land on opposite sides. The service's documented contract is that a
LastEvaluatedKeymay be returned whenLimitis reached whether or not more items exist, and that a client must follow it until it is absent; the raw service follows that contract on both tables and indexes, with the consistent-read path always returning the key.ExtendDB returns the key on every such call, on tables and indexes alike. That is the consistent-read behaviour and one of the outcomes the service produces on every call, so there is nothing to change. A client that stops paging because a page is short, or that counts pages, is relying on something the service does not promise.
Measurement script and raw output:
probe-434-shapes.py,service-434-shapes.jsonlandservice-434-repeat.jsonlin the measurement set for this issue (kept with the other Query parity probes).LeeroyHannigan commentedon Oct 24, 2026
Follow-up measurement from the #440 work (2026-10-24, ten repeats per row) that narrows this: the trailing
LastEvaluatedKeyis non-deterministic only at the end of a partition with no sort key condition. With a sort key condition on a base table the service is deterministic, 10 of 10 per row:sk = mwithLimit1;sk BETWEEN aa AND mending atm;sk <= mending atm; reversesk >= mending atm;begins_with(sk, a)withLimit3 overa, aa, ab(the next rowd1fails the condition);sk < zwithLimit7 over the seven matches.BETWEEN aa AND mwithLimit3 over six matches), and on a GSI page over three identical index keys withLimit3.sk >= mwithLimit2 overm, zwherezis the last item of the partition (both answers seen in ten calls).#483 drops the key when a base table page ends at a bound by value (
=,BETWEENupper,<=, reverse>=, reversebegins_with), which also keeps ExtendDB's own keys acceptable under the new start key check. The two rows where the next row exists but fails the condition (sk < zwithLimit7, forwardbegins_withwithLimit3) are not covered: matching them needs a look-ahead read of one more row. They are listed in #489 with the other measured divergences. The partition-end case stays as closed here.