Repository navigation
fix(query): classify and order key conditions on a multi-attribute GSI the way the service does - #429
Open
LeeroyHannigan wants to merge 2 commits into
Open
LeeroyHannigan wants to merge 2 commits into
LeeroyHannigan wants to merge 2 commits into
Conversation
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 21, 2026 23:23
…I the way the service does
On a global secondary index with one HASH attribute and several RANGE attributes, Query ordered results by the first RANGE attribute and then the table key, refused every key condition with three or more clauses as "Query condition missed key schema element: h", and accepted a condition that skipped the first RANGE attribute while returning no items. KeyCondition::resolve_multipart returned early when the schema had one HASH attribute, so the parser's assignment stood: the first equality became the partition key and the rest sat where the engine's HASH-count check refused them.
Measured against the service on 2026-10-21, with the issue's schema (HASH h, RANGE a, b, c, d) and with two HASH attributes:
- Results are ordered over every RANGE attribute in key-schema order, then the base table key; ScanIndexForward = false is the exact mirror. LastEvaluatedKey carries every key attribute.
- A key condition is served when the HASH attributes all have an equality and the RANGE conditions cover a leading run of the RANGE attributes with an equality on every one before the last; the last may use <, <=, >, >=, BETWEEN or begins_with. Clause order in the expression is free.
- Refusals, in the order the service applies them: a second condition on one key attribute ("KeyConditionExpressions must only contain one condition per key"); more conditions than key attributes ("The number of query conditions (6) exceeds the number of key attributes defined in the schema (5)"); a HASH attribute without an equality ("HASH key attributes h1, h2 must have equality conditions specified in the query", every missing one named); a RANGE gap ("RANGE key attributes a must have equality conditions specified in the query because a condition is present on key attribute b"); a value whose type differs from the attribute definition ("One or more parameter values were invalid: Condition parameter type does not match schema type"); a condition on an attribute outside the key schema, named by the first RANGE attribute without a condition ("Query condition missed key schema element: a"). begins_with on a number attribute is "Incorrect operand type for operator or function; operator or function: begins_with, operand type: N". The single-attribute wording "Query condition missed key schema element" stays on single-attribute schemas.
resolve_multipart now classifies every clause by name against the HASH and RANGE attribute lists and applies those rules in that order; the value type check runs inside it through resolve_multipart_with, with the engine supplying the attribute definitions. The engine's single-attribute path gets the same type check ahead of its missing-key check, which turns a 500 on a wrong-typed sort key value into the service's refusal. Legacy KeyConditions on a multi-attribute schema go through the same classifier. The SQLite and PostgreSQL backends apply every RANGE condition to its column and order and page over all RANGE columns; the cursor keeps its shape. The MongoDB and Cassandra backends filter the first RANGE attribute only and now refuse a condition on a later one instead of applying it to the wrong column. On Cassandra every Query on an index with several RANGE attributes answered 500 before this change: the index table clusters on every RANGE column and CQL rejects an ORDER BY that skips one, so the ORDER BY now names every index RANGE column in declared order, and a Query with an ExclusiveStartKey on such an index is refused instead of answering 500, because the cursor compares the first RANGE attribute only. docs/differences-from-dynamodb.md records both backends.
tests/test_gsi_multi_range_query.py pins 60 cases, all measured on the service: order, pages and LastEvaluatedKey, every accepted shape, every refusal and its text, legacy KeyConditions, and the MongoDB and Cassandra refusal. 60 pass on the service and on SQLite and PostgreSQL; on MongoDB 39 pass and the 22 served-result cases skip; on Cassandra the same 22 skip. On main 30 of the 30 then-existing cases failed. Unit tests in core cover the classifier rules; the storage plan and cursor tests cover the ORDER BY and cursor columns.
Closes #381
LeeroyHannigan
force-pushed
the
fix/gsi-multi-range-query
branch
from
October 22, 2026 01:02
b3343a7 to
4ec5f66
Compare
Conflicted file and resolution: docs/differences-from-dynamodb.md: main appended two rows (GSI throughput on a PROVISIONED table, projected attributes across all indexes) and the Backup and Restore section after the multi-part keys row; this branch appended one row on Query over a GSI with several RANGE attributes at the same place. The merged table keeps main's two rows, then this branch's row, then main's Backup and Restore section. Auto-merged files checked by hand: crates/storage/src/util/mod.rs: main's backup_id and catalog_version re-exports and this branch's refuse_conditions_beyond_first_range re-export both kept. crates/storage-mongodb/src/data_engine.rs: main's new MongoEngine test fixture fields and this branch's range-condition refusal in the query path do not overlap.
This was referenced Oct 24, 2026
Collaborator
Author
|
Measured on the service on 2026-10-23 while working #427 (each row twice, identical answers), for the placement of the type check this PR adds:
#479 conflicts with this branch in |
This was referenced Oct 24, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #381.
What the service does
Measured on 2026-10-21 against live DynamoDB with the issue's schema (GSI with HASH
hand RANGEa,b,c,d) and a two-HASH index:ScanIndexForward = falsereturns the exact mirror.LastEvaluatedKeycarries every key attribute.<,<=,>,>=,BETWEENorbegins_with. Clause order is free.KeyConditionExpressions must only contain one condition per key); more conditions than key attributes (The number of query conditions (6) exceeds the number of key attributes defined in the schema (5)); a HASH attribute without an equality (HASH key attributes h1, h2 must have equality conditions specified in the query); a RANGE gap (RANGE key attributes a must have equality conditions specified in the query because a condition is present on key attribute b); a value whose type differs from the attribute definition (One or more parameter values were invalid: Condition parameter type does not match schema type); a condition on an attribute outside the key schema, named by the first RANGE attribute without a condition (Query condition missed key schema element: a).begins_withon a number attribute isIncorrect operand type for operator or function; operator or function: begins_with, operand type: N.What ExtendDB did
KeyCondition::resolve_multipartreturned early for one HASH attribute, so the parser's assignment stood: results were ordered by the first RANGE attribute then the table key, every key condition with three or more clauses was refused asQuery condition missed key schema element: h, andh = :h AND b = :bwas accepted and returned nothing.Changes
crates/core/src/expression/key_condition.rs:resolve_multipartclassifies every clause by name and applies the rules above in the service's order.resolve_multipart_withtakes a per-clause value check, which the engine supplies with the attribute definitions, so a wrong-typed value is reported after the structural rules and before the non-key rule, as measured.crates/engine/src/query.rs: the multi-attribute path calls the classifier with the type check; the single-attribute path runs the same type check ahead of its missing-key check (a wrong-typed sort key value answered 500 before).begins_withon a number attribute is refused with the measured wording.crates/engine/src/legacy_filter.rs:KeyConditionson a multi-attribute schema go through the same classifier.crates/storage-sqliteandcrates/storage-postgres: every RANGE condition is applied to its column;ORDER BYand the row-comparison cursor cover all RANGE columns.crates/storage/src/util/key.rs,crates/storage-mongodb,crates/storage-cassandra: these two backends filter the first RANGE attribute of such an index only. A condition on a later RANGE attribute is refused with aValidationExceptionnaming the attribute and the backend, instead of being applied to the wrong column.crates/storage-cassandra/src/data/query.rs: onmain, everyQueryon a Cassandra index with two or more RANGE attributes answers 500 (measured onec81e9d4: hash-only, first-RANGE equality and first-RANGE range conditions all returnOrder by currently only supports the ordering of columns following their declared order in the PRIMARY KEYfrom Cassandra). The index table clusters on every RANGE column ahead ofbase_pk, so theORDER BYnow names every index RANGE column in declared order; those queries answer 200 with the page in clustering order. AQuerywith anExclusiveStartKeyon such an index also answered 500 and is now refused with aValidationException(ExclusiveStartKey on index gsi: the Cassandra backend pages a Query over an index with one RANGE attribute only), because the cursor compares the first RANGE attribute only.docs/differences-from-dynamodb.mdrecords both backends.Verification
tests/test_gsi_multi_range_query.py, 60 cases, every expectation measured on the service: order, pages andLastEvaluatedKey, each accepted shape, each refusal and its text, legacyKeyConditions, the value type rule, and the MongoDB and Cassandra refusal.Queryanswered 500 as onmain); with theORDER BYchange the fixture'sQueryanswers 200 locally. GSI propagation did not run in the local Cassandra setup used for that check (the queue rows stayed pending onmainas well), so the full run on Cassandra is the CI job'smainatec81e9d4, SQLiteRegression on the branch servers:
test_multipart_gsi.py,test_query_scan.py,test_conditional_operator.py,test_expression_param_usage.py,test_item_operations.pyand the othertest_query*.pyfiles, 309 cases on SQLite and 180 on PostgreSQL, all pass. A 67-request probe comparing the branch with the service case by case leaves 13 differences, all present onmainand outside this change: the parser's wording for two non-equality clauses (#427), the 2048-byte hash key and 1024-byte aggregate range key limits on Query, and an extra page with aLastEvaluatedKeywhenLimitlands on the last item.Negative controls on the committed tree, each restored with
git checkout: the eight classifier mutations (missing-HASH order, RANGE duplicate detection, non-equality duplicate detection, RANGE gap window, non-equality excluded from the gap, early return widened, non-key refusal disabled, count refusal disabled) each fail their unit test; the type check disabled fails 6 wire cases; thebegins_withrule disabled fails 1; the MongoDB refusal disabled fails 18 wire cases on wrong items or order.Gates:
cargo fmt --check;cargo +1.97.0 clippy --all-targets -D warningson core, engine, storage and the four storage crates;cargo +1.88.0 buildon the same; unit tests core 521, engine 103, storage 72, storage-sqlite 101, storage-postgres 46, storage-mongodb 96, storage-cassandra 52. No dependency change.Not in this change
Pre-existing, filed separately: #427 (the parser refuses a lone non-equality, or two non-equalities, before the key schema is known, so the wording differs from the service); Query does not enforce the key value size limits the service applies (
Size of hashkey has exceeded the maximum size limit of2048 bytes,Aggregated size of all range keys has exceeded the size limit of 1024 bytes); aLimitthat lands on the last item returns aLastEvaluatedKeyand one more empty page where the service returns none. Full support for several RANGE attributes on the MongoDB and Cassandra backends (typed fields and clustering for every RANGE attribute) is a separate change.Overlap with other open PRs
Condition parameter type does not match schema typerefusal on the expression path, at the position measured on 2026-10-24 (after the filter, projection, unused value andSelectchecks, before theExclusiveStartKeycheck).check_key_value_typehere runs earlier, at key condition classification, so with both merged a request that carries both faults would get the type sentence where the service reports the other fault. Whichever merges second drops this copy and keeps fix(query): check Query values against the key schema and the number rules #544's.parse_key_conditionandresolve_pk_skfor every key schema. The duplicate check in the multi-attribute classifier here is then needed only for the alias case (#a = :x AND h = :ywith#anamingh).KeyConditionstype check inlegacy_filter.rs. The two PRs conflict inlegacy_filter.rsandquery.rs, so the legacy path is settled in the same rebase.