Skip to content

fix(query): refuse key condition values over the key size limits - #481

Open
LeeroyHannigan wants to merge 3 commits into
mainfrom
fix/435-query-key-size
Open

LeeroyHannigan wants to merge 3 commits into
mainfrom
fix/435-query-key-size

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #435.

Summary

The service refuses a Query whose key condition values exceed the key size limits before it reads anything; ExtendDB ran the Query. This change adds the check with the service's sentences and ordering.

Measured on the service on 2026-10-23 (every row repeated, identical answers):

Shape Service answer
HASH value of 2049 bytes (table, LSI, GSI, legacy KeyConditions) One or more parameter values were invalid: Size of hashkey has exceeded the maximum size limit of2048 bytes (no space after of)
HASH value of 2048 bytes accepted
Two HASH attributes, 1025 + 1024 bytes, or 2048 + 2048 One or more parameter values were invalid: Aggregated size of all hash keys has exceeded the size limit of 2048 bytes
Two HASH attributes, 1024 + 1024 bytes accepted
Two HASH attributes, one value of 2049 bytes the single hash sentence
RANGE value of 1025 bytes, any operator; or RANGE values adding up to 1025 (512 + 513, 600 + 1 + 600, 600 + BETWEEN with a 600-byte larger bound) One or more parameter values were invalid: Aggregated size of all range keys has exceeded the size limit of 1024 bytes
RANGE total of exactly 1024 bytes; BETWEEN with two 600-byte bounds accepted (BETWEEN contributes its larger bound)
Oversized HASH and RANGE in one request the hash sentence (single or aggregated)
683 U+20AC characters (2049 bytes) in a HASH value; 342 (1026 bytes) in a RANGE value refused by UTF-8 byte length; 682 plus xx and 341 plus x accepted
Undefined #name or :value in the key condition with an oversized value the undefined-reference sentence first

Ordering, measured with a 2049-byte value: filter syntax, unused names and values, index name, ConsistentRead on a GSI, missing table, Limit, projection syntax and overlap, the Select conflicts and the filter operand type are reported before the size check; the ExclusiveStartKey checks after it.

Changes

  • crates/core/src/validation/key_size.rs: the check, with the limits from the configuration and the sentences naming them. Unit tests in validation/key_size/tests.rs.
  • handle_query calls it after the expression validations and before the start key checks.
  • docs/dynamodb-limits.md: the partition key cell names the HASH sum.

Tests

Related

  • The undefined-name row (pk = :v AND #n = :w, #n undefined) is unit level only: ExtendDB has no undefined-name refusal for a sort key clause today; fix(query): refuse a key condition with no HASH equality the way the service does #479 adds it, and with both merged the row matches the service.
  • Cassandra, two things the new rows exposed, both reproduced on a cassandra:4.1 node:
    • The Cassandra backend wrapped a missing index name as Index nope not found inside IndexNotFound, so the Query sentence read The table does not have the specified index: Index nope not found on Cassandra only. Corrected to pass the bare name like the other backends (one line in storage-cassandra/src/data/index.rs).
    • The Cassandra backend cannot read a multi-RANGE index at all: every Query on such an index is refused by the node (Order by currently only supports the ordering of columns following their declared order in the PRIMARY KEY, because the data table's clustering key is sk_s, sk2_s, ..., base_pk and the Query emits ORDER BY sk_s, base_pk) and surfaces as InternalServerError, with or without a RANGE clause, for any value size. Not a one-line change (the single sort key model also drives the pagination restrictions and the start key read). The idx accept rows skip on that backend through the same capability probe, with the reason named; the defect will be filed on its own.

Pre-existing, measured, not changed here

  • A BETWEEN whose lower bound is above the upper bound: the service refuses it (The BETWEEN condition was provided a range where the lower bound is greater than the upper bound), ExtendDB runs it.
  • Two conditions on one key: KeyConditionExpressions must only contain one condition per key on the service, 200 here.
  • A Number with 39 or more digits: Attempting to store more than 38 significant digits in a Number on the service, 200 here.
  • A FilterExpression on a key attribute: Filter Expression can only contain non-primary key attributes: Primary key attribute: pk on the service, 200 here.

These will be filed as one issue together with the ones from #479.

Closes #435.

The service refuses a Query whose key condition values exceed the key
size limits before it reads anything; ExtendDB ran the Query. Measured on
2026-10-23, every row repeated with identical answers:

- A HASH value over 2048 bytes: "One or more parameter values were
  invalid: Size of hashkey has exceeded the maximum size limit of2048
  bytes" (no space after "of", as the service writes it). 2048 bytes is
  accepted.
- HASH values that each fit but add up to more than 2048 bytes (an index
  with two HASH attributes): "One or more parameter values were invalid:
  Aggregated size of all hash keys has exceeded the size limit of 2048
  bytes". A sum of exactly 2048 is accepted. A single oversized value
  gets the single sentence even when the sum is also over.
- RANGE values adding up to more than 1024 bytes, one clause or several:
  "One or more parameter values were invalid: Aggregated size of all
  range keys has exceeded the size limit of 1024 bytes". 1024 is
  accepted. Every operator is sized (=, <, <=, >, >=, begins_with);
  BETWEEN contributes its larger bound, not the sum of both.
- The HASH checks are reported before the RANGE check.
- Strings are sized by UTF-8 bytes (683 U+20AC characters, 2049 bytes,
  refused; 682 plus "xx", 2048 bytes, accepted) and binary values by
  byte length. Base tables, LSIs, GSIs and legacy KeyConditions behave
  the same.
- An undefined #name or :value in the key condition is reported before
  the size check.

Ordering, measured with a 2049-byte value: filter syntax, unused names
and values, index name, ConsistentRead on a GSI, missing table, Limit,
projection syntax and overlap, the Select conflicts and the filter
operand type are all reported first; the ExclusiveStartKey checks after.

The check is crates/core/src/validation/key_size.rs, called from
handle_query after the expression validations and before the start key
checks. The limits come from the configuration and the sentences name
them.

Tests: unit tests in validation/key_size/tests.rs and the engine;
tests/python/test_query_key_size.py, test_query_key_size_values.py and
test_query_key_size_order.py (64 rows) pass against the service. On
ExtendDB alone, five rows with two or more RANGE clauses on one index
skip until the multi-attribute range change in #429 merges (the handler
refuses the shape earlier today); they pass on the merged tree. The
undefined-name row stays unit level because the undefined-name refusal
for a sort key clause arrives with #479.

Pre-existing, measured, not changed here: a BETWEEN with the lower bound
above the upper bound is refused by the service with a bound-order
sentence; two conditions on one key are refused with "KeyConditionExpressions
must only contain one condition per key"; a Number with 39 or more digits
is refused with "Attempting to store more than 38 significant digits in a
Number"; a FilterExpression on a key attribute is refused with "Filter
Expression can only contain non-primary key attributes". ExtendDB accepts
all four.
… multi-RANGE index

The Cassandra backend answers InternalServerError to any Query on an index
with several RANGE attributes: its ORDER BY lists sk_* then base_pk and skips
the sk2_* to skN_* clustering columns, which Cassandra refuses with
"Order by currently only supports the ordering of columns following their
declared order in the PRIMARY KEY". require_multi_range now also skips on
InternalServerError, so test_one_600_byte_range_value_is_accepted carries the
probe like the other idx rows.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Query does not enforce the key value size limits: 2048 bytes for the hash key, 1024 bytes for all range keys together

2 participants