Skip to content

fix(scan): check the numbers in ExpressionAttributeValues as the service does - #562

Open
LeeroyHannigan wants to merge 3 commits into
fix/query-key-value-typesfrom
fix/scan-number-check
Open

LeeroyHannigan wants to merge 3 commits into
fix/query-key-value-typesfrom
fix/scan-number-check

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Closes #545. Stacked on #544 (target branch fix/query-key-value-types), which adds the request-time number check for Query that this change reuses.

What changed

Scan did not check that the N values in ExpressionAttributeValues are numbers. A filter on {"N": "bob"}, a bob nested in an L, an M or an NS, 1e200, a 39-digit value or an empty N answered 200 with Count: 0. The service refuses each with a ValidationException before any row is read, whether or not an expression reads the value.

The early Scan checks now run in one function on the raw request body, validate_scan_request_body in crates/engine/src/scan_request.rs, in the order measured against DynamoDB on 2026-10-25 (us-east-1, 09:23 to 09:28, 10:38, 10:55 and 11:31 UTC, each request sent twice):

  1. The range constraints on Limit (at least 1), Segment (0 to 999999) and TotalSegments (1 to 1000000), each checked when present, with or without the other segment field, the first fault in that order. A Limit below 1 inside the int32 range answers 1 validation error detected: Value at 'Limit' failed to satisfy constraint: Member must have value greater than or equal to 1; below int32 the service saturates the value and answers the older form, Value '-2147483648' at 'limit' ..., and so does the engine. The Segment and TotalSegments texts are the previous release's (see the gaps below).
  2. The expression and non-expression parameter mix (AttributesToGet, ScanFilter, ConditionalOperator against ProjectionExpression, FilterExpression).
  3. The ReturnConsumedCapacity enum.
  4. Every number in ExpressionAttributeValues, NS members and values nested in L and M included, through validate_expression_attribute_value_numbers from query_key_values.rs, which accepts a Unicode decimal digit (general category Nd) as the digit it stands for, as the service does. The filter then compares the value as the request wrote it, as on main; the service reads it as the digit there too, which is left to a separate issue. The NS members are checked when the value map deserializes; an NS with a duplicate member is reported by the deserializer (Input collection ... contains duplicates) ahead of a malformed member in the same set, where the service reports the malformed member. The texts: The parameter cannot be converted to a numeric value: bob, Number overflow. Attempting to store a number with magnitude larger than supported range, Number underflow. Attempting to store a number with magnitude smaller than supported range, Attempting to store more than 38 significant digits in a Number, and The parameter cannot be converted to a numeric value: for an empty N.
  5. The Select enum.

Each step is reported ahead of the next, ahead of the table lookup, and ahead of the checks the handler makes after deserializing the body (Segment pairing, IndexName, ConsistentRead on a GSI, Select COUNT with ProjectionExpression, ExclusiveStartKey, undefined placeholder, unused value, filter syntax). A field of the wrong JSON type anywhere in the body (a string ExpressionAttributeValues beside Limit: 0, a string TotalSegments beside Segment: -1, a string ExpressionAttributeNames beside a malformed N, an object AttributesToGet beside ProjectionExpression) answers SerializationException ahead of all of them: when a check fails, the body is deserialized as the typed input and a SerializationException from that is returned instead, through the helper #544 adds for Query. The helper leaves an AttributeValue with an unknown type descriptor to the check: Limit: 0 beside {"X": "1"} answers the Limit ValidationException, as on the service. A body that passes every check is deserialized once, by the handler.

Two behaviours changed beyond the number check, each because the check order above requires it and each measured on 2026-10-25:

  • Limit below 1 inside the int32 range answers Value at 'Limit' with no value (Value '0' at 'limit' before), and the Segment and TotalSegments ranges are checked at request time, each when present, where the engine checked them after the table lookup and only when both fields were present. Their texts are unchanged; tests/test_query_scan.py is unchanged.
  • Both enums invalid answers the ReturnConsumedCapacity error alone. The handler aggregated the two with Select first (measured 2026-08-24); the service does not do that today.

crates/engine/src/scan.rs drops the enum pre-scan, the typed parameter mix check, the range arms of the segment match and the Limit check (520 to 462 lines). The Segment pairing checks and their text stay where they were.

Tests

  • crates/engine/src/scan_request/tests.rs: seven unit tests covering the measured texts and orders above, the int32 boundary pair, the shape-fault order with its controls and the Unicode digit rows. cargo test -p extenddb-engine --lib: 208 passed.
  • tests/test_scan_number_values.py: 69 wire tests, dual-target. The Segment and TotalSegments range rows assert that a ValidationException naming the field is reported ahead of the number, not a text. Against the binary built from 4a4e92f (the head of fix(query): check Query values against the key schema and the number rules #544 before its fold round): 49 failed, 20 passed (the twenty are rows the base already satisfied: the parameter mix, the ReturnConsumedCapacity enum, the segment range positions, the four shape-fault rows, the well-formed and Unicode digit rows that pass with no number check). Against this branch on SQLite: 69 passed.
  • Collected by .github/workflows/integration.yml, jobs run-integration and run-integration-sqlite, step Run integration tests.

Gates

Run locally on SQLite at the head: cargo +1.97.0 fmt --all -- --check (pass); cargo +1.97.0 clippy -p extenddb-engine --all-targets -- -D warnings (pass); cargo test -p extenddb-engine --lib (208 passed); every wire file in tests/ that mentions Scan, 24 files (818 passed, 7 skipped, 0 failed). On the earlier head of this branch: the eight files in tests/python/ that mention Scan (157 passed). The host's disk did not allow a workspace test build, so the full workspace suite, the 1.88.0 run and the PostgreSQL wire run are left to CI. No dependency changed.

Pre-existing gaps surfaced, not changed here

Measured on 2026-10-25 against the service, each present on main:

  • Segment not below TotalSegments: the service answers The Segment parameter is zero-based and must be less than parameter TotalSegments with no suffix; the engine appends : Segment: 5 is not less than TotalSegments: 2.
  • Segment without TotalSegments, and the reverse, are reported before the table lookup on the service; the engine checks them after it.
  • Select COUNT with a ProjectionExpression: service 1 validation error detected: Cannot specify the ProjectionExpression when choosing to get only the Count; engine Cannot specify the ProjectionExpression when Select is COUNT.
  • The Segment and TotalSegments range texts: the service answers Value at 'Segment' and Value at 'TotalSegments' with no value for integers inside int32, aggregates several range faults into one N validation errors detected message in the order Limit, Segment, TotalSegments, and uses the older Value '<n>' at '<field>' form with the value saturated when any field is outside int32 or a float; the engine keeps Value '<n>' at 'segment', Value '<n>' at 'totalSegments' and The parameter TotalSegments should be greater than or equal to 1, and reports one fault.
  • The service counts an empty ScanFilter as present for the parameter mix (ScanFilter: {} with a FilterExpression answers the mix) and refuses AttributesToGet: [] with Value at 'AttributesToGet' failed to satisfy constraint: Member must have length greater than or equal to 1 ahead of the number check; the engine accepts both.
  • A malformed N in a legacy ScanFilter AttributeValueList or in ExclusiveStartKey answers the number text on the service; the engine does not check those.
  • ExpressionAttributeValues nested 32 levels or deeper answers Nesting Levels have exceeded supported limits on the service; the engine does not check the depth on Scan or Query.
  • The write paths (PutItem, UpdateItem, the batch and transaction writes) refuse a Unicode decimal digit in an N that the service accepts and stores as the ASCII value; GetItem and ExclusiveStartKey by such a key answer InternalServerError; and a filter written with a Unicode digit compares the text as written where the service reads the digit.

Unmeasured

  • Which of two bad numbers the service reports: the answer varied between two runs of the same request (one then two). The engine reports the first in placeholder order; the wire test accepts either.
  • Whether a principal without Scan permission gets AccessDeniedException or the number ValidationException (no restricted principal was available). The engine runs the check in the operation handler only.
  • The order of three range faults in the older form, and the order of a shape fault against a ValidationException the deserializer raises (an empty set, a duplicate, an invalid AttributeValue): the engine reports the check's error for the latter.

…time

Scan did not check that the N values in ExpressionAttributeValues are
numbers, so a filter on {"N": "bob"}, a nested one in an L, M or NS, an
overflow, an underflow, a 39 digit value or an empty N answered 200 with
Count 0. The service refuses each with the same ValidationException text
Query uses, whether or not an expression reads the value.

Move the early Scan checks to a request-time function on the raw body,
in the order measured on 2026-10-25: the Limit, Segment and TotalSegments
ranges (one aggregated message, "Value at 'Limit'" with no value echoed),
the expression and non-expression parameter mix, the ReturnConsumedCapacity
enum, every number in ExpressionAttributeValues, then the Select enum.
Both enums invalid now answers the ReturnConsumedCapacity error alone, as
the service does; the earlier aggregation of the two no longer matched it.
The Segment pairing checks stay in the handler after the table lookup.

The two Scan tests that asserted the old segment range text now assert the
measured text.
…tion

Every N value shape the service refuses on Scan (a word, a nested L, M or
NS member, overflow, underflow, 39 digits, empty, a leading space), read by
the filter or by nothing, and the order against every other early check:
the Limit, Segment and TotalSegments ranges, the parameter mix and the
ReturnConsumedCapacity enum ahead of it; the Select enum, the Segment
pairing, the table lookup, IndexName, ConsistentRead, Select with
ProjectionExpression, ExclusiveStartKey, undefined placeholder and filter
syntax checks behind it. Expected texts measured on 2026-10-25.
… accept Unicode decimal digits

The range checks on Limit, Segment and TotalSegments reported every fault in
one aggregated message with a capitalised field name and no value for all
three fields. The service uses that form for a Limit inside the int32 range
and saturates the value into the older form below it (Limit -2147483649
answers Value '-2147483648' at 'limit'); the Segment and TotalSegments texts
and the aggregation are a separate question. The checks now report the first
fault in the order Limit, Segment, TotalSegments; Limit follows the measured
int32 rule and the two segment fields keep the texts of the previous release.
The two assertions in tests/test_query_scan.py go back to the previous text.

A field of the wrong JSON type in another field (a string
ExpressionAttributeValues beside Limit 0, a string TotalSegments beside
Segment -1, a string ExpressionAttributeNames beside a malformed N, an object
AttributesToGet beside ProjectionExpression) answered the check's
ValidationException where the service and the previous release answer
SerializationException. When a check fails the body is deserialized as the
typed input and a SerializationException from that is returned instead, with
the helper the Query path uses.

A Unicode decimal digit in an N value passes the request-time check shared
with Query, as it does on the service. The filter compares the value as the
request wrote it, as on the previous release.

The unit test for the parameter mix no longer asserts that an empty
ScanFilter beside a FilterExpression passes, which the service refuses as a
parameter mix; the engine's handling of an empty ScanFilter and an empty
AttributesToGet is unchanged and covered by a separate issue.

Measured against the service on 2026-10-25 (10:38, 10:55 and 11:31 UTC),
each request twice.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants