Repository navigation
fix(limits): size items and updates the way the service does - #409
LeeroyHannigan wants to merge 1 commit into
Conversation
38bcc5d to
ec8998c
Compare
| /// | ||
| /// Returns `DynamoDbError::ValidationException` with the update wording when | ||
| /// either the statement figure or the finished item exceeds `max_bytes`. | ||
| pub fn validate_update_item_size( |
There was a problem hiding this comment.
Since this appears to be the hairiest code in the PR, would it be possible to add at least some basic unit tests for it? (Or is it being covered by unit tests elsewhere?)
There was a problem hiding this comment.
It had no direct unit test; it was covered end to end by the wire suite (tests/test_item_size_at_limit.py, 58 cases through the storage layers). Added direct ones: an update_size_tests module in the same file with eight cases that pin the statement figure for a plain SET, a two-clause SET, a nested-map path, a list-index target and a REMOVE, plus the over-limit refusal and its wording, the finished-item refusal, and the legacy AttributeUpdates form. They run under cargo test with no server. The byte figures are the ones measured on the service, the same numbers the wire cases assert.
| | Limit | DynamoDB Value | Status | Notes | | ||
| |-------|---------------|--------|-------------------------------------------------------------------------------| | ||
| | Maximum item size | 400 KB (409,600 bytes) | Enforced | `LimitsConfig::max_item_size_bytes`, validated on PutItem and post-UpdateItem | | ||
| | Maximum item size | 400 KB (409,600 bytes) | Enforced | `LimitsConfig::max_item_size_bytes`, applied by the engine to PutItem, BatchWriteItem, the transaction Put path and ImportTable, and by the PostgreSQL and SQLite storage layers to UpdateItem and the transaction Update path. The Cassandra and MongoDB storage layers do not read the configured value on their update paths (Cassandra applies a fixed 409,600 to the finished item with the put wording; MongoDB applies no limit). An item is sized as attribute names plus values; a number costs `1 + ceil(integer digits / 2) + ceil(fraction digits / 2)` bytes plus one for a sign, the two halves rounded up separately (`1.5` is 3 bytes, `3.14159` is 5). UpdateItem is sized by the statement, not the finished item: 3, plus 18 per SET or ADD clause, plus a skeleton of the written paths (map entries name plus one, containers 3, list elements one more, untouched siblings left out) whose SET or ADD leaves cost the clause's value placeholders, plus one per clause through a list index, plus 26 and the name length per function call, 34 per `+`, 37 per `-`, and 15 plus the name (14 for a nested attribute, 15 for an index) per document path used as an operand; REMOVE and DELETE cost their path skeleton only. The legacy AttributeUpdates form costs 3 plus the name and finished value of each attribute it names that remains. The key and untouched attributes are left out, and the finished item never exceeds the limit either way. A transacted Update is sized on the finished item alone. Both update paths refuse with `Item size to update has exceeded the maximum allowed size`. Measured against the service in us-east-1 on 2026-10-19. | |
There was a problem hiding this comment.
We should document the calculation in one place and just have a pointer here. Most of this is implementation detail; not needed to understand that there's a limit and it is enforced.
There was a problem hiding this comment.
Done. The calculation now lives in one place, the Item Size section of docs/differences-from-dynamodb.md (how an item is sized, the number cost, and the statement-based UpdateItem rule). This row is back to the limit and where it is enforced, with a pointer to that section.
jcshepherd
left a comment
There was a problem hiding this comment.
Generally good: a couple questions about testing and documentation.
ec8998c to
7507e71
Compare
…es the way the service does On the Cassandra backend, UpdateItem and the Put and Update members of TransactWriteItems did not check secondary-index key attributes on the item they were about to write. An UpdateExpression that set a GSI or LSI key to an empty string, an empty binary, or a value of another type was accepted with HTTP 200 and stored, leaving an item whose index attribute no longer matched the index's declared type; later writes to that item failed inside the index row write with HTTP 500. Fixes #410. Measuring the fix against the service showed that the other backends, which did check the written item, did so in a different order and with different precedence from the service, so the measured rules are now applied on every backend from shared code. Every rule below was measured verbatim on live DynamoDB; the probe scripts and their recorded output live under conf-run in the working tree, not in the repository. UpdateItem: - A literal operand assigned to a secondary-index key (SET k = :v, ADD k :v, DELETE k :v, list_append of literals) is checked on the statement, ahead of the ConditionExpression and ahead of applying any action, so it is reported even when the condition would fail or another action could not be applied. Wording: "Type mismatch for Index Key <attr> Expected: <T> Actual: <U> IndexName: <index>". ADD of a number onto a number key passes; ADD or DELETE of a set, or ADD of a number onto a string or binary key, is a mismatch. - A value computed from the existing item (an attribute reference, the fallback of if_not_exists when the attribute is absent) is checked after the condition passes and the statement is applied, with the wording "The update expression attempted to update the secondary index key to unsupported type". if_not_exists(k, :bad) with k present is accepted, since nothing bad is written. - A key the statement leaves empty is refused after the condition passes, with the update-expression wording. - A failed condition is reported ahead of a statement that cannot be applied. TransactWriteItems: - One member: a wrong-typed index key is a positional ValidationError cancellation reason; an empty index key is a top-level ValidationException with no CancellationReasons. On a Put member both checks run ahead of the member's condition. On an Update member the literal type check runs ahead of the condition, and the computed-value and empty checks after it. Within one statement the type mismatch wins. - Across members: a wrong-typed key on any member, table key or secondary-index key, cancels the transaction with the first such member's ValidationError in position and None in every other position, whatever those members' own faults were. Otherwise an empty index key on any member is the top-level ValidationException, outranking failed conditions and unapplicable statements elsewhere. An empty index key on a member whose own condition fails is reported as that member's ConditionalCheckFailed. Changes: - core: validate_index_key_literal_operands (the statement-level pass), validate_index_key_types_after_update (the computed-value wording), and on CancellationReason two internal constructors plus resolve_key_faults, which applies the cross-member precedence and decides between a cancellation and a top-level ValidationException. The internal codes never reach a client. - engine: UpdateItem runs the literal pass before dispatching to storage, so every backend reports it in the same position. - storage-cassandra: update_item.rs checks the written item after apply; transactions.rs gains compute_transact_update, used by the first PREPARE pass and by the create-race retry so the two cannot drift, runs both checks ahead of the condition on a Put member, tags table-key failures in all four arms, and resolves the reasons after rollback. index_key_refs replaces the put path's inline copy. - storage-sqlite, storage-postgres and storage-mongodb: every member's key checks (table key, Put item index keys, Update literal operands) run in a pre-pass before any row is read or locked; a fault there decides the request under the precedence without touching a row, so a request that is already decided never waits on a row another transaction holds (the contract crates/storage-postgres/tests/twi_conflict.rs pins). The Update arms then run the computed-value check after apply; the per-op Validation variant that aborted on the first empty key is gone, and the reasons are resolved after every member has run. Standalone UpdateItem on all three now reports a computed value with the service's wording. - docs: the rules and the two known shared differences, in the Secondary Indexes table. Depends on #409 (fix/item-size-parity), which adds the PostgreSQL standalone UpdateItem index-key checks this change reuses; the branch is rebased on it. tests/test_index_key_validation_on_update.py adds 64 dual-target cases: each path, each fault, each index kind, the positional reason, the no-write guarantee, the ordering against a condition on Put and Update members, the cross-member precedence including table keys, if_not_exists, and the literal versus computed split. They pass against live DynamoDB and against servers built from this commit on Cassandra, SQLite, PostgreSQL and MongoDB, with no backend-specific skips. On main 26 of the first 31 fail on Cassandra. Workspace lib tests (1,232), fmt, clippy on 1.97.0 with -D warnings, and the 1.88 MSRV build are clean. Measured and left as they are, all shared across backends and pre-existing: an empty index key through the legacy AttributeUpdates parameter uses the update-expression wording where the service names the index; a request whose own value exceeds 400 KB is refused for size by the service before any index-key check; a transacted Update whose Key has the wrong type answers the Type mismatch wording where the service says the key element does not match the schema; the UpdateItem reserved-keyword message lacks the service's "1 validation error detected: " prefix; and the service's position of the literal-empty check relative to the condition varied across tables when measured repeatedly (one table of five answered empty-first), so the two tests pinning empty-after-condition can disagree with live DynamoDB on a table that lands the other way.
Five behaviours at the 409,600-byte item limit, each measured against Amazon DynamoDB (us-east-1, 2026-10-19) by an accept/refuse pair either side of the limit, and each wrong on main. Number byte cost. A number costs 1 + ceil(integer significant digits / 2) + ceil(fraction significant digits / 2), plus one for a sign. The two halves are rounded up separately. main counted all significant digits together, so every number with a fraction was one byte short: 1.5 costs 3 (main 2), 3.14159 costs 5 (main 4), 100.5 costs 4 (main 3). Sixteen measured literals are pinned in a unit test. Item limit on updates. The PostgreSQL and SQLite storage factories hardcoded 400_000 as the bound on the finished item of an UpdateItem or a transacted Update, while the engine applied the configured 409,600 to puts. The configured limit now reaches those two factories through ServerComponentsOptions, so on PostgreSQL and SQLite every write path enforces the one configured figure. The Cassandra and MongoDB factories do not read the option; Cassandra keeps its fixed 409,600 on the finished item and MongoDB applies no limit on updates, as before. UpdateItem is sized by the statement. The service sizes a standalone UpdateItem by the statement rather than the finished item: a base of 3; 18 per SET or ADD clause and nothing per REMOVE or DELETE; a skeleton item holding only the written paths (each map entry its name plus one, each map or list 3, each list element one more, untouched siblings left out), where a SET or ADD leaf costs the clause's value placeholders and a REMOVE or DELETE leaf costs nothing; one more for a clause through a list index; and, inside a SET operand, 26 plus the name length per function call, 34 per plus, 37 per minus, and for a document path operand 15 plus the name for its first element, 14 plus the name for a later attribute and 15 for an index, with the attribute's stored value not charged. The legacy AttributeUpdates form costs 3 plus the name and finished value of each attribute it names that remains. The key and untouched attributes are left out, so at a one-byte key a plain SET can produce a finished item of at most 409,581 bytes, and a long key or an untouched attribute buys the headroom back up to 409,600, which the finished item never exceeds. An attribute reached through an alias is charged as the attribute. A transacted Update is sized on the finished item alone: exactly 409,600 is stored and one byte over is a cancellation. Both update paths word the refusal "Item size to update has exceeded the maximum allowed size"; main used the put wording. validate_update_item_size and validate_transact_update_item_size in core carry the rules, called from the PostgreSQL and SQLite update paths; the engine marks a desugared AttributeUpdates request on the expression maps so storage sizes it as the legacy form. Empty binary index key on UpdateItem (PostgreSQL). UpdateItem setting a GSI key to an empty binary was written on PostgreSQL; SQLite and the service refuse it with "The update expression attempted to update a secondary index key to a value that is not supported. The AttributeValue for a key attribute cannot contain an empty binary value." #401 has since added those index key checks to the PostgreSQL standalone path, with the index attribute definitions from the write's own catalog read; this change keeps that version and adds no second copy. tests/test_item_size_at_limit.py carries 58 dual-target cases, 32 of them the largest accepted padding for one statement shape each (plain and multi-clause SET, same-map clauses, nested paths, list indexes with siblings, if_not_exists, list_append, nested functions, addition, subtraction, ADD, REMOVE, DELETE and six legacy AttributeUpdates forms). 58 of 58 pass against the service and against both backends with this change; 49 fail on main (PostgreSQL) and 48 (SQLite). The statement sizer also carries a direct unit test in core (validation::update_size_tests): eight cases pinning the byte figure for the common shapes, the over-limit refusal and its wording, the finished-item refusal, and the legacy form, so the arithmetic runs under cargo test without a server. docs/differences-from-dynamodb.md holds the sizing rules in one place (item bytes, number cost, the statement rule) and records the two regions (eu-south-2, me-central-1) that size an UpdateItem on the finished item instead; docs/dynamodb-limits.md states the limit and points there. The update-specific cases run on PostgreSQL and SQLite and against live DynamoDB; they are skipped on Cassandra and MongoDB, whose storage layers size updates differently as recorded in docs/differences-from-dynamodb.md. devtools/run-tests exports EXTENDDB_TEST_BACKEND so the file can gate those classes. Number sizing runs on every backend.
2737970 to
827edb9
Compare
|
Rebased on main at abf8082; conflicts in crates/storage-postgres/src/data/transactions.rs, resolved by keeping main's pub(super) on index_key_refs, which #401 already calls from update_item.rs. #401 added the same secondary-index key checks to the PostgreSQL standalone UpdateItem that this branch added, with the index attribute definitions taken from the write's own catalog read. Git merged update_item.rs without a conflict and kept both blocks, so this branch's copy and its import are removed; #401's block runs after the new update sizing. The commit message now says the checks come from #401, and the earlier merge of main is folded into the commit. tests/test_item_size_at_limit.py: 58 passed on PostgreSQL. |
…es the way the service does On the Cassandra backend, UpdateItem and the Put and Update members of TransactWriteItems did not check secondary-index key attributes on the item they were about to write. An UpdateExpression that set a GSI or LSI key to an empty string, an empty binary, or a value of another type was accepted with HTTP 200 and stored, leaving an item whose index attribute no longer matched the index's declared type; later writes to that item failed inside the index row write with HTTP 500. Fixes #410. Measuring the fix against the service showed that the other backends, which did check the written item, did so in a different order and with different precedence from the service, so the measured rules are now applied on every backend from shared code. Every rule below was measured verbatim on live DynamoDB; the probe scripts and their recorded output live under conf-run in the working tree, not in the repository. UpdateItem: - A literal operand assigned to a secondary-index key (SET k = :v, ADD k :v, DELETE k :v, list_append of literals) is checked on the statement, ahead of the ConditionExpression and ahead of applying any action, so it is reported even when the condition would fail or another action could not be applied. Wording: "Type mismatch for Index Key <attr> Expected: <T> Actual: <U> IndexName: <index>". ADD of a number onto a number key passes; ADD or DELETE of a set, or ADD of a number onto a string or binary key, is a mismatch. - A value computed from the existing item (an attribute reference, the fallback of if_not_exists when the attribute is absent) is checked after the condition passes and the statement is applied, with the wording "The update expression attempted to update the secondary index key to unsupported type". if_not_exists(k, :bad) with k present is accepted, since nothing bad is written. - A key the statement leaves empty is refused after the condition passes, with the update-expression wording. - A failed condition is reported ahead of a statement that cannot be applied. TransactWriteItems: - One member: a wrong-typed index key is a positional ValidationError cancellation reason; an empty index key is a top-level ValidationException with no CancellationReasons. On a Put member both checks run ahead of the member's condition. On an Update member the literal type check runs ahead of the condition, and the computed-value and empty checks after it. Within one statement the type mismatch wins. - Across members: a wrong-typed key on any member, table key or secondary-index key, cancels the transaction with the first such member's ValidationError in position and None in every other position, whatever those members' own faults were. Otherwise an empty index key on any member is the top-level ValidationException, outranking failed conditions and unapplicable statements elsewhere. An empty index key on a member whose own condition fails is reported as that member's ConditionalCheckFailed. Changes: - core: validate_index_key_literal_operands (the statement-level pass), validate_index_key_types_after_update (the computed-value wording), and on CancellationReason two internal constructors plus resolve_key_faults, which applies the cross-member precedence and decides between a cancellation and a top-level ValidationException. The internal codes never reach a client. - engine: UpdateItem runs the literal pass before dispatching to storage, so every backend reports it in the same position. - storage-cassandra: update_item.rs checks the written item after apply; transactions.rs gains compute_transact_update, used by the first PREPARE pass and by the create-race retry so the two cannot drift, runs both checks ahead of the condition on a Put member, tags table-key failures in all four arms, and resolves the reasons after rollback. index_key_refs replaces the put path's inline copy. - storage-sqlite, storage-postgres and storage-mongodb: every member's key checks (table key, Put item index keys, Update literal operands) run in a pre-pass before any row is read or locked; a fault there decides the request under the precedence without touching a row, so a request that is already decided never waits on a row another transaction holds (the contract crates/storage-postgres/tests/twi_conflict.rs pins). The Update arms then run the computed-value check after apply; the per-op Validation variant that aborted on the first empty key is gone, and the reasons are resolved after every member has run. Standalone UpdateItem on all three now reports a computed value with the service's wording. - docs: the rules and the two known shared differences, in the Secondary Indexes table. Depends on #409 (fix/item-size-parity); the branch is rebased on it. The PostgreSQL standalone UpdateItem index-key checks this change reuses came to main with #401. On PostgreSQL and SQLite the storage-side index key checks, the transaction pre-pass included, take the index attribute definitions from the write's own catalog read, as #401 does for the rest of the write path. #401's storage-level test update_item_rejects_a_wrong_typed_gsi_key calls the storage engine past the engine's statement pass, so it now expects the wording of the check on the written item. tests/test_index_key_validation_on_update.py adds 64 dual-target cases: each path, each fault, each index kind, the positional reason, the no-write guarantee, the ordering against a condition on Put and Update members, the cross-member precedence including table keys, if_not_exists, and the literal versus computed split. They pass against live DynamoDB and against servers built from this commit on Cassandra, SQLite, PostgreSQL and MongoDB, with no backend-specific skips. On main 26 of the first 31 fail on Cassandra. Workspace lib tests (1,232), fmt, clippy on 1.97.0 with -D warnings, and the 1.88 MSRV build are clean. Measured and left as they are, all shared across backends and pre-existing: an empty index key through the legacy AttributeUpdates parameter uses the update-expression wording where the service names the index; a request whose own value exceeds 400 KB is refused for size by the service before any index-key check; a transacted Update whose Key has the wrong type answers the Type mismatch wording where the service says the key element does not match the schema; the UpdateItem reserved-keyword message lacks the service's "1 validation error detected: " prefix; and the service's position of the literal-empty check relative to the condition varied across tables when measured repeatedly (one table of five answered empty-first), so the two tests pinning empty-after-condition can disagree with live DynamoDB on a table that lands the other way.
Summary
Five behaviours at the 409,600-byte item limit, each measured against Amazon DynamoDB (us-east-1, 2026-10-19) by an accept/refuse pair either side of the limit, and each wrong on main. The new dual-target test
tests/test_item_size_at_limit.pypasses 58 of 58 against the service and against PostgreSQL and SQLite with this change; 49 of its cases fail on main with PostgreSQL and 48 with SQLite.Number byte cost
A number costs
1 + ceil(integer significant digits / 2) + ceil(fraction significant digits / 2), plus one for a sign, the two halves rounded up separately. main counted all significant digits together, so every number with a fraction was one byte short.1.51.234100.53.1415915,0.15,1E125Sixteen literals are pinned in a core unit test; fifteen are also asserted on the wire.
Item limit on updates
The PostgreSQL and SQLite storage factories hardcoded
400_000as the bound on the finished item of an UpdateItem or a transacted Update, while the engine applied the configured 409,600 to puts.ServerComponentsOptionsnow carrieslimits.max_item_size_bytesto those two factories, so on PostgreSQL and SQLite every write path enforces one configured figure. The Cassandra and MongoDB factories do not read the option: Cassandra keeps its fixed 409,600 on the finished item, MongoDB applies no limit on updates. Both unchanged by this PR and noted indocs/dynamodb-limits.md.UpdateItem is sized by the statement
The service sizes a standalone UpdateItem by the statement rather than by the finished item. Measured figure:
+, 37 per-; a document path operand costs 15 plus the name for its first element, 14 plus the name for a later attribute, 15 for an index, and the attribute's stored value is not charged;AttributeUpdatesform costs 3 plus the name and finished value of each attribute it names that remains.The key and untouched attributes are left out, so at a one-byte key a plain SET can produce a finished item of at most 409,581 bytes, and a long key or an untouched attribute buys the headroom back up to 409,600, which the finished item never exceeds. Largest accepted padding at key
k, service and this branch identical on both backends:SET p = :vSET p = :v, q = :wSET m.a = :v, m.b = :wSET p[1] = :vbeside a 50-byte siblingSET p = if_not_exists(p, :v), p present or absentSET p = list_append(:v, :w)SET p = list_append(p, :v)onto a one-element listSET p = list_append(if_not_exists(p, :e), :v)SET p = :v, n = n + :wSET p = :v, n = :w - :xSET p = :v DELETE s :wAttributeUpdatesPUT pA transacted Update is sized on the finished item alone: exactly 409,600 is stored and one byte over cancels. Both update paths word the refusal
Item size to update has exceeded the maximum allowed size; main used the put wording and refused at 400,001.Two commercial regions (eu-south-2, me-central-1) size an UpdateItem on the finished item like a put; ExtendDB follows the other 31, recorded in
docs/differences-from-dynamodb.md.Empty binary index key on UpdateItem (PostgreSQL)
UpdateItem setting a GSI key to an empty binary was written on PostgreSQL. SQLite and the service refuse it with
The update expression attempted to update a secondary index key to a value that is not supported. The AttributeValue for a key attribute cannot contain an empty binary value.The PostgreSQL standalone path now runs the index key checks its transacted path and SQLite already ran.Behaviour change for users
Verification
tests/test_item_size_at_limit.py: 58 of 58 against the service, 58 of 58 on PostgreSQL and SQLite with this branch, 49 and 48 failures on main.cargo fmt --check,cargo +1.97.0 clippy --all-targets -- -D warnings,cargo +1.88.0 check --workspace --locked,cargo test --workspace, license notice checks: clean.Backend scope
The update-sizing rule and the empty-binary refusal are enforced by the PostgreSQL and SQLite storage layers. Cassandra sizes every update on the finished item against a fixed 409,600 bytes and MongoDB applies no update limit, both recorded in
docs/differences-from-dynamodb.md. The new test file's update-specific classes run on PostgreSQL, SQLite and against live DynamoDB, and are skipped on Cassandra and MongoDB;devtools/run-testsexportsEXTENDDB_TEST_BACKENDfor the gate. The number-cost cases run on every backend.Overlap with other open PRs
#401 adds the same secondary-index key check after the item-size check in
crates/storage-postgres/src/data/update_item.rs(validate_index_key_typesandvalidate_index_key_not_empty), validated against the definitions read with the indexes, which a GSI added during its online build needs. A merge of the two keeps both blocks and runs the check twice. Whichever merges second drops this block and keeps #401's.