Repository navigation
fix(mongodb): reach map members under their M field in a pushed-down condition - #582
Open
LeeroyHannigan wants to merge 3 commits into
Open
LeeroyHannigan wants to merge 3 commits into
LeeroyHannigan wants to merge 3 commits into
Conversation
condition.rs was 759 lines. The test module moves unchanged to condition_tests.rs, loaded with #[path] as backup_engine.rs loads its test files, so the next change can add tests without the file passing 600 lines.
…condition On a table with no index and no stream, while the server's per-process cache holds a fresh entry recording that the table has no index, a pushable UpdateItem or DeleteItem condition is compiled to a MongoDB filter. The compiler turned `#m.#k` into `item_data.m.k`, but each map's members are stored under its `M` field (`item_data.m.M.k`), so a condition on a map member never matched: `attribute_exists(#m.#k)` and `#m.#k = :v` answered ConditionalCheckFailedException when they held, and `attribute_not_exists` and `<>` on a present member let the write through when they failed. A member name after the top-level attribute now gets an `M` segment, as a list index already got an `L` segment. Amazon DynamoDB evaluates these conditions as expected on map members, list elements inside maps and map members inside list elements (measured 2026-10-25 at 20:51 and 20:52 UTC, each request twice).
Dual-target wire tests from measurements on 2026-10-25 at 20:51 and 20:52 UTC: attribute_exists, attribute_not_exists, comparisons, begins_with, contains and attribute_type on map members, list elements inside maps and map members inside list elements, on UpdateItem and DeleteItem, holding and failing. Each test puts its own item right before the conditional write, so on MongoDB the write runs while the cache entry that enables the pushdown is fresh.
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 25, 2026 22:22
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #572.
On MongoDB, a condition on a map member was compiled to a filter path that skipped the map's
Mfield, so it never matched. A condition that held was refused, andattribute_not_existsor<>on a member that was present let the write through.What happened on main
On a table with no index and no stream, while the server's cache holds a fresh entry recording that the table has no index (60 seconds after a write), an UpdateItem or DeleteItem condition that the analyzer accepts is compiled to a MongoDB filter and checked by MongoDB. Items are stored in their typed form, so a map's members sit under its
Mfield and a list's elements under itsLfield:#m.#kcompiled toitem_data.m.k; the member is stored atitem_data.m.M.k.#m.#l[0]compiled toitem_data.m.l.L.0; the element is stored atitem_data.m.M.l.L.0.#l[0].#kcompiled toitem_data.l.L.0.k; the member is stored atitem_data.l.L.0.M.k.attribute_exists, comparisons,begins_with,containsandattribute_typeon those paths answered ConditionalCheckFailedException when they held.attribute_not_exists(#m.#k)and#m.#k <> :von a present member let the write through. Top-level attributes and list indexes were compiled correctly.The change
A member name after the top-level attribute now gets an
Msegment, as a list index already got anLsegment. The rule reaches the stored value for maps in maps, lists in maps, maps in lists and lists in lists. A member namedM,L,Sor0is looked up inside the parent'sMdocument, so it cannot be read as a type tag or an index.What the service answers
Measured on 2026-10-25 at 20:51, 20:52 and 21:13 UTC in us-east-1, each request sent twice with the same answer: 21 shapes on map members, list elements inside maps and map members inside list elements, on UpdateItem and DeleteItem, holding and failing, plus 52 further shapes covering missing and wrong-type intermediates, out-of-range indexes, members named
M,L,S,0,x$,$xanda.b, empty maps, binary, BOOL and NULL members, sets and lists inside maps, and<and>on nested strings. On the 50 condition rows of the second set that ExtendDB can also run, SQLite matches the service on all 50 and MongoDB at the head on 47; the 3 it misses are the U+0000 rows under "Not changed here". MongoDB on the build before the fix matches 29.Tests
crates/storage-mongodb/src/condition_tests.rs: unit tests for the compiled paths of map members, list elements inside maps and map members inside list elements. Removing theMsegment, adding it before the first name, or adding it before an index each fails a test.tests/test_condition_nested_paths.py: 21 dual-target wire tests from the measurements. Each test puts its own item right before the conditional write, so on MongoDB the write runs while the cache entry is fresh and reaches the compiled filter.Run locally on 2026-10-25:
EXTENDDB_TEST_MONGODB_CONTAINER; CI sets it, and with it set they are the expected failures main already marks.a,b,M,L,S,0,N), sent with retries off: MongoDB at the head and SQLite gave the same answer for all 6,000, and MongoDB's profiler shows each one was checked by the compiled filter. The build before the fix differed from SQLite on 472.cargo +1.97.0 fmt --all -- --check,cargo +1.97.0 clippy -p extenddb-storage-mongodb --all-targets -- -D warnings, andcargo test -p extenddb-storage-mongodb --lib condition(17 passed).CI runs the new file in
run-integration,run-integration-sqlite,mongodb-pytestandcassandra-pytest.Not changed here
A map member whose name holds U+0000 answers InternalServerError on MongoDB, in PutItem and in a condition on it, where the service accepts both. That is on main and unchanged here; #368 escapes such names in storage.
Files
condition.rswas 759 lines on main. The first commit moves its test module unchanged intocondition_tests.rs, loaded with#[path]asbackup_engine.rsloads its test files.condition.rsis now 505 lines.Merging
git merge-tree --write-treeis clean against #571 and #368, and against the other open PRs that touchcrates/storage-mongodb. #571 counts AND, OR and NOT nesting; theMsegment lengthens a field name and adds no nesting, so its count is unchanged.