Skip to content

fix(mongodb): reach map members under their M field in a pushed-down condition - #582

Open
LeeroyHannigan wants to merge 3 commits into
mainfrom
fix/mongodb-nested-map-path
Open

LeeroyHannigan wants to merge 3 commits into
mainfrom
fix/mongodb-nested-map-path

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Fixes #572.

On MongoDB, a condition on a map member was compiled to a filter path that skipped the map's M field, so it never matched. A condition that held was refused, and attribute_not_exists or <> on a member that was present let the write through.

What happened on main

On a table with no index and no stream, while the server's cache holds a fresh entry recording that the table has no index (60 seconds after a write), an UpdateItem or DeleteItem condition that the analyzer accepts is compiled to a MongoDB filter and checked by MongoDB. Items are stored in their typed form, so a map's members sit under its M field and a list's elements under its L field:

  • #m.#k compiled to item_data.m.k; the member is stored at item_data.m.M.k.
  • #m.#l[0] compiled to item_data.m.l.L.0; the element is stored at item_data.m.M.l.L.0.
  • #l[0].#k compiled to item_data.l.L.0.k; the member is stored at item_data.l.L.0.M.k.

attribute_exists, comparisons, begins_with, contains and attribute_type on those paths answered ConditionalCheckFailedException when they held. attribute_not_exists(#m.#k) and #m.#k <> :v on a present member let the write through. Top-level attributes and list indexes were compiled correctly.

The change

A member name after the top-level attribute now gets an M segment, as a list index already got an L segment. The rule reaches the stored value for maps in maps, lists in maps, maps in lists and lists in lists. A member named M, L, S or 0 is looked up inside the parent's M document, so it cannot be read as a type tag or an index.

What the service answers

Measured on 2026-10-25 at 20:51, 20:52 and 21:13 UTC in us-east-1, each request sent twice with the same answer: 21 shapes on map members, list elements inside maps and map members inside list elements, on UpdateItem and DeleteItem, holding and failing, plus 52 further shapes covering missing and wrong-type intermediates, out-of-range indexes, members named M, L, S, 0, x$, $x and a.b, empty maps, binary, BOOL and NULL members, sets and lists inside maps, and < and > on nested strings. On the 50 condition rows of the second set that ExtendDB can also run, SQLite matches the service on all 50 and MongoDB at the head on 47; the 3 it misses are the U+0000 rows under "Not changed here". MongoDB on the build before the fix matches 29.

Tests

  • crates/storage-mongodb/src/condition_tests.rs: unit tests for the compiled paths of map members, list elements inside maps and map members inside list elements. Removing the M segment, adding it before the first name, or adding it before an index each fails a test.
  • tests/test_condition_nested_paths.py: 21 dual-target wire tests from the measurements. Each test puts its own item right before the conditional write, so on MongoDB the write runs while the cache entry is fresh and reaches the compiled filter.

Run locally on 2026-10-25:

  • MongoDB (mongo:7 replica set): the new file 21 passed at the head; on the commit before the fix 13 failed and 8 passed, the same rows in two runs. The 8 that pass on the old build are rows whose answer the old path also gives (existence tests below a map member that answer "absent", list indexes, failing conditions).
  • MongoDB: 281 other condition, update and delete wire tests pass at the head. Three transaction isolation rows fail without EXTENDDB_TEST_MONGODB_CONTAINER; CI sets it, and with it set they are the expected failures main already marks.
  • SQLite and PostgreSQL: the new file 21 passed on each.
  • 6,000 generated conditions on nested items (paths and values drawn from the item, AND and OR of up to three leaves, names a, b, M, L, S, 0, N), sent with retries off: MongoDB at the head and SQLite gave the same answer for all 6,000, and MongoDB's profiler shows each one was checked by the compiled filter. The build before the fix differed from SQLite on 472.
  • cargo +1.97.0 fmt --all -- --check, cargo +1.97.0 clippy -p extenddb-storage-mongodb --all-targets -- -D warnings, and cargo test -p extenddb-storage-mongodb --lib condition (17 passed).

CI runs the new file in run-integration, run-integration-sqlite, mongodb-pytest and cassandra-pytest.

Not changed here

A map member whose name holds U+0000 answers InternalServerError on MongoDB, in PutItem and in a condition on it, where the service accepts both. That is on main and unchanged here; #368 escapes such names in storage.

Files

condition.rs was 759 lines on main. The first commit moves its test module unchanged into condition_tests.rs, loaded with #[path] as backup_engine.rs loads its test files. condition.rs is now 505 lines.

Merging

git merge-tree --write-tree is clean against #571 and #368, and against the other open PRs that touch crates/storage-mongodb. #571 counts AND, OR and NOT nesting; the M segment lengthens a field name and adds no nesting, so its count is unchanged.

condition.rs was 759 lines. The test module moves unchanged to
condition_tests.rs, loaded with #[path] as backup_engine.rs loads its test
files, so the next change can add tests without the file passing 600 lines.
…condition

On a table with no index and no stream, while the server's per-process cache
holds a fresh entry recording that the table has no index, a pushable
UpdateItem or DeleteItem condition is compiled to a MongoDB filter. The
compiler turned `#m.#k` into `item_data.m.k`, but each map's members are
stored under its `M` field (`item_data.m.M.k`), so a condition on a map
member never matched: `attribute_exists(#m.#k)` and `#m.#k = :v` answered
ConditionalCheckFailedException when they held, and `attribute_not_exists`
and `<>` on a present member let the write through when they failed.

A member name after the top-level attribute now gets an `M` segment, as a
list index already got an `L` segment. Amazon DynamoDB evaluates these
conditions as expected on map members, list elements inside maps and map
members inside list elements (measured 2026-10-25 at 20:51 and 20:52 UTC,
each request twice).
Dual-target wire tests from measurements on 2026-10-25 at 20:51 and 20:52
UTC: attribute_exists, attribute_not_exists, comparisons, begins_with,
contains and attribute_type on map members, list elements inside maps and map
members inside list elements, on UpdateItem and DeleteItem, holding and
failing. Each test puts its own item right before the conditional write, so
on MongoDB the write runs while the cache entry that enables the pushdown is
fresh.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MongoDB: a condition on a nested map member fails when it holds

3 participants