[coderdan]

Summary

EQL v4 stores an encrypted JSON document in a public.eql_v4_json_search column. On a plaintext jsonb column, an application that compiles a filter language into SQL (an AIP-160 filter engine, for example) emits one call, jsonb_path_match(col, $1, $2): a jsonpath predicate and its variables, two parameters. On the encrypted column the database cannot evaluate a path or a value, so each question is its own operator (@>, json_tree_path_exists, (col -> $sel) < $leaf, json_tree_element_match), and the same filter had to be assembled by hand, one parameter per leaf.

This PR gives the encrypted column a one-parameter form again. Stack Encrypt's new query operator match takes {"predicate", "vars"}, the same two things jsonb_path_match takes, parses the jsonpath subset the encrypted terms can answer, and compiles it into one operand: a boolean tree whose leaves are the operands the existing operators already take. The new SQL function eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate), or col @@ $1::eql_v4.query_json_predicate, evaluates the tree. Over the same documents it returns the rows jsonb_path_match returns over the plaintexts, and answers as it does row by row, apart from five documented cases (one of which, a NULL for false, never changes the rows). No stored format changes.

The trade-off is in the plan: the tree is evaluated inside a function, so json_tree_match filters a scan and engages no index. The per-operator form stays the fast path.

Changes

EQL (packages/eql)

• src/v3/json_tree/predicate.sql (new, v4 only):
  • the domain eql_v4.query_json_predicate and its recursive CHECK. Each node is an object with one key: and, or, not, contains (a needle), exists (a selector), cmp (one or two ordering bounds at one selector) or elements (an element-scoped predicate). Each leaf is validated by its own domain's validator: the needle's, the element predicate's, and a new shared bounds validator;
  • eql_v4.json_tree_match(e, p) RETURNS boolean, which follows SQL three-valued logic. Each leaf answers as its operator does on the column;
  • the @@ overload on (public.eql_v4_json_search, eql_v4.query_json_predicate). The jsonpath blocker stays.
• src/v3/json_tree/types.sql: eql_v4_internal.json_tree_is_valid_bounds. The element member's CHECK now calls it too, so one function defines the bounds shape.
• src/v3/json_tree/elements.sql:
  • The body of json_tree_element_match moved into eql_v4_internal.json_tree_element_match_opened(iv, sv, tags, p). Both functions call it, so a tree with several element leaves opens and indexes the row once.
  • New: eql_v4_internal.json_tree_within_bounds(entry, bounds), an inlinable SQL function that tests an entry against gt/gte/lt/lte. An element member and a cmp leaf both call it, so one function defines how bounds are evaluated.
  • json_tree_element_match's answers are unchanged, and its tests pass as before.
• Catalog: eql_domains::JSON_TREE gains the PredicateQuery role (JsonSearchPredicateQuery, eql_v4.query_json_predicate). Generated from it: the eql-bindings v4 inventory, and the Go type eql.JSONSearchPredicateQuery in languages/golang/encrypt/eql/eql_gen.go. Hand-written: JsonTreePredicateQuery and JsonTreeBounds in eql-bindings, and Target::predicate_query_sql_domain().
• Docs: a new "A compiled predicate in EQL v4" section in docs/reference/eql-functions.md: the node table with the per-operator equivalent of each node, the planner cost, the cast, and where the answer and jsonpath's differ. Also a leakage bullet in json-tree-leakage.md, and the file and domain in packages/eql/AGENTS.md.
• Changeset: @cipherstash/eql minor (.changeset/eql-json-tree-match.md).

Engine (packages/stack-encrypt)

• dynamic/json_predicate.rs (new): a hand-written recursive-descent parser and a compiler. No new dependency.
• dynamic/json_query.rs: match in the operator table, and a "Compiled predicates" module section (grammar, mapping, deviations, refusals). derive is split into reading and derive_input, so every leaf is derived by its own operator's existing code.
• dynamic/query.rs, target.rs, mod.rs: OpSpec::Match (wire name match, input predicate, fidelity exact), TargetDescriptor::with_predicate_query_sql_domain, a new error variant Error::Predicate { field, reason, construct, at }, and six new reasons.
• json/path.rs: JsonPath::of(segments) (crate-private).

The grammar and the compiled shape

Input: {"predicate": "<jsonpath>", "vars": "<JSON text of an object>"?, "wildcard_bound": N?}. vars is the third argument of jsonb_path_match, verbatim, so a compiler passes the same two values it passes today.

Accepted, as PostgreSQL reads them:

• paths $, .key, ."quoted key" (JSON escapes), [*];
• values: a variable ($v, $"v") or a JSON literal (a string, a number, true, false, null);
• comparisons ==, != (or <>), <, <=, >, >= between a path and a value, either side first;
• exists(path), and exists(path ? (cond && …)), each condition a comparison on @ or @.member;
• &&, ||, !(…) and parentheses.

How each comparison compiles:

jsonpath	compiled to
P == v, P == null	contains: eq_at's needle
P != v, P != null	and[exists(P), not(contains(P:v))]
P < v … P >= v	and[exists(P), not(contains(P:null)), cmp(P < v)]
P < v with a wildcard in P	elements: element_match on the array through P's last [*]
exists(P)	exists: the selector
exists(P ? (@.m == v))	contains at P.m (exact)
exists(P ? (@.m == v && @.n > w))	elements

The guards make an absent path and a JSON null answer as jsonpath answers them. Without them, !($.age < 3) on a row with no age would be NULL, where jsonpath says true.

Refusals #1443 must mirror. Each is reported as Error::Predicate, whose payload is reason, construct (a fixed string from the grammar, never text from the predicate) and at (the byte offset in the caller's predicate text):

• json_predicate_syntax: text the grammar does not read;
• json_predicate_unsupported: arithmetic, like_regex, starts with, is unknown, methods, strict/lax, .*, .**, slices, last, a comparison of two paths or two values, a filter outside exists, and ||, !, != or exists inside a filter;
• json_predicate_unknown_variable;
• json_predicate_value_type: an object or array anywhere; under an ordering, or in a filter of more than one condition, anything but a number or a string;
• json_predicate_not_equal_at_wildcard;
• json_predicate_too_deep: more than 32 levels of nesting, or a path deeper than a document;
• existing reasons, now with an offset: json_path_position ([2]), json_path_no_wildcard (a filter on a path with no [*]), json_path_wildcard (a [*] in a filter's member).

Unchanged, from the leaves' own operators: TermError::JsonPathNotOrdered, for a range on a path the column does not order, and json_wildcard_bound_zero, json_input and json_syntax for the input object itself.

Verification

All of this ran locally on macOS, against a private PostgreSQL 17 container on port 7641, after mise run build in packages/eql.

Engine (packages/stack-encrypt)

• mise x --env test -- cargo nextest run -p stack-encrypt --all-features: 1028 passed. New tests:
  • 12 parser and compiler unit tests in dynamic/json_predicate.rs. They cover each accepted form and what it compiles to, and every refusal by reason, construct and byte offset (about 90 cases). One test checks that a refusal never quotes text from the predicate, and another that a compiled tree's Debug shows only its operators.
  • 9 tests in tests/json_operators.rs (compiled_match):
    • each leaf equals the operand its own operator derives;
    • the boolean structure survives compilation;
    • a Rust model of json_tree_match over sealed documents, compared by a 64-case proptest with an independent jsonpath oracle over the plaintexts (three-valued, over absent fields, nulls, wildcards and filters);
    • fixed three-valued cases;
    • refusals through check_query, with the field and the offset;
    • the catalog listing, both a field's and a type's own operators().
  • The heap-hygiene watch covers match, with a marked literal and a marked variable.
• Root workspace, after the rebase onto feat(eql): test equality and ranges on one array element of a json_tree document #1445's new head: cargo clippy --workspace --all-targets -D warnings under default features and --all-features, mise run wasm:wasi-check, mise run wasm:no-http-test, cargo fmt --all --check, doctests (91 across the workspace, including a new example on OpSpec::Match that runs predicates through check_query), mise run doc:stack-encrypt, and stack-encrypt nextest --all-features (1028 passed): clean.
• cargo-mutants, cargo mutants -p stack-encrypt --in-diff <diff against #1445's head>: 274 mutants, 222 caught, 51 unviable, 1 timeout, 0 missed.
  • The first full run left 15 missed, all in the parser: nesting depth that only grows, a second filter after a filter, the last and mode-word operands, a quoted key after . , number literals with a fraction or a signed exponent, the depth limit on a path and on a filter's joined path, and Node's Debug.
  • For each, I added a test that pins it. A filtered re-run of those 15 against the new tests: 15 caught.
  • The timeout is += replaced by -= in the parser's cursor. That loops forever, which the run stops.
  • The unviable mutants are Default::default() replacements of types that have no Default.
  • After the review fixes: the 19 mutants on the lines the fixes change (--in-diff against the previous head, 6a3419d): 14 caught, 5 unviable, 0 missed.
  • After the rebase onto feat(eql): test equality and ranges on one array element of a json_tree document #1445's new head, a fresh --in-diff run was stopped part way on the owner's call to leave mutants to CI: 118 of 278 tested, 100 caught, 18 unviable, 0 missed.

EQL (packages/eql)

• SQLx lint_tests (12), build_validation_tests (5), selector_literal_guard (10) and v3_public_surface_tests (7): passed.

• mise run test:encryption:postgres on a fresh database: all suites passed. json_tree is 36 tests, 8 of them new here, every one over real Stack Encrypt documents and predicates compiled by match through eql-bindings' target table. EQL tests must run over real encrypted data and the SQLx suite has no way to produce a JSON tree document, so the SQL's contract, which a SQLx file with a SQL stand-in for the JSON index pinned before, is pinned here. Six of the new tests cover:

  • the domain CHECK, node kind by node kind (9 valid, 21 invalid, over a compiled tree and its parts), plus the element member CHECK through the shared bounds validator;
  • for 18 predicates over 7 documents: json_tree_match equals the WHERE written from the compiled tree's own leaves (@>, json_tree_path_exists, -> with the orderings, json_tree_element_match), @@ equals json_tree_match, and all are true of exactly jsonb_path_match's rows over the plaintext, answering as it does row by row save NULL for false where an ordering over a null sends no null needle;
  • NULL leaves under and, or and not, and an ordering leaf alone (NULL where the path is absent or null);
  • contains equals @>, including the root scalar's alt;
  • an elements leaf equals json_tree_element_match; a leaf the answer does not depend on is not read and does not raise, one it does depend on raises 22023;
  • @@ with a typed operand or a bound parameter works, @@ '…'::jsonpath still raises, an untyped literal is refused with 42725, and a NULL argument gives NULL.

  The other two:

  • for 24 predicates (with variables, wildcards, filters, !, ||, and !=/== null/absent cases): json_tree_match and @@ are true of exactly jsonb_path_match(plain, predicate, vars)'s rows, with the same answers save that NULL for false;
  • a mixed predicate equals the WHERE written from each leaf operator's own operand (json_tree_path_exists, @>, -> with >=, and json_tree_element_match), and also equals jsonb_path_match;
  • refusals carry their offsets through the data plan.

• mise run test:crates (498 tests), test:encryption (543), types:check, typescript:check and codegen:parity: passed.

• docs:validate:required-tags, docs:validate:coverage, docs:validate:documented-sql (against my own container), test:doc-anchors and test:docs_v3_grep: passed.

• The v3 and v4 bundles install side by side, and eql_v3.lints() and eql_v4.lints() both return 0 rows. The v3 bundle contains none of the new SQL.

Go

• mise run wasm:guest:test: passed, including the new guest test above.
• All five guests rebuilt: wasm:guest:build, wasm:auth-guest:build, wasm:guest:build:deterministic, wasm:guest:build:eql and wasm:guest:build:eql:deterministic. Then mise run go:test passed for every package.

Not run

• v3_v4_coexistence_tests does not compile here. It reads the fixtures that need CipherStash credentials, and this environment has none (as on feat(eql): test equality and ranges on one array element of a json_tree document #1445). I checked its lint assertion by hand, as above.
• pnpm run code:check: no JS or TS changed.

Related

Closes #1441

Stacked on #1445 (which is on #1444). Refs #1443 (the Go Match call on top).

Review notes

• The answer matches jsonpath except in five places. Four come from the stored document or the ordering terms, not from the predicate; the fifth (a NULL where jsonpath says false) never changes the rows. All five are documented in eql-functions.md and the module docs:

  1. A path is read as written, so lax mode's array unwrapping (an array where the predicate has no [*]) is not applied.

  2. Lax mode's wrapping is not applied either. Where the predicate names [*] and the document holds a value that is not an array, jsonpath wraps it, and the tree finds nothing. For example, $.tags[*] == "a" over {"tags": "a"} is true to jsonpath and false here, because $.tags[*] and $.tags have different selectors.

  3. A comparison of a string with a number is unknown (NULL) to jsonpath, but the terms do not record a value's kind. So it is false under ==, true under !=, and decided by the terms under an ordering.

  4. Orderings, and filters with more than one condition (which compile to element_match), compare the folded ordering terms.

  5. An ordering of a JSON null is NULL rather than false, except under an odd number of !. That picks the same rows, so a WHERE is unaffected; a SELECT of the boolean can show NULL. See the next note.

  The parity tests use documents where the first four do not apply. They require the same rows as jsonb_path_match, and the same answers save the fifth case's NULL for false (assert_jsonpath_rows).

• The null needle is sent only under an odd number of ! (owner decision). jsonpath answers false of an ordering over a JSON null, where the ordering's term comparison is NULL. Only a ! turns that difference into different rows: and and or are monotone, so a child's NULL cannot become true, and under an even number of ! the rows are the same. So an ordering compiles to and(exists, cmp), and under an odd number of ! to and(exists, not(contains null), cmp). The needle tells the database which rows hold null at the path, so it now goes only where it changes an answer's rows.

  • Tests: an_ordering_sends_the_null_needle_only_under_an_odd_number_of_negations pins the placement, including !!, a ! over ||, mixed depths, and element leaves (which never carry it). The proptest oracle (now 256 cases) requires jsonpath's rows for every random predicate and document, and allows only NULL for false in the answer. The PostgreSQL parity tests check the same against jsonb_path_match, with negated, doubly negated and mixed predicates added.

• Refusals for an undeclared path carry no offset. A range on a path the column does not order still fails with the existing TermError::JsonPathNotOrdered, as the brief asked. That error carries neither the path text (the crate keeps key names out of errors) nor an offset. Grammar refusals carry an offset. If you want the index-level refusals to carry the offset too, that is a small follow-up: wrap them in Error::Predicate with the path's offset.

• TargetDescriptor::operators() now filters JSON operators by named domain. Without that, a type's own list showed match even when the type named no predicate domain. The Go guest reads that list, and failed with unknown operator "match" in four Go tests. element_match never hit this, because it also needs an ordered path. Now the type list and the field list agree, and a test pins it. This is the one change outside the new operator.

• The untyped @@ operand. Before this PR, col @@ '…' with an untyped literal reached the jsonpath blocker and raised. Now it raises 42725 (operator is not unique), still loudly. The docs say to cast.

• json_tree_match opens the row once. It decompresses the document once and reads q once. It builds the tag index only when the tree has an element leaf, which it checks with one $.**.elements walk of the predicate per row (the predicate is small). A flag in the operand would remove that walk, but would change the operand shape; I left the shape as it is. A contains leaf applies json_tree_contains's two-line rule to the projection it has already read. Calling json_tree_contains itself would re-read the TOASTed column for every leaf. Wrapping the rule in a shared SQL helper would stop the planner inlining @> into the GIN-indexed form, because the argument is costly and used twice.

• What Go SDK: add Match(ctx, cipher, predicate, vars) on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443 needs (the Go guest does not name the domain, so the Go SDK sees no new operator today):

  • in languages/golang/encrypt/guest/src/targets.rs: with_predicate_query_sql_domain(target.predicate_query_sql_domain()), and with_elements_query_sql_domain(...) for Go SDK: no way to test an equality and a range on the same array element of a JsonSearch field — add an element-scoped Match on top of #1440 #1442. A new guest test, a_document_names_the_leaf_domain_and_not_the_predicate_domains, pins that the guest names neither domain today, so it fails until Go SDK: no way to test an equality and a range on the same array element of a JsonSearch field — add an element-scoped Match on top of #1440 #1442/Go SDK: add Match(ctx, cipher, predicate, vars) on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443 update it alongside the Go side;
  • "match" in parseOperatorName (encrypt/checker.go) and in stashgen's jsonOperators;
  • the input shape "predicate" in checker.go's list of shapes;
  • Error::Predicate in the guest's status_for_dynamic. Its catch-all maps it to STATUS_INTERNAL until then;
  • the reasons listed above. The payload keys are reason, construct and at.

• JsonSubtreeKey::below (from feat(eql): test equality and ranges on one array element of a json_tree document #1445) is now gated on dynamic there, and the one helper this PR adds (JsonPath::of) is gated the same way, so clippy passes with default features and with --all-features.

https://claude.ai/code/session_01MrxG724qYKcAWJdcABBFNY

[changeset-bot]

🦋 Changeset detected

Latest commit: c6da7e2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@cipherstash/eql	Minor
stash	Patch
@cipherstash/stack-prisma	Patch
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/prisma-example	Patch
@cipherstash/stack-drizzle	Patch
@cipherstash/stack-supabase	Patch
@cipherstash/stack	Patch
@cipherstash/wizard	Patch
@cipherstash/bench	Patch
@cipherstash/test-kit	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI; Enforced overrides: Organization
• Review profile: CHILL
• Plan: Advanced
• Run ID: 276e5c7f-bd39-479d-b9c6-cff121e2784e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

[github-actions]

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught	missed	unviable	timeout
227	0	51	0

Every mutant in the changed lines was caught by a test.

[coderdan]

Force-pushed after the mutation run: the stack-encrypt commit now also carries the parser tests that kill the 15 mutants the first run missed (nesting depth, a second filter, last and mode words as operands, a quoted key after . , fractional and signed-exponent numbers, the depth limits, and Node's Debug). No behaviour changed. The description has the final counts: 274 mutants, 222 caught, 51 unviable, 1 timeout, 0 missed.

[coderdan]

A single-equality filter whose base + member path exceeds MAX_DEPTH returns term(TermError::JsonTooDeep) (an Error::Cipher), while the PR/module docs say a path deeper than a document is Error::Predicate { reason: JsonPredicateTooDeep, at }. member_of has the same shape for the multi-condition filter. A client mirroring the refusals (#1443) gets a different error type with no offset for this one case; use the Refusal { JsonPredicateTooDeep, C_NESTING, at: path.at } form as steps() does.

[coderdan]

Fixed in 6a31ad7.

• The compiler checks the depth of base + member for every condition of a filter, before anything is derived.
• A filter's joined path deeper than a document is now Error::Predicate { reason: JsonPredicateTooDeep, construct: "nesting", at: <the filter's path> }. This holds for the single-equality filter and the multi-condition (element_match) filter alike, so neither surfaces TermError::JsonTooDeep any more.
• Tests cover both forms, and the path exactly as deep as a document still compiles.

[coderdan]

Rebased onto #1445's new head (15e9c56); new head 500aa58.

Why: #1444 and #1445 changed under review (#1444's tests moved to real Stack Encrypt documents and gained upgrade note U-004; #1445 got the same test move and the clippy fix that gates JsonSubtreeKey::below on dynamic).

This PR's SQLx file v4_json_tree_match_tests.rs sealed its documents with a SQL stand-in for the JSON index, which the rule on real encrypted data does not allow, so its six tests moved into packages/eql/tests/encryption/tests/json_tree/predicate.rs, over documents Stack Encrypt seals and trees match compiles: the domain CHECK node by node, the WHERE written from each compiled tree's own leaves for every predicate, three-valued logic, contains with alt, elements leaves, and @@. The SQL commit no longer adds the SQLx file; the test commit carries them. JsonPath::of was already gated on dynamic, and clippy passes under default features and --all-features.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T10:45:55.702803Z	500aa58	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-30T10:48:31.744744Z	500aa58	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[cipherstash-agent]

Review attestation: automated code review (medium) plus cross-check against existing PR feedback, at head 500aa581.

Result: no critical or blocking issues. No new findings.

What was reviewed, read at head 500aa581:

• packages/eql/src/v3/json_tree/predicate.sql, with the types.sql and elements.sql refactors. I checked the three-valued and / or / not, the short-circuit that keeps an unread element leaf from raising, the cmp leaf routed through json_tree_within_bounds, the contains rule against json_tree_contains, and the @@ overload.
• The compiler half of packages/stack-encrypt/src/dynamic/json_predicate.rs: comparison, filter, member and simplify. The != and ordering guards, the not(not x) and dedupe rewrites, and the depth checks preserve the stated answers.
• The eql-bindings, TargetDescriptor and Go-guest wiring diffs.

Cross-check:

• The 7 existing review threads (doctests and docs, guest wiring, the JsonTooDeep refusal, keyword case, lax wrapping, the per-row @? walk, json_tree_within_bounds reuse) have all been answered with a fix or a stated deferral, and the head still matches those replies. Nothing is re-raised here.
• The changeset covers @cipherstash/eql. No skills/* file mentions the json_tree operators, so no skill drift.

Not verified: I did not run the Rust or PostgreSQL suites, and read the ~1.8k-line parser and the test files only selectively. The PR description's results and the cargo-mutants report are the evidence for those.

Non-blocking observation: json_tree_is_valid_predicate_payload recurses without a depth bound. The engine caps nesting at 32, but a hand-built operand nested thousands deep would fail with a stack-depth error (54001) rather than a CHECK violation. It fails loudly, so this is optional hardening.

---

Generated by Claude Code

[coderdan]

New head: 6109e38. The stack-encrypt commit now refuses a path step past MAX_DEPTH inside the step loop, before the step is read. That turns the steps mutant that used to time out (+= replaced with -= on the name length, which made the parser re-read one key forever) into a refusal that the tests catch. It also adds a test for that refusal. A targeted local cargo-mutants run over the earlier survivors (ascend, the exists ? arm, the operand guards, steps, number_len, Debug for Node, Compiler::filter) caught all 73 mutants: 71 caught, 2 unviable, 0 missed, 0 timeouts.

https://claude.ai/code/session_01MrxG73fnX1p7oiizh3GRG5

[coderdan]

The typed Rust interface for the operator this PR adds is #1459: today it is reachable only through stack_encrypt::dynamic, with no compile-time checks.