stack-encrypt: element match and jsonpath match have no typed Rust API — only the binding layer, with no compile-time checks #1459

Background

Stack Encrypt (packages/stack-encrypt) is the Rust engine every CipherStash language SDK is built on. It encrypts values and derives the search terms that EQL (the SQL library installed in the customer's PostgreSQL database) compares. For an encrypted JSON document column (public.eql_v4_json_search), a program asks for a query value through the column's plan, then binds that value as a SQL parameter:

let plan = Plan::value::<serde_json::Value>()
    .context("users/profile")
    .encrypt_with_layout(Json::new(JsonOptions::default().ordering(["$.age"])?))
    .build()?;

let q = cipher.query(op::AtPath(op::Gte), &("$.age", 18)).on(&plan).await?;

The crate has two layers:

• The typed API (cipher.query(op, &input).on(&plan), with operators from stack_encrypt::op). This is what a Rust program is meant to use.
• stack_encrypt::dynamic, behind the dynamic feature. Here an operator is chosen by name, and its inputs travel as JSON. It exists for language bindings (the Go SDK's WASI guest, and others to come), whose values arrive as wire data with no Rust type to name. Its module docs say so.

Problem

Two JSON query operations were added in the "JSONB path match on encrypted JSON" milestone:

• Element match: one element of an array holds several member predicates together (json_tree_element_match, json_tree: an equality and a range cannot be tied to one array element — return the token index from json_tree_subtree and derive per-element reads in the engine #1440 / feat(eql): test equality and ranges on one array element of a json_tree document #1445).
• Jsonpath match: a whole jsonpath filter, compiled into one operand (json_tree_match and @@, json_tree: a compiled predicate operand and eql_v4.json_tree_match, so a filter engine binds one predicate as one parameter #1441 / feat(eql): one-parameter jsonpath predicates on json_tree documents (json_tree_match, @@, match) #1446).

Both exist only in stack_encrypt::dynamic. There is no typed Rust interface for either. A Rust program that needs them writes the binding layer's calls:

use stack_encrypt::dynamic::{self, json_frame, OpSpec};

// One element of $.visits[*] holds both members (json_tree_element_match).
let term = dynamic::query_term(&keyset, &plan, "profile", OpSpec::ElementMatch,
    json_frame(&json!({"path": "$.visits[*]", "predicates": [
        {"member": "place", "op": "eq", "json": "\"home\""},
        {"member": "at",    "op": "gt", "json": "15"},
    ]}))?, &Documents)?.await?;

// A whole jsonpath filter (json_tree_match / @@).
let term = dynamic::query_term(&keyset, &plan, "profile", OpSpec::Match,
    json_frame(&json!({"predicate": "$.age > $min", "vars": "{\"min\": 30}"}))?,
    &Documents)?.await?;

What the compiler checks there is almost nothing:

• the field is a string ("profile");
• the operator is picked from an enum by name;
• the array path, the member names, and every value are strings or JSON text inside JSON ("json": "\"home\"");
• the caller passes a binding's target table (&Documents) and unpacks a QueryTerm, instead of getting back the typed query value every other operator returns.

A misspelt member, a member the field does not order, a value of the wrong type, or a member from a different array is caught only when the query runs, if at all.

The typed API's single-path operator, op::AtPath, is better: the compiler checks the call's shape (AtPath(op::Between) takes a pair of bounds, and AtPath(op::Contains) is refused with a compiler message). But it has the same gap underneath. The path is a string, parsed and checked against the field's ordered paths at run time. The value is not checked against the path's type, because the plaintext is a serde_json::Value.

Why this matters now

The Go SDK is getting compile-time checks for exactly these operations (#1451): generated, typed paths and member predicates, so that a misspelt member, a wrong value type, or a member of the wrong array does not compile. Once that lands, Go, a language SDK built on this engine, will give stronger guarantees than the engine's own Rust API.

That inverts docs/sdk-design-principles.md: Rust is the reference, the other SDKs take their words from it, and principle 3 says each check runs "at the earliest stage the language allows". In Rust the earliest stage is compile time, and we use it for neither operation.

A Rust customer with the same need as the Go one that motivated this milestone (a filter engine turning user filters into SQL over an encrypted JSON column) has no Rust interface to build on but the binding layer.

What is needed

Strict compile-time guarantees are a requirement, not a nice-to-have. A Rust program should be able to derive both operands through the typed API, and the compiler, not the query at run time, should refuse:

• a path or member the document does not have, or one the field does not order;
• a value whose type does not match the path's;
• a member predicate given to an array it does not belong to;
• an operation the path cannot answer (for example, an element match on a path that is not an array).

The calls should return typed EQL query values, as the other operators do (JsonSearchElementsQuery, JsonSearchPredicateQuery), rather than a QueryTerm.

How to get there is open, and is the main work of this issue. It covers both the two new operations and whether op::AtPath's string paths should move to the same model. One constraint to keep in view: some callers build predicates at run time from user input (the filter engine above). They still need a path to the same operands, checked as early as their input allows.

Relationship to other work

• json_tree: an equality and a range cannot be tied to one array element — return the token index from json_tree_subtree and derive per-element reads in the engine #1440 / feat(eql): test equality and ranges on one array element of a json_tree document #1445 (element match) and json_tree: a compiled predicate operand and eql_v4.json_tree_match, so a filter engine binds one predicate as one parameter #1441 / feat(eql): one-parameter jsonpath predicates on json_tree documents (json_tree_match, @@, match) #1446 (jsonpath match): the operations, available today only through stack_encrypt::dynamic.
• Go SDK: no way to test an equality and a range on the same array element of a JsonSearch field — add an element-scoped Match on top of #1440 #1442 / feat(go): test several members of one array element of a JsonSearch field #1451 and Go SDK: add Match(ctx, cipher, predicate, vars) on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443 / feat(go): bind a jsonpath predicate on a JsonSearch field as one parameter #1452: the Go interfaces for them, including the compile-time-checked paths in feat(go): test several members of one array element of a JsonSearch field #1451.
• docs/sdk-design-principles.md: principles 1 and 3 (every SDK), and Go principles 1 and 13.