Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/eql-json-tree-match.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
"@cipherstash/eql": minor
---

EQL v4 `json_tree`: a whole filter as one parameter, `eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate)` and `col @@ $1::eql_v4.query_json_predicate` (#1441).

An application that compiles a filter language into SQL targets `jsonb_path_match(col, $1, $2)` on a plaintext column: one predicate, one set of values. On a `public.eql_v4_json_search` column each question is its own operator, so the same filter had to be assembled by hand, one operand per leaf. The new domain `eql_v4.query_json_predicate` is a boolean tree (`and`, `or`, `not`) whose leaves are the operands those operators already take (a `contains` needle, an `exists` selector, `cmp` ordering bounds and an `elements` predicate), each checked by its own domain's rule, and `eql_v4.json_tree_match` evaluates it with SQL's three-valued logic, the answer the hand-written WHERE gives. Stack Encrypt's `match` query operator compiles a jsonpath predicate in the subset the terms can answer into the tree. The tree is evaluated inside a function, so it filters a scan and engages no index; the operators written out per leaf are the fast path. The operand must carry its type: an untyped operand on the right of `@@` is refused as ambiguous, and `@@` with a `jsonpath` is still blocked.

`eql_v4.json_tree_element_match` now reads an opened document through `eql_v4_internal.json_tree_element_match_opened`, which `json_tree_match` shares; its answers are unchanged.
23 changes: 23 additions & 0 deletions languages/golang/encrypt/eql/eql_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions languages/golang/encrypt/guest/src/targets.rs
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,35 @@ mod tests {
));
}

/// The JSON document's query domains this guest names, and so the JSON
/// operators the Go SDK is offered: the leaf operand, for the orderings
/// at a path, and not the element-scoped or compiled predicate, whose
/// operators (`element_match`, `match`) the Go side does not parse yet.
/// Naming one is a change to the Go SDK's operator list: update this
/// test with the Go side that takes the operator.
#[cfg(feature = "eql")]
#[test]
fn a_document_names_the_leaf_domain_and_not_the_predicate_domains() {
let targets = resolver().targets();
let document = targets
.iter()
.find(|t| t.name == "JsonSearch")
.expect("JsonSearch");
assert_eq!(
document.leaf_query_sql_domain.as_deref(),
Some("eql_v4.query_json_leaf")
);
assert_eq!(document.elements_query_sql_domain, None);
assert_eq!(document.predicate_query_sql_domain, None);
let operators = document.operators();
for absent in [
stack_encrypt::dynamic::OpSpec::ElementMatch,
stack_encrypt::dynamic::OpSpec::Match,
] {
assert!(!operators.contains(&absent), "{absent:?}: {operators:?}");
}
}

#[cfg(feature = "eql")]
#[test]
fn eql_refusals_convert_to_the_engines_with_the_field_left_to_the_lowering() {
Expand Down
2 changes: 1 addition & 1 deletion packages/eql/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Where the two versions' SQL must differ, the source holds both forms in **versio

#### Files confined to one EQL version

A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Three surfaces do today. The generated range surface: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7). And the hand-written JSON tree family under `src/v3/json_tree/` (`types.sql`, `functions.sql`, `operators.sql`, `elements.sql`, `blockers.sql`): `public.eql_v4_json_search`, `public.eql_v4_json_entry`, `eql_v4.query_json`, `eql_v4.query_json_leaf` and `eql_v4.query_json_elements`, the document Stack Encrypt's JSON index writes, under the names v3 gives its searchable JSON document but not a port of the SteVec surface (design: `docs/plans/2026-10-17-json-tree-index.md` at the repository root; bindings: `crates/eql-bindings/src/v4/json_tree.rs`, hand-written like the SteVec structs; catalog: `eql_domains::DOCUMENT_FAMILIES`, whose stored type is the target `JsonSearch`). Both are `-- ONLY EQL VERSION: 4`. The third surface goes the other way: the SteVec SQL under `src/v3/json/` and the generated `json_entry_<T>` cross files under `src/v3/scalars/<T>/` are `-- ONLY EQL VERSION: 3`, because in v4 their domain names belong to the JSON tree family. EQL v3 is what cipherstash-client writes, nothing there produces a range or a JSON tree document, and its SQL must not change: the v3 bundle differs from an unconfined build only by those directive comment lines. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs the whole v4 install order (`build/render/eql_v4/deps-ordered.txt`) in one rolled-back transaction, so every statement of the confined files runs after what it requires, besides running each file of the v3 render on its own. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the files confined to either version; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Three surfaces do today. The generated range surface: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7). And the hand-written JSON tree family under `src/v3/json_tree/` (`types.sql`, `functions.sql`, `operators.sql`, `elements.sql`, `predicate.sql`, `blockers.sql`): `public.eql_v4_json_search`, `public.eql_v4_json_entry`, `eql_v4.query_json`, `eql_v4.query_json_leaf`, `eql_v4.query_json_elements` and `eql_v4.query_json_predicate`, the document Stack Encrypt's JSON index writes, under the names v3 gives its searchable JSON document but not a port of the SteVec surface (design: `docs/plans/2026-10-17-json-tree-index.md` at the repository root; bindings: `crates/eql-bindings/src/v4/json_tree.rs`, hand-written like the SteVec structs; catalog: `eql_domains::DOCUMENT_FAMILIES`, whose stored type is the target `JsonSearch`). Both are `-- ONLY EQL VERSION: 4`. The third surface goes the other way: the SteVec SQL under `src/v3/json/` and the generated `json_entry_<T>` cross files under `src/v3/scalars/<T>/` are `-- ONLY EQL VERSION: 3`, because in v4 their domain names belong to the JSON tree family. EQL v3 is what cipherstash-client writes, nothing there produces a range or a JSON tree document, and its SQL must not change: the v3 bundle differs from an unconfined build only by those directive comment lines. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs the whole v4 install order (`build/render/eql_v4/deps-ordered.txt`) in one rolled-back transaction, so every statement of the confined files runs after what it requires, besides running each file of the v3 render on its own. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the files confined to either version; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.

Anything that needs the source as real SQL reads the rendered tree, not `src/`: `docs:validate:documented-sql` runs psql over `build/render/eql_v3/src`, and the Doxygen input filter (`tasks/docs/doxygen-filter.sh`) reads each file's rendered copy from `build/render/eql_v3/`. A new tool that parses `src/v3` must do the same, and depend on `build`; do not substitute the placeholders a second way.

Expand Down
12 changes: 12 additions & 0 deletions packages/eql/crates/eql-bindings/src/encryption/targets.rs
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,18 @@ impl Target {
.map(|_| crate::v4::json_tree::JsonTreeElementsQuery::sql_domain_static())
}

/// The PostgreSQL domain of the type's compiled predicate, for a type
/// that stores a JSON document (`JsonSearch`:
/// `eql_v4.query_json_predicate`); `None` for every other type. Stack
/// Encrypt binds `match` as it
/// (`TargetDescriptor::with_predicate_query_sql_domain`), and answers
/// `match` only on a type that names it.
pub fn predicate_query_sql_domain(&self) -> Option<&'static str> {
use crate::v4::DomainType;
document(self.name)
.map(|_| crate::v4::json_tree::JsonTreePredicateQuery::sql_domain_static())
}

/// The options this type's match index is built under, for a type that
/// carries one (`TextMatch`): [`EqlMatch`](super::EqlMatch)'s, the one
/// match configuration every EQL type's filter is built under. `None`
Expand Down
1 change: 1 addition & 0 deletions packages/eql/crates/eql-bindings/src/v4/inventory.rs
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ pub fn all_query() -> Vec<Box<dyn DomainType>> {
Box::new(PhantomData::<super::json_tree::JsonTreeQuery>),
Box::new(PhantomData::<super::json_tree::JsonTreeLeafQuery>),
Box::new(PhantomData::<super::json_tree::JsonTreeElementsQuery>),
Box::new(PhantomData::<super::json_tree::JsonTreePredicateQuery>),
]
}
/// Every v4 RANGE query operand (`eql_v4.query_<family>_range_<term>`),
Expand Down
93 changes: 93 additions & 0 deletions packages/eql/crates/eql-bindings/src/v4/json_tree.rs
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,57 @@ pub struct JsonTreeElementsQuery {
pub m: Vec<JsonTreeElementsMember>,
}

/// The bounds of an ordering at one path: one or two ordering operands at
/// the path's selector, `gt` or `gte` the lower and `lt` or `lte` the upper.
/// The `cmp` leaf of a [`JsonTreePredicateQuery`]. The domain's CHECK also
/// holds what this type does not: at least one bound, at most one of each
/// side, and every bound at one selector.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)]
#[serde(deny_unknown_fields)]
pub struct JsonTreeBounds {
#[serde(default, skip_serializing_if = "Option::is_none")]
#[ts(optional)]
pub gt: Option<JsonTreeLeafQuery>,
#[serde(default, skip_serializing_if = "Option::is_none")]
#[ts(optional)]
pub gte: Option<JsonTreeLeafQuery>,
#[serde(default, skip_serializing_if = "Option::is_none")]
#[ts(optional)]
pub lt: Option<JsonTreeLeafQuery>,
#[serde(default, skip_serializing_if = "Option::is_none")]
#[ts(optional)]
pub lte: Option<JsonTreeLeafQuery>,
}

/// `eql_v4.query_json_predicate`: a compiled predicate, a boolean tree
/// whose leaves are the family's other query operands, which
/// `eql_v4.json_tree_match` and `col @@ $1` evaluate. What Stack Encrypt's
/// `match` operator compiles a jsonpath predicate into. Each node is an
/// object of one key naming its kind: `{"and": [node, …]}`, `{"or": […]}`,
/// `{"not": node}`, `{"contains": needle}`, `{"exists": selector}`,
/// `{"cmp": bounds}` and `{"elements": predicate}`. The domain's CHECK also
/// holds what this type does not: at least one child under `and` and `or`.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)]
#[serde(rename_all = "snake_case", deny_unknown_fields)]
pub enum JsonTreePredicateQuery {
/// True when every child is; false when one is false; else NULL.
And(Vec<JsonTreePredicateQuery>),
/// True when one child is; false when every child is false; else NULL.
Or(Vec<JsonTreePredicateQuery>),
/// The child's negation, NULL of NULL.
Not(Box<JsonTreePredicateQuery>),
/// The document contains the needle, `col @> needle`.
Contains(JsonTreeQuery),
/// The document has a node at the path, `json_tree_path_exists`.
Exists(JsonTreeSelector),
/// The first entry at the bounds' selector against each bound,
/// `(col -> s) > gt AND (col -> s) < lt`.
Cmp(JsonTreeBounds),
/// One element of an array satisfies every member,
/// `json_tree_element_match`.
Elements(JsonTreeElementsQuery),
}

/// The entry `col -> selector` returned, with the selector that asked for
/// it: opens to the scalar the entry holds, or to the whole document for a
/// sealed root asked for at `$`. The only constructor,
Expand Down Expand Up @@ -330,6 +381,7 @@ json_tree_domain_type!(JsonTreeEntry, "public.eql_v4_json_entry");
json_tree_domain_type!(JsonTreeQuery, "eql_v4.query_json");
json_tree_domain_type!(JsonTreeLeafQuery, "eql_v4.query_json_leaf");
json_tree_domain_type!(JsonTreeElementsQuery, "eql_v4.query_json_elements");
json_tree_domain_type!(JsonTreePredicateQuery, "eql_v4.query_json_predicate");

// =============================================================================
// Conversions to and from the engine's types, through their one wire form
Expand Down Expand Up @@ -594,6 +646,47 @@ mod tests {
JsonTreeElementsQuery::sql_domain_static(),
"eql_v4.query_json_elements"
);
assert_eq!(
JsonTreePredicateQuery::sql_domain_static(),
"eql_v4.query_json_predicate"
);
}

#[test]
fn a_predicate_query_is_a_tree_of_one_key_nodes() {
let leaf = serde_json::json!({"s": b64(0xaa), "op": "AQI="});
let needle = serde_json::json!({"sv": [{"s": b64(1), "vs": b64(2)}], "el": []});
let elements = serde_json::json!({"m": [{"t": [b64(1)], "gt": leaf}]});
let query = serde_json::json!({"and": [
{"exists": b64(0xaa)},
{"not": {"cmp": {"gte": leaf, "lt": leaf}}},
{"or": [{"contains": needle}, {"elements": elements}]},
]});
let parsed: JsonTreePredicateQuery = serde_json::from_value(query.clone()).unwrap();
let JsonTreePredicateQuery::And(children) = &parsed else {
panic!("the root is an and: {parsed:?}")
};
assert!(
matches!(children[0], JsonTreePredicateQuery::Exists(_)),
"{children:?}"
);
assert_eq!(
serde_json::to_value(&parsed).unwrap(),
query,
"a tree reads and writes as itself"
);
for wrong in [
serde_json::json!({"exists": b64(1), "not": {"exists": b64(1)}}),
serde_json::json!({"xor": []}),
serde_json::json!({"cmp": {"eq": leaf}}),
serde_json::json!({"exists": 1}),
serde_json::json!([]),
] {
assert!(
serde_json::from_value::<JsonTreePredicateQuery>(wrong.clone()).is_err(),
"{wrong} should not deserialize"
);
}
}

#[test]
Expand Down
1 change: 1 addition & 0 deletions packages/eql/crates/eql-codegen/src/bindings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1084,6 +1084,7 @@ pub fn render_v4_inventory_rs() -> String {
eql_domains::DocumentRole::Query,
eql_domains::DocumentRole::LeafQuery,
eql_domains::DocumentRole::ElementsQuery,
eql_domains::DocumentRole::PredicateQuery,
]);
quote! { #query_entries #documents }
};
Expand Down
4 changes: 4 additions & 0 deletions packages/eql/crates/eql-codegen/src/go_eql.rs
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,9 @@ fn document_types(out: &mut String) {
DocumentRole::ElementsQuery => {
"element-scoped predicate, which eql_v4.json_tree_element_match tests"
}
DocumentRole::PredicateQuery => {
"compiled predicate, a boolean tree of the other operands, which eql_v4.json_tree_match and `@@` evaluate"
}
DocumentRole::Entry => {
"entry: one node of a document, as `->` and the walks return it"
}
Expand Down Expand Up @@ -275,6 +278,7 @@ mod tests {
"JSONSearchQuery",
"JSONSearchLeafQuery",
"JSONSearchElementsQuery",
"JSONSearchPredicateQuery",
"JSONEntry",
] {
assert_eq!(
Expand Down
15 changes: 13 additions & 2 deletions packages/eql/crates/eql-domains/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -567,6 +567,9 @@ pub enum DocumentRole {
/// The element-scoped predicate: fields of one array element, each
/// with its read tokens and its bounds.
ElementsQuery,
/// The compiled predicate: a boolean tree whose leaves are the other
/// query operands.
PredicateQuery,
/// One node, as `->` and the walks return it.
Entry,
}
Expand Down Expand Up @@ -625,8 +628,10 @@ impl DocumentFamily {
/// `JsonSearch` over `public.eql_v4_json_search`, its needle
/// `JsonSearchQuery` (`eql_v4.query_json`), its ordering operand
/// `JsonSearchLeafQuery` (`eql_v4.query_json_leaf`), its element-scoped
/// predicate `JsonSearchElementsQuery` (`eql_v4.query_json_elements`) and
/// its entry `JsonEntry` (`public.eql_v4_json_entry`). The storage-only `Json`
/// predicate `JsonSearchElementsQuery` (`eql_v4.query_json_elements`), its
/// compiled predicate `JsonSearchPredicateQuery`
/// (`eql_v4.query_json_predicate`) and its entry `JsonEntry`
/// (`public.eql_v4_json_entry`). The storage-only `Json`
/// (`public.eql_v4_json`) is the `json` family's. The Rust structs keep the
/// implementation's name, as v3's `SteVecDocument` does.
pub const JSON_TREE: DocumentFamily = DocumentFamily {
Expand Down Expand Up @@ -658,6 +663,12 @@ pub const JSON_TREE: DocumentFamily = DocumentFamily {
rust: "JsonTreeElementsQuery",
sql_domain: "eql_v4.query_json_elements",
},
DocumentDomain {
role: DocumentRole::PredicateQuery,
name: "JsonSearchPredicateQuery",
rust: "JsonTreePredicateQuery",
sql_domain: "eql_v4.query_json_predicate",
},
DocumentDomain {
role: DocumentRole::Entry,
name: "JsonEntry",
Expand Down
Loading
Loading