docs(eql): say what native jsonb functions do on the v4 JSON tree domains

packages/eql/docs/reference/json-tree-leakage.md

@@ -6,6 +6,7 @@
 
 - [What a stored row shows](#what-a-stored-row-shows)
 - [What a query shows](#what-a-query-shows)
+- [Native jsonb functions read the envelope](#native-jsonb-functions-read-the-envelope)
 - [What is authenticated](#what-is-authenticated)
 - [Compared with EQL v3 (`public.eql_v3_json_search`)](#compared-with-eql-v3-publiceql_v3_json_search)
 - [The two options that change the picture](#the-two-options-that-change-the-picture)
@@ -51,6 +52,19 @@ A query hands the database a term derived from the plaintext it asks about, and
   - **The root.** A read of `$` sends the root's token, and a read of anything below `$` does not. With the sealed root on (the default) that token covers no document's structure: it finds the root entry, whose ciphertext is the document, and nothing beneath it, since the top-level fields chain from nothing. With it off, the root's token covers every row's whole document.
   - **`json_tree_children` and `json_tree_child_count`** take one token, of any node a path names, and send nothing else. Under the sealed root the root's token enumerates nothing (its count is NULL), for the same reason.
 
+## Native jsonb functions read the envelope
+
+`public.eql_v4_json_search` and `public.eql_v4_json_entry` are PostgreSQL domains over `jsonb`, so PostgreSQL's own `jsonb` operators and functions accept them. On a column they see the stored envelope (`{v, i, h, sv, q}`), not the document. They reveal nothing beyond what a stored row already shows, but their answers are wrong.
+
+- **Operators are blocked.** `?`, `?|`, `?&`, `@?`, `@@`, `#>`, `#>>`, `-`, `#-`, `||`, `->` and `->>` by position, root-document comparisons, and containment against a plain `jsonb` operand raise `operator … is not supported for public.eql_v4_json_search`. `col @@ '…'` with an untyped right operand is refused as ambiguous (`operator is not unique`).
+- **Functions are not blocked, and fail silently.** `jsonb_path_match(col, '$.score == 3')` returns false for every row, with no error, and so does `jsonb_path_exists(col, '$.score')`, whether the path is a literal, cast, or bound as a parameter. `jsonb_extract_path(col, 'score')` returns NULL. A key of the envelope returns the envelope's own field instead: `jsonb_extract_path(col, 'v')` is `4`, and `jsonb_object_keys(col)` lists `h`, `i`, `q`, `sv` and `v`. The jsonpath functions find nothing on an entry extracted with `->` either. Only `jsonb_array_elements`, `jsonb_array_elements_text` and `jsonb_array_length` raise, because the envelope and an entry are objects.
+
+Query the document with EQL instead: `@>` with an `eql_v4.query_json` needle for equality, `col -> selector` compared with an `eql_v4.query_json_leaf` for ranges, and `col -> selector`, `eql_v4.json_tree_path_query` or `eql_v4.json_tree_subtree` to extract, then decrypt in the client.
+
+EQL cannot make these functions raise. An overload taking the domain as its first argument would be a second candidate for every call whose first argument has no type, such as `jsonb_typeof($1)` from a driver that sends untyped parameters, so installing EQL would make those calls ambiguous on plain `jsonb` too, in the whole database. And it would still lose to the native function whenever any other argument is untyped, as the path is in `jsonb_path_match(col, '$.a == 1')`. A type of its own that `jsonb` functions do not accept would close this, but PostgreSQL lets only a superuser create one, and EQL installs without superuser.
+
+To read the stored envelope on purpose, cast the column: `jsonb_pretty(col::jsonb)`.
+
 ## What is authenticated
 
 Only the ciphertext `c`. Opening an entry verifies that it was sealed under this keyset, for this column, under the data key its row's header names (a fresh one for every row), with the selector `s` and the slot `j` it is opened with, and over the AEAD tags of exactly the children it is opened with: each entry's AAD binds a digest of its children's tags, in order. A ciphertext moved into another row or column, or opened under another selector, slot or set of children, fails. Nothing in it names a row, so an entry that travels with its own row's header still opens; see below. The selectors, the projection, the parent tags, the self tags, the ordering terms, the header's sealed-root flag, and which entries a row holds are not authenticated, by the same rule every EQL stored value follows (ADR 0010 in `packages/stack-encrypt/docs/adr`).

packages/eql/tests/encryption/tests/json_tree.rs

@@ -511,6 +511,11 @@ fn postgres_containment_over_the_encrypted_column_is_jsonb_containment() {
 #[path = "json_tree/generated.rs"]
 mod generated;
 
+/// Native jsonb functions on a stored document and on an entry read the
+/// envelope, and installing EQL leaves them unambiguous on plain jsonb.
+#[path = "json_tree/native_functions.rs"]
+mod native_functions;
+
 /// Needs an EQL v4 installation. Extraction by selector opens to the leaf;
 /// the subtree walk by a client-derived token returns exactly the nodes
 /// under it and reassembles; the array functions count and list elements;

packages/eql/tests/encryption/tests/json_tree/native_functions.rs

@@ -0,0 +1,204 @@
+//! Native jsonb functions on EQL v4's JSON tree domains (#1439).
+//!
+//! `public.eql_v4_json_search` and `public.eql_v4_json_entry` are domains
+//! over `jsonb`, so a native function such as `jsonb_path_match` accepts
+//! them and runs over the stored envelope (`{v, i, h, sv, q}`), not the
+//! document. EQL does not overload these functions. An overload taking the
+//! domain would make every plain-jsonb call whose first argument is untyped
+//! (`jsonb_typeof($1)`) ambiguous in the whole database, and would still
+//! lose to the native function whenever another argument is untyped
+//! (`jsonb_path_match(col, '$.a == 1')`).
+//!
+//! These tests pin what `docs/reference/json-tree-leakage.md` says instead:
+//!
+//! - a native jsonpath function finds nothing of the document, however its
+//!   arguments are written, and an envelope key returns the envelope's own
+//!   field;
+//! - the array functions raise on both domains;
+//! - installing EQL leaves a plain-jsonb call with an untyped first
+//!   argument resolving to the native function, and puts no overload of a
+//!   native jsonb function in `public`.
+//!
+//! The document is the corpus's [`fixture`], sealed by Stack Encrypt's JSON
+//! index under the fake key source, and the entry is the one `->` extracts
+//! at the selector Stack Encrypt derives for `$.hello`.
+
+use super::*;
+
+use postgres::types::Type;
+
+/// Raise on both domains: the envelope and an entry are objects.
+const ARRAY_FUNCTIONS: &[&str] = &[
+    "jsonb_array_elements",
+    "jsonb_array_elements_text",
+    "jsonb_array_length",
+];
+
+/// A connection with the sealed [`fixture`] stored as row 1 of the temp
+/// table `docs`, and the selector of `$.hello` in it.
+struct Stored {
+    db: postgres::Client,
+    selector: String,
+}
+
+/// Seal [`fixture`] and store it in `docs.doc`.
+fn setup() -> Stored {
+    let runtime = tokio::runtime::Builder::new_current_thread()
+        .enable_all()
+        .build()
+        .unwrap();
+    let (cipher, _) = wait(&runtime, common::cipher());
+    let stored = wait(&runtime, seal(&cipher, &fixture()));
+    let selector = wait(
+        &runtime,
+        cipher
+            .default_keyset()
+            .json_selector(&JsonPath::parse("$.hello").unwrap(), column()),
+    )
+    .unwrap()
+    .to_string();
+    let mut db = connect();
+    db.batch_execute("CREATE TEMP TABLE docs (id integer, doc public.eql_v4_json_search)")
+        .unwrap();
+    db.execute(
+        "INSERT INTO docs VALUES (1, $1::text::jsonb::public.eql_v4_json_search)",
+        &[&serde_json::to_string(&stored).unwrap()],
+    )
+    .unwrap();
+    Stored { db, selector }
+}
+
+/// The two domain values a caller can reach, as SQL over `docs`.
+fn values(selector: &str) -> [String; 2] {
+    ["doc".to_string(), format!("(doc -> '{selector}'::text)")]
+}
+
+/// `$.hello` is in the document, and a native jsonpath function on either
+/// domain says it is not: typed, untyped, or bound as the filter compiler
+/// in #1439 binds it. No error tells the caller.
+#[test]
+#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"]
+fn postgres_a_native_jsonpath_function_finds_nothing_of_the_document() {
+    let Stored { mut db, selector } = setup();
+
+    for value in values(&selector) {
+        for sql in [
+            format!("SELECT jsonb_path_exists({value}, '$.hello'::jsonpath) FROM docs"),
+            format!("SELECT jsonb_path_exists({value}, '$.hello') FROM docs"),
+            format!(
+                r#"SELECT jsonb_path_match({value}, '$.hello == "world-1"'::jsonpath) FROM docs"#
+            ),
+            format!(r#"SELECT jsonb_path_match({value}, '$.hello == "world-1"') FROM docs"#),
+        ] {
+            let found: bool = db.query_one(&sql, &[]).unwrap().get(0);
+            assert!(!found, "{sql}: the native function reads the envelope");
+        }
+    }
+
+    let sql = "SELECT jsonb_path_match(doc, $1::text::jsonpath, $2::text::jsonb) FROM docs";
+    let found: bool = db
+        .query_one(sql, &[&"$.hello == $h", &r#"{"h": "world-1"}"#])
+        .unwrap()
+        .get(0);
+    assert!(!found, "{sql}: bound parameters read the envelope too");
+}
+
+/// A key of the document returns nothing, and a key of the envelope
+/// returns the envelope's own field.
+#[test]
+#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"]
+fn postgres_a_native_extraction_reads_the_envelope_keys() {
+    let Stored { mut db, .. } = setup();
+
+    let row = db
+        .query_one(
+            "SELECT jsonb_extract_path(doc, 'hello')::text,
+                    jsonb_extract_path(doc, 'v')::text,
+                    (SELECT array_agg(k ORDER BY k) FROM jsonb_object_keys(doc) AS k)
+               FROM docs",
+            &[],
+        )
+        .unwrap();
+    let (hello, version, keys): (Option<String>, Option<String>, Vec<String>) =
+        (row.get(0), row.get(1), row.get(2));
+    assert_eq!(hello, None, "a document key finds nothing");
+    assert_eq!(
+        version.as_deref(),
+        Some("4"),
+        "an envelope key finds its field"
+    );
+    assert_eq!(keys, ["h", "i", "q", "sv", "v"]);
+}
+
+/// The array functions raise natively on both domains: the envelope and an
+/// entry are objects, not arrays.
+#[test]
+#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"]
+fn postgres_the_array_functions_raise_on_both_domains() {
+    let Stored { mut db, selector } = setup();
+
+    for value in values(&selector) {
+        for f in ARRAY_FUNCTIONS {
+            let sql = format!("SELECT {f}({value}) FROM docs");
+            let error = db.query(&sql, &[]).expect_err(&format!("{sql} must raise"));
+            let message = error.as_db_error().map(|e| e.message().to_owned());
+            assert!(
+                message.as_deref().is_some_and(|m| m.starts_with("cannot ")),
+                "{sql} must raise the native error: {error}"
+            );
+        }
+    }
+}
+
+/// With EQL installed, a plain-jsonb call whose first argument is untyped
+/// (a bare literal, or a parameter the driver sends with no type) still
+/// resolves to the native function, because nothing in `public` overloads
+/// it. An overload of any of these names fails the catalog check.
+#[test]
+#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"]
+fn postgres_installing_eql_leaves_untyped_plain_jsonb_calls_unambiguous() {
+    let mut db = connect();
+
+    let row = db
+        .query_one(
+            r#"SELECT jsonb_typeof('{}'),
+                      jsonb_path_exists('{"a": 1}', '$.a'),
+                      (SELECT count(*) FROM jsonb_each_text('{"a": "b"}'))"#,
+            &[],
+        )
+        .unwrap();
+    let (kind, exists, pairs): (String, bool, i64) = (row.get(0), row.get(1), row.get(2));
+    assert_eq!(kind, "object");
+    assert!(exists);
+    assert_eq!(pairs, 1);
+
+    let statement = db.prepare("SELECT jsonb_typeof($1)").unwrap();
+    assert_eq!(
+        statement.params(),
+        [Type::JSONB],
+        "an untyped parameter resolves to the native function's jsonb"
+    );
+
+    let overloads: Vec<String> = db
+        .query(
+            "SELECT p.oid::regprocedure::text
+               FROM pg_proc p
+              WHERE p.pronamespace = 'public'::regnamespace
+                AND EXISTS (
+                  SELECT 1 FROM pg_proc n
+                   WHERE n.pronamespace = 'pg_catalog'::regnamespace
+                     AND n.proname = p.proname
+                     AND n.pronargs > 0
+                     AND n.proargtypes[0] = 'jsonb'::regtype)
+              ORDER BY 1",
+            &[],
+        )
+        .unwrap()
+        .iter()
+        .map(|row| row.get(0))
+        .collect();
+    assert!(
+        overloads.is_empty(),
+        "public overloads a native jsonb function: {overloads:?}"
+    );
+}