feat(go): test several members of one array element of a JsonSearch field by coderdan · Pull Request #1451 · cipherstash/stack

coderdan · 2026-10-30T02:11:48Z

Summary

A Go program can now ask "does one array element of this encrypted JSON document pass several tests at once?", for example "a contact who is a sibling and was called more than five times". Before, the Go SDK could only ask each half on its own (EqAt and an ordering at a path), and each half could be answered by a different contact, so the application had to read the array back, decrypt it and re-apply the filter by hand.

Terms used below:

A JsonSearch field is a Go struct, map or json.RawMessage field stored as one encrypted JSON document in a public.eql_v4_json_search column.
An ordered path is a path declared in the field's tag (ordered=$.contacts[*].calls) whose numbers and strings get an ordering term, so the database can compare them without decrypting.
stashgen is the Go generator that writes the typed Fields value from the struct tags.

This is the Go half of #1440. The engine and EQL work it builds on (#1444, #1445, #1446) is merged into next, so this PR's diff is its own two commits. It does not touch the engine or the EQL SQL.

Changes

Public Go API

On a generated JsonSearch field, Fields.<Field>.Elements.<Array>.Match(ctx, cipher, opts, predicates...): one accessor per array that has at least one ordered member beneath its wildcard, named as At names a path ($.extra.emergency_contacts[*] is Elements.ExtraEmergencyContacts; a root array $[*] is Elements.Root). Two arrays whose keys give one name, and an array whose keys give none, get no accessor: stashgen still generates the field and its At unchanged, exits 0, and prints a notice naming the ordered paths as written, which the generated file repeats in a comment. So no tag that generated before this PR stops generating.
Match returns eql.JSONSearchElementsQuery (already generated by feat(eql): one-parameter jsonpath predicates on json_tree documents (json_tree_match, @@, match) #1446), which binds as the one operand of WHERE eql_v4.json_tree_element_match(col, $1::eql_v4.query_json_elements).
Typed members, checked by the compiler. stashgen writes a Paths tree beside Fields (named as Fields is: CustomerPaths under -name Customer), nested as the JSON nests: Paths.Profile.Visits.Place is an encrypt.Path[ProfileFieldElementsVisits, string], its first type parameter naming the array and its second the member's Go type, read from the document struct as encoding/json reads it (any where the struct cannot say). encrypt.AtPath(path) gives Eq, Lt, Lte, Gt, Gte and Between, which take that Go type and return an encrypt.ElementPredicate[E] for that array alone:
```
q, err := Fields.Profile.Elements.Visits.Match(ctx, cipher, encrypt.JSONReadOptions{},
    encrypt.AtPath(Paths.Profile.Visits.Place).Eq("home"),
    encrypt.AtPath(Paths.Profile.Visits.At).Gt(15),
)
```
So a misspelt member, a value of the wrong type (AtPath(Paths.Profile.Visits.At).Gt("15")) and a member of another array do not compile. Paths holds only fields and the array accessor only methods, so no JSON key is reserved: a member named match is Paths.….Match, a field.
Members named at run time. Fields.<Field>.Elements.<Array>.Member(name) gives the same comparisons over any values, for a filter compiler that only knows member names as text; it makes the same ElementPredicate[E], and a member the field does not order is refused when the operand is derived.
Values are Go values written to JSON by the same gensupport.JSONText the orderings use. A predicate prints and logs its member and comparison, never its values; Values() returns a copy, so a built predicate cannot be changed.

Refusals

Before the engine is asked: no predicates, a zero ElementPredicate or Path, a member with a wildcard (all encrypt.ErrEncoding), and a member whose full path the field does not order (encrypt.ErrTerm). The last one names the member, the array and the members the field orders there, because the engine's own json_path_not_ordered names none.
Everything else (a value that is not a number or string, a bad member path, a member too deep, …) is the engine's *encrypt.Diagnostic, wrapping ErrEncoding for a data-plan reason and ErrTerm for an index refusal, as Subtree and At.<P>.Lt already surface theirs.
JSONReadOptions.WildcardBound keeps the meaning it has for Subtree: 0 sends no bound and the engine's default, 64, applies. So the engine's json_wildcard_bound_zero cannot be reached through Match; it is still tested through Cipher.QueryTerm.

Wiring

Guest (languages/golang/encrypt/guest/src/targets.rs): the EQL resolver names eql_v4.query_json_elements. The compiled-predicate domain stays unnamed (match is Go SDK: add Match(ctx, cipher, predicate, vars) on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443). The guest test now asserts the elements domain is named, element_match is listed on a field that orders a path, and match is not listed.
parseOperatorName and the operator input parser accept element_match / path_predicates; stashgen's jsonOperators too.
record.TargetOptions.ElementArrays groups ordered paths by the array their last wildcard names; stashgen uses it to decide which accessors to write, and gensupport uses it to check members at run time.
Guest status mapping: checked, no change. element_match's data-plan refusals are Error::Source (encoding) and its index refusals TermError (term); none falls into the internal catch-all.

Tests

The compiler refuses the two mistakes the types exist to catch. TestElementMatchTypeChecks (encrypt) loads the real generated testusers package with an overlay file through go/packages. A correct program compiles; AtPath(contact.Age).Gt("15") (a string for an int64 member) does not, and neither does passing one type's array predicate to another type's Match. TestRunTypesTheElementMatch (cmd/stashgen) does the same through go build of a freshly generated module against the stub SDK. It also checks the member types stashgen reads from the struct: an int64 through a pointer, a named string type, time.Time through a promoted field, an untagged field, a map step, and any for a path the struct does not hold.
Hermetic (encrypt): the operand's member tokens equal the Subtree tokens of the full path at the same bound (bound + 1, the probe last); eq is gte and lte at one value and equals At.<P>.Gte; gt equals At.<P>.Gt; between equals At.<P>.Between; Go values and their JSON text derive the same operand; the default bound gives 65 tokens. Typed paths and Member(name) derive the same operand (TestElementMatchIsTheReadsAndTheOrderingsOfEachMember, TestAtPathMakesThePredicatesOfItsMember). Every pre-engine refusal returns its sentinel with no Diagnostic, a zero Path included. Every engine refusal listed in feat(eql): test equality and ranges on one array element of a json_tree document #1445 comes back through the guest with its sentinel and its reason or code. Predicates print and log without their values, and copy json.RawMessage and []byte values.
PostgreSQL (go:test:postgres, new TestJSONElementMatchInPostgres): one family has a sibling and, separately, a contact called more than five times; another has one contact who is both. Match returns only the second; the EqAt + ordering superset returns both. Members named at run time find the same row. Also a range, three members, folded equality ("sibling" matches, EqAt does not), a bound of 1 raising SQLSTATE 22023, and an unordered member refused before the engine.
stashgen (TestRunGeneratesJSONSearchFields): accessor and Paths entries generated for an array with an ordered member, not for one without ($.tags[*], $.age), named as At names paths, a root array named Root; colliding or unnameable arrays skipped with a notice (exit 0, At unchanged), including the Paths-only cases (a key that names nothing on its own, a member above a member, a member above an array, two keys giving one Paths name); and an engine that answers no element_match refused.
The stub SDK (stashgen/testdata/stubsdk) gains encrypt.Path, AtPath, MemberTest and ElementPredicate[E], and its eql stub the JSON search types, so generated JSON code compiles against it.
New test type testusers.Entity, mirroring Go SDK: no way to test an equality and a range on the same array element of a JsonSearch field — add an element-scoped Match on top of #1440 #1442's example; customer_stash.go and subscriber_stash.go are regenerated.

Docs

cmd/stashgen/README.md "Encrypted JSON documents": a Match row with its SQL, a section on Paths and encrypt.AtPath, the run-time Member form, when Elements exists, the folded equality and the 22023 bound; Paths in "What stashgen writes" and in the -name flag row.
stashgen/doc.go and the -name help text in cmd/stashgen/main.go: Paths is named as Fields is.
encrypt/README.md, encrypt/doc.go, languages/golang/CHANGELOG.md (the Go module records changes there, not in changesets).
The typed-document example (encrypt/example/json) now orders each visit's at and place, and finds the one customer with a visit at home after 15. Its generated file is regenerated.
Skills: none mention the Go SDK's JSON queries; nothing to change.

Verification

mise run wasm:guest:test (run before the typed-path change, which did not touch the guest): fmt, clippy (8 feature/target combinations), rustdoc, nextest 73 passed (default) and 77 passed (eql).
After the typed-path change, with the deterministic guests rebuilt: go vet ./..., go test ./... (every package ok), gofmt -l . (empty), mise run go:lint (0 issues), and go generate ./... changes nothing.
mise run go:test:postgres against a local PostgreSQL 15 with the EQL v4 bundle built from this tree: all 6 tests passed, including TestJSONElementMatchInPostgres. These use the deterministic guests, so no CipherStash credentials were needed.
The same Go suite and PostgreSQL tests passed again at the top of the stack (feat(go): bind a jsonpath predicate on a JsonSearch field as one parameter #1452) after each later restack.
Not run: the example program against real ZeroKMS (no credentials here). It compiles, and its query is the same shape the PostgreSQL test runs. No stack-encrypt or EQL SQL lines changed, so their gates and mutation testing do not apply.

Review notes

Typed members, after review. A reviewer suggested generated per-member builders; the owner chose typed paths in a separate Paths tree with encrypt.AtPath, keeping a run-time form (Member(name)) for filters built from text. Match takes the options before the variadic predicates, because Go requires the variadic parameter last and the wildcard bound must stay expressible.
Where Paths has no entry. A key that names nothing on its own (the 1 in $.a.1[*].x), or one Paths name standing for two paths, gives the affected arrays no Elements accessor and a notice, as the existing Elements name collision does, so no tag that generated before stops generating.
Where to look first: encrypt/gensupport/json.go (JSONElementMatch), stashgen/emit.go (planJSON, jsonElements), internal/record/record.go (ElementArrays).
The member check compares the member text with the ordered paths as the tag writes them (relationship, address.city). A member spelled another way (["relationship"]) is refused as not ordered, though the engine would accept it. Tag paths are plain JSONPath, so the tag spelling is the only one a caller needs.
The unordered-member error names the member and the declared array path. Other JSON read errors hide paths, because a path's keys can be customer data. Here both come from the caller's code and the generated declaration, and naming them was the issue's ask.
Left for Go SDK: add Match(ctx, cipher, predicate, vars) on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443: the match operator and its predicate input shape, naming eql_v4.query_json_predicate in the guest, mapping Error::Predicate to an input status in the guest's status.rs (it falls into the internal catch-all today), and flipping the guest test's match assertion.

changeset-bot · 2026-10-30T02:11:53Z

⚠️ No Changeset found

Latest commit: ae71ed2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

coderabbitai · 2026-10-30T02:11:57Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI; Enforced overrides: Organization
Review profile: CHILL
Plan: Advanced
Run ID: ddf84087-dea5-4bd4-899b-234d80c95db7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

Trigger review

github-actions · 2026-10-30T02:13:18Z

Mutation testing (cargo-mutants, `--in-diff`, stack-auth + stack-encrypt)

No mutants were generated for the changed lines.

coderdan · 2026-10-30T10:14:01Z

Rebased onto #1446's new head (500aa58); new head cb6a190. Rebase only: no change to this PR's own commits.

Why: the three EQL PRs below changed under review (#1444's feedback, the move of the JSON tree SQLx tests onto documents Stack Encrypt seals, and the clippy fix in #1445 that gates JsonSubtreeKey::below on the dynamic feature).

chatgpt-codex-connector · 2026-10-30T10:44:23Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T10:45:32.440225Z	`cb6a190`	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-30T10:46:38.517583Z	`cb6a190`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

cipherstash-agent

Review attestation — automated /code-review medium plus cross-check against existing PR feedback, at cb6a190.

Critical / blocking: none.
New findings: none.
Adds to existing: one minor reply on the Values() thread (slices.Clone is shallow, so reference-typed values stay mutable).
Dropped: four candidate findings. The len(values) guard was already raised and removed in this PR. The member-spelling refusal is disclosed in the PR's review notes. The per-call ElementArrays() regroup and the strings.Trim cutset in jsonArrayName are unreachable or negligible today.

Not checked: CI results, and the Codex security review, which was still running.

Generated by Claude Code

coderdan · 2026-10-30T13:23:58Z

New head: 52802ef, rebased onto #1446 at 6109e38. ElementPredicate now copies the bytes of a json.RawMessage or []byte value in its constructors and in Values(), and the docs now describe that narrower guarantee (see the review thread).

https://claude.ai/code/session_01MrxG73fnX1p7oiizh3GRG5

…ield A filter such as "a contact who is a sibling and was called more than five times" had no Go method. An EqAt and an ordering at the contacts' paths can each be answered by a different contact, so the application had to read the array back and re-apply the predicate itself. The engine and EQL already answer it: element_match derives one eql_v4.query_json_elements operand, and json_tree_element_match is true of a row where one element passes every predicate. This wires it into the Go SDK. The EQL guest's resolver now names the elements domain, so the engine lists element_match (input path_predicates) on a field that orders a path, and the checker and stashgen read the operator. The compiled predicate domain stays unnamed: match is #1443's. stashgen groups the tag's ordered paths by the array their last wildcard names (record.TargetOptions.ElementArrays) and writes one accessor per array under Elements, named as At names a path, with Match(ctx, cipher, opts, predicates...). An array with no ordered member beneath it gets none, and a root array ($[*]) is Root. The members are compile-time names. stashgen writes Paths (NPaths with -name), a tree nested as the JSON nests it, of encrypt.Path[E, T] values: Paths.Profile.Visits.Place. E is a type stashgen writes for each array, and T is the member's Go type, read from the document's struct as encoding/json reads it, or any where the type names none (a json.RawMessage, say). encrypt.AtPath(path) gives Eq, Lt, Lte, Gt, Gte and Between, each taking a T and returning an encrypt.ElementPredicate[E], and an array's Match takes only its own E. So a value of another type, or a predicate on another array's member, does not compile. Paths holds fields and no methods, so no JSON key is reserved. For a filter built at run time, the array accessor's Member("place") gives the same comparisons with any values. An array gets no accessor, and a notice naming its ordered paths, when its keys give no accessor name, when another array's keys give the same one, or when Paths cannot name its members: a key that gives no Go name on its own, or one name that would stand for two paths. A tag that generated before still generates, with its At unchanged. Match writes each value with gensupport.JSONText as the orderings do, and returns an eql.JSONSearchElementsQuery. The member check runs in the SDK, against the members the declaration orders beneath the array, before the guest is called: the engine's json_path_not_ordered names no member, so the first error a caller sees names the member, the array and the members it orders. It wraps ErrTerm, as the engine's refusal does; no predicates, a zero predicate, a zero Path and a member with a wildcard wrap ErrEncoding. Every other refusal is the engine's Diagnostic. A predicate prints and logs its member and comparison, never its values, and copies the bytes of a json.RawMessage or []byte value. Tests: the hermetic suite pins the operand against the subtree reads and the orderings of the same members, the run-time form against the typed one, every pre-engine refusal, and each engine refusal's sentinel and code through the guest. A type-checking test loads the generated testusers package with a probe file and holds the two compile errors, a value of another type and another array's predicate; the stashgen test does the same against a generated module, and pins the member types and every case that gets no accessor. The PostgreSQL suite (go:test:postgres) finds the row where one contact passes both predicates, by path and by name, and not the row where two contacts each pass one, which the EqAt and ordering superset returns, and raises SQLSTATE 22023 past the bound. The guest test now pins the elements domain named and match not listed. Closes #1442 Claude-Session: https://claude.ai/code/session_01MrxG73fnX1p7oiizh3GRG5

The stashgen README's "Encrypted JSON documents" table gets the Match row and its SQL, with when Elements and Paths are generated, how encrypt.AtPath makes a predicate from a path, what a path's type checks, the run-time Member form, what is refused before the engine, which arrays get no accessor, the folded equality and the 22023 bound. Its list of what stashgen writes, and -name, name Paths. The SDK README and package doc show the call, and the CHANGELOG records it under Unreleased. The typed-document example orders each visit's at and place, and finds the customer with one visit at home after 15, with the paths from Paths: Ada has a visit at home and a visit after 15, but no one visit that is both. Its generated file is regenerated from the new tag. Refs #1442 Claude-Session: https://claude.ai/code/session_01MrxG73fnX1p7oiizh3GRG5