Repository navigation
test(stack-encrypt): add declaration grammar and shared conformance #1460
+5,110
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| # Stack Encrypt conformance | ||
|
|
||
| This directory holds a shared test contract for Stack Encrypt bindings. The | ||
| [implementation plan](../../docs/plans/2026-10-31-stack-encrypt-conformance.md) | ||
| sets out the wider work; [coverage.md](coverage.md) maps existing evidence and | ||
| gaps. These profiles are **experimental**, not certification of an entire SDK. | ||
|
|
||
| ## Run | ||
|
|
||
| From the repository root, with the pinned mise tools installed: | ||
|
|
||
| ```sh | ||
| mise run wasm:guest:build:deterministic | ||
| mise run test:conformance | ||
| ``` | ||
|
|
||
| No credentials or database are required. The task refuses a missing test guest. | ||
| It runs native Rust against frozen vectors, Go against the same vectors, then | ||
| native Rust against the ciphertexts Go has just produced. It never regenerates | ||
| expectations. CI also runs the Go cases on its other supported host platforms. | ||
| The same task runs the shared declaration acceptance/rejection corpus in both | ||
| languages, with native observations of normalization. | ||
|
|
||
| The Rust runner is `packages/stack-encrypt/tests/conformance.rs`; the Go runner | ||
| is `languages/golang/encrypt/conformance_test.go`. Go calls the record interface | ||
| used by generated code. Testing public SDK declarations is a separate layer. | ||
| Python and TypeScript runners are pending; their absence is not a passing result. | ||
|
|
||
| ## `record-declarations-v1` contract | ||
|
|
||
| The [declaration grammar](declaration-grammar.md) defines the engine's current | ||
| record-plan syntax, defaults, context rules, kind/index table and null policies. | ||
| `v1/declarations.json` contains 201 cases, including malformed frames and all | ||
|
Contributor
Author
There was a problem hiding this comment. Case counts are hard-coded prose in several places: 201 here, "91 accepted and 110 refused" in |
||
| 21 kinds against five scalar indexes. The resolver is explicitly `none`; EQL | ||
| targets need their own profile. | ||
|
|
||
| The runners are `packages/stack-encrypt/tests/declarations.rs` and | ||
| `languages/golang/encrypt/declaration_conformance_test.go`. Both call the engine | ||
| without a cipher. Valid-frame declaration diagnostics agree exactly; malformed | ||
| frames document the different native-parser and WASI-boundary classifications. | ||
| Native Rust also checks normalization observations. Go's checker does not expose | ||
| the parsed plan, so it asserts acceptance and diagnostics. | ||
|
|
||
| ## `record-scalars-v1` contract | ||
|
|
||
| `v1/cases.json` has a corpus format `version`, a semantic `profile`, a public | ||
| deterministic-provider `seed`, a record `context`, a `field`, an exact list of | ||
| `required_capabilities`, and uniquely identified `cases`. Unknown fields, | ||
| versions, profiles, capabilities and input types fail. Every case must run. | ||
|
|
||
| The profile fixes a one-field record with ciphertext (`c`) and equality (`eq`) | ||
| outputs. Its field context is the record context followed by the field name. | ||
| Each case declares the field's `kind` and a typed `input`. `input.text` is the | ||
| literal string value, canonical base-10 integer, `true`/`false`, or empty text | ||
| for null, as determined by `input.type`. Runners construct real typed host | ||
| values, preserving integer width. JSON numbers are deliberately not used for | ||
| values: future TypeScript runners must not pass 64-bit integers through Number. | ||
|
|
||
| The initial positives cover int32 and int64 bounds, uint64 max, an integer | ||
| above JavaScript's safe-integer limit, Unicode text and empty text. For each: | ||
|
|
||
| 1. The encoded typed frame must equal the frozen frame, including its type. | ||
| 2. Record encryption and opening must preserve both value and type. | ||
| 3. An equality query must produce one exact `eq` operand, encoded as `bytes`, | ||
| without an EQL domain. Its bytes must equal the frozen equality term and | ||
| the term stored with the record. | ||
| 4. The frozen native owned-scalar ciphertext must open through the record | ||
| path. Native Rust additionally opens it through the owned scalar API. | ||
| 5. Fresh Go ciphertext must open in Rust through both paths. Ciphertext bytes | ||
| themselves are not compared: encryption is randomized. | ||
| 6. Opening with `wrong` appended to the record context (before the field name) | ||
| must refuse key retrieval as forbidden. | ||
| This expectation uses `stack_kms::DeterministicSource`, whose descriptor | ||
| binding models ZeroKMS's refusal. It is distinct from an AEAD failure and | ||
| does not imply every provider returns the same error shape. | ||
|
|
||
| Negative cases have `expected_error` and exercise the equality query only. | ||
| They require the exact engine diagnostic `code`, `reason`, and field name; | ||
| Go must also expose `ErrEncoding`. Native Rust verifies rejection occurs | ||
| before data-key generation or retrieval. String/bool inputs for an integer | ||
| field and a null query are currently covered. Error prose is not contractual. | ||
|
|
||
| Boolean equality is not admitted by the current engine. Boolean scalar | ||
| encryption, nullable records, other numeric kinds, bytes, JSON, EQL targets, | ||
| other operators, declarations, cancellation and provider failures need later | ||
| profiles or cases. The full SQL behavior is not part of this initial profile. | ||
|
|
||
| ## Artifacts and evolution | ||
|
|
||
| `v1/native.json` is a frozen artifact set produced by the Rust owned scalar API. | ||
| It records `version`, `profile`, `producer`, and a case-ID map containing hex | ||
| `frame`, `ciphertext` and `equality` bytes. The map must contain exactly every | ||
| positive case; missing or extra cases fail. Every key and plaintext here is | ||
| public test material. The provider seed is not a production key mechanism. | ||
|
|
||
| The runners accept `STACK_CONFORMANCE_OUTPUT` to export fresh artifacts; Rust | ||
| accepts `STACK_CONFORMANCE_INPUT` to validate another producer. The task isolates | ||
| these variables and sets `STACK_CONFORMANCE_REQUIRED=1`, preventing a missing | ||
| Go test guest from turning conformance into a skipped test. Ordinary Go test | ||
| runs may skip this suite when the guest has not been built. | ||
|
|
||
| To intentionally create or update native artifacts: | ||
|
|
||
| ```sh | ||
| STACK_CONFORMANCE_UPDATE=1 mise x --env test -- cargo nextest run --locked \ | ||
| -p stack-encrypt --all-features --test conformance | ||
| mise run test:conformance | ||
| ``` | ||
|
|
||
| Review the case and artifact diff together. Never regenerate artifacts merely | ||
| to make a regression pass. Existing IDs retain their meaning; incompatible | ||
| expectations require a new profile. Corpus version 1 is a test format, not a | ||
| production wire version. Ciphertext/storage versions remain owned by their | ||
| existing codecs. | ||
|
|
||
| Both runners currently share the Rust engine. This checks host value conversion, | ||
| native/WASI interoperability and regressions against frozen evidence; it cannot | ||
| detect every bug shared by both paths. Independent SQL oracles, broader negative | ||
| cases, release-version matrices and richer version reports remain in the plan. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # Coverage map | ||
|
|
||
| Snapshot: PR #1404 at `b684a217e`, including the JSON matching work through #1452. | ||
| This is an inventory of evidence, not a claim that every suite was rerun here. | ||
| Paths are relative to the repository root. New profile cases run from `v1/`. | ||
|
|
||
| | Requirement | Existing evidence | Remaining work | | ||
| |---|---|---| | ||
| | Scalar value widths and lossless transport | `packages/stack-encrypt/tests/value_kinds.rs`; `languages/golang/encrypt/value_kinds_test.go`; initial shared scalar cases | Expand shared cases to decimals, dates, timestamps, floats, bytes and all widths; Python/TS mappings | | ||
| | Owned opaque ciphertext and storage envelope | `packages/stack-encrypt/tests/scalar.rs`, `frozen_bytes.rs`; initial cross-opening cases | Blob profile; encoded targets; supported version matrix | | ||
| | Record declaration and lowering | [Declaration grammar](declaration-grammar.md); `v1/declarations.json` shared by native Rust and Go; native normalization observations; existing record lowering suites | EQL-resolver profile; SDK-to-declaration mappings; runtime source/read shapes | | ||
| | Context binding and composition | `packages/stack-encrypt/tests/target.rs`, `descriptor.rs`, `record_lowering.rs`; `languages/golang/encrypt/context_field_test.go`; initial wrong-context cases | Shared composition vectors; expected-context and required-input matrix | | ||
| | Query terms and diagnostics | `packages/stack-encrypt/tests/query_operators.rs`; `languages/golang/encrypt/query_test.go`; initial equality/query-error cases | Common rejection corpus for every operator, capability and input shape | | ||
| | Missing, SQL NULL and sealed null | `packages/stack-encrypt/src/dynamic/record.rs` null tests; `languages/golang/encrypt/null_test.go` | Explicit language mappings and common missing/null fixtures | | ||
| | Fresh cross-language storage | `.github/workflows/tests-golang.yml` exports fresh Go EQL rows for Rust; initial scalar artifact exchange | Every producer/consumer/backend pair and supported released versions | | ||
| | JSON queries, extraction and tampering | `packages/stack-encrypt/tests/json_tree_vectors.rs`; `packages/eql/tests/encryption/tests/json_tree.rs`; `languages/golang/encrypt/json_tree_test.go`, `json_postgres_test.go` | Formal operation profile; JSON vectors consumed by all runners | | ||
| | Element and predicate matching | Go `json_elements_test.go`, `json_match_test.go` and corresponding PostgreSQL tests; Rust/SQL predicate suites | Map exact supported grammar and documented semantic differences to case IDs | | ||
| | Generated SDK declarations | Go stashgen and generated-code fixture tests | Python and TypeScript public-API runners; compiler/type-checker cases | | ||
| | Validation before key I/O and batching | Rust `tests/common::CountingSource` and record/query tests; guest preflight tests | Requirement-by-requirement coverage; common provider observations | | ||
| | Binding lifecycle and cleanup | `packages/stack-guest-abi`; Go `cancel_test.go`, `step_test.go`, guest memory tests | Native Python/TS and WASM backend-specific matrices; no blanket host-erasure claim | | ||
| | Version/capability negotiation | Existing ABI status table and target/operator discovery | Normative supported-version rules; reject unknown required capabilities | | ||
|
|
||
| Python and future TypeScript runners have no evidence in this checkout. Their | ||
| absence is a delivery gap, not a test skip that counts toward certification. | ||
|
|
||
| ## Consolidation policy | ||
|
|
||
| Shared semantic expectations should be maintained once and executed by each | ||
| language runner. Conformance can replace duplicate case tables and fixture | ||
| maintenance. It does not eliminate the need to execute the Rust and Go paths. | ||
| No existing test has been deleted in these first two slices. | ||
|
|
||
| | Existing evidence | Relationship to conformance | Consolidation action | | ||
| |---|---|---| | ||
| | Rust `dynamic/record.rs` tests `a_malformed_plan`, `a_null_is_refused_as_its_key`, `a_missing_or_null_index_payload_is_the_default` | Many inputs and diagnostic outcomes are now shared declaration cases | Move the remaining cases and retain any unique display/low-level assertions before removing duplicate rows | | ||
| | Rust kind/index admission tests | The corpus has the complete 21-kind/five-index acceptance matrix | Share case data; retain direct tests that compare declaration admission with actual term execution | | ||
| | Rust `value_kinds.rs`, Go `value_kinds_test.go` | Already share `value_kinds.json`; cover more types and generated SDK code than the initial scalar profile | Enroll that existing fixture under named requirements instead of copying it; keep generated-code assertions | | ||
| | Rust `query_operators.rs`, Go `query_test.go` | Already share operator vectors, including verification and target limits | Reuse their fixture and cases when adding an operator profile; preserve host/generated query mapping tests | | ||
| | Go checker and host validation tests in `guest_test.go` | Exercise host rejection and diagnostic forwarding through public checker methods | Keep adapter-specific assertions; raw-frame conformance bypasses host validation deliberately | | ||
| | Native scalar/record round trips and cross-opening | Some behavior overlaps, but existing tests cover more types and API paths | Replace only exact duplicates after matching requirements and execution paths | | ||
|
|
||
| Keep Rust compile-fail, ownership/zeroization, batching, fuzz/property, and | ||
| implementation regression tests. Keep Go generation, host type conversion, | ||
| error wrapping, cancellation and guest lifecycle tests. Keep independent | ||
| PostgreSQL oracles and live-provider tests. These observe guarantees a shared | ||
| engine corpus cannot prove by itself. | ||
|
|
||
| For each removal, link the old assertion to stable case IDs, compare coverage | ||
| and failure behavior, and run both runners in CI. Preserve unrelated assertions | ||
| from mixed tests. Enroll Python and TypeScript against the existing corpus so | ||
| adding a language does not create another copy of the expected behavior. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Efficiency: this repeats work.
mise run go:testin the previous step already runsTestStackConformanceandTestDeclarationConformance(amd64 and 386, withSTACK_CONFORMANCE_REQUIRED=1).tests-crates.ymlalready runs the Rustconformanceanddeclarationstests. This step runs both Go tests again, plus a Rust host build and run of both suites, before reaching the one new check (the RustSTACK_CONFORMANCE_INPUTconsumer). Consider exportingSTACK_CONFORMANCE_OUTPUTfrom thego:teststep and running only the Rust consumer here.