diff --git a/.github/workflows/tests-python.yml b/.github/workflows/tests-python.yml index 1a91ff6bf..54d4ad4da 100644 --- a/.github/workflows/tests-python.yml +++ b/.github/workflows/tests-python.yml @@ -41,13 +41,18 @@ on: # suite that reads them. - packages/eql/crates/eql-codegen/** - packages/eql/crates/eql-domains/** + # The EQL types the engine check resolves (#1387). The core crate links + # eql-bindings by path, and compiles the Go guest's resolver file + # in place, so a change to either reruns this suite. + - packages/eql/crates/eql-bindings/** + - languages/golang/encrypt/guest/src/targets.rs # The Rust version the workspace builds under. The workspace's own # mise.toml is under languages/python/** above. - mise.toml - .github/workflows/tests-python.yml # Unfiltered, like tests-rust.yml. Every push to main runs the suite, so - # main runs a superset of what pull requests run. Recorded in - # scripts/__tests__/workflow-paths-filter-parity.test.mjs. + # main runs a superset of what the filtered `pull_request` trigger above + # runs. Recorded in scripts/__tests__/workflow-paths-filter-parity.test.mjs. push: branches: [main] workflow_dispatch: {} diff --git a/AGENTS.md b/AGENTS.md index 011505c00..cbf61e4e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,7 +93,7 @@ Every npm package except EQL lives under `languages/typescript/`: packages in `l - `packages/stack-auth`, `packages/stack-profile`, `packages/stack-kms`, `packages/stack-encrypt`, `packages/stack-encrypt-derive`, `packages/stack-guest-abi`: The Rust crates imported from `cipherstash/cipherstash-suite` with their history — `stack-auth` and `stack-profile` (published to crates.io, one version group), `stack-kms` (published to crates.io from 0.1.0, its own version group, re-exported by `stack-encrypt` as `stack_encrypt::kms`), `stack-encrypt` and `stack-encrypt-derive` (published to crates.io from 0.1.0, one version group; `eql-bindings`' `stack-encrypt` feature depends on them from the registry), and `stack-guest-abi` (`publish = false`). They are the members of the **root Cargo workspace**, with the three node binding crates below. See "Working on the Rust crates". - `languages/typescript/packages/auth`, `languages/typescript/packages/profile`, `languages/typescript/packages/stack-auth-wasm`: The node bindings of those crates. `@cipherstash/auth` (napi-rs v2) and its six `platforms/*` packages are published to npm from this repository by `release.yml` (`auth-artifacts`, `publish-auth`); a change to what it ships, the `stack-auth` crate included, needs an `@cipherstash/auth` changeset (`require-auth-npm-changeset.yml`). `@cipherstash/profile` and its platforms are private and never published; `@cipherstash/stack-auth-wasm` is private and builds the wasm that `@cipherstash/auth` ships. Their `build` and `test` scripts never invoke cargo; `build:native`, `build:debug` and `test:cargo` do. - `languages/golang`: The Go module — the SDK `encrypt` with its generated-code support `encrypt/gensupport` and the policy packages `encrypt/policy` and `encrypt/policy/protosource`; the credential package `auth`; the generator `stashgen` and its command `cmd/stashgen`; and `internal` (the shared guest plumbing and the `record` wire model). A wazero host with no cgo. Its two WASI guests (`encrypt/guest`, `auth/guest`) are detached Cargo workspaces built by `mise run wasm:guest:build` and `mise run wasm:auth-guest:build`; `mise run wasm:guest:build:deterministic` builds the seeded TEST build the hermetic Go tests use; the `.wasm` files they embed are gitignored. The stack-encrypt guest has a second build with the EQL types (`mise run wasm:guest:build:eql`, cargo feature `eql`, linking `eql-bindings` by path; `wasm:guest:build:eql:deterministic` is its test build), embedded by `encrypt/eql` and registered on import, so a program whose generated code names an EQL type runs it (ADR-0007, amended). `encrypt/eql`'s types and `Types` table are generated from the EQL catalog by `eql-codegen` (`eql_gen.go`, written by `mise run types:generate` in `packages/eql` and drift-gated by `types:check` and `cargo test -p eql-codegen`). Generated `*_stash.go` files are committed and CI fails when `go generate ./...` changes one. Releases are cut by hand with `.github/workflows/release-golang.yml` (`gh workflow run release-golang.yml -f version=X.Y.Z[-pre] --ref `): because the guests are gitignored on main and Go fetches a version as the files committed at its tag, the workflow builds them, commits them on a release-only commit no branch points at, tags it `languages/golang/vX.Y.Z` and pushes the tag alone, then attaches the EQL v4 SQL bundle to the GitHub release. The ref must carry the EQL v4 bundle, or the build stops at "Build the EQL SQL bundles". `scripts/go-release-version.mjs` refuses, before anything is built, every version Go would refuse (non-canonical semver, `+build` metadata, a v2+ major on a module path with no `/vN` suffix, a version shaped like a pseudo-version), a final version dispatched from anywhere but `main`, and a final version while `languages/golang/go.mod` pins a `github.com/cipherstash/vitaminc/…` module at a pseudo-version (vitaminc's Go bindings have no tags, and a TEMPORARY pin on an unreleased commit must be gone before the module is tagged); a pre-release may come from a branch and may carry such a pin. The build job also refuses a version the Go proxy already lists in `@v/list`, which catches a tag deleted after the proxy fetched it; it never asks the proxy for the version itself before the tag exists, because the proxy would cache "not found" for up to 30 minutes and `verify` would fail after the push. The build checks the `Cargo.lock` of the EQL workspace and both guest workspaces with `--locked` first, since none of those builds passes it. It runs in a job with no write token and hands the guests to a `publish` job that holds `contents: write` and runs only git and gh. `publish` fetches the commit with plain unauthenticated git rather than `actions/checkout`, whose `token` input defaults to the job's writable token, and refuses a release commit whose tree is not the one the build tested. `publish` takes the paths it commits from the workflow's fixed `GUESTS` list, never from the artifact, and refuses a `guests.sha256` that is not exactly one SHA-256 line for each of those three guests: the build job runs third-party code after writing that manifest, so a path read from it could put any file on the tag (`scripts/__tests__/go-release-artifact.test.mjs`). A version is immutable once the Go proxy has fetched it, so a tag is never moved or deleted: if `publish` or `verify` fails after the tag is pushed, use "Re-run failed jobs" on the same run, which keeps a tag that run pushed and creates or completes the release; a new dispatch of the same version is refused. Never tag a main commit by hand: `go get` would fetch a module with no guests. -- `languages/python`: The Python SDK, `stackencrypt`, a pyo3 extension over `stack-encrypt`, built by maturin. **The fourth Cargo workspace**, excluded from the root one. It has two crates. `crates/core` has no pyo3: the engine-side logic, linked and tested without a Python interpreter. `crates/engine` is the `_engine` cdylib, `abi3-py39`, one wheel per platform for Python 3.9 to 3.14. The `extension-module` feature is set only by maturin in `pyproject.toml`, never in a crate manifest, so `cargo test` links libpython. Its tasks live in its own `mise.toml` and run from that folder. `mise run build` builds the wheel into `.venv`. `mise run test` runs pytest. `mise run test:cargo` runs plain `cargo test --locked` for both crates, with libpython on the loader path. It does not use nextest, and its CI job sets no `MISE_ENV`. The nextest rule under "Working on the Rust crates" is for the root workspace, whose `stack-kms` tests set environment variables. No test in this workspace does, and nextest comes from `mise.test.toml`, which this workspace's CI does not load. The wheel's version is `crates/engine/Cargo.toml`'s. `stackencrypt.__engine__` is the commit of this checkout, recorded by `crates/core/build.rs`. Its `Cargo.lock` is seeded from the root one, so every crate it shares with the root workspace resolves to the version the root tests. After a root dependency moves, re-seed it the same way: copy the root `Cargo.lock` over it and run `cargo metadata --offline` there. That command prunes the copy to what the workspace uses. Until then, `tests-python.yml` fails on `--locked`. Nothing is published. CI is `tests-python.yml`. It runs the Rust checks and pytest on Python 3.9 and 3.14, where `MISE_PYTHON_VERSION` moves a leg off the local version. It then builds one abi3 wheel per FFI platform, uploads them as artifacts, and writes each build time to the job summary. `scripts/__tests__/python-ci.test.mjs` compares its path filter with the workspace's `Cargo.lock` and its wheel matrix with the protect-ffi platform packages. There is no Python lockfile yet. The package has no runtime dependency, so only the `test` extra could be locked: pytest, plus attrs, Pydantic and SQLAlchemy for the declaration tests. Those three have no upper bounds, so the tests meet each new release when users do. `python/stackencrypt/_eql.py`, the EQL type markers, is generated from the EQL catalog by `eql-codegen py-eql`. To regenerate it, run `mise run types:generate` in `packages/eql`. Never edit it by hand: `tests-python.yml` runs its parity tests and fails on a hand edit. Design: Python ADR-0001 under `languages/python/docs/adr/`, the spec in #1341. +- `languages/python`: The Python SDK, `stackencrypt`, a pyo3 extension over `stack-encrypt`, built by maturin. **The fourth Cargo workspace**, excluded from the root one. It has two crates. `crates/core` has no pyo3: the engine-side logic, linked and tested without a Python interpreter. `crates/engine` is the `_engine` cdylib, `abi3-py39`, one wheel per platform for Python 3.9 to 3.14. `crates/core` compiles the Go guest's EQL resolver, `languages/golang/encrypt/guest/src/targets.rs`, in place through `#[path]`. So both SDKs run one resolver, and a change to that file is a change to both. The `extension-module` feature is set only by maturin in `pyproject.toml`, never in a crate manifest, so `cargo test` links libpython. Its tasks live in its own `mise.toml` and run from that folder. `mise run build` builds the wheel into `.venv`. `mise run test` runs pytest. `mise run test:cargo` runs plain `cargo test --locked` for both crates, with libpython on the loader path. It does not use nextest, and its CI job sets no `MISE_ENV`. The nextest rule under "Working on the Rust crates" is for the root workspace, whose `stack-kms` tests set environment variables. No test in this workspace does, and nextest comes from `mise.test.toml`, which this workspace's CI does not load. The wheel's version is `crates/engine/Cargo.toml`'s. `stackencrypt.__engine__` is the commit of this checkout, recorded by `crates/core/build.rs`. Its `Cargo.lock` is seeded from the root one, so every crate it shares with the root workspace resolves to the version the root tests. After a root dependency moves, re-seed it the same way: copy the root `Cargo.lock` over it and run `cargo metadata --offline` there. That command prunes the copy to what the workspace uses. Until then, `tests-python.yml` fails on `--locked`. Nothing is published. CI is `tests-python.yml`. It runs the Rust checks and pytest on Python 3.9 and 3.14, where `MISE_PYTHON_VERSION` moves a leg off the local version. It then builds one abi3 wheel per FFI platform, uploads them as artifacts, and writes each build time to the job summary. `scripts/__tests__/python-ci.test.mjs` compares its path filter with the workspace's `Cargo.lock` and its wheel matrix with the protect-ffi platform packages. There is no Python lockfile yet. The package has no runtime dependency, so only the `test` extra could be locked: pytest, plus attrs, Pydantic and SQLAlchemy for the declaration tests. Those three have no upper bounds, so the tests meet each new release when users do. `python/stackencrypt/_eql.py`, the EQL type markers, is generated from the EQL catalog by `eql-codegen py-eql`. To regenerate it, run `mise run types:generate` in `packages/eql`. Never edit it by hand: `tests-python.yml` runs its parity tests and fails on a hand edit. Design: Python ADR-0001 under `languages/python/docs/adr/`, the spec in #1341. - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) - `languages/typescript/examples/*`: Working apps (basic, prisma, supabase-worker) - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo). diff --git a/languages/golang/encrypt/guest/src/targets.rs b/languages/golang/encrypt/guest/src/targets.rs index 0754a9908..7549ba4c8 100644 --- a/languages/golang/encrypt/guest/src/targets.rs +++ b/languages/golang/encrypt/guest/src/targets.rs @@ -18,6 +18,12 @@ //! The Go module embeds both: `encrypt` the build without EQL types, //! `encrypt/eql` the build with them, registered on import, so a program //! that names an EQL type links the build that has it. +//! +//! The Python SDK compiles this file too, in place, from +//! `languages/python/crates/core` through `#[path]`, with its own `eql` +//! feature. A change here is a change to both SDKs, and +//! `tests-python.yml` runs on it. A crate this file starts to use must also +//! go into that crate's `Cargo.toml`. #[cfg(not(feature = "eql"))] use stack_encrypt::dynamic::NoTargets; diff --git a/languages/python/Cargo.lock b/languages/python/Cargo.lock index 5cba58c17..14a2a5550 100644 --- a/languages/python/Cargo.lock +++ b/languages/python/Cargo.lock @@ -671,6 +671,12 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" +[[package]] +name = "dyn-clone" +version = "1.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" + [[package]] name = "either" version = "1.15.0" @@ -680,6 +686,22 @@ dependencies = [ "serde", ] +[[package]] +name = "eql-bindings" +version = "3.0.6" +dependencies = [ + "base64", + "base85-simd", + "hex", + "schemars", + "serde", + "serde_json", + "stack-encrypt", + "thiserror 2.0.18", + "ts-rs", + "zeroize", +] + [[package]] name = "equivalent" version = "1.0.2" @@ -1224,6 +1246,12 @@ dependencies = [ "zeroize", ] +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + [[package]] name = "leb128fmt" version = "0.1.0" @@ -1834,6 +1862,26 @@ dependencies = [ "thiserror 1.0.69", ] +[[package]] +name = "ref-cast" +version = "1.0.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f354300ae66f76f1c85c5f84693f0ce81d747e2c3f21a45fef496d89c960bf7d" +dependencies = [ + "ref-cast-impl", +] + +[[package]] +name = "ref-cast-impl" +version = "1.0.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.114", +] + [[package]] name = "regex" version = "1.12.3" @@ -1916,6 +1964,31 @@ version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" +[[package]] +name = "schemars" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "687274d293b6cdc6e73e0fee520bf2049650090d7164f87672d212a3c530cf4a" +dependencies = [ + "dyn-clone", + "ref-cast", + "schemars_derive", + "serde", + "serde_json", +] + +[[package]] +name = "schemars_derive" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d98c67716b46af2f0b8cf752abc930f6f9aecfbf671ecfb531db8a31dbe4e2ba" +dependencies = [ + "proc-macro2", + "quote", + "serde_derive_internals", + "syn 3.0.6", +] + [[package]] name = "scopeguard" version = "1.2.0" @@ -1978,6 +2051,17 @@ dependencies = [ "syn 3.0.6", ] +[[package]] +name = "serde_derive_internals" +version = "0.30.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f852137cce035d6a4df67ccce505ff6b3e9fd3a10e3e52b24dc71e650bb1a9bd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.6", +] + [[package]] name = "serde_json" version = "1.0.151" @@ -2219,6 +2303,10 @@ dependencies = [ name = "stackencrypt-core" version = "0.0.0" dependencies = [ + "eql-bindings", + "miette", + "serde", + "serde_json", "stack-encrypt", ] @@ -2300,6 +2388,15 @@ version = "0.13.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" +[[package]] +name = "termcolor" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755" +dependencies = [ + "winapi-util", +] + [[package]] name = "thiserror" version = "1.0.69" @@ -2486,6 +2583,29 @@ dependencies = [ "once_cell", ] +[[package]] +name = "ts-rs" +version = "10.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e640d9b0964e9d39df633548591090ab92f7a4567bc31d3891af23471a3365c6" +dependencies = [ + "lazy_static", + "thiserror 2.0.18", + "ts-rs-macros", +] + +[[package]] +name = "ts-rs-macros" +version = "10.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e9d8656589772eeec2cf7a8264d9cda40fb28b9bc53118ceb9e8c07f8f38730" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.114", + "termcolor", +] + [[package]] name = "typenum" version = "1.19.0" @@ -3116,6 +3236,15 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "winapi-x86_64-pc-windows-gnu" version = "0.4.0" diff --git a/languages/python/crates/core/Cargo.toml b/languages/python/crates/core/Cargo.toml index 156667112..d72622f5b 100644 --- a/languages/python/crates/core/Cargo.toml +++ b/languages/python/crates/core/Cargo.toml @@ -10,5 +10,25 @@ edition.workspace = true publish = false repository.workspace = true +[features] +default = ["eql"] +# Not a switch. `src/lib.rs` compiles the EQL resolver from the Go guest's +# own file (`languages/golang/encrypt/guest/src/targets.rs`), and that file +# reads a feature of exactly this name. Every build of the SDK holds the EQL +# types, so the crate refuses to compile without it. +eql = [] + [dependencies] +# By path, like every other eql-bindings dependency in the repository: the +# SQL that stores a payload and the Rust that emits it must agree +# (scripts/lint-no-eql-registry-pins.mjs). +eql-bindings = { path = "../../../../packages/eql/crates/eql-bindings", features = ["targets"] } +# For the engine error's diagnostic code. +miette = "7.5.0" +# For the plan's frame: the fields in declaration order. +serde = "1" stack-encrypt = { workspace = true } + +[dev-dependencies] +# The shared resolver file's own tests use it. +serde_json = "1" diff --git a/languages/python/crates/core/src/lib.rs b/languages/python/crates/core/src/lib.rs index 3c7d1c952..12d8b34c9 100644 --- a/languages/python/crates/core/src/lib.rs +++ b/languages/python/crates/core/src/lib.rs @@ -1,8 +1,30 @@ //! The Python SDK's engine-side logic, with no pyo3 in it, so that it can be //! linked and tested without a Python interpreter. //! -//! Today it holds one fact: the engine commit the build recorded (`build.rs`). -//! The Python SDK spec (#1341, "Layout") says what else belongs here. +//! It holds the engine commit the build recorded (`build.rs`), and the +//! engine's check of a model's plan ([`plan`]). The Python SDK spec (#1341, +//! "Layout") says what else belongs here. + +pub mod plan; + +// The resolver file below holds the EQL types only under this feature. +#[cfg(not(feature = "eql"))] +compile_error!( + "stackencrypt-core needs its `eql` feature: every build of the SDK holds the EQL types" +); + +// The EQL resolver: the Go guest's own file, compiled here too, so the two +// SDKs run one resolver and cannot drift apart. It switches on this crate's +// `eql` feature. Its tests run here as well as in the guest. `HOLDS_EQL` is +// the guest's alone, so dead code is allowed. +// +// This crate must have every crate the file uses. Today that is +// stack-encrypt and eql-bindings, plus serde_json for its tests. A Go change +// that adds one fails this crate's build in tests-python.yml. Then add it +// to Cargo.toml here too. +#[allow(dead_code)] +#[path = "../../../../golang/encrypt/guest/src/targets.rs"] +mod targets; /// The commit of the `stack-encrypt` engine this build was taken from: a /// full 40-character lower-case hex SHA, with `-dirty` appended when any diff --git a/languages/python/crates/core/src/plan.rs b/languages/python/crates/core/src/plan.rs new file mode 100644 index 000000000..cdfccf626 --- /dev/null +++ b/languages/python/crates/core/src/plan.rs @@ -0,0 +1,161 @@ +//! A model's plan, checked by the engine when the class is declared. +//! +//! The Python decorator reads the fields and their markers. This module +//! writes them as the plan frame every binding sends, and runs the engine's +//! own reader over it (`record::plan_with`) with the EQL resolver. So the +//! engine sees the same declaration a Rust or Go caller would, and its +//! refusal reaches Python unchanged: its code, message, field and reason. + +use miette::Diagnostic as _; +use serde::ser::{Serialize, SerializeMap, SerializeStruct, Serializer}; +use stack_encrypt::dynamic::{self, record}; +use stack_encrypt::ErrorPayload as _; + +use crate::targets; + +/// What one plan field stores. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Column<'a> { + /// A field sealed as the EQL type of this name, such as `TextEq`. + Eql(&'a str), + /// A field stored as it is. + Passthrough, +} + +/// The engine's refusal of a plan, in the four fields every SDK error has. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Refusal { + /// The engine's diagnostic code, with `::` replaced by `_`: + /// `stack_encrypt_dynamic_plan`. + pub code: String, + /// The engine's message. + pub message: String, + /// The plan field at fault, where the engine names one. + pub field: Option, + /// The engine's reason token, such as `no_fields`, where it gives one. + pub reason: Option, +} + +/// Check the plan of a model stored in `table`, with `fields` in +/// declaration order. Each field's context is `[table, field]`. +/// +/// # Errors +/// +/// The engine's refusal, as a [`Refusal`]. +pub fn check(table: &str, fields: &[(&str, Column<'_>)]) -> Result<(), Refusal> { + // A frame of strings and lists of strings always encodes. + let frame = dynamic::frame::serde::to_vec(&Wire { table, fields }) + .expect("a plan of strings encodes as a frame"); + record::plan_with(&frame, &targets::resolver()) + .map(drop) + .map_err(|error| refusal(&error)) +} + +/// An engine diagnostic code as the SDK's `code`: `stack_encrypt::aead` +/// becomes `stack_encrypt_aead`. +pub fn python_code(engine_code: &str) -> String { + engine_code.replace("::", "_") +} + +fn refusal(error: &dynamic::Error) -> Refusal { + let payload = error.payload(); + let text = |key: &str| payload.get(key).and_then(|v| v.as_str()).map(str::to_owned); + Refusal { + code: error.code().map_or_else( + || "stack_encrypt_dynamic".to_owned(), + |code| python_code(&code.to_string()), + ), + message: error.to_string(), + field: text("field"), + // Only the engine's own reason tokens. A resolver's reason, such as + // the catalog's for an unproducible type, is prose and stays in the + // message. + reason: error.reason().map(|reason| reason.as_str().to_owned()), + } +} + +/// The plan frame: `{field: {"context": [table, field], "target": name}}` +/// for an EQL field, `"outputs": ["passthrough"]` for a passthrough field. +struct Wire<'a> { + table: &'a str, + fields: &'a [(&'a str, Column<'a>)], +} + +impl Serialize for Wire<'_> { + fn serialize(&self, serializer: S) -> Result { + let mut map = serializer.serialize_map(Some(self.fields.len()))?; + for (name, column) in self.fields { + map.serialize_entry( + name, + &Spec { + context: [self.table, name], + column: *column, + }, + )?; + } + map.end() + } +} + +struct Spec<'a> { + context: [&'a str; 2], + column: Column<'a>, +} + +impl Serialize for Spec<'_> { + fn serialize(&self, serializer: S) -> Result { + let mut spec = serializer.serialize_struct("Spec", 2)?; + spec.serialize_field("context", &self.context)?; + match self.column { + Column::Eql(name) => spec.serialize_field("target", name)?, + Column::Passthrough => spec.serialize_field("outputs", &["passthrough"])?, + } + spec.end() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_plan_of_eql_and_passthrough_fields_is_accepted() { + let fields = [ + ("email", Column::Eql("TextEq")), + ("age", Column::Eql("IntegerOrd")), + ("note", Column::Passthrough), + ]; + assert_eq!(check("users", &fields), Ok(())); + } + + #[test] + fn an_empty_plan_is_refused_with_the_engine_code_and_reason() { + let refused = check("users", &[]).unwrap_err(); + assert_eq!(refused.code, "stack_encrypt_dynamic_plan"); + assert_eq!(refused.field, None); + assert_eq!(refused.reason.as_deref(), Some("no_fields")); + } + + #[test] + fn an_unknown_eql_type_is_refused_by_the_resolver() { + let refused = check("users", &[("email", Column::Eql("Nope"))]).unwrap_err(); + assert_eq!(refused.code, "stack_encrypt_target_unknown"); + assert!(refused.message.contains("Nope"), "{refused:?}"); + assert_eq!(refused.reason, None); + } + + #[test] + fn an_unproducible_eql_type_is_refused_by_the_resolver() { + let refused = check("users", &[("doc", Column::Eql("Json"))]).unwrap_err(); + assert_eq!(refused.code, "stack_encrypt_target_unproducible"); + } + + #[test] + fn engine_codes_replace_the_path_separator() { + assert_eq!(python_code("stack_encrypt::aead"), "stack_encrypt_aead"); + assert_eq!( + python_code("stack_kms::load_keyset_forbidden"), + "stack_kms_load_keyset_forbidden" + ); + } +} diff --git a/languages/python/crates/engine/src/lib.rs b/languages/python/crates/engine/src/lib.rs index 4cf64543c..89c8ca1d7 100644 --- a/languages/python/crates/engine/src/lib.rs +++ b/languages/python/crates/engine/src/lib.rs @@ -4,6 +4,7 @@ //! the boundary. Logic belongs in `stackencrypt-core` and `stack-encrypt`. use pyo3::prelude::*; +use stackencrypt_core::plan::{self, Column}; /// The two build facts the package exposes: `(version, engine_commit)`. /// @@ -19,6 +20,29 @@ fn build_info() -> (&'static str, &'static str) { ) } +/// The engine's check of a model's plan, run when the class is declared. +/// +/// `fields` is `(name, eql_type)` per plan field in declaration order, with +/// `eql_type` `None` for a passthrough field. Returns `None` when the engine +/// accepts the plan, or its refusal as `(code, message, field, reason)`. The +/// Python side raises it as a `DeclarationError`. +#[pyfunction] +fn check_plan( + table: &str, + fields: Vec<(String, Option)>, +) -> Option<(String, String, Option, Option)> { + let fields: Vec<(&str, Column<'_>)> = fields + .iter() + .map(|(name, eql_type)| { + let column = eql_type.as_deref().map_or(Column::Passthrough, Column::Eql); + (name.as_str(), column) + }) + .collect(); + plan::check(table, &fields) + .err() + .map(|refused| (refused.code, refused.message, refused.field, refused.reason)) +} + /// Gate fact 2 (#1384): block for `millis` on a multi-thread tokio runtime /// with the GIL released, the way an engine call will. Returns the /// milliseconds the call blocked for, measured in Rust. @@ -42,6 +66,7 @@ fn _probe_detached_sleep(py: Python<'_>, millis: u64) -> u64 { #[pymodule] fn _engine(module: &Bound<'_, PyModule>) -> PyResult<()> { module.add_function(wrap_pyfunction!(build_info, module)?)?; + module.add_function(wrap_pyfunction!(check_plan, module)?)?; #[cfg(feature = "gate-probe")] module.add_function(wrap_pyfunction!(_probe_detached_sleep, module)?)?; Ok(()) diff --git a/languages/python/python/stackencrypt/_declaration.py b/languages/python/python/stackencrypt/_declaration.py index c826dd71a..7dd01d57d 100644 --- a/languages/python/python/stackencrypt/_declaration.py +++ b/languages/python/python/stackencrypt/_declaration.py @@ -1,9 +1,10 @@ -"""The ``model`` decorator: reads a class's fields and checks rule A. +"""The ``model`` decorator: reads a class's fields and checks what they mean. Rule A: every field has exactly one stash marker, in the ``Annotated`` directly around its value type. A plain field has ``Passthrough``, and an attribute that is not a column has ``Ignore``. The decorator raises -``DeclarationError`` at the class statement when a field breaks the rule. +``DeclarationError`` at the class statement when a field breaks the rule, or +when one of the checks below fails. The fields are the library's own list where one exists (dataclasses, attrs, Pydantic, ``NamedTuple``), and otherwise the class's type hints minus @@ -12,13 +13,23 @@ class around ``Annotated`` is read by structure, so ``Mapped[Annotated[str, TextEq]]`` reads as ``TextEq``, with no SQLAlchemy import. A list, ``Final`` and the other generics of the standard library are not read through. -The checks of each field's meaning come later: the value type against the -marker, the constructor, and the engine's own check of the plan. +Then, in this order: + +- Each marker is one the engine produces, and the value type is exactly the + Python type of the marker's kind. A ``Passthrough`` value type is one of + the eight types the engine stores as they are. +- The constructor takes every plan field as a keyword argument, and needs no + ``Ignore`` field. A decorator applied below ``@dataclass`` finds no such + constructor, because the class has none yet. +- The engine reads the plan, with the EQL types, as it does for every SDK. """ import dataclasses +import inspect import sys from dataclasses import dataclass +from datetime import date, datetime +from decimal import Decimal from typing import ( Annotated, Any, @@ -35,7 +46,7 @@ class around ``Annotated`` is read by structure, so ``Mapped[Annotated[str, get_type_hints, ) -from stackencrypt import _eql +from stackencrypt import _engine, _eql from stackencrypt._errors import DeclarationError from stackencrypt._markers import Ignore, Marker, Passthrough @@ -55,6 +66,24 @@ class around ``Annotated`` is read by structure, so ``Mapped[Annotated[str, {"builtins", "collections", "collections.abc", "types", "typing"} ) +# The one Python type of each marker kind. A value type must be exactly this +# type: not Optional, not a union, and not a subclass. +_KIND_TYPES: Dict[str, type] = { + "string": str, + "int16": int, + "int32": int, + "int64": int, + "float32": float, + "float64": float, + "bool": bool, + "decimal": Decimal, + "date": date, + "timestamp": datetime, +} + +# The types the engine stores as they are, for a Passthrough field. +_PASSTHROUGH_TYPES = (str, int, float, bool, bytes, date, datetime, Decimal) + @dataclass(frozen=True) class DeclaredField: @@ -64,6 +93,15 @@ class DeclaredField: marker: Type[Marker] +@dataclass(frozen=True) +class _ReadField: + """One field as the reader found it, ``Ignore`` fields included.""" + + name: str + marker: Type[Marker] + value_type: Any + + @dataclass(frozen=True) class Declaration: """What the decorator read from a class: the table, and every field that @@ -95,8 +133,14 @@ def decorate(cls: T) -> T: f"class {cls.__name__!r} is already declared, with table " f"{existing.table!r}. Apply stackencrypt.model once.", ) - declaration = Declaration(table=table, fields=tuple(_read_fields(cls))) - setattr(cls, DECLARATION_ATTRIBUTE, declaration) + read = _read_fields(cls) + planned = [field for field in read if field.marker is not Ignore] + for field in planned: + _check_value_type(field) + _check_constructor(cls, read) + _check_plan(table, planned) + fields = tuple(DeclaredField(name=f.name, marker=f.marker) for f in planned) + setattr(cls, DECLARATION_ATTRIBUTE, Declaration(table=table, fields=fields)) return cls return decorate @@ -112,7 +156,7 @@ def declaration_of(cls: type) -> Optional[Declaration]: return found if isinstance(found, Declaration) else None -def _read_fields(cls: type) -> List[DeclaredField]: +def _read_fields(cls: type) -> List[_ReadField]: try: hints = get_type_hints(cls, include_extras=True) except (NameError, TypeError) as error: @@ -121,12 +165,11 @@ def _read_fields(cls: type) -> List[DeclaredField]: from_hints = names is None if names is None: names = [name for name, hint in hints.items() if not _is_class_level(hint)] - declared = [] + read = [] for name in names: - marker = _marker_of(name, hints.get(name), from_hints=from_hints) - if marker is not Ignore: - declared.append(DeclaredField(name=name, marker=marker)) - return declared + marker, value_type = _marker_of(name, hints.get(name), from_hints=from_hints) + read.append(_ReadField(name=name, marker=marker, value_type=value_type)) + return read def _library_field_names(cls: type) -> Optional[List[str]]: @@ -233,8 +276,8 @@ def _is_wrapper(hint: Any) -> bool: ) -def _marker_of(name: str, hint: Any, *, from_hints: bool) -> Type[Marker]: - """The one marker of a field, or a ``DeclarationError``. +def _marker_of(name: str, hint: Any, *, from_hints: bool) -> Tuple[Type[Marker], Any]: + """The one marker of a field and its value type, or a ``DeclarationError``. ``from_hints`` says the field list came from the type hints, as on a SQLAlchemy class. Only there is a wrapper inside the marker's @@ -270,7 +313,7 @@ def _marker_of(name: str, hint: Any, *, from_hints: bool) -> Type[Marker]: field=name, reason="rule_a", ) - return markers[0] + return markers[0], value_type def _no_marker(name: str) -> DeclarationError: @@ -304,6 +347,147 @@ def _marker_not_outermost( ) +def _check_value_type(field: _ReadField) -> None: + """Check that the engine produces the marker. Then check the value type + against the marker's kind.""" + name, marker, value_type = field.name, field.marker, field.value_type + if marker is Passthrough: + if not any(value_type is allowed for allowed in _PASSTHROUGH_TYPES): + raise DeclarationError( + "declaration_type_mismatch", + f"field {name!r} is declared {_type_name(value_type)}. " + "Passthrough stores only str, int, float, bool, bytes, date, " + "datetime, or Decimal.", + field=name, + reason="passthrough", + ) + return + # The engine refuses this marker too, as stack_encrypt_target_unproducible. + # Python refuses it first, with the catalog's reason, so the engine's code + # never reaches a Python caller. The spec asks for this (#1341, Markers). + if not marker.producible: + raise DeclarationError( + "declaration_unproducible", + f"field {name!r} is declared {marker.__name__}. The engine " + f"cannot produce {marker.__name__} yet: {marker.reason}.", + field=name, + ) + kind = marker.plaintext + expected = _KIND_TYPES.get(kind) if kind is not None else None + if kind is None or expected is None: + raise DeclarationError( + "declaration_type_mismatch", + f"field {name!r} is declared {marker.__name__}. Its plaintext kind, " + f"{kind}, has no Python type in this version of the SDK.", + field=name, + reason=kind, + ) + if value_type is not expected: + raise DeclarationError( + "declaration_type_mismatch", + f"field {name!r} is declared {_type_name(value_type)}. " + f"{marker.__name__} takes exactly {expected.__name__}: " + f"Annotated[{expected.__name__}, {marker.__name__}].", + field=name, + reason=kind, + ) + + +def _check_constructor(cls: type, read: List[_ReadField]) -> None: + """Check that the constructor takes every plan field as a keyword argument, + and requires no Ignore field. + + ``decrypt`` builds an instance with one keyword argument per plan field, + and passes no ``Ignore`` field. + """ + try: + parameters = inspect.signature(cls).parameters + except (TypeError, ValueError): + parameters = inspect.Signature().parameters + any_keyword = any(p.kind is p.VAR_KEYWORD for p in parameters.values()) + # A plain class with no constructor: a decorator below this one, such as + # @dataclass, has not made the class yet. + unbuilt = _library_field_names(cls) is None and not any( + "__init__" in vars(base) or "__new__" in vars(base) + for base in cls.__mro__ + if base is not object + ) + for field in read: + parameter = parameters.get(field.name) + if field.marker is Ignore: + if parameter is not None and _required(parameter): + raise DeclarationError( + "declaration_ignore_required", + f"field {field.name!r} is marked Ignore, but the constructor " + f"of class {cls.__name__!r} requires it. Decrypt never " + "passes it, so give it a default or a default factory.", + field=field.name, + ) + elif _takes(cls, field.name, parameter, any_keyword): + continue + elif unbuilt: + raise DeclarationError( + "declaration_not_outermost", + f"class {cls.__name__!r} has no constructor yet. Put " + "@stackencrypt.model above @dataclass or @attrs.define, so " + "that it runs last.", + field=field.name, + ) + else: + raise DeclarationError( + "declaration_constructor_missing_field", + f"the constructor of class {cls.__name__!r} does not take field " + f"{field.name!r} as a keyword argument. Decrypt passes every " + "field by its name, so the constructor must take that name. " + "Look for init=False, a private attrs name, or an alias.", + field=field.name, + ) + + +def _takes( + cls: type, name: str, parameter: Optional[inspect.Parameter], any_keyword: bool +) -> bool: + """Whether the constructor takes field ``name`` as a keyword argument.""" + fields = getattr(cls, "__pydantic_fields__", None) + info = fields.get(name) if isinstance(fields, dict) else None + if info is not None: + # Pydantic's signature shows a field's alias, not its name. Its config + # says whether the name is taken too. With extra="allow" the signature + # has **kwargs, and a name that is not taken lands in the extras. + alias = getattr(info, "validation_alias", None) or getattr(info, "alias", None) + if alias is None or alias == name: + return True + config = getattr(cls, "model_config", {}) + return bool(config.get("populate_by_name") or config.get("validate_by_name")) + return _keyword(parameter) or any_keyword + + +def _keyword(parameter: Optional[inspect.Parameter]) -> bool: + return parameter is not None and parameter.kind in ( + inspect.Parameter.POSITIONAL_OR_KEYWORD, + inspect.Parameter.KEYWORD_ONLY, + ) + + +def _required(parameter: inspect.Parameter) -> bool: + variadic = (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD) + return parameter.kind not in variadic and parameter.default is parameter.empty + + +def _check_plan(table: str, planned: List[_ReadField]) -> None: + """The engine's own check of the plan, with the EQL types.""" + refused = _engine.check_plan( + table, + [ + (field.name, None if field.marker is Passthrough else field.marker.name) + for field in planned + ], + ) + if refused is not None: + code, message, field, reason = refused + raise DeclarationError(code, message, field=field, reason=reason) + + def _type_name(hint: Any) -> str: """A type as the user writes it: ``str``, or ``Optional[str]``.""" if isinstance(hint, type) and not get_args(hint): diff --git a/languages/python/python/stackencrypt/_engine.pyi b/languages/python/python/stackencrypt/_engine.pyi index 9004e0fe1..4ee2cbe2f 100644 --- a/languages/python/python/stackencrypt/_engine.pyi +++ b/languages/python/python/stackencrypt/_engine.pyi @@ -1,4 +1,12 @@ """Type stub for the native extension. Kept by hand, beside the Rust source.""" +from typing import List, Optional, Tuple + def build_info() -> tuple[str, str]: """``(version, engine_commit)`` as recorded at build time.""" + +def check_plan( + table: str, fields: List[Tuple[str, Optional[str]]] +) -> Optional[Tuple[str, str, Optional[str], Optional[str]]]: + """The engine's check of a model's plan: ``None``, or its refusal as + ``(code, message, field, reason)``.""" diff --git a/languages/python/tests/_declaration_cases.py b/languages/python/tests/_declaration_cases.py index ff17e5604..35c6a1705 100644 --- a/languages/python/tests/_declaration_cases.py +++ b/languages/python/tests/_declaration_cases.py @@ -7,7 +7,10 @@ """ import dataclasses +import datetime +import decimal import pickle +import uuid from typing import Annotated, ClassVar, Final, List, NamedTuple, Optional import attrs @@ -17,8 +20,25 @@ from sqlalchemy import ForeignKey import stackencrypt -from stackencrypt import DeclarationError, Ignore, Passthrough, Text, TextEq -from stackencrypt._declaration import declaration_of +from stackencrypt import ( + BigintEq, + Boolean, + DateEq, + DeclarationError, + DoubleEq, + Ignore, + IntegerEq, + JsonSearch, + NumericEq, + Passthrough, + RealEq, + SmallintEq, + Text, + TextEq, + TimestampEq, +) +from stackencrypt import _eql +from stackencrypt._declaration import _KIND_TYPES, declaration_of EXPECTED_FIELDS = [("email", TextEq), ("age", Passthrough)] @@ -197,12 +217,21 @@ class User: ) +class KeywordConstructor: + """A plain class whose fields come from its hints, as a SQLAlchemy + class's do, with a constructor that takes any keyword argument.""" + + def __init__(self, **kwargs): + vars(self).update(kwargs) + + def test_a_collection_value_type_is_not_a_misplaced_wrapper(): @stackencrypt.model("users") - class User: - tags: Annotated[List[str], Passthrough] + class User(KeywordConstructor): + email: Annotated[str, TextEq] + tags: Annotated[List[str], Ignore] - assert planned(User) == [("tags", Passthrough)] + assert planned(User) == [("email", TextEq)] @pytest.mark.parametrize("kind", ["dataclass", "plain"]) @@ -274,6 +303,26 @@ class User: assert isinstance(error.__cause__, NameError) +class LaterBase(DeclarativeBase): + pass + + +def test_a_relationship_to_a_class_defined_later_is_refused_with_its_field(): + # Author is defined after Book, as a relationship target often is. + def build(): + @stackencrypt.model("books") + class Book(LaterBase): + __tablename__ = "books" + id: Mapped[Annotated[int, Passthrough]] = mapped_column(primary_key=True) + title: Mapped[Annotated[str, TextEq]] + authors: Mapped[Annotated[List["Author"], Ignore]] = relationship() # noqa: F821 + + error = refusal(build) + assert error.code == "declaration_hints_unresolved" + assert error.field == "authors" + assert isinstance(error.__cause__, NameError) + + def test_a_class_is_declared_once(): def build(): @stackencrypt.model("users") @@ -338,7 +387,7 @@ def test_model_refuses_an_empty_table_name(): stackencrypt.model("") -def test_declaring_a_class_makes_no_engine_call_and_needs_no_credentials(monkeypatch): +def test_declaring_a_class_needs_no_credentials(monkeypatch): for name in ("CS_WORKSPACE_CRN", "CS_CLIENT_ID", "CS_CLIENT_KEY", "CS_CLIENT_ACCESS_KEY"): monkeypatch.delenv(name, raising=False) @@ -348,3 +397,261 @@ class User: email: Annotated[str, TextEq] assert planned(User) == [("email", TextEq)] + + +# The value type against the marker's kind. + + +@pytest.mark.parametrize( + "value_type, marker", + [ + (str, TextEq), + (int, SmallintEq), + (int, IntegerEq), + (int, BigintEq), + (float, RealEq), + (float, DoubleEq), + (bool, Boolean), + (decimal.Decimal, NumericEq), + (datetime.date, DateEq), + (datetime.datetime, TimestampEq), + ], +) +def test_each_kind_takes_its_one_python_type(value_type, marker): + User = dataclasses.make_dataclass("User", [("value", Annotated[value_type, marker])]) + + assert planned(stackencrypt.model("users")(User)) == [("value", marker)] + + +def test_every_producible_kind_but_json_has_a_python_type(): + # A new kind in the catalog fails here until the table names its type. + # JSON has no Python type in v1 on purpose (#1341, Out of Scope). + kinds = {marker.plaintext for marker in _eql.TYPES if marker.producible} + assert kinds - set(_KIND_TYPES) == {"json"} + + +def test_a_value_type_that_is_not_the_kinds_is_refused(): + def build(): + @stackencrypt.model("users") + @dataclasses.dataclass + class User: + email: Annotated[int, TextEq] + + error = refusal(build) + assert error.code == "declaration_type_mismatch" + assert error.field == "email" + assert error.reason == "string" + assert "'email'" in str(error) + assert "TextEq" in str(error) + assert "Annotated[str, TextEq]" in str(error) + + +@pytest.mark.parametrize( + "value_type, marker", + [ + (Optional[str], TextEq), + (bool, IntegerEq), + (datetime.datetime, DateEq), + (int, DoubleEq), + ], +) +def test_a_value_type_must_be_exactly_the_kinds(value_type, marker): + User = dataclasses.make_dataclass("User", [("value", Annotated[value_type, marker])]) + + error = refusal(lambda: stackencrypt.model("users")(User)) + assert error.code == "declaration_type_mismatch" + + +@pytest.mark.parametrize( + "value_type", + [str, int, float, bool, bytes, datetime.date, datetime.datetime, decimal.Decimal], +) +def test_a_passthrough_field_takes_the_eight_types(value_type): + User = dataclasses.make_dataclass("User", [("value", Annotated[value_type, Passthrough])]) + + assert planned(stackencrypt.model("users")(User)) == [("value", Passthrough)] + + +def test_a_passthrough_uuid_is_refused(): + def build(): + @stackencrypt.model("users") + @dataclasses.dataclass + class User: + id: Annotated[uuid.UUID, Passthrough] + email: Annotated[str, TextEq] + + error = refusal(build) + assert error.code == "declaration_type_mismatch" + assert error.field == "id" + assert error.reason == "passthrough" + + +def test_a_json_field_has_no_python_type_yet(): + def build(): + @stackencrypt.model("users") + @dataclasses.dataclass + class User: + profile: Annotated[str, JsonSearch] + + error = refusal(build) + assert error.code == "declaration_type_mismatch" + assert error.reason == "json" + + +def test_an_unproducible_marker_is_refused_with_the_catalog_reason(): + def build(): + @stackencrypt.model("users") + @dataclasses.dataclass + class User: + profile: Annotated[str, _eql.Json] + + error = refusal(build) + assert error.code == "declaration_unproducible" + assert error.field == "profile" + assert _eql.Json.reason in str(error) + + +# The constructor. + + +def test_the_decorator_below_dataclass_is_refused(): + def build(): + @dataclasses.dataclass + @stackencrypt.model("users") + class User: + email: Annotated[str, TextEq] + + error = refusal(build) + assert error.code == "declaration_not_outermost" + assert error.field == "email" + assert "above @dataclass" in str(error) + + +def test_the_decorator_below_attrs_is_refused(): + def build(): + @attrs.define + @stackencrypt.model("users") + class User: + email: Annotated[str, TextEq] + + assert refusal(build).code == "declaration_not_outermost" + + +def build_without_the_field_in_the_constructor(kind): + if kind == "init=False field": + + @stackencrypt.model("users") + @dataclasses.dataclass + class User: + email: Annotated[str, TextEq] = dataclasses.field(init=False) + + elif kind == "init=False class": + + @stackencrypt.model("users") + @dataclasses.dataclass(init=False) + class User: + email: Annotated[str, TextEq] + + elif kind == "attrs private name": + + @stackencrypt.model("users") + @attrs.define + class User: + _email: Annotated[str, TextEq] + + elif kind == "pydantic alias": + + @stackencrypt.model("users") + class User(pydantic.BaseModel): + email: Annotated[str, TextEq] = pydantic.Field(alias="Email") + + else: + + @stackencrypt.model("users") + class User(pydantic.BaseModel): + model_config = pydantic.ConfigDict(extra="allow") + email: Annotated[str, TextEq] = pydantic.Field(alias="Email") + + +@pytest.mark.parametrize( + "kind", + [ + "init=False field", + "init=False class", + "attrs private name", + "pydantic alias", + "pydantic alias with extra=allow", + ], +) +def test_a_field_the_constructor_does_not_take_is_refused(kind): + error = refusal(lambda: build_without_the_field_in_the_constructor(kind)) + assert error.code == "declaration_constructor_missing_field" + assert "above @dataclass" not in str(error) + + +def test_a_pydantic_alias_is_accepted_when_the_name_is_taken_too(): + @stackencrypt.model("users") + class User(pydantic.BaseModel): + model_config = pydantic.ConfigDict(populate_by_name=True) + email: Annotated[str, TextEq] = pydantic.Field(alias="Email") + + assert planned(User) == [("email", TextEq)] + + +def build_with_a_required_ignore(kind): + if kind == "dataclass": + + @stackencrypt.model("users") + @dataclasses.dataclass + class User: + email: Annotated[str, TextEq] + note: Annotated[str, Ignore] + + elif kind == "attrs": + + @stackencrypt.model("users") + @attrs.define + class User: + email: Annotated[str, TextEq] + note: Annotated[str, Ignore] + + else: + + @stackencrypt.model("users") + class User(pydantic.BaseModel): + email: Annotated[str, TextEq] + note: Annotated[str, Ignore] + + +@pytest.mark.parametrize("kind", ["dataclass", "attrs", "pydantic"]) +def test_a_required_ignore_field_is_refused(kind): + error = refusal(lambda: build_with_a_required_ignore(kind)) + assert error.code == "declaration_ignore_required" + assert error.field == "note" + assert "default" in str(error) + + +def test_an_ignore_field_the_constructor_does_not_name_is_omittable(): + @stackencrypt.model("users") + class User(KeywordConstructor): + email: Annotated[str, TextEq] + note: Annotated[str, Ignore] + + assert planned(User) == [("email", TextEq)] + + +# The engine's own check of the plan. + + +def test_a_plan_the_engine_refuses_is_a_declaration_error(): + # Every field is Ignore, so the plan has no field, and the engine refuses + # an empty plan. + def build(): + @stackencrypt.model("users") + @dataclasses.dataclass + class User: + note: Annotated[str, Ignore] = "" + + error = refusal(build) + assert error.code == "stack_encrypt_dynamic_plan" + assert error.reason == "no_fields" diff --git a/scripts/__tests__/cargo-lock-freshness.test.mjs b/scripts/__tests__/cargo-lock-freshness.test.mjs index ddaacfc96..74ebd575d 100644 --- a/scripts/__tests__/cargo-lock-freshness.test.mjs +++ b/scripts/__tests__/cargo-lock-freshness.test.mjs @@ -170,18 +170,20 @@ describe('Cargo.lock records this tree’s crates at their real versions', () => // pushing it out of sync: `scripts/sync-lockstep-versions.mjs` writes its // `Cargo.toml` on every release. If this crate ever drops out of the pair // set, the check that matters most has silently stopped running. It is - // in the three locks whose workspaces build it: EQL's own, protect-ffi's, - // and the stack-encrypt Go guest's, whose `eql` build links it by path - // (ADR-0007, amended). The stack-* workspaces do not depend on it. The - // bump's own set is discovered by walking (`cargoLockWorkspaces`), so a - // lock added here is one it already rewrites; this pins that the walk - // still sees all three. + // in the four locks whose workspaces build it: EQL's own, protect-ffi's, + // the stack-encrypt Go guest's, whose `eql` build links it by path + // (ADR-0007, amended), and the Python SDK's, whose core crate links it by + // path for the EQL resolver (#1387). The stack-* workspaces do not depend + // on it. The bump's own set is discovered by walking + // (`cargoLockWorkspaces`), so a lock added here is one it already + // rewrites; this holds that the walk still sees all four. expect( PAIRS.filter(({ name }) => name === 'eql-bindings') .map(({ lock }) => lock) .sort(), ).toEqual([ 'languages/golang/encrypt/guest/Cargo.lock', + 'languages/python/Cargo.lock', 'languages/typescript/packages/protect-ffi/Cargo.lock', 'packages/eql/Cargo.lock', ]) diff --git a/scripts/__tests__/python-ci.test.mjs b/scripts/__tests__/python-ci.test.mjs index 69fc5b537..f27814809 100644 --- a/scripts/__tests__/python-ci.test.mjs +++ b/scripts/__tests__/python-ci.test.mjs @@ -1,5 +1,5 @@ import { existsSync, readdirSync, readFileSync } from 'node:fs' -import { join } from 'node:path' +import { join, posix } from 'node:path' import { describe, expect, it } from 'vitest' import { filterCovers } from './lib/paths-filter.mjs' import { REPO_ROOT } from './lib/repo-root.mjs' @@ -39,7 +39,8 @@ const read = (rel) => readFileSync(join(REPO_ROOT, rel), 'utf8') /** * The directories outside `languages/python` whose crates the workspace * compiles: every `[[package]]` in its Cargo.lock without a `source` line is a - * path dependency, and the stack-* ones live at `packages/`. + * path dependency. The stack-* ones live at `packages/`, and the EQL + * crates at `packages/eql/crates/`. */ function pathDependencyTrees() { const lock = read(`${PYTHON_ROOT}/Cargo.lock`) @@ -49,10 +50,12 @@ function pathDependencyTrees() { const name = /^name = "([^"]+)"/m.exec(block)?.[1] if (!name) throw new Error(`Cargo.lock package with no name:\n${block}`) if (name.startsWith('stackencrypt-')) continue - const dir = `packages/${name}` - if (!existsSync(join(REPO_ROOT, dir, 'Cargo.toml'))) { + const dir = [`packages/${name}`, `packages/eql/crates/${name}`].find((d) => + existsSync(join(REPO_ROOT, d, 'Cargo.toml')), + ) + if (!dir) { throw new Error( - `${PYTHON_ROOT}/Cargo.lock lists ${name} as a path dependency, and ${dir}/Cargo.toml does not exist; teach this test where it lives`, + `${PYTHON_ROOT}/Cargo.lock lists ${name} as a path dependency, and neither packages/${name} nor packages/eql/crates/${name} holds its Cargo.toml; teach this test where it lives`, ) } trees.add(dir) @@ -68,6 +71,32 @@ function pathDependencyTrees() { const coversTree = (pattern, file) => pattern.endsWith('/**') && file.startsWith(pattern.slice(0, -2)) +/** + * The files the workspace's crates compile through a `#[path = "..."]` + * module. Cargo.lock does not record them, so they are read from every `.rs` + * file under `crates/`. A path is resolved from the directory of the file + * that names it, which is Rust's rule outside an inline `mod { }` block. + */ +function pathModuleFiles() { + const files = [] + const walk = (dir) => { + for (const entry of readdirSync(join(REPO_ROOT, dir), { + withFileTypes: true, + })) { + const rel = `${dir}/${entry.name}` + if (entry.isDirectory()) walk(rel) + else if (entry.name.endsWith('.rs')) { + const attribute = /#\[\s*path\s*=\s*"([^"]+)"\s*\]/g + for (const match of read(rel).matchAll(attribute)) { + files.push(posix.normalize(`${dir}/${match[1]}`)) + } + } + } + } + walk(`${PYTHON_ROOT}/crates`) + return files.sort() +} + describe('tests-python.yml', () => { it('runs on pull requests touching every tree the Python workspace compiles', () => { const on = triggers(readWorkflow(WORKFLOW)) @@ -90,6 +119,21 @@ describe('tests-python.yml', () => { ).toEqual([]) }) + it('runs on pull requests touching a file the core crate compiles by #[path]', () => { + const paths = triggers(readWorkflow(WORKFLOW)).pull_request?.paths ?? [] + const files = pathModuleFiles() + expect(files).toContain('languages/golang/encrypt/guest/src/targets.rs') + const exact = (pattern, file) => pattern === file + const missing = files.filter( + (file) => + !filterCovers(paths, file, (p, f) => exact(p, f) || coversTree(p, f)), + ) + expect( + missing, + `pull_request.paths does not select these files, which crates/core compiles by #[path]:\n${missing.join('\n')}`, + ).toEqual([]) + }) + it('runs the pure-Python suite on the oldest supported Python and the one mise.toml names', () => { // The floor is `requires-python` in pyproject.toml; the local version is the // interpreter `mise run test` links locally. A wheel tagged abi3-py39 is a