stack-encrypt: data keys leave copies on the stack, and long-lived keys are not in Locked — hold every key in one place, by reference #1458

Background

Stack Encrypt (packages/stack-encrypt) is the Rust engine every CipherStash SDK runs on. It handles two kinds of key material in memory:

• Per-row data keys. Every row is sealed under a fresh data key that ZeroKMS (CipherStash's key service) issues or returns. A bulk operation handles one key per row, so thousands at a time, and each lives only for the operation. The type is stack_kms::DataKey: two plain byte arrays (iv: [u8; 16], key: [u8; 32]), wiped on drop through the zeroize crate's ZeroizeOnDrop.
• Long-lived keys. These live as long as the cipher: the client key in ClientKey (a V1KeySet), each keyset's PRF key in KeysetState (cached across calls in an Arc), and the cipher's JSON seed (a Protected<[u8; 32]>).

vitaminc, our secrets library, has two storage types for this:

• Protected<T> stores the value inline, wipes it on drop, and gates access (risky_ref), so a value is not printed, compared or copied by accident. It is the right shape for a value that lives for one call.
• Locked<T> writes the value once into its own memory region and never moves it: only a small handle moves, and the value is reached by reference (with, risky_ref). The region is locked against swapping (mlock), left out of core dumps on Linux (MADV_DONTDUMP), fenced by guard pages, and overwritten on drop with stores the compiler cannot remove. Locked::generate builds a secret inside the region, so it never touches the stack. vitaminc's docs describe it as the right shape for secrets that outlive a call, since "nothing drops a static, a leaked Arc, or anything at all on SIGTERM, SIGKILL or process::exit".

A Rust move is a bitwise copy that leaves the source bytes behind, and ZeroizeOnDrop wipes only a value's final location. So a secret that travels by value leaves a copy in every temporary slot it passes through.

Problem

1. Data keys are handed out by value, and leave copies on the stack

The queue of fetched data keys (Responses and Queue in packages/stack-encrypt/src/target/request.rs) keeps the originals in its buffer, so that they are wiped where they sit, and hands each caller a clone:

fn pop_front(&mut self) -> Option<T> {
    let item = self.items.get(self.next).cloned()?;
    self.next += 1;
    Some(item)
}

pub fn next_retrieved_key(&mut self) -> Result<DataKey, Error> {
    self.retrieved.pop_front().ok_or(Error::ResponseShape)
}

The clone is moved into an Option, then into Some(item), then through ok_or into a Result, and then out to the caller (json/derive.rs draws keys this way). drain_retrieved and drain_generated hand keys out by value the same way, to target/core.rs. Each move leaves its source bytes unwiped.

Evidence (macOS arm64, debug, found while investigating the heap-hygiene failure on #1445). A temporary probe scrubbed the stack before a JSON subtree open, then scanned the dead stack afterwards for the data key:

• it found three copies of the key, whether or not the fake key source wiped its own frames;
• scrubbing in dispatch after the join left all three, and scrubbing in settle after fulfil cleared all three, so the fulfilment writes them;
• their offsets are those of Option<DataKey> temporaries, which have alignment 1.

The heap-hygiene suite (tests/heap_hygiene.rs) does not see these, because it watches freed heap blocks, not the stack. With its spy fixed on #1445, no watched operation shows them reaching the heap. But dead stack is reused, and a future built over it is copied into its heap box together with the bytes it never writes, which is exactly how the spy's own stray copy reached a box on #1445. So these copies are one step from the heap.

2. Workarounds whose need is now unknown

dispatch boxes its retrieve_keys and load_keysets futures, its error and each per-keyset load, and keyset_state boxes its future, each with a comment saying the box keeps residue out of the heap. With the fixed spy, the suite also passed (debug) with every one of those boxes removed.

That does not prove the boxes are useless: the original, faulty spy also passed against the stripped code, because the residue landed somewhere it did not watch. The faulty spy and those boxes landed on the same day, so the boxes may have been chasing the same artefact. Which of them still earn their place is open.

3. DataKey is Clone, and not a vitaminc type

DataKey derives Clone, with a FIXME saying it should not (the JSON indexer uses one key many times). It gets vitaminc's helpers piecemeal (TimingSafeEq for constant-time equality, OpaqueDebug) but none of vitaminc's storage types. So nothing stops a key being cloned or moved by value, which is how problem 1 arises.

4. Long-lived keys are not held in Locked

The client key, each keyset's cached PRF key, and the JSON seed live for the life of the cipher. They are exactly what Locked is for, and none of them is held in one. No crate in this repository uses Locked today. Their bytes can therefore reach swap or a core dump, and every address a move left them at outlives the process. (Not checked yet: how V1KeySet and the PRF type wipe their own contents.)

5. vitaminc has no Locked for a batch of per-row keys

Locked<T> gives each value its own mapping: a system call and at least a page of memory, plus guard pages, per value. That suits a few long-lived keys. It does not suit per-row data keys, which arrive by the thousand per batch and live for one operation. It also covers inline types only, so a Locked<Vec<DataKey>> would protect the vector's header and not the keys.

What fits a batch is a batch variant of the Locked concept: one locked, fenced, non-dumped region that holds a whole batch's fixed-size secrets, written in place, never moved, reached by reference, and wiped when the batch drops. vitaminc does not have one. Designing and adding it to vitaminc is in scope for this issue; a separate vitaminc issue can be split out once the shape is clear.

What is needed

• A data key exists in one place for its whole life in the engine, from the moment it is derived or received to the moment it is wiped. It is reached by reference, never copied or moved by value through temporaries, and the type system makes that the only way to use it.
• Long-lived keys are held in memory that cannot be swapped, dumped or left behind by a move.
• A way to hold a batch of per-row keys with the same guarantees, without a mapping per key: the batch variant of Locked in vitaminc.
• A test that sees what the heap-hygiene suite cannot: stack residue after an operation.
• A decision, with evidence, on each residue-motivated box in dispatch and keyset_state.

Options the #1445 investigation considered. These are recorded, not decided:

1. Hand keys out by reference. Responses returns &DataKey from its buffer, and the retrieval fills pre-zeroed slots in place. This removes the copies at their source. next_retrieved_key and next_generated_key are public (tests/target.rs uses them), and derive.rs draws the next keyset while it holds a key, so that borrow needs restructuring. The drain_* consumers change too.
2. Wipe engine-owned future boxes when they are freed. This closes the heap side whatever the layout, at one wipe per box, but leaves the dead stack.
3. Scrub a bounded region of stack after fulfil. In the probe this cleared all three copies. It is cheap, but best-effort, since the compiler may put copies elsewhere.
4. The batch variant of Locked, above, as the storage the by-reference handout points into.

Whatever the approach, using a key can still make the compiler place its bytes on the stack, for example during AES key expansion. The aim is to remove every copy the engine makes itself, and to measure what is left.

Relationship to other work

• feat(eql): test equality and ranges on one array element of a json_tree document #1445: where this was found. Its heap-hygiene failure was the test spy's own copy of the key, fixed there in the test. It no longer boxes the keyset loads.
• docs/sdk-design-principles.md: every SDK runs on this engine, so its handling of keys is every SDK's.