feat(eventing): make event signing deployable on a cluster

[mrsabath]

Draft — about half the plan. The key material is generated, mountable and applied at deploy time; the preflight check, e2e assertions, the forged-response demo stage and the README_PHASE1 rewrite are still to come (list at the bottom). Opening early because the Secret-handling and manifest decisions are where review is cheapest, and because one of them deviates from a documented convention on purpose.

The problem

#879 wired signing in both directions, but none of it could be switched on in a cluster. The ConfigMap shipped ER_/EB_VERIFY_KEYSET_PATH and *_SIGNING_KID as empty strings and configmap.yaml:54 said seed paths "belong in the Deployment beside the volume" — except no Deployment had one. The only volume in either was data: emptyDir, and grep -rn "secretName\|configMap:" k8s/ returned nothing. So the feature worked with local file paths on a laptop and was unreachable in Kind or OpenShift.

That blocks the demo track: S6, S8 and S9 (#2596/#2598/#2599) are each defined as "reproducible on a local Kind cluster", and S9 is the forged-event teaching moment.

There was also no way to make a key. shared/signing.py has public_key(seed) but nothing anywhere generated one — enabling signing meant hand-rolling os.urandom(32).hex() and deriving public keys in a REPL.

What's here

scripts/gen_signing_keys.py — seeds at mode 0600 under a 0700 directory, and an agents.json whose public keys are derived from those seeds rather than typed, so the keyset cannot disagree with the keys it authorizes. Never prints a seed on either stream; only kids and public keys, because a pasted terminal transcript is the most likely way a demo key escapes. Refuses to overwrite an existing seed without --force, since that key may already be deployed and in an approved keyset.

Both Deployments mount a seed Secret at /etc/ce-signing and the keyset ConfigMap at /etc/ce-keyset, both optional: true — without that, every signing-less overlay would block in ContainerCreating waiting for a Secret that will never exist.

k8s/base/keyset-configmap.yaml ships empty. "There is never a key in this file" is checkable; "only public keys in this file" is not, and a 32-byte seed is indistinguishable from a public key by length.

k8s/overlays/kind-signed/ flips enforcement on. Separate overlay rather than a patch on kind, because S3/S5 are explicitly unsigned milestones and the before/after contrast is the demo.

k8s_deploy.py --signing-keys DIR applies one seed Secret per service plus the keyset ConfigMap, via stdin so the seed never reaches ps through argv — modelled on ensure_ntfy_secret, including the no-op-when-absent behaviour so a redeploy can't silently switch enforcement off and strand a signed topic. It validates first, because every one of these produces a cluster that refuses all traffic with no local symptom: the directory and each named seed must exist, agents.json must parse and approve at least one kid, and the kid each service signs under must be in that keyset.

One deliberate deviation, and one bug caught before it shipped

Per-service Secrets, not the documented single one. Both Deployments initially mounted the same event-signing-key, which would have given the two services one shared seed. That collapses the asymmetry the design rests on — EventBridge would hold the key it verifies runner responses with and could forge them, which is the exact property that made HMAC unacceptable (DESIGN_PHASE2.md §4.3). Now eventbridge-signing-key and eventrunner-signing-key, with a test asserting they never match.

This deviates from event-signing-key as documented at README_PHASE1.md:704-722. That section needs rewriting regardless — it tells the operator to write a seed to /tmp/seed.hex and rm it, contradicting the stdin-only rule stated 230 lines earlier at :475, and it says "have the publisher sign with the same seed", describing the symmetric setup §4.3 explicitly rejected. The rewrite is in the remaining work; flagging it now so the name change isn't a surprise.

Logging asymmetry is intentional. Seeds are logged as a length only — not even a prefix, unlike the API token, because 4 hex chars of a 32-byte seed is 16 bits of a private key and there's nothing to gain from identifying it. Kids are logged in full, because "the operator deployed yesterday's key set" is otherwise invisible until it surfaces as an unexplained rejection.

Also fixed a pre-existing leak: --purge listed eventbridge-ntfy in neither the doomed preview nor the delete tuple, so a "purged" namespace kept an ntfy topic — a capability that lets anyone who learns it both read and publish your notifications.

Testing

568 passed, 5 skipped (5 cluster-gated). 17 new tests. ruff check and ruff format --check clean.

The three manifest tests this could have broken all still pass, and they're why the design looks the way it does: test_no_credential_is_ever_rendered_into_a_manifest (so the Secret is applied by the script, never declared — no secretGenerator), test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic (so enforcement stays out of the shared overlays), and test_no_manifest_sets_run_as_user (so no defaultMode: 0400, which would be root-only given OpenShift assigns the UID).

Verified by rendering rather than assumed:

• data is still volume index 0 in all three existing overlays, so the demo overlay's op: replace on /spec/template/spec/volumes/0 still targets the PVC swap it means to.
• No overlay renders a kind: Secret.
• kind stays fully off; kind-signed renders enforcement plus both key paths, and those values map onto the real Cfg fields.
• Keys from the generator verify through verify_with_keyset after a to_kafka_binary/from_kafka_binary round trip.

Not verifiable in my sandbox, and not claimed: kubectl apply --dry-run=client needs a reachable cluster to fetch the OpenAPI schema and fails even with --validate=false, and Kind can't run here (podman has no VM; creating one fails on the proxy's download cap). So schema validity of the new volume stanzas and the actual mount behaviour are unproven — covered by test_every_object_is_accepted_by_the_api_server when run against a real cluster.

Still to do on this branch

• k8s_preflight.py check 14 — Secret present, keyset parses, every kid the ConfigMap names is in it
• k8s_e2e_test.py — assert the terminal response carries ce_signature and the startup banner says signing is ON
• k8s_demo_flow.py --forge — publish an unsigned event to the responses topic and assert it lands as phase="error" with the payload retained under data["rejected"]. Mechanism already validated against a live broker: kafka-console-producer.sh --property parse.headers=true yields a CloudEvent the consumer parses and the verifier rejects (binary mode matters — a structured-mode body gives attrs == {})
• Rewrite README_PHASE1.md §"Signed events"

Question for review

Is kind-signed the right shape? The alternative is a --signed flag on the existing kind overlay. I went with a separate overlay so the unsigned path keeps working for S3/S5 and so the test overlay's no-credential assertion keeps meaning something — but the name is about to get baked into three more files (preflight, e2e, demo flow), so it's cheap to change now and annoying later.

Separately: S6 (#2596) specifies ES256 + a SPIFFE-derived ce-source + rotation, and this is Ed25519 + a static keyset. The eventing design docs never commit to ES256 — only the issue does, and §4.3 argues Ed25519 on merits with SPIRE as a later swap that leaves verification logic unchanged. Worth an explicit decision rather than a surprise at demo time; I'll comment on #2596.

Assisted-By: Claude Code