Skip to content

feat(eventing): Phase 3 steps 1-4 + T5 — tenancy keys, per-user stores, declarative agents - #883

Draft
mrsabath wants to merge 11 commits into
mainfrom
feat/eventing-phase3-tenancy-foundation
Draft

mrsabath wants to merge 11 commits into
mainfrom
feat/eventing-phase3-tenancy-foundation

Conversation

@mrsabath

@mrsabath mrsabath commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Implements agentdocs/DESIGN_PHASE3.md §9 rollout steps 1–4, plus T5 — the
foundation for per-user isolation. Extends the Phase 2 identity work (#879) rather than
replacing any of it.

The design is a 12-step, 22-task phase, far larger than Phase 0/1/2 were individually.
§9 singles out the first four steps as the part that is safe to merge while multi has
never been switched on anywhere: "the first four steps are refactors with tests, and
everything risky is behind a flag that defaults off."
T5 is the keystone that everything
from step 5 on depends on, so it is here too.

What landed

Task What
T1 shared/tenancy.py — userkey(), TopicSet, ntfy_topic()
T2 TopicSet threaded through the producer, responses consumer and both mirrors
T3 Caller, the user registry, ce_userkey stamped on requests
T4 SIGNED_ATTRS += userkey, depth — one change, per §8.3
T5 Per-user stores, the global correlationid → userkey index, Minter uniqueness
T13 eventrunner/agentspec.py + build_cmd as a function of the spec
T22 The Secret-vs-ConfigMap rule pinned in test_manifests.py

Not implemented, deliberately: T6 (ensure_subscribed), T7 (owner-scoped reads), T8
(transcript auth), T9–T12 (capability keys, ntfy isolation, k8s_tenant.py), T14–T19
(triggers, fetched skills), T20 (Kafka ACLs), T21 (deletion).

The additive guarantee

EB_TENANCY_MODE=single is the default and reproduces Phase 2 byte for byte. The 656
tests that existed before this branch pass untouched
— three test fakes were widened
to match a producer signature, but no assertion was weakened or removed. That is the
check §9 step 2 asks for, and it is why the largest diff in the phase is also the least
risky.

The default AgentSpec is held to the same standard:
test_default_spec_reproduces_phase2_argv_exactly compares against a literal
transcription
of the pre-Phase-3 build_cmd, not a call into the current
implementation. Single-tenant mode also keeps its SQLite files exactly where Phase 2 left
them — directly in the bridge root, not under shared/ — so an upgraded deployment's
sessions and transcripts keep showing up instead of appearing to vanish.

Decisions worth a reviewer's eye

  • The digest in userkey is the security control, not decoration. _slug is lossy so
    keys stay readable in kubectl get kafkatopics; that is safe only because 32 bits of
    SHA-256 over the canonical form are appended. a.b@x.com, a_b@x.com and a-b@x.com
    all slug to a-b-x-com and must not merge. The issuer is inside the hashed bytes too —
    otherwise a second configured issuer impersonates the first.
  • Separate store FILES, not one table with a userkey column (§6.1). store.py has
    28 methods and every one touches tenant data; a WHERE userkey = ? remembered 28 times
    is one that will eventually be forgotten. Separate files make the mistake structurally
    unavailable — the connection is the tenant.
  • The ownership index is UNIQUE on correlationid alone (§2.6). (userkey, correlationid) is right inside a tenant's store and exactly wrong here: two tenants
    minting one id produce two distinct tuples, never conflict, and the collision passes
    silently. That constraint is also what lets sessionuuid stay unsalted.
  • An explicit topic list, not subscribe(pattern=…) (§3.2). A pattern is picked up by
    metadata refresh (5 min default), so a new user's first response can be published before
    anyone is subscribed — and with auto_offset_reset=latest it is never read, with no
    error anywhere.
  • A request cannot widen the tool policy. max_turns/model still come from the
    request (Phase 0 wire parameters); permission mode, tool lists and system prompt are
    spec-only.

Bugs found in self-review and fixed

Each was confirmed by executing the real code, and each has a regression test that fails
without the fix:

  1. /continue returned a WSGI 500 where the owning tenant could not be resolved —
    reachable via a deregistered user, a NULL submitter, or a mirror-backfilled
    correlation. Now a 503 (or 404) before anything is written.
  2. Eviction closed stores that callers still held. Any multi-call read path
    (get_html makes five calls) could have its store closed by a concurrent request for
    another tenant → ProgrammingError: Cannot operate on a closed database. Eviction now
    drops the reference and lets CPython close at the last holder. The first attempt at
    this fix only pinned the SSE path — which closed the easiest case to see and left the
    class open; the bug was in the eviction contract, not one caller.
  3. GroupMirror never learned about tenancy — restart-orphaned groups were settled
    against the shared store (so never settled at all, with silence as the symptom), and
    replayed pre-Phase-3 events, which carry no ce_userkey, put member rows in shared/
    while the group row sat in the tenant's store. That is the normal upgrade path.
  4. except Exception hid two things: a signature mismatch that silently stopped every
    group event from publishing, and a broken ownership index reported as "correlation ID
    space exhausted".

Two limitations stated rather than implied

  1. multi mode is not yet isolation anyone should present as such. Per-user stores
    exist and reads route to the owning tenant's store, but nothing checks that the
    caller is that owner — T7/T8 are not in, so reads, /continue and PUT /transcript
    are as open as Phase 2's. §10 is also blunt that even once those land, separately-named
    topics without Kafka ACLs are organisation, not isolation.
  2. GET /v0/groups now requires an authenticated caller in multi mode, because the
    list must be scoped to someone. §6.2's route table does not mention it; flagging
    because it is a route that was open in Phase 2 and is the HTML page's own poller.
    multi already refuses anonymous callers at resolve_caller, so the practical blast
    radius is small — but say so if you would rather it waited for T7.

Testing

  • 840 passed, 5 skipped. Counted properly this time: +217 unique test functions
    (525 on upstream/main to 742), 845 collected with parametrisation. An earlier version
    of this body said "656 baseline + 139 new", which was wrong — it diffed collected
    counts across a moving baseline. ruff check and ruff format --check clean.
  • Both services verified to start in every tenancy configuration; all startup refusals
    print readable messages rather than tracebacks; the on-disk layout verified
    upgrade-safe.
  • Test strength checked by mutation, not assumed. One gap found and closed: deleting the
    LRU's pinning check passed every test, because they asserted object identity rather than
    the notification path that is the rule's actual purpose. Removing it now fails 4 tests.

Review round 1 (#883, @aslom) is addressed in 6ebdd80: five must-fixes, including a
single-mode regression where the ownership index was never seeded from an existing store
(so exists() reported live correlation ids as free and a mint could reissue one), an
unauthenticated cross-tenant group mutation, agent missing from SIGNED_ATTRS while it
selects the tool-policy sandbox, and a userkey reaching a filesystem path join
unvalidated from an inbound Kafka header. Each reproduced before fixing; each has a
regression test verified by reverting its fix. Six design-doc contradictions corrected in
the same commit. Deferred, by agreement: the T22 subset check,
RequestsMirror/NtfyPublisher stores=, a handler-to-topic integration test, the
forget() tombstone decision, and the four phase documents.

Draft because this is the first slice of a 12-step phase — review comments here should
shape T6/T7/T8 before they are written.

Design: eventing/agentdocs/DESIGN_PHASE3.md (#881). Reading order for review: §9 for
why this scope, §2.3 for why the key ends in a hash, §6.1 for the per-user store trade,
§2.6 for the uniqueness guarantee.

🤖 Generated with Claude Code

…pecs

DESIGN_PHASE3.md T1 and T13, the two tasks with no dependencies. Nothing here
changes behaviour: `single` tenancy is the default everywhere and the `default`
AgentSpec reproduces Phase 2's argv byte for byte.

shared/tenancy.py (T1)
  `userkey(issuer, userid)` is the only place an identity becomes a name. The
  slug is deliberately lossy so keys stay readable in `kubectl get kafkatopics`;
  that is safe only because 32 bits of SHA-256 over the canonical form are
  appended, so two identities that slug identically still differ. The issuer is
  inside the hashed bytes as well as the prefix — without it, configuring a
  second issuer is a way to impersonate an identity on the first. The digest
  input is joined with \x1f so the issuer/userid boundary is unambiguous.

  Canonicalisation is per-issuer and minimal: GitHub folds case (and must agree
  with `ghauth.is_allowed`, which a test pins), email lower-cases only the
  domain per RFC 5321, static is byte-exact. Erring toward two tenants for one
  human is wasteful; the other direction is a leak.

  `TopicSet` is the one place §3.1's layout is written down, and refuses to name
  a topic without a userkey in multi mode rather than falling back to a shared
  one. `ntfy_topic` is HMAC-derived so the name carries no identifier (§4.1).

eventrunner/agentspec.py (T13)
  One TOML file per agent, carrying the tool policy that §7.4 depends on — which
  is why §9 puts this before triggers. Validation refuses rather than coerces: a
  policy that silently drops the clause it could not parse would hand a
  trigger-driven agent the tool the operator meant to remove. A missing
  non-default agent is an error event, never a silent fallback to `default`.

  `build_cmd` takes a spec. The request still wins on `max_turns`/`model` (Phase
  0 wire parameters), but the tool policy, permission mode and system prompt are
  spec-only: a request that could widen the sandbox is a request that can escape
  it.

Tests: 105 new, 656 total passing, no existing test modified. The argv-equality
test compares against a literal transcription of Phase 2's `build_cmd` rather
than a call into the current one, so it catches a regression instead of
following one.

The `claude --help` flag assertion covers only the flags Phase 3 adds.
`--max-turns` is supported but undocumented in `--help` on 2.1.270, so asserting
the pre-existing flags would fail against a working deployment.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…igned tenancy

DESIGN_PHASE3.md T2, T3, T4 and T22. Still no behaviour change in the default
configuration: `EB_TENANCY_MODE=single` is the default, and the 656 tests that
existed before this branch all pass untouched. That is the property §9 asks steps
1-4 to preserve — refactors with tests, everything risky behind a flag that is off.

T2 — TopicSet threaded through (§3.2)
  One set, built once in `__main__` and handed to the producer, the responses
  consumer, both mirrors and the config, so §3.1's layout appears exactly once.
  `publish_request` takes a `userkey` and resolves the topic from it; one producer
  can send to any topic, so there is no second producer and no connection per user.

  The explicit-topic-list option, not `subscribe(pattern=...)`. A pattern is picked
  up by metadata refresh (5 min by default), so a new user's first response can be
  published before anyone is subscribed and with `auto_offset_reset=latest` it is
  never read — the page stays empty and no error appears anywhere. The blocking
  `ensure_subscribed` that closes the remaining race is T6.

  GroupMirror takes the full topic list directly: it is one-shot at startup, so
  there is no new-user race to lose, and a tenant whose topic is not provisioned
  yet is skipped with a log line rather than costing everyone their group history.

T3 — the registry and Caller (§2.4, §2.5)
  `Caller` carries the tenancy key so no caller derives it twice. `submitter_iss`
  is deliberately not the same as `issuer`: a static identity needs an issuer for
  its key to separate from a GitHub login of the same name, but Phase 2 made an
  absent `ce_submitteriss` mean "an operator typed this", and promoting it would
  upgrade an assertion into a verified claim.

  The registry refuses rather than skips — the opposite of `auth.parse_tokens`,
  because a skipped user is one whose topics exist and whose events have nowhere to
  go. A recorded userkey that disagrees with this build's derivation is a startup
  refusal naming both values, and two identities colliding on one key is refused,
  which is what covers §2.3's 32-bit birthday bound.

  `multi` mode refuses anonymous with 401 and an unregistered user with 403. Both
  disagree with the single-tenant defaults on purpose; `single` is byte-identical
  to Phase 2.

T4 — SIGNED_ATTRS += userkey, depth (§8.3)
  One change, both attributes, because adding one changes canonicalisation and a
  signer and verifier on different versions disagree about every signature. Both
  have to be covered rather than merely present: a mutable `userkey` lets anything
  with topic write access file events into another user's history, and a resettable
  `depth` is not a hop limit.

T22 — the Secret-vs-ConfigMap rule, pinned in test_manifests.py
  The secret-path variables are declared ahead of the tasks that read them (T8-T16)
  so the guard is in place when they land. The classification test only asserts the
  ConfigMap side against live code, and says so, rather than looking stronger than
  it is.

Known gap, stated rather than hidden: `/continue` resolves the owning tenant from
the recorded submitter so a resume reaches the right runner, but does not yet check
that the caller IS the owner — owner-scoped reads and `?k=` keys are §6.2/§4.4
(T7/T9, step 5). Until those land, `multi` mode's `/continue` is as open as Phase
2's. The lookup also cannot distinguish a GitHub `alice` from a static `alice`;
`multi` mode refuses both anonymous and static submissions, so GitHub is the only
issuer that reaches it today, and this moves to T5's global index when that exists.

Tests: 66 new (722 total), ruff clean. Verified both services start in every
tenancy configuration and that all four refusals print readable messages rather
than tracebacks.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…till design

The Phase 3 design doc said "nothing in this document is implemented yet" and the
agentdocs index said "Design only — not implemented". Both are now wrong in a way
that matters: a reader who trusts them will either re-implement steps 1-4 or assume
`EB_TENANCY_MODE=multi` gives them isolation it does not yet give.

So the status lines name the tasks that landed (T1, T2, T3, T4, T13, T22 = §9 steps
1-4) and say plainly that everything from step 5 on — per-user stores, owner-scoped
reads, transcript auth, `k8s_tenant.py`, ntfy isolation, triggers, fetched skills,
Kafka ACLs — is still design only.

The component README gains a Phase 3 configuration table covering only the six
variables whose code exists, with two notes that are the point of including it at
all: `multi` mode is not yet isolation anyone should present as such (the HTTP reads
and `/continue` are still as open as Phase 2's, because steps 5-8 are not in), and
adding a user is an operator action because EventBridge has no Kubernetes client.

Verified against the code rather than assumed: only `POST /v0/agents` and
`POST /v0/groups` call the auth path, so every GET, `/continue` and `/transcript`
is open in both modes today.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…rashing

Found reviewing the previous commit. In `multi` mode `_owner_userkey` returned
`None` both when no key was needed (single mode) and when the owning tenant could
not be determined. The second `None` flowed into `publish_request` ->
`TopicSet.requests(None)`, which raises `ValueError` by design — publishing a resume
to a shared topic would run it on another tenant's runner, under that tenant's
credential — but nothing caught it, so the request died as a WSGI 500 with a stack
trace instead of a status code.

Three ways to reach it, none of them abuse:

  * a user removed from the registry;
  * a `prompts` row whose `submitter` is NULL — the column is nullable and pre-dates
    auth, so these rows exist in any store that outlived Phase 2;
  * a correlation back-filled from the topic by `RequestsMirror`, which records no
    submitter at all.

`_owner_userkey` now returns `(userkey, unresolved_reason)` with exactly one set, so
the two `None` cases are distinguishable, and both `/continue` paths check it before
writing anything — no touched session row and no orphan prompt row claiming a turn
that never ran. The JSON route answers `503` naming what could not be resolved,
matching how §2.5 treats a user whose topics do not exist. The HTML form route
answers `503` with a plain-text body rather than redirecting: a silent `303` back to
the transcript page would look like the turn was accepted and then vanished, which is
the Phase 2 §6.1 symptom class this phase keeps trying to avoid.

Only reachable with `EB_TENANCY_MODE=multi`, so the default path was never affected.

13 new tests (735 total). Verified they are load-bearing by reintroducing the bug:
the four refusal tests fail without the guard and pass with it.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
… index

DESIGN_PHASE3.md T5 (§6.1, §2.6), the keystone the rest of step 5+ depends on: T7
(owner-scoped reads), T8 (transcript auth), T15 (triggers) and T21 (deletion) are all
blocked on it. `EB_TENANCY_MODE=single` remains the default and reproduces Phase 2.

eventbridge/owner_index.py — the one deliberately cross-tenant table
  `correlationid -> userkey`, UNIQUE on `correlationid` **alone**. That is the part to
  get right: `(userkey, correlationid)` is the correct primary key inside a tenant's
  store and exactly the wrong constraint here, because two tenants minting one id
  produce two distinct tuples, never conflict, and the collision passes through
  silently — which is the failure the guarantee exists to prevent.

  It has to be cross-tenant: the lookup that decides which tenant owns a correlation
  must happen before a per-user store can be chosen. It is also what lets
  `sessionuuid = uuid5(NAMESPACE, correlationid)` stay UNSALTED (§2.6) — salting it
  would break every existing session for a collision this constraint removes.

  `owner_of` returns `(userkey, known)` because a bare `None` conflates "the shared
  tier owns this" with "never seen", and §6.2's 404 rule needs them distinguished.

eventbridge/store_registry.py — per-user stores, LRU-bounded
  Separate FILES, not one database with a `userkey` column. `store.py` has 28 methods
  and every one touches tenant data; a `WHERE userkey = ?` remembered 28 times is one
  that will eventually be forgotten, with a user reading someone else's conversation
  as the symptom. Separate files make the mistake structurally unavailable — the
  connection IS the tenant.

  Two connections per tenant in WAL mode (main + -wal + -shm) bounds this near 150-200
  tenants under a 1024 soft RLIMIT_NOFILE, hence `EB_MAX_OPEN_STORES=64`. A store with
  live SSE subscribers is PINNED: evicting one makes an open page stop updating with no
  error anywhere. If everything is pinned the cache exceeds its ceiling rather than
  breaking a viewer — over-budget degrades and is visible on /healthz; a dead stream
  does not recover.

Minter now consults the index (§2.6)
  One indexed SELECT per mint replaces the startup seeding loop. With N per-user stores
  the old loop meant N SQLite opens before the socket binds — 100 tenants x 10,000
  correlations for a guarantee one SELECT gives directly. The claim happens inside the
  retry loop, so a candidate lost to a concurrent worker retries instead of both callers
  believing they own it.

  `mint_for()` passes `userkey` only to minters that accept it, by signature inspection
  rather than by catching TypeError around the call — a working `mint()` can raise
  TypeError from its own body, and swallowing that would silently drop the ownership
  claim.

Routing
  Responses are filed by the event's OWN signed `ce_userkey`. One with no userkey in
  multi mode lands in `shared/` and increments `unattributed` (now on /healthz): never
  guessed into a tenant's store, never dropped. `/continue`, the reads, the group routes
  and the deadline sweeper all route through the index — the sweeper across every tenant,
  because a stalled batch belonging to an untouched tenant is exactly what deadlines are
  for and also the least likely to be in the LRU.

  This retires the submitter-derived owner lookup from the previous commit, and with it
  the limitation that it could not tell a GitHub `alice` from a static `alice`.

Single-tenant mode keeps its files exactly where Phase 2 left them — directly in the
bridge root, not under `shared/` — so an upgraded deployment's sessions and transcripts
keep showing up in the UI instead of appearing to vanish.

Two bugs found and fixed while building this:

  * **Eviction could return a closed store.** With every older entry pinned, the LRU
    took the store it had just opened and handed the caller a closed one — surfacing
    later as `ProgrammingError: Cannot operate on a closed database` from somewhere that
    looks unrelated to caching. Fixed with `protect=`, with a regression test.
  * **`except Exception` hid a signature mismatch.** Adding `userkey=` to
    `publish_group_event` made every group event stop publishing, and the broad handler
    around it turned that into a batch that silently never completed. TypeError now
    re-raises at both call sites.

`Store.close()` added (it had none), and it wakes subscribers first so a viewer blocked
on `Event.wait()` notices shutdown instead of hanging to its own timeout.

Tests: 49 new (785 total), ruff clean. Verified both modes start and that the on-disk
layout is upgrade-safe.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
… them

Found reviewing T5. `StoreRegistry` eviction called `Store.close()`, which created a
use-after-close race on every multi-call read path: a handler resolves a store, and
while it is still making calls a *concurrent request for a different tenant* pushes
the cache over `EB_MAX_OPEN_STORES` and closes it underneath. The next call fails with

    sqlite3.ProgrammingError: Cannot operate on a closed database

from a line that looks nothing to do with caching. `get_html` alone makes five calls on
one store, so the window is wide rather than theoretical, and reproducing it needs only
`max_open=1` and two tenants.

Eviction now drops the registry's reference and lets CPython close the connections when
the last holder goes away — exactly the lifetime that is safe. File descriptors are
reclaimed slightly later than an explicit close; that is the right trade, because the
ceiling is a soft budget while a closed connection under an in-flight request is a 500.

`_sse_generator` additionally now subscribes BEFORE the replay read. Pinning is what
keeps a subscribed store in the cache so later readers get the same object and see the
subscriber's notifications, and the old order left the store unpinned between
`_store_of()` and `subscribe()`. Reordering is harmless: the Event only ever means
"there may be something new", every read is `since_seq=sent`, and an already-set Event
costs one extra query.

Worth noting what the first fix attempt got wrong, since it is the more interesting
mistake: pinning only the SSE path would have closed the case that was easiest to see
while leaving every ordinary read exposed. The bug was in the eviction contract, not in
one caller.

Two regression tests, one asserting an evicted-but-held store stays usable for both
reads and subscribe, one pinning the subscribe-before-replay order. 787 passing.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
… failures

Three bugs from an adversarial review of T5. All three were confirmed by driving the
real code, and all three are multi-tenant-only — the default `single` path is unaffected.

**1. Restart-orphaned groups were never settled (`_settle`).** It called
`maybe_complete(gid)` with no `userkey`, so it looked in the SHARED store for a group
that lives in a tenant's store, found nothing, and returned False. Every group a restart
left unfinished stayed unfinished forever, and the failure mode is silence — which §21.6
calls the worst one. The mirror now resolves the owner per group from the ownership index.

**2. Replayed member events landed in the shared store.** `on_member_event` reads
`ce_userkey` off the event, and any response published *before* Phase 3 has none — so
member rows went to `shared/` while the group row sat in the tenant's store, leaving the
group's counts permanently wrong and `_decorate_member` reading from a third place.
Pre-existing Kafka history is exactly what this mirror exists to replay, so that is the
normal upgrade path, not an edge case.

The mirror now back-fills a missing `userkey` from the index. It only ever FILLS: an event
carrying its own `userkey` keeps it, because that attribute is signed (§2.6) and the index
is not authoritative over a signed assertion.

Both of these had one root cause — the mirror knew about per-user topics but not about
which tenant a replayed group belonged to.

**3. A broken index surfaced as "correlation ID space exhausted".** `mint()` caught
`Exception` around `claim` to treat a lost race as a retry, so a genuinely broken index
(disk full, database locked, schema missing) was retried 2000 times and then reported a
message that sends the reader to the word lists rather than the database. Now catches
`Collision` specifically; everything else belongs to the caller.

Also closes a test-strength gap the review found: deleting the `_has_subscribers` pinning
check entirely still passed every `test_store_registry.py` test. The existing tests
asserted object identity and counters, not the notification path that is the actual reason
for the rule — now that eviction only drops a reference, a held store keeps working, so
what actually breaks is that the *writer* gets a new Store whose subscriber map is empty
and the viewer is never notified. The new test asserts that end to end; removing the
pinning check now fails 4 tests instead of 0.

Tests: 8 new (795 total), ruff clean.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…ve you

The status lines said steps 5-12 were design only; T5 (per-user stores and the global
ownership index) is now in the tree, so they were wrong in the direction that matters —
a reader would either re-implement it or, worse, read "per-user stores" as "isolation".

The distinction the docs now draw explicitly: each tenant's sessions, responses and
transcripts DO live in their own SQLite files, and a read routes to the owning tenant's
store — but nothing yet checks that the *caller* is that owner, because owner-scoped
reads (T7) and transcript auth (T8) are not implemented. So `multi` mode's reads,
`/continue` and `PUT /transcript` remain as open as Phase 2's.

Also adds `EB_MAX_OPEN_STORES` to the component README's Phase 3 table with the
descriptor arithmetic that explains the default, and to `test_manifests.py`'s
classification list so T22's "every Phase 3 variable is on one side of the
Secret-vs-ConfigMap rule" check keeps covering it.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
@mrsabath mrsabath changed the title feat(eventing): Phase 3 steps 1-4 — tenancy keys, per-user topic plumbing, declarative agents feat(eventing): Phase 3 steps 1-4 + T5 — tenancy keys, per-user stores, declarative agents Oct 2, 2026

@aslom aslom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Careful, well-reasoned work — 37 files, +4618/−153, implementing §9 steps 1–4 plus T5,
T13 and T22, with four fix(...) commits for defects found during the work and docstrings
that argue for their decisions rather than restating them. The two modules the design is
most specific about are faithful to it: userkey() puts the issuer inside the hashed bytes
with a \x1f separator for the right reason, and owner_index.py declares
correlationid TEXT PRIMARY KEY with a comment on why it must not be
(userkey, correlationid) — the #881 correction landing exactly as specified.

Requesting changes for five things, two of which I would not want merged in any form
and one of which affects the default configuration.

The single-mode regression is the serious one. __main__.py deletes the
Minter seeding loop and nothing backfills its replacement — OwnerIndex has no
seed-from-store path. On the first start after an upgrade the index is empty while
sessions.sqlite is full, so exists() reports ids as free that are in use. The space is
50 × 50 × 10,000 = 25M, so with ~1,000 existing correlations a mint has ~1-in-25,000 odds
of reissuing a live id, and the result is upsert_session overwriting a session and the
new prompt appending to somebody's old conversation. Phase 2's odds were zero. This
contradicts the headline guarantee, and no test covers the state that causes it because
the fixtures build index and store together.

Then two that make multi incorrect rather than merely incomplete. create_group
passes userkey to submit_members but not to groups.create(), so the group row lands
in the shared store while its members land in the tenant's — the batch reports 0 members
forever and never completes, and a retried POST returns members: []. And
close_group/cancel_group perform no authentication at all (_caller is called in only
three places in that file, neither of them these), then resolve any tenant's group from the
global index and mutate it. T7 covers reads staying open; it does not cover unauthenticated
cross-tenant mutation, which is new here.

Two more. agent is absent from SIGNED_ATTRS while ce_agent selects the AgentSpec
that supplies --permission-mode and the tool allowlists — so a forged attribute widens
the sandbox and the signature still verifies, which makes the signed configuration worse
than the unsigned one. And userkey reaches a filesystem path join unvalidated, from an
inbound Kafka header: .. escapes users/ into another tenant's store or out of the tree
entirely, verified by construction. tenancy.py is the only place that knows the key's
shape and exports no validator, which is the root of both.

Everything above was reproduced against 16158e50 rather than inferred. Six suggestions
and four nits follow, and two documentation comments.

On design conformance, which was asked for: the code is not fully reflected in
DESIGN_PHASE3.md.
Six gaps, two of them contradictions — §6.1 still places
single-tenant stores in shared/ where the code uses the bridge root, and still asserts
"Closing is safe" about an LRU that commit 49aed05 proved unsafe. owner_index.py
appears in the document exactly once, in the status header this PR adds. Inline on that
header.

I have also set out what IMPLEMENTATION_REPORT3.md and README_PHASE3.md should contain,
and why the two Phase 2 documents are still owed — DESIGN_PHASE2.md §6 is a
test-results-and-findings report filed inside a design document, and its §7 "Running it?"
points at README_PHASE1.md for a phase that added the device flow, the approved-user
list, the break-glass path, a 300 s revocation window and the keyset rollout.

Author: mrsabath (MEMBER — maintainer, normal review posture)
Areas reviewed: Python (24 files), tests (12), Markdown (3). No YAML, Helm/K8s, Dockerfile, shell or CI in the diff.
Agent/IDE config (.claude/.vscode): none — §3.5a gate run for both +++ b/ and rename to forms, no match.
Secrets scan: clean. No CI workflow, dependency manifest, pyproject.toml, uv.lock or Dockerfile touched.
Commits: 8, every one Signed-off-by, DCO passing, all conventional prefixes.
CI status: all 11 checks passing — which is worth noting against the findings above: the suite cannot see the two multi defects, because every multi-mode group test calls GroupService.create directly with the userkey the handler fails to supply, and test_continue_tenancy.py mocks the producer so no topic string is ever asserted.

Checked and clear

  • "No assertion was weakened" — true. test_auth.py and test_groups.py have one identical hunk each, widening a fake producer's publish_group_event signature and recording the new argument. Nothing removed or relaxed; test_manifests.py is purely additive.
  • Hidden skips — none. The only skip is test_agentspec.py's disclosed skipif(shutil.which("claude") is None). Worth knowing that this is the test pinning §5.5's CLI flags, so it does not run in CI.
  • Test strength generally — test_tenancy.py's collision corpus, test_owner_index.py's test_the_constraint_is_on_correlationid_alone and its concurrency test, test_store_registry.py's pinning and eviction regressions, and test_agentspec.py's literal Phase-2 argv transcription are all real and would fail on a regression. The two gaps are the ones noted inline.
  • §8.3's coordination warning is narrower than written, in your favour. canonical() omits absent attributes and sorts by name, and nothing sets depth yet, so for every event a default deployment signs today the canonical form is byte-identical across this change. The two services do not have to move together unless multi is on.
  • The GET /v0/groups question in the PR body is already answered by §6.2, which has that row — and also commits to filtering the list by owner, which this PR does not yet do.
  • The test arithmetic needs reconciling: the body says "656 baseline + 139 new", but this PR adds 192 def test_ functions and removes none, and parametrisation only raises the collected count.

return self._root
if not userkey:
return self._root / SHARED
return self._root / "users" / userkey

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[must-fix] userkey reaches this path join with no validation, and in multi mode it
arrives from an inbound Kafka header (for_event → event.get(ce.EXT_USERKEY), line 141).
A userkey containing .. escapes users/:

'gh-alice-a1b2c3d4'            -> /data/eventbridge/users/gh-alice-a1b2c3d4
'../users/gh-victim-0bad0bad'  -> /data/eventbridge/users/gh-victim-0bad0bad   # another tenant
'../../../../tmp/pwned'        -> /tmp/pwned                                   # outside the tree
'..'                           -> /data/eventbridge                            # the bridge root

Store.__init__ does base.mkdir(parents=True, exist_ok=True), so the directory is
created, not rejected. A forged ce_userkey therefore files events directly into a
victim's responses.sqlite, or drops SQLite files anywhere the process can write.

Two reasons this is reachable today rather than theoretical:

  • userkey is in SIGNED_ATTRS, but EB_REQUIRE_RESPONSE_SIGNATURE defaults to
    false, and in audit mode the event is stored unchanged. Verification is what would
    catch the forgery and it is off by default.
  • §3.6 already concedes that in Tier A anything with network access to the shared broker
    can write to any topic. That is an accepted property for reading events; it must not
    also be a filesystem write primitive.

Note the invalid key is truthy, so it bypasses the unattributed branch above and
goes straight to the join.

The fix belongs in tenancy.py, which is "the ONLY place an identity becomes a name" and
currently exports no validator — so no consumer can check the shape the producer
guarantees:

# shared/tenancy.py
USERKEY_RE = re.compile(r"^[a-z]{2}-[a-z0-9][a-z0-9-]{0,23}-[0-9a-f]{8}$")

def is_valid_userkey(value: str | None) -> bool:
    return bool(value) and bool(USERKEY_RE.fullmatch(value))

Then treat an invalid key exactly like a missing one — shared/ plus unattributed —
so a forged value is counted and visible rather than acted on. Worth a round-trip test
asserting is_valid_userkey(userkey(iss, uid)) over the same adversarial corpus
test_tenancy.py already builds.

For what it is worth, agentspec.py gets this right 150 lines away: _resolve_under
refuses a path that "escapes the agent directory" and reasons explicitly about
../../../etc/shadow, and validate_name enforces a DNS-1123 label. The same gate is
missing on the one input that is genuinely attacker-supplied.

Comment thread eventing/eventbridge/owner_index.py Outdated
Comment on lines +107 to +110
self._c.execute(
"INSERT INTO correlation_owner(correlationid,userkey,created_utc) "
"VALUES (?,?,strftime('%Y-%m-%dT%H:%M:%SZ','now'))",
(correlationid, userkey))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The uniqueness guarantee is enforced by the PRIMARY KEY, but the
error type callers depend on comes only from the in-process lock. If two writers ever
race past it, the bare INSERT raises sqlite3.IntegrityError, not Collision — and
Minter.mint catches Collision specifically, with a comment explaining (correctly) why
it must not catch Exception. So that path would surface a 500 rather than retrying.

Today the lock holds: one process, RLock, single-replica. It is load-bearing in a way
the docstring attributes to the constraint, though, and a second writer is exactly what
§6.5's deletion path and any future sweeper add. Letting the database be the authority
removes the dependency:

cur = self._c.execute(
    "INSERT INTO correlation_owner(correlationid,userkey,created_utc) "
    "VALUES (?,?,strftime('%Y-%m-%dT%H:%M:%SZ','now')) "
    "ON CONFLICT(correlationid) DO NOTHING", (correlationid, userkey))
if cur.rowcount == 0:
    row = self._c.execute("SELECT userkey FROM correlation_owner WHERE correlationid=?",
                          (correlationid,)).fetchone()
    if row[0] != userkey:
        raise Collision(...)

That keeps the idempotent-for-same-owner behaviour and makes Collision the only way a
conflict can present, lock or no lock.

Comment thread eventing/eventbridge/owner_index.py Outdated
if userkey is None:
rows = self._c.execute(
"SELECT correlationid FROM correlation_owner WHERE userkey IS NULL "
"ORDER BY created_utc DESC LIMIT ?", (limit,)).fetchall()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] created_utc is written at second granularity
(strftime('%Y-%m-%dT%H:%M:%SZ','now')), so ORDER BY created_utc DESC LIMIT ? is not a
deterministic "newest first" — a batch submitted inside one second has arbitrary order
among its rows, and with LIMIT smaller than that batch it returns an arbitrary subset.
GroupService.submit_members publishes a whole batch in a loop, so a 100-member group is
precisely the case that lands in one or two seconds.

Everything else on the event path uses millisecond precision (ce.now_iso() sets
timespec="milliseconds"), so this is also an inconsistency rather than only an
imprecision. Either match that, or add rowid as a tiebreaker (ORDER BY created_utc DESC, rowid DESC), which is free and total.

Comment thread eventing/eventbridge/owner_index.py Outdated
not the first.
"""
with self._lock:
self._c.execute("DELETE FROM correlation_owner WHERE correlationid=?",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The docstring contradicts the code: "Note what this does NOT do: freeing the id
for reuse is deliberate" — but the DELETE does free the id, since exists() is the
Minter's only check and it will no longer see it.

Worth settling before T21 builds on this, because reuse has a consequence beyond tidiness:
sessionuuid = uuid5(NAMESPACE, correlationid) is unsalted by design (§2.6), so a reused
correlationid derives the same session uuid as the deleted one. A claude
transcript left on a runner's volume, or a checkpoint that outlived the delete, is then
resumable by the new correlation — a different user's conversation continuing into
somebody else's agent. Either keep a tombstone so ids are never reissued, or say
explicitly that deletion must also purge every transcript keyed on that uuid.

if "@" in s:
local, _, domain = s.rpartition("@")
return f"{local}@{domain.lower()}"
return s

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The docstring says static identities are "byte-exact: an operator typed them",
but the @ branch above this one catches them first — a static EB_AUTH_TOKENS name of
Ops@Example.COM canonicalises to Ops@example.com, not byte-exact. Harmless in
practice (and arguably the nicer behaviour), but the third bullet claims something the
code does not do, and this function's whole contract is that each branch is a deliberate
per-issuer claim. Either reorder on issuer == "static" or say the email rule applies to
any address-shaped identifier regardless of issuer.

and returned False — leaving every restart-orphaned group unfinished forever, with
silence as the symptom (§21.6's worst failure mode)."""
svc, owners, stores, _ = _svc(tmp_path)
gid, _ = svc.create(label="b", expected=1, userkey=UK)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This is the suite's multi-mode blind spot, and it is why the
create_group bug above survived 192 new test functions.

Every multi-mode group test calls GroupService.create(label=..., userkey=UK) directly
(also 113, 130), supplying by hand the one parameter Handlers.create_group fails to pass.
So the suite proves the tenanted path works when userkey is given, which is not the
property that matters — nothing drives Handlers.create_group in multi at all.

test_continue_tenancy.py:51 has the matching gap from the other side: producer = MagicMock(), and every "publishes to the owning tenant's topic" assertion checks
publish_request.call_args[1]["userkey"] — never a topic name. The file's stated hazard is
that "publishing to a shared topic would run the turn on another tenant's runner", and no
assertion anywhere covers it; test_topicset_wiring.py checks TopicSet and Producer
separately but never the handler → producer → topic chain.

One test per route that goes through Handlers with a real Producer against a fake Kafka
and asserts the topic string would have caught both the create_group defect and the
continue_agent 500. That is a better investment than more unit tests on the pieces, which
are already well covered.

# costs one GitHub call rather than one each.
self.logins = ghauth.LoginCache(cfg.github_cache_ttl_s)

def _agent_for(self, caller, body) -> str | None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] str(named).strip() accepts any JSON value — a dict, a list, "../../etc" —
and ships it as ce_agent. agentspec.NAME_RE and validate_name already exist and
enforce a DNS-1123 label; the HTTP boundary is where that check belongs.

As it stands a bad agent name is answered 202 Accepted and surfaces minutes later as an
asynchronous phase=error event, which the submitter may never look at, instead of a
400 on the call that made the mistake.

Comment thread eventing/eventbridge/config.py Outdated
f"got {cfg.tenancy_mode!r}. See DESIGN_PHASE3.md §8.1.")
cfg.topic_prefix = e("EB_TOPIC_PREFIX", cfg.topic_prefix)
cfg.user_registry_path = e("EB_USER_REGISTRY_PATH", cfg.user_registry_path)
cfg.max_open_stores = int(e("EB_MAX_OPEN_STORES", str(cfg.max_open_stores)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Unguarded int(), so EB_MAX_OPEN_STORES=lots is a bare ValueError traceback
at startup while EB_TENANCY_MODE seven lines above raises a readable SystemExit. The
whole startup path is otherwise careful to print messages rather than tracebacks (the PR
body says so explicitly), so this is the odd one out.

Line 196 is the related gap: EB_TOPIC_PREFIX is unvalidated, so a prefix containing _,
. or a space produces exactly the illegal or JMX-colliding topic names that
tenancy.py's _slug docstring warns about — a topic a.b_c and a topic a_b.c collide
in Kafka's metric names and one silently overwrites the other's. The prefix is the one
component of those names _slug never sees.

Comment thread eventing/eventrunner/runner.py Outdated
session = event["sessionuuid"]
mode = event.get("mode", "start")
prompt = payload.get("prompt", "")
max_turns = payload.get("max_turns") or spec.max_turns

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] payload.get("max_turns") or spec.max_turns replaced
payload.get("max_turns", 3), so a request with max_turns: 0 now silently gets the
spec's value instead of 0. The spec side rejects zero
(test_a_nonsensical_max_turns_is_refused), so the two paths disagree about the same
nonsensical input — one refuses it, the other rewrites it. is not None makes them agree,
or validate it at the HTTP boundary alongside the agent name above.

Comment thread eventing/tests/test_tenancy.py Outdated
"""§4.1 is the whole design: the name travels to a phone."""
t = T.ntfy_topic("kev1", T.userkey("github", "mrsabath"), SECRET)
assert "mrsabath" not in t
assert "gh" not in t.removeprefix("kev1-")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This assertion is flaky by construction. t.removeprefix("kev1-") is 26
characters of lower-cased base32, whose alphabet is a-z2-7 — so it contains g and h,
and the chance that "gh" appears somewhere in 26 characters is roughly
25 × (1/32)² ≈ 2.4%. It passes deterministically for this fixture's secret and
userkey, and will fail for about one in forty other combinations — including whatever
secret a deployment actually uses, if this ever runs against one.

It also adds nothing: assert "mrsabath" not in t on the line above is the property §4.1
cares about. The issuer prefix is two characters, so testing for its absence in a
130-bit digest is testing that a random string does not happen to contain a common bigram.
Dropping the line is the fix.

Every finding reproduced against 16158e5 before fixing, and every fix has a regression
test in `tests/test_review_fixes.py` that fails without it (verified by mutation, one
fix at a time). They are filed together because they share a cause worth naming: all
five were invisible to 192 test functions, because the fixtures assembled the
collaborators by hand with the arguments the production call sites fail to supply.

**1. `single`-mode regression: the ownership index was never seeded.** The serious one,
and against the exact guarantee this phase claims. Dropping Phase 2's `Minter` seeding
loop left nothing to populate its replacement, so on the first start after an upgrade
`owners.sqlite` is empty while `sessions.sqlite` is full and `exists()` reports live ids
as free. With ~1,000 existing correlations a mint has roughly 1-in-25,000 odds of
reissuing one, and then `upsert_session` overwrites a session and the new prompt appends
to somebody's existing conversation. Phase 2's odds were zero.

`OwnerIndex.seed_from()` plus a one-time backfill from every store at startup.
§2.6 is right that per-start seeding is the wrong trade; it accidentally argued away the
one-time backfill too, and §6.1a now records the distinction.

**2. `create_group` did not pass the userkey.** `submit_members` got it and
`groups.create()` did not, so in `multi` the group row and the groupid's owner claim
landed in the shared store while every member row landed in the tenant's: 0 members
forever, `maybe_complete` never fires, the batch never completes and never notifies. A
retried POST with an Idempotency-Key read the tenant store `create` never wrote and
returned `members: []` — worse than a second batch, because it looks like success.

**3. `close_group`/`cancel_group` were unauthenticated cross-tenant mutations.** Neither
called `_caller`; both resolved any tenant's group from the global index and acted on it.
T7 covers *reads* staying as open as Phase 2's — Phase 2 had no tenants to cross and no
cross-tenant mutation anywhere, so this was a capability being introduced, not an openness
preserved. Now gated by `_authorize_mutation`, answering `404` for another tenant's group
per §6.2's enumeration rule. Single-tenant mode is untouched.

**4. `agent` was outside `SIGNED_ATTRS`** while `ce_agent` selects the AgentSpec that
supplies `--permission-mode` and the tool allowlists — §5.1's "sandbox". A forged value
widened the sandbox and the signature still verified, making a signed deployment worse
than an unsigned one because an operator believes it is attested. Added in the same
change as `userkey`/`depth`, per §8.3's one-canonicalisation-break rule.

**5. `userkey` reached a filesystem path join unvalidated**, from an inbound Kafka
header, and `Store.__init__` calls `mkdir(parents=True)` — so `..` created and wrote to a
directory inside another tenant's store, or outside the tree. `tenancy.is_valid_userkey()`
is the gate, exported from the module that is "the ONLY place an identity becomes a name"
so a consumer can check the shape the producer guarantees. An invalid key is treated as a
missing one — `shared/` plus `unattributed` — so a forgery is counted and visible rather
than acted on. Note it is *truthy*, so the pre-existing `not userkey` check let it past.

Also from the review:

- **A known-but-unowned correlation 500'd on `/continue`.** `owner=None,
  unresolved=None` is falsy, so the handler proceeded, wrote a prompt row, then
  `TopicSet.requests(None)` raised — a traceback with an orphan row already committed, so
  the conversation showed a turn that was never submitted. Reading and publishing have
  different requirements, so `_publishable_owner` is now separate from `_owner_userkey`:
  shared-tier data stays readable, and a turn for it is refused before anything is written.
- `claim()` uses `ON CONFLICT DO NOTHING` + a follow-up read, so `Collision` is the only
  way a conflict presents whether or not two writers race the in-process lock — the bare
  `INSERT` would have raised `IntegrityError`, which `Minter.mint` deliberately does not
  catch.
- `created_utc` at millisecond precision (matching `ce.now_iso()`) with a `rowid`
  tiebreaker; at second granularity a 100-member batch had arbitrary order among its rows.
- A bad `agent` name is a `400` on the call that made the mistake, not a `202` followed by
  an async `phase=error` the submitter may never read.
- `max_turns: 0` is no longer silently rewritten to the spec's value while the spec path
  rejects zero.
- `EB_TOPIC_PREFIX` validated (it is the one topic-name component `_slug` never sees, and
  `.`/`_` there reproduce the JMX metric collision) and `EB_MAX_OPEN_STORES` refuses
  readably instead of raising a bare `ValueError`.
- Dropped a flaky assertion: `"gh" not in` 26 characters of base32 fails ~2.4% of the
  time by construction, and the `"mrsabath" not in` line above it is the real property.
- `_canonicalise`'s docstring no longer claims static identities are byte-exact when the
  `@` branch catches address-shaped ones first.

Design doc, where the code had disproved it (§9's "the code is the authority" rule):
§6.1's layout corrected to put single-tenant stores in the bridge root; "Closing is safe"
replaced with what commit 49aed05 established, including that the first fix attempt only
pinned the SSE path and left the class open; new §6.1a documents `owner_index.py`; §2.6's
"one new extension attribute" corrected to four, with which three are signed and why; §3.1
records the single-mode `events`/`dead` names. The status header now warns that `multi`
does not work end to end until T6 — no response is consumed, which is more useful than
"not yet isolated".

Tests: 45 new (840 total), ruff clean.

Still outstanding from the review and deliberately not in this commit: the T22 subset
check, `RequestsMirror`/`NtfyPublisher` `stores=`, the handler-to-topic integration test,
the `forget()` tombstone decision, and the four documents.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
@mrsabath

mrsabath commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — this was a genuinely useful review. All five must-fixes are in 6ebdd80, and I
reproduced each one against 16158e50 before touching it rather than taking the report on
trust. Every fix has a regression test in the new tests/test_review_fixes.py, and I
verified each test is load-bearing by reverting its fix one at a time.

The single-mode regression was the one that mattered, and your reasoning about it was
exactly right. I confirmed it directly: a store with 50 pre-existing sessions plus a fresh
index reports 50 of 50 live ids as free. OwnerIndex.seed_from() plus a one-time
backfill from every store at startup closes it.

Worth recording why I walked into it, because the design contributed: §2.6 argues
convincingly that one SELECT per mint beats re-seeding a seen set on every start — and
in arguing that, it reads as though the index needs no seeding at all. Those are different
claims. §2.6 now distinguishes per-start seeding (correctly rejected) from one-time
backfill
(required for correctness), and new §6.1a documents the index properly.

Finding Fix
Index never seeded (single-mode regression) seed_from() + startup backfill from every store
create_group dropped userkey passes it; group row and members now colocate
close/cancel unauthenticated cross-tenant mutation _authorize_mutation, 404 per §6.2
agent outside SIGNED_ATTRS added alongside userkey/depth, one canonicalisation break
userkey → path join unvalidated tenancy.is_valid_userkey(); invalid ⇒ shared/ + unattributed

On the path traversal — I took your suggestion as written, including treating an invalid
key exactly like a missing one so the forgery is counted and visible rather than
rejected. Your note that an invalid key is truthy and therefore bypassed the existing
not userkey branch was the part I would have missed; the validator lives in tenancy.py
for the reason you gave, and there is a round-trip test asserting
is_valid_userkey(userkey(iss, uid)) over the adversarial corpus.

On close/cancel: I agree "T7 is not in yet" does not cover it, and your framing is the
right one — reads staying open preserves a Phase 2 property, whereas unauthenticated
cross-tenant mutation is a capability this PR would have introduced. Also fixed the
three-routes-disagree problem you noted; all of them now keep the second return value.

Also addressed from your suggestions and nits: the known-but-unowned /continue 500 (I
reproduced the orphan prompt row — reading and publishing turn out to need different
answers, so _publishable_owner is now separate from _owner_userkey, keeping shared-tier
data readable); ON CONFLICT DO NOTHING so Collision is the only way a conflict
presents; millisecond created_utc with a rowid tiebreaker; agent name validated at
the HTTP boundary; max_turns: 0; EB_TOPIC_PREFIX and EB_MAX_OPEN_STORES validation;
the _canonicalise static/@ docstring.

You were right about the flaky test, including the arithmetic. "gh" not in 26
base32 characters fails about 2.4% of the time by construction. Dropped — the
"mrsabath" not in line above it is the property §4.1 actually cares about.

And right about my test count. It is +217 unique test functions (525 → 742; 845
collected with parametrisation), not the 139 I claimed. I had been diffing collected
counts across a changing baseline. Corrected in the PR body.

On design conformance

Fixed in-tree, under §9's "the code is the authority where implemented" rule:

  • §6.1's layout — single-tenant stores are in the bridge root, not shared/, with the
    upgrade-safety reasoning that motivated it. I also fixed the same drift inside
    store_registry.py, whose module docstring reproduced the design's version while
    _dir_for did the opposite 40 lines below.
  • "Closing is safe" — replaced with what 49aed05 established, including the detail
    you singled out: the first attempt pinned only the SSE path, which fixed the visible
    case and left the class open. The bug was in the eviction contract, not one caller.
  • §6.1a — new, documenting owner_index.py: the correlationid-alone PRIMARY KEY,
    nullable userkey, Collision vs idempotent re-claim, owner_of's two-value return,
    the seeding requirement, and mint_for's honest degradation.
  • §2.6 — "one new extension attribute" → four, with which three are signed and why.
  • §3.1 — records the single-mode events/dead names.
  • Status header — now says multi does not work end to end until T6, because no
    response is consumed. You are right that this is a more useful warning than "not yet
    isolated", which implies a working-but-unisolated mode.

Two corrections accepted: §6.2 does have the GET /v0/groups row, so requiring an
authenticated caller is the design's own position and needs no dispensation — and it also
commits to filtering the list by owner, which I have noted for T7. And §8.3's coordination
warning being narrower than written is a useful thing to know.

Deliberately not in this commit

Your remaining suggestions, which I would rather do as a reviewable second pass than bury
in this one: the T22 subset-direction check (you are right that it cannot currently fail
for the reason it exists), RequestsMirror/NtfyPublisher missing stores= — which do
look like the sweep in 77807b6 not reaching them — the handler→topic integration test,
and the forget() tombstone decision. That last one I want to think about rather than
answer quickly, since the unsalted sessionuuid makes id reuse a cross-user transcript
resume, which is a worse consequence than the docstring implies.

The four documents I have not started. Your argument that the findings are freshest now is
the right one and I would rather write IMPLEMENTATION_REPORT3.md while the eviction and
except Exception findings are still attributable to specific commits, so I will take it
next unless you would rather see the code suggestions land first.

All 11 checks passing on 6ebdd80.

…etion

Round two of the review, covering what was deferred from 6ebdd80.

**T22 now checks the direction that enforces the rule.** `CONFIGMAP_VARS ⊆ found`
catches a stale entry and nothing else; the direction that catches a *new* variable
classified as neither is `found ⊆ CONFIGMAP_VARS ∪ SECRET_PATH_VARS`, which is the case
the test exists for. Scoped to Phase 3's variables — the 36 that pre-date §8.3 are listed
explicitly, with `test_the_baseline_list_is_still_accurate` keeping that list honest
rather than letting it silently widen the exemption.

Verified by deliberate failure, as asked: injecting an unclassified `e("EB_UNCLASSIFIED_X")`
read into `config.py` turns the test red with the variable named. Also replaced the
hardcoded ten-space indentation in the inlined-secret check with `\s*`, matching the
precedent elsewhere in the same file — the old pattern would miss the same variable
inlined in an initContainer, a sidecar or a deeper patch.

**`RequestsMirror` and `NtfyPublisher` now take `stores=`.** Both knew the per-tenant
topic names and still wrote to the shared store, which does look like the sweep in
77807b6 not reaching them. The mirror back-filled every prompt into `shared/`, so a
correlation whose request the bridge did not originate showed a BLANK PROMPT on its
owner's page — the exact gap the mirror exists to close. ntfy's `_last_prompt` and
`_last_assistant_text` read only the shared store, so a tenant's notification arrived
without the prompt or the reply, which §4.1 says is the content that lets a phone that
cannot reach EventBridge still see the result. Its group-existence check is routed too.

**A handler→topic integration test**, which the review argued was a better investment
than more unit tests on the pieces. It is: faking at `KafkaProducer` rather than at
`Producer` means the real handler, the real `Producer` and the real `TopicSet` all run,
and the assertion is *which topic the bytes went to*. Confirmed by reverting each fix that
it catches both defects it was written for — the `create_group` missing `userkey` and the
`/continue` 500 on an unowned correlation.

That test immediately earned itself twice over: the mutation run surfaced that
`_publish_started`/`maybe_complete` swallow `ValueError` from `TopicSet` into a log line,
so a caller forgetting a `userkey` produced a group that silently never announced itself —
the same shape as the `TypeError` already guarded there. Both guards now cover both.

**Deletion tombstones rather than freeing the id.** This is the one I said needed thought
rather than a quick answer, and the answer is that reuse is unsafe. Verified: `forget()`
used to `DELETE` the row, `exists()` is the `Minter`'s only uniqueness check, and
`sessionuuid = uuid5(NAMESPACE, correlationid)` is unsalted by design — so a reissued
`correlationid` derives *the same session uuid*, and a `claude` transcript left on a
runner's volume becomes resumable by the new correlation. That is one user's conversation
continuing inside somebody else's agent.

So `forget()`/`forget_tenant()` keep the row, set `deleted_utc` and clear `userkey` — the
id is reserved forever, and the tombstone discloses nothing about whose it was. The
alternative is guaranteeing every transcript keyed on that uuid is purged everywhere,
including volumes this process does not own, which is not a guarantee EventBridge can
make. Cost is one short row per deleted correlation. Migrated with an `ALTER TABLE`, the
same idiom `store.py` uses for `prompts.submitter`, and a test covers an index created
before the column existed.

The reviewer was also right that the docstring contradicted the code: it claimed "freeing
the id for reuse is deliberate" about a `DELETE` nobody had reasoned about.

Design doc: §6.1a records the tombstone decision and why, since the index is what
enforces it.

Tests: 855 passing (+13), ruff clean.

Still outstanding: the four phase documents (`IMPLEMENTATION_REPORT3.md`,
`README_PHASE3.md`, and the two owed Phase 2 documents).

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
@mrsabath

mrsabath commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Round two is in 3278d75 — the four remaining code items. All 11 checks passing, 855
tests.

T22 now checks the direction that enforces the rule

You were right that it could not fail for its stated reason. CONFIGMAP_VARS ⊆ found
catches a stale entry and nothing else; found ⊆ CONFIGMAP_VARS ∪ SECRET_PATH_VARS is the
direction that catches a new variable classified as neither.

Scoped to Phase 3's variables, with the 36 that pre-date §8.3 listed explicitly — and
test_the_baseline_list_is_still_accurate keeps that list from silently widening the
exemption, which was my worry about hardcoding it at all.

I took the deliberate-failure suggestion literally: injecting an unclassified
e("EB_UNCLASSIFIED_X") read into config.py turns the test red with the variable named.
The hardcoded ten-space indentation is now \s*, per the precedent you pointed at in the
same file.

RequestsMirror and NtfyPublisher

Both take stores= now. Your read looks right that these are where the 77807b6 sweep
stopped rather than a decision — Consumer and GroupMirror got it and these did not,
with no reasoning anywhere for the difference. The mirror's case was the worse of the two:
back-filling into the shared store means a correlation the bridge did not originate shows
a blank prompt on its owner's page, which is the exact gap the mirror exists to close.
ntfy's group-existence check needed the same treatment, which I would have missed if I had
only fixed the two functions you named.

The integration test was the right call, and it paid for itself twice

Faking at KafkaProducer rather than Producer means the real handler, the real
Producer and the real TopicSet all run, and the assertion is the topic string. I
confirmed it catches both defects it was written for by reverting each fix in turn.

Then it found a third thing nobody had flagged. Mutating create_group back to its buggy
form produced this in the captured output:

[groups] could not publish group.started for loud-weasel-0357:
    ValueError("userkey is required to name the 'responses' topic in multi mode")

_publish_started and maybe_complete were swallowing ValueError from TopicSet into
a log line — so a caller forgetting a userkey produced a group that silently never
announced itself. Exactly the shape of the TypeError I had already guarded there after
the first round, and I had not thought to widen it. Both guards now cover both, and a
genuine broker failure is still handled.

Your framing — "one test per route that goes through Handlers with a real Producer
against a fake Kafka, asserting the topic string, is a better investment than more unit
tests on the pieces" — is the most useful single suggestion in the review.

forget(): tombstone, never free the id

This is the one I said I wanted to think about rather than answer quickly, and the thought
changed the answer. I verified the mechanism instead of reasoning about it:

after forget, exists(): False
sessionuuid before     : 9189c66c-9eff-5cb5-92b2-0565ca420c33
sessionuuid if reissued : 9189c66c-9eff-5cb5-92b2-0565ca420c33
SAME session uuid: True

So your reading was right and the consequence is worse than the docstring implied.
exists() is the Minter's only uniqueness check, sessionuuid is unsalted by design,
and therefore a reissued correlationid derives the same session uuid — a claude
transcript left on a runner's volume, or a checkpoint that outlived the delete, becomes
resumable by the new correlation. One user's conversation continuing inside somebody else's
agent.

forget() and forget_tenant() now keep the row, set deleted_utc and clear userkey:
the id is reserved forever and the tombstone discloses nothing about whose it was, which
matters on a deletion path. I chose that over your other option — purging every transcript
keyed on that uuid — because that is a guarantee EventBridge cannot make: the volumes are
not all its own. Cost is one short row per deleted correlation. Migrated with ALTER TABLE, the same idiom store.py uses for prompts.submitter, with a test covering an
index created before the column existed.

And you were right about the docstring: it asserted "freeing the id for reuse is
deliberate" about a DELETE nobody had reasoned about. §6.1a now records the decision and
why, since the index is what enforces it.


Next: IMPLEMENTATION_REPORT3.md

Taking this next, and your argument for doing it now rather than later is the one I found
most persuasive in the review — the findings are attributable to specific commits today
and will not be in a month. Following Phase 1's report as the model: measured numbers,
failures with root causes, and an explicit list of what is not verified.

What I plan to put in it, so you can redirect me before I write rather than after:

  • The findings, each with its commit and the symptom that hid it. The use-after-close
    eviction bug (49aed05) with the detail you singled out — first attempt pinned only the
    SSE path, fixed the visible case, left the class open, and the bug was in the eviction
    contract rather than one caller. The except Exception pair: a signature mismatch that
    silently stopped every group event from publishing, and a broken ownership index
    surfacing as "correlation ID space exhausted". Plus this round's ValueError variant,
    which is the same lesson a third time.
  • Your five must-fixes as findings in their own right, including the one that matters
    most for a report: a single-mode regression shipped behind a headline claiming the
    opposite, and the suite could not see it because the fixtures supplied by hand the
    arguments the production call sites omitted.
  • The deltas from the design as decisions, not drift — single-tenant store location
    chosen for upgrade safety, eviction by reference-drop, tombstoned deletion, the
    single-mode events/dead names.
  • What is not verified, which for this phase is a long and more useful list than the
    verified one: no cluster run, no two-tenant deployment, multi not functioning end to
    end until T6, and the per-tenant throughput ceiling §10 asks for still unmeasured.

On the two owed Phase 2 documents — I agree they are owed, and your diagnosis of why the
gap shows is hard to argue with: §6 of DESIGN_PHASE2.md is a test-results-and-findings
report filed inside a design document, and §7 sends people to README_PHASE1.md for a
phase that added the device flow, the approved-user list, the break-glass path, a 300 s
revocation window and the keyset rollout. I would rather do those as a separate PR than
grow this one further, unless you would prefer them here.

Asked for in the #883 review, and the argument for writing it now rather than later was
the most persuasive one there: the findings are attributable to specific commits today and
will not be in a month. 361 lines, following `IMPLEMENTATION_REPORT1.md`'s contract —
measured figures rather than estimates, failures with root causes, and an explicit account
of what is NOT verified.

Phase 0 and Phase 1 each ship three documents; Phase 2 shipped none, and the gap shows in
that there is nowhere to put the §6.1 split-consumer trap except inside a design document.
This starts closing that for Phase 3.

**Measured rather than asserted.** §6.1 estimates the per-tenant descriptor cost and
derives a ceiling; measured it is 6.0 descriptors per open store (10 tenants, 60 fds),
which puts ~680 tenants under this host's 4096 soft limit and ~170 under the 1024 the
design assumed — so squarely inside its "150-200" once the limit is held constant. The
ownership index costs 27.4 µs per `claim`, 1.7 µs per `exists`, and 270 ms to seed 10,000
ids. That last figure is the one that matters: it settles §2.6's objection to seeding,
because the *one-time* backfill it appeared to rule out costs 270 ms once, while the
*per-start* seeding it correctly rejects is what the index exists to avoid.

**Seven deltas from the design, recorded as decisions rather than drift** — the
single-tenant store location, eviction by reference-drop, tombstoned deletion, the
single-mode `events`/`dead` names, `agent` joining `SIGNED_ATTRS`, the exported userkey
validator, and prefix validation.

**The findings, ordered by what they cost to find.** Three are worth the document on their
own:

- The `single`-mode regression shipped behind a headline claiming it could not happen,
  with the reproduction (50 of 50 live ids reported free), the 1-in-25,000 arithmetic over
  a verified 50x50x10,000 id space, and the three reasons it is instructive: it was in the
  default configuration, the suite could not see it because no fixture simulated an
  upgrade, and §2.6's wording contributed.
- The eviction contract, where the lesson is the *first* fix: pinning only the SSE path
  addressed the case that was easy to picture and left the class open.
- `except Exception` converting three separate programming errors into log lines, on three
  separate occasions, which is why it is filed as one finding with the pattern extracted.

Also records the suite's structural blind spot as a finding in its own right — a fake
placed at the boundary under test cannot test that boundary — since that is what let two
defects survive 200-odd new test functions.

**§5 is the longest section, deliberately.** `multi` does not work end to end until T6 (no
response is consumed); no cluster, no two-tenant deployment, no Kafka; reads not
owner-scoped; Tier A is not isolation; the per-tenant throughput ceiling §10 asks for is
still unmeasured; the `claude --help` flag assertion does not run in CI; `seed_from` is
capped at 10,000 per store; and `is_valid_userkey` is shape validation, not authorisation.

Every figure in the report was measured on this branch, every cited commit verified to be
on it, and every cited design section verified to exist — §21.6 turned out to be
`DESIGN_PHASE1.md`'s and is now qualified as such. Two numbers in my own PR description
were wrong and are corrected here: the test delta is +232 unique functions (525 to 757),
not the 139 or 217 claimed earlier.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New/ToDo

Development

Successfully merging this pull request may close these issues.

3 participants