docs(security-model): describe the credential and policy caches

docs/design/05-component-auth-v3.md

@@ -12,7 +12,7 @@ The `auth` crate provides pluggable authentication and authorization for extendd
 
 The crate depends on `extenddb-core` (for types and errors) and `async_trait` (for object-safe async trait dispatch). It has no HTTP framework or storage dependencies.
 
-> **No caching.** All credential lookups, policy fetches, and identity resolution read directly from Postgres on every request. The latency is sub-millisecond for indexed lookups, and this approach eliminates stale-cache bugs and cache invalidation complexity entirely. Multiple extenddb instances sharing the same catalog see consistent data without coordination. If profiling under production load shows database round-trips are a bottleneck, caching can be added as a transparent layer without changing any interfaces.
+> **Caching.** Credential lookups and policy fetches go through in-memory caches with write-through invalidation and a TTL: the credential cache in this crate (`credential_cache.rs`) and the authorization caches in the server crate (`authz_cache.rs`). `docs/design/12-auth-authz-cache.md` describes that design and replaces the "no caching" decision this document originally recorded.
 
 ## 2. Key Design Decisions (from 2026-04-15 review session)
 

docs/design/12-auth-authz-cache.md

@@ -36,7 +36,7 @@ This document specifies an in-memory cache with stale-while-revalidate semantics
 - Eliminate the per-request catalog roundtrips for credentials, policies, boundaries, principal tags, resource tags, and TableKeyInfo on the steady-state hot path.
 - Eliminate the per-request `PolicyDocument::from_json` parse cost.
 - Refresh stale entries **without** making the user request wait (refresh-ahead, a.k.a. stale-while-revalidate).
-- Bound memory with an LRU upper limit per cache.
+- Bound memory with an upper limit on entries per cache.
 - Allow operators to disable the cache entirely as a kill switch.
 - Propagate self-induced changes (admin API mutations) instantly within a single process via write-through invalidation hooks.
 
@@ -192,15 +192,15 @@ enabled            = true
 ttl_seconds        = 60   # hard TTL — entries beyond this are full misses
 soft_ttl_seconds   = 30   # entries beyond soft_ttl trigger background refresh on access
 negative_ttl_seconds = 5  # how long to cache "not found" results
-max_entries        = 10000 # per-cache LRU size limit
+max_entries        = 10000 # maximum entries per cache
 ```
 
 Default rationale:
 
 - **60 s hard TTL** mirrors AWS IAM's "policy propagation" expectation (~30 s–2 min). Long enough to absorb the perf bottleneck (60 s of caching → 99.99%+ hit rate per hot key under any meaningful load), short enough to bound the blast radius of off-instance credential revocation.
 - **30 s soft TTL** = TTL / 2 keeps refreshes from clustering at the boundary.
 - **5 s negative TTL** is short enough that newly-issued keys work quickly, long enough to absorb a dictionary attack against `lookup_credential`.
-- **10 000 entries per cache** is a generous default for typical deployments; under memory pressure the LRU evicts cold entries.
+- **10 000 entries per cache** is a generous default for typical deployments; beyond it, moka's TinyLFU policy drops the entries it rates least likely to be reused, which can be the new one.
 - **`enabled = false`** is a kill switch for incident response.
 
 The cache is configured statically (`extenddb.toml`). Runtime tunability is not added — cache TTL changes are rare and a server restart is acceptable. Adding it to the existing `settings` poller table is an option for a follow-up if operationally needed.
@@ -416,12 +416,12 @@ Each phase is a separate PR. After each phase the test suite runs green and the
 > **Propagation timing for invalidations**:
 >
 > - **Single-key invalidations** (e.g. `DeleteAccessKey`, `PutUserPolicy`) bump an internal epoch counter and drop the moka entry immediately. The next request blocks on a fresh load.
-> - **Fanout invalidations** (e.g. `DeleteAccount`, `DeleteRole`'s session sweep, `DeleteGroup`'s member fanout) use moka's `invalidate_entries_if`, which is **asynchronous** — predicate evaluation runs on moka's internal worker after the management call returns. There is a small window (~ms) between the API responding 200/204 and the matching entries actually being evicted from the underlying store. The accompanying epoch bump ensures any in-flight refresh on those keys is dropped on completion, but a request that reads the slot during this window can still see the pre-invalidation value. Operators should treat fanout invalidation as "eventually consistent within a few ms"; for hard cutover (e.g. revoking a compromised key), prefer the single-key invalidations (`DeleteAccessKey`) over the cascade ones.
+> - **Fanout invalidations** (e.g. `DeleteAccount`, `DeleteRole`'s session sweep, `DeleteGroup`'s member fanout) take effect before the management call returns. The account and role sweeps use moka's `invalidate_entries_if`, which registers the predicate and returns; moka removes the entries from its map during later cache operations, but `get` applies registered predicates, so matching entries are not served once the call returns. The accompanying epoch bump drops any refresh in flight on that cache when it completes. `DeleteGroup` awaits the invalidation of each member. If a group's members cannot be listed, those members' entries age out at TTL.
 > - **Off-instance changes** (a separate process modifying the catalog directly, or a different instance in a multi-node deployment) take up to `auth.cache.ttl_seconds` to propagate. The cross-process invalidation channel described in Appendix B is not yet implemented.
 >
 > Self-induced single-key changes via the admin API or the web console propagate instantly on the local instance via write-through invalidation; both paths share the same `AuthCacheRegistry`.
 >
-> To force-flush the cache without restart: not currently supported. To disable: set `auth.cache.enabled = false` in `extenddb.toml` and restart.
+> To force-flush the cache without a restart, use the manual invalidation API (§6.1) on each instance. To disable: set `auth.cache.enabled = false` in `extenddb.toml` and restart.
 
 ---
 

docs/getting-started.md

@@ -1426,12 +1426,12 @@ ttl_seconds        = 60     # hard TTL — entries older than this are full miss
 soft_ttl_seconds   = 30     # entries older than this serve cached AND trigger
                             #   a background refresh (request does not block)
 negative_ttl_seconds = 5    # how long "not found" results are remembered
-max_entries        = 10000  # per-cache LRU cap
+max_entries        = 10000  # maximum entries per cache
 ```
 
 **Self-induced changes** (admin API and console mutations: `CreateAccessKey`, `PutUserPolicy`, `TagResource`, `CreateTable`, etc.) propagate instantly via write-through invalidation on the local instance.
 
-**Fanout invalidations** (`DeleteAccount`, `DeleteRole` session sweep, `DeleteGroup` member fanout) complete asynchronously; there is a brief (~ms) window between the API responding success and the eviction landing. For hard cutover, prefer single-key invalidations (`DeleteAccessKey`) over cascades.
+**Fanout invalidations** (`DeleteAccount`, `DeleteRole` session sweep, `DeleteGroup` member fanout) also take effect when the API returns. If a group change cannot list the group's members, those members' entries wait for `ttl_seconds`.
 
 **Off-instance changes** (multi-node deployments, or direct DB writes outside the management API) take up to `ttl_seconds` to propagate.
 

docs/manuals/02-design-guide.md

@@ -106,7 +106,7 @@ extenddb uses SigV4 signature verification with a local IAM credential store. Th
 Full SigV4 signature verification:
 
 1. Extract `Authorization` header components (credential, signed headers, signature)
-2. Look up access key in the credential store (database-backed, credential lookup per request; encryption key cached at startup)
+2. Look up access key in the credential store (cached in memory, database on a miss; encryption key cached at startup)
 3. Reconstruct the canonical request and string-to-sign
 4. Derive the signing key: `HMAC-SHA256(HMAC-SHA256(HMAC-SHA256(HMAC-SHA256("AWS4" + secret, date), region), service), "aws4_request")`
 5. Compare computed signature with the provided signature (constant-time comparison)
@@ -128,7 +128,7 @@ Policy conditions support all IAM condition operators: `StringEquals`, `StringNo
 
 Access key secrets are encrypted at rest using AES-256-GCM. The encryption key is generated during `extenddb init` and stored in the `encryption_keys` table. Each access key record stores the encrypted secret and a unique nonce.
 
-Credential lookups (access key → encrypted secret) read directly from the database on every request — there is no in-process cache for credentials. The encryption key used to decrypt secrets is cached at startup because it is immutable after `extenddb init` (see Caching Design below).
+Credential lookups (access key ID → decrypted secret) go through each server's in-memory credential cache; a miss reads the database. See Credential and Policy Caching in the Security Model. The encryption key used to decrypt secrets is cached at startup because it is immutable after `extenddb init` (see Caching Design below).
 
 ## DynamoDB Streams Internals
 
@@ -181,7 +181,7 @@ Item size includes attribute names and values, matching DynamoDB's size calculat
 
 ## Caching Design
 
-extenddb caches a small set of operational settings in memory to avoid per-request database queries on hot paths. Catalog state (table metadata, auth policies, tags, GSI definitions) is never cached.
+extenddb caches a small set of operational settings in memory to avoid per-request database queries on hot paths; this section covers those settings and the risks of caching catalog state. Credentials, policies, boundaries, tags and table key definitions are cached separately by the auth/authz caches (see Credential and Policy Caching in the Security Model, and `docs/design/12-auth-authz-cache.md`).
 
 ### What Is Cached
 
@@ -194,18 +194,18 @@ extenddb caches a small set of operational settings in memory to avoid per-reque
 
 All cached values are operational tuning knobs where a briefly-stale value does not affect correctness.
 
-### What Is NOT Cached (and Why)
+### Risks of Caching Catalog State
 
-Catalog state is never cached because correctness requires every request to see the current state:
+The settings cache above holds no catalog state. The auth/authz caches hold some (credentials, IAM policies, boundaries, tags, and table key definitions), and a stale copy can be wrong in the ways listed below. On the server that makes a change, they drop the affected entries before the call returns. For a change made elsewhere, the TTL (`auth.cache.ttl_seconds`) bounds how long a stale entry can be used (see `docs/design/12-auth-authz-cache.md`).
 
-- **Table metadata** (key schema, attribute definitions, status, billing mode): A stale cache could serve the wrong key schema after a table is deleted and recreated with the same name but different schema. The new table has a different `table_id`, different key schema, and different indexes — stale cache serves wrong schema, writes corrupt data, reads return garbage.
+- **Table metadata** (table ID, key schema, attribute definitions, indexes, stream specification): A stale cache could serve the wrong key schema after a table is deleted and recreated with the same name but different schema. The new table has a different `table_id`, different key schema, and different indexes — stale cache serves wrong schema, writes corrupt data, reads return garbage.
 - **IAM policies and credentials**: A revoked Deny policy still cached as absent creates a security gap. A deleted access key still cached as valid allows unauthorized access.
 - **Tags**: Tag-based authorization (`aws:ResourceTag/*`) requires current tag values.
 - **GSI definitions**: Stale GSI metadata could route writes to wrong index tables.
 
 ### The Table-Name-Reuse Problem
 
-The fundamental reason catalog state cannot be cached safely:
+The main risk in caching catalog state:
 
 1. Client calls `DeleteTable("Orders")`
 2. Client immediately calls `CreateTable("Orders")` with a different key schema
@@ -214,15 +214,15 @@ The fundamental reason catalog state cannot be cached safely:
 5. Writes use wrong column layout → data corruption
 6. Reads return items with wrong attribute interpretation → garbage
 
-No safe TTL exists because delete-recreate can happen within milliseconds. Cross-instance invalidation (e.g., PostgreSQL LISTEN/NOTIFY) would be a prerequisite for any future catalog caching.
+On the server that runs the table operation, `CreateTable`, `UpdateTable`, `DeleteTable`, `RestoreTableFromBackup` and `ImportTable` drop the table's cached key and index definitions before the call returns, and a cache refresh that started before the change discards its result (the epoch counter in `crates/cache/src/swr.rs`). Across servers no TTL is safe, because a delete and recreate can happen within milliseconds: after a table is recreated through one server, another server keeps the old `table_id` for up to `ttl_seconds`. Cross-instance invalidation (for example PostgreSQL LISTEN/NOTIFY, Appendix B of `docs/design/12-auth-authz-cache.md`) is a prerequisite for running several servers on one catalog.
 
 ### Multi-Instance Considerations
 
-On SQLite, `extenddb serve` enforces one instance per database file with an exclusive lock on `<database>.lock`. On PostgreSQL and MongoDB nothing enforces it, and running several instances against one catalog is not supported: the credential and policy caches are per instance with no cross-instance invalidation, and every instance runs every background worker (see the deployment guide). Any in-process cache of catalog state would likewise be invisible to other instances. PostgreSQL's own buffer pool provides memory-resident access to hot rows, making application-level caching unnecessary for most workloads.
+On SQLite, `extenddb serve` enforces one instance per database file with an exclusive lock on `<database>.lock`. On PostgreSQL and MongoDB nothing enforces it, and running several instances against one catalog is not supported: the credential and policy caches are per instance with no cross-instance invalidation, and every instance runs every background worker (see the deployment guide).
 
 ### Future Considerations
 
-Caching of operational settings is currently unconditional. If issues arise (e.g., a setting change must take effect immediately for safety reasons), a runtime toggle (`extenddb settings set caching_enabled false`) should be added. Catalog caching remains prohibited without a cross-instance invalidation design and explicit human approval.
+Caching of operational settings is currently unconditional. If issues arise (e.g., a setting change must take effect immediately for safety reasons), a runtime toggle (`extenddb settings set caching_enabled false`) should be added. Running several servers on one catalog needs the cross-instance invalidation in Appendix B of `docs/design/12-auth-authz-cache.md` first.
 
 ---
 

docs/manuals/05-admin-guide.md

@@ -133,14 +133,14 @@ In-memory stale-while-revalidate caches eliminate the per-request catalog roundt
 | `ttl_seconds` | `60` | Hard TTL — entries older than this trigger a fresh, request-blocking load. |
 | `soft_ttl_seconds` | `30` | Stale-but-usable threshold; older entries still serve immediately while a background task refreshes them. |
 | `negative_ttl_seconds` | `5` | Negative-cache TTL for "not found" results. |
-| `max_entries` | `10000` | Per-cache LRU cap. |
+| `max_entries` | `10000` | Maximum entries per cache. |
 
 Statistics are exposed at `/management/auth-cache-metrics` (JSON, admin-authenticated). The per-cache `pass_through` flag distinguishes "cache disabled" from "cache cold." The report includes every authorization sub-cache, among them the two principal creation-time caches (`user_created_at`, `role_created_at`) consulted on each user and role-session request to derive `aws:userid` and `aws:TokenIssueTime`.
 
 **Invalidation timing**:
 
 - **Single-key invalidations** (e.g. `DeleteAccessKey`, `PutUserPolicy`) drop the cached entry immediately; the next request sees the post-mutation state.
-- **Fanout invalidations** (e.g. `DeleteAccount`, `DeleteRole` session sweep, `DeleteGroup` member fanout) are **asynchronous** — there is a small window (~ms) between the API returning success and the matching entries being evicted. For hard cutover (e.g. revoking a compromised key), prefer the single-key path (`DeleteAccessKey`) over the cascade.
+- **Fanout invalidations** (e.g. `DeleteAccount`, `DeleteRole` session sweep, `DeleteGroup` member fanout) also take effect when the API returns. The account and role sweeps register a predicate with moka, and moka's `get` applies it, so matching entries are not served once the call returns; moka removes them from its map during later cache operations. The group fanout drops each member's entries in turn. If a group change cannot list the group's members, the server logs a warning and those members' entries wait for `ttl_seconds`.
 - **Off-instance changes** (multi-node deployments, or direct catalog writes) wait up to `ttl_seconds` to be observed locally.
 
 #### [server.tls]