[robinnsc]

What

The security model said there was no in-process credential cache and that every request read the catalog, and the design guide said catalog state was never cached. Neither has been true since the auth/authz cache landed. This adds a Credential and Policy Caching section to the security model and corrects the related text in four other documents.

The new section covers:

• what each server caches (decrypted credentials, parsed inline policies, boundaries, principal and resource tags, role-session data, user and role creation times, table key definitions) and what it does not (authorization decisions, item data);
• the [auth.cache] settings, their defaults and the limits that stop a server from starting, and that they are read only at startup;
• which changes take effect at once (management API, console, TagResource/UntagResource, on the server that makes them) and which wait up to ttl_seconds or negative_ttl_seconds (other servers on the same catalog, direct catalog writes);
• how to make a revocation take effect immediately: change the catalog first, then flush every other server with extenddb manage cache invalidate using the scope that matches the change, or restart them. AssumeRole session credentials cannot be deleted one at a time; deleting the role's inline policies stops its sessions.

Other corrections:

• Security model, Fail-Closed Design: policies are read from the catalog when they are not in the server's cache.
• Security model, Logging: only some management operations are audit-logged at WARN, and the list is now given. Cache flushes are logged at INFO, so they are not recorded at warn or error level. Log records can contain access key IDs.
• Design guide: credential lookups go through the credential cache. The Caching Design section now says it covers the settings cache only and points to the auth/authz cache design, and its What Is NOT Cached text says the auth/authz caches do hold some catalog state and how they limit the risk.
• Admin guide, getting-started guide and docs/design/12-auth-authz-cache.md §15: fanout invalidations (DeleteAccount, DeleteRole, DeleteGroup) were described as asynchronous with a window in which stale entries could be served. They take effect when the call returns. The account and role sweeps use moka's invalidate_entries_if, which registers a predicate that get applies, so only the removal of entries from moka's map happens later; the group fanout awaits each member. If a group's members cannot be listed, those members' entries wait for ttl_seconds. §15 also said a forced flush was not supported; it now points to the manual invalidation API (§6.1).
• Admin guide and getting-started guide: max_entries is described as a maximum entry count rather than an LRU cap, since moka does not evict by LRU alone.

Why

An operator reading the security model would expect a deleted key or a removed policy to stop working on every server at once. With the cache, that holds only on the server that made the change; other servers sharing the catalog keep the old entry for up to ttl_seconds (60 s by default). Running more than one server on a catalog is not supported in this release, but direct catalog writes hit the same delay, and the revocation steps need to say how to close it.

Testing done

Every statement in the new text was checked against the code on main (2287d17), with file and line references. That covers the cached set, the defaults and validation in crates/cache/src/swr.rs and crates/config, the write-through calls in the management, console and tagging paths, the manage cache invalidate scopes and flags in crates/app/src/manage_types.rs, the CLI's endpoint and TLS handling in manage_http.rs, the log levels, and moka 0.12.15's invalidate_entries_if and get. The Markdown was rendered to HTML to check the tables and code block. This branch adds no conflicts to #704, #718, #727, #734, #737, #773 or #228, which touch the same files. It conflicts with #355, which replaces the same Credential Storage bullet, and adds conflicts to the documentation sweeps #491, #523, #530 and #537, which already conflict with main.

Not changed here

• Web Console Security is left as it is because fix(console): end console sessions when the principal is deleted or its password changes #704 rewrites the session bullet. On main, console sessions are not checked against the catalog after login, so deleting an admin or user or changing a password does not end them. fix(console): end console sessions when the principal is deleted or its password changes #704 fixes that.
• The console's DeleteUser and DeleteRole handlers do not drop the cached user and role creation times, while the management API handlers do (console/pages/user_pages.rs:354-379, role_pages.rs:316-337). A principal recreated with the same name through the console can be given the old aws:userid for up to ttl_seconds. The text here describes the intended behaviour; tracked in server: console user and role deletes leave the cached creation time #787.
• A background refresh that finishes at the same moment as an invalidation can put the pre-change value back (crates/cache/src/swr.rs:494-514: the epoch and identity checks are not atomic with the insert). The window is very short, and it affects single-key and fanout invalidations alike; tracked in cache: a background refresh can reinsert a value loaded before an invalidation #788.
• fix(auth): check the scope and session token, canonicalize repeated headers, and time out stalled requests #355 also replaces the "No in-process credential cache" bullet. Whichever lands second should keep one bullet; this branch's points to the new section.
• Deleting a user, group or role can leave its inline policies or boundary for a later principal with the same name #693 (inline policies and boundaries left after a principal is deleted) affects how permanent "delete the user" is as a revocation step; that is for Deleting a user, group or role can leave its inline policies or boundary for a later principal with the same name #693 to fix.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace): documentation only
• Code is formatted (cargo fmt --check): documentation only
• Clippy is clean (cargo clippy -- -W clippy::pedantic): documentation only
• I have added or updated tests for new functionality: n/a
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a

Breaking changes

None.

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

[LeeroyHannigan]

This PR replaces the "no in-process credential cache" claims with a Credential and Policy Caching section in the security model and corrects the fanout invalidation and max_entries text in four other documents. I checked each statement against the code at 8c17f33, including every write-through call site, the CLI scopes and TLS handling, the log levels, and moka 0.12.15 applying registered predicates in get (future/base_cache.rs:281). Apart from the findings below, they hold. The PR changes documentation only, so there are no tests to check.

Major, docs/manuals/02-design-guide.md:197-225. Line 199 says the auth/authz caches handle the risks of caching catalog state with write-through invalidation and "a TTL for changes made elsewhere". The rest of the section says the opposite: "What Is NOT Cached" heads a list of things TableKeyInfo holds (crates/core/src/types/key_schema.rs:75-105), line 217 calls cross-instance invalidation a prerequisite for "any future catalog caching", line 221 calls application-level caching unnecessary, and line 225 says catalog caching "remains prohibited". Rename the heading, and say in line 199 that the TTL bounds the delay. Rewrite line 217: table operations drop the table's entry before the call returns and the epoch check discards a refresh that started earlier, but across servers no TTL is safe, since a table recreated through a second server keeps its old table_id here for up to ttl_seconds. Drop the last sentence of line 221, and reword line 225 to say several servers per catalog need Appendix B of doc 12 first.

Minor, docs/manuals/10-security-model.md:77. The list of changes that apply at once omits table operations, so "Every other change" reads as covering them. CreateTable, UpdateTable, DeleteTable, RestoreTableFromBackup and ImportTable drop the table's key and index definitions, and CreateTable and DeleteTable drop its resource tags (crates/engine/src/create_table.rs:93,104, delete_table.rs:31,40, update_table.rs:196, backup.rs:990, import_export.rs:96). Add them.

Minor, docs/manuals/10-security-model.md:99-107. The scope table lacks the CLI's table-key-info --account-id <id> --table-name <name> (crates/app/src/manage_types.rs:466), and the role row omits a deleted role, which the role scope covers by dropping the role's session data and session credentials (crates/server/src/management/cache_invalidate.rs:195). Add a row for a recreated or altered table, and "or a deleted role".

Minor, docs/manuals/10-security-model.md:77. The line says console deletes drop every affected entry. The console's DeleteUser and DeleteRole do not drop the cached creation times (console/pages/user_pages.rs:354-379, role_pages.rs:316-337), so a principal recreated with the same name through the console resolves to the old aws:userid for up to ttl_seconds, and a condition on aws:userid matches the old id in that window. Fix #787 first, or name the gap and #787 in a clause.

Minor, docs/manuals/10-security-model.md:109. The certificate name check applies without --endpoint too: the CLI checks the certificate against 127.0.0.1 or server.bind_addr (crates/app/src/manage_http.rs:57, :165-167). The extenddb init certificate lists both (crates/app/src/init_helpers.rs:47). A CA-signed certificate, which line 299 recommends, usually lists neither, so the default connection fails the handshake. Say the certificate must cover the address in both cases, and that a CA-signed one needs --endpoint https://<name in the certificate>:<port>.

Minor, docs/manuals/10-security-model.md:292. The catalog's log_level setting overrides logging.level while the server runs (crates/server/src/workers.rs:33-45). Write "when the log level (logging.level, or the log_level setting, which overrides it) is warn or error".

Minor, docs/manuals/02-design-guide.md:131. The cache holds the decrypted secret (crates/auth/src/lib.rs:58), so "(access key → encrypted secret)" should read "(access key ID → decrypted secret)".

Optional, docs/manuals/10-security-model.md:88. Deleting a role's inline policies also blocks its new sessions. An inline Deny on the role with DateLessThan on aws:TokenIssueTime, which is resolved for role sessions (docs/manuals/05-admin-guide.md:544), ends only the sessions issued before that time.

Optional, docs/manuals/10-security-model.md:93. Add unset EXTENDDB_PASSWORD after the flush so the password does not stay in the shell's environment.

Optional, docs/design/12-auth-authz-cache.md:195 and :203 still call max_entries an LRU limit, and docs/design/05-component-auth-v3.md:15 still says "No caching". Update both, or point doc 05 at doc 12.

Overlap with #228: git merge-tree --write-tree origin/pr-786 origin/pr-228 is clean, and the merged Credential Storage list reads correctly. This PR rewrites the "No in-process credential cache" bullet and 02-design-guide.md:131, two items raised in review on #228. With git merge-file, a #228 rewrite of either line conflicts with this PR, while #228 fixes to 02-design-guide.md:40 and :129 and to the "cannot be retrieved afterward" bullet merge cleanly. I will make those three in #228 and leave the other two here. Neither PR says that a running server holds the encryption key and recently used decrypted secrets in process memory, which makes a core dump of it credential-equivalent. That belongs in the Trust Boundaries list in #228.

Not checked:

• Running the CLI flush against a server, and rendering the Markdown.
• The conflict statements for PRs other than #228.
• Whether a stale TableKeyInfo on a second server causes errors or misplaced writes.