cache: a background refresh can reinsert a value loaded before an invalidation #788

A background refresh in SwrCache can write a value loaded before an invalidation back into the cache after that invalidation. The stale value is then served until it expires (up to auth.cache.ttl_seconds, 60 s by default), on the server that made the change.

Mechanism

On main (2287d17), the soft-TTL refresh path in crates/cache/src/swr.rs runs the loader, then:

1. checks the epoch (line 494) and drops the result if an invalidation happened during the load;
2. awaits cache.get(&key) and checks that the slot still holds the entry the refresh started from (lines 505-508);
3. calls cache.insert(key, new_entry) (line 514).

These steps are not atomic. If an invalidation (single-key invalidate, or a predicate from invalidate_if / invalidate_if_value) lands after step 2 and before step 3, the insert writes the pre-invalidation value with a fresh timestamp. A registered predicate does not match it, because moka applies predicates only to entries last modified before the predicate was registered. The hard-miss path (lines 414-431) does not have this gap, because it checks the epoch after the insert and invalidates the key if it moved.

The window is very short and needs a refresh that finishes at the same moment as the change. The docs (security model, admin guide, docs/design/12-auth-authz-cache.md §15) say that changes made through the management API or console take effect on that server when the call returns, so a revocation that hits this race takes up to one TTL instead.

Possible fix

Check the epoch again after insert and invalidate the key if it moved, as the hard-miss path does. Invalidations bump the epoch before they touch moka, so one that ran between the first check and the insert is then seen.

Reproduction

Not reproduced on a binary. A unit test can hold the refresh loader on a barrier, release it, and run an invalidation between the identity check and the insert, for example by adding a test hook there.

Found while writing #786.