caching: scan the cache in steps, mtime fallback, more tests (follow-up to #223)

[ThomasWaldmann]

Follow-up to #223 (#183), addressing findings of a critical review of that PR. 3 commits:

1. scan the cache in steps while the store is in use

A store that has put more than size / 4 bytes into a size limited namespace scans it again to see what other clients did. That scan listed the whole namespace within one store() / load() call, holding the store's lock - with a lot of cached items, all threads were blocked for seconds, and there was nothing in the logs about it.

Now each item that is put into the cache continues the scan for about 5 ms (CACHE_SCAN_STEP_TIME), the store uses the cache as usual in between. No threads, no lock changes - it relies on the same thing Store.list() already relies on (an open backend listing interleaved with other operations).

• What the store changes while a scan is in progress is tracked (CacheScan.changed) and not overridden by what the scan has (or has not) seen: it might have listed an item before the store evicted it, or have listed a directory before the store added an item there. See CacheIndex.finish_scan.
• The listing also yields the directories, so a step stays short if there are many directories, but few items.
• If a scan found nothing that must be sorted into the index (as usual if the cache is not shared), the index is not rebuilt.
• Store.open() and Store.close() still scan in one go. close() drops a scan in progress and starts over, so it sees the latest changes of other clients.
• A finished scan logs (debug): item count, items added / removed by other clients, time, steps.

Longest single store() call while rescanning, levels [2], local APFS (same total run time):

cache	main	this PR
100k -> 200k items	4508 ms	114 ms
65k directories, 1000 items	1505 ms	15 ms

p99 is 5.5 ms (= the step time). What remains is finishing a scan (comparing the listing with the index) and one directory listing - a flat directory with 100k names still is one 390 ms step, because posixfs.list does sorted(path.iterdir()) (an os.scandir based list would be about 3x faster in general, not part of this PR).

4 processes hammering one shared cache for 8 s (68k verified loads): 0 wrong reads, 0 cache errors, 0 warnings, peak within size * (1 + N / 4), under size after the last close - also with a step time of 0 (one listing entry per step, scans interleaved with everything).

2. use the mtime if the cache backend has no atime

For items the store has not used itself, max_age and the LRU order depend on ItemInfo.atime. Without atime support (s3), max_age removed all these items and size evicted them in no particular order. ItemInfo.mtime (when the item was put into the cache) is a much better guess than 0. Docs: list which backends have atime / mtime (sftp has atime, too).

3. add missing tests, simplify the eviction loops

Found by mutating the code of #223:

• no test noticed if a scan did not keep the more recent last access of a known item (own vs. listed) - new test, also covering a changed size.
• no test noticed if a cache miss did not drop the index entry - new test (item deleted by another client from primary and cache).
• the eviction loops removed the index entry although _cache_delete does that anyway - removed.
• the test for failing evictions also passed without any eviction - it now checks that evictions really failed and that each one was counted.
• new: size limit in mirror mode, failing scan while the store is in use, close() with a scan in progress, clients using a shared cache at the same time (threads, each with its own Store).

All 13 mutants of the new / changed code are caught by the tests now.

Docs: while multiple clients put items into a shared cache, its size usually is above the limit, not just temporarily (measured medians for clients that continuously insert: +14 % for 2, +40 % for 4, +62 % for 8 clients). It is within size again when the last of them has closed the store.

Not in this PR

• Items bigger than size are not cached since caching: continuous cache eviction, fixes #183 #223, and a partial load() of such an item reads the full item from the primary backend each time (measured: 20 x 100 B from a 5 MB item with a 1 MB limit = 100 MB read; before caching: continuous cache eviction, fixes #183 #223: 5 MB). Needs a separate fix.
• The index needs about 325 B per cached item (not documented yet).

No CHANGES.rst entry yet.

🤖 Generated with Claude Code