Skip to content

Cache native decoded blocks within an explicit byte budget - #11

Merged
initial-d merged 4 commits into
mainfrom
codex/native-cache-20261022
Oct 22, 2026
Merged

initial-d merged 4 commits into
mainfrom
codex/native-cache-20261022

Conversation

@initial-d

Copy link
Copy Markdown
Owner

Repeated native gathers previously decompressed the same blocks on every call. Add an optional per-handle decoded-block LRU with an explicit payload-byte budget and a 64-entry cap. Opening still disables caching; files, original row IDs and dtype bytes stay unchanged. Raw/oversized blocks bypass caching, failed decodes are not inserted, and budget changes clear entries.

Expose additive C cache configuration/usage/hit counters and native_cache_bytes through NativeRows, Engram factory/gather and ordinary embedding loaders. Zero-budget loading stays compatible with older compiled libraries; opting in requires a rebuilt library. Handles remain immutable and single-worker. Scratch, metadata and query buffers are additional memory beyond the decoded payload budget.

Validation:

  • 35 tests pass with PyTorch, including LRU eviction, budget reset, tail blocks, oversize/raw bypass, failed decode, invalid-query recovery, 64-entry cap, old-library loading and special float bit patterns. System Python passes with 17 optional tests skipped.
  • ASan/UBSan exercise enabled caches, repeated calls, changing budgets, versions 1/2 and 128 deterministic mutated/truncated files.
  • Official Engram and proposed factory each preserve full output bytes in nine tiny random CPU cases, including cached repeated forwards. These are interface fixtures, not trained Engram evidence.
  • All 81 trained-table/trace/budget cases preserve original rows/query bytes and respect payload limits. A 2 MiB cache improves warm corpus-replay p50 ~6.9× at order 4 and ~1.7× at orders 8/12; 256 KiB gets no hits on that replay and adds overhead. The deliberately favorable eight-block hotspot control is reported separately, without a model-speedup claim.

Raw samples, hashes, commands, initial pilot and negative capacity results are retained in docs/native_cache_evidence_20261022/, summarized in docs/native_cache_20261022.md. Small English SGNS tables and short repeated traces; no real Engram workload, cold I/O, total process RSS, GPU or task-quality claim.

@initial-d
initial-d merged commit deab253 into main Oct 22, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant