Skip to content

Stream embedding exports and accelerate lossless XOR packing - #10

Merged
initial-d merged 3 commits into
mainfrom
codex/streaming-export-20261021
Oct 20, 2026
Merged

initial-d merged 3 commits into
mainfrom
codex/streaming-export-20261021

Conversation

@initial-d

Copy link
Copy Markdown
Owner

Embedding export previously materialized a whole-table CPU copy and Python bytes object, while native XOR prediction looped over every byte in Python. Stage weights in bounded row slices with incremental SHA-256, and compute each block's identical XOR residual through C-backed Python integer operations. Format, source IDs/dtype, compression candidates and C queries are unchanged.

Validation:

  • 31 tests pass with PyTorch, including noncontiguous float32/float16/bfloat16 sources, block-sized byte staging, streamed hashes, packed-file identity for Python/native backends, and XOR zeros/odd-width/special-bit cases. System Python passes with 17 optional cases skipped.
  • The pinned official Engram module and draft factory each pass all nine full-output cases. Tiny random weights are interface fixtures, not trained-model evidence.
  • All 27 baseline/optimized file pairs on nine existing trained SGNS tables are byte-identical and reconstruct every row through C. Median XOR export improves about 1.9×.
  • A separate 64 MiB fixture repeats trained rows for process-memory measurement: median export drops from 4.121 s to 2.207 s; additional process peak RSS drops from 67.19 MiB to 2.70 MiB. This fixture is not representative compression evidence. Native no-dedup descriptors, dedup dictionaries, Python-builder memory and temporary disk-space limits remain documented.

Exact baseline/code commits, commands, source/library/data/artifact hashes, all timing samples and limitations are retained in docs/export_evidence_20261021/ and summarized in docs/streaming_export_20261021.md. No new data download or retraining. CPU only; no trained Engram, GPU, task-quality or query-speed claim.

@initial-d
initial-d merged commit 8154c40 into main Oct 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant