Repository navigation
Guard legacy export with structural cosine and norm checks - #16
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The legacy
.zexporter previously merged on norm ratio alone: opposite vectors with equal norms could share a row, and trie traversal did not verify structural ancestry. Both export overloads now require whole-key prefix/suffix ancestry plus cosine >= 0.99 and norm ratio >= 0.99. Every original row is checked against its final representative after both passes to reject transitive drift. The first visited key retains its own ID, and row strides/copies follow the model dimension.Production merge callbacks are extracted into a testable header. Sanitized regressions cover direction, magnitude, structural ancestry, UTF-8 reversal, first-key IDs, 1/2/64/128/256 dimensions and final-representative drift. Linux CI also syntax-checks the complete exporter translation unit. Documentation maps paper-related claims to implementation and outstanding evidence.
Validation: clean Python 3.12.14/PyTorch 2.14.1 on macOS/Apple M5 passed all 36 unit tests; native C library/example build passed. The callbacks pass ASan/UBSan. Full legacy syntax requires a temporary macOS malloc-header compatibility shim locally; unmodified headers are checked on Linux CI. A full linked legacy exporter/reader run is not claimed.
The format is unchanged, but regenerated row counts/vectors can change. This remains approximate compression and does not reproduce paper-scale savings or downstream quality. The final guard adds an O(rows + key bytes) builder map and another traversal. Legacy serving helpers still contain 128-element buffers. Existing Engram byte-exact storage remains independent.