Repository navigation
scripts/musebench/leak_map.py can annotate against a lineage of training manifests (a references JSON listing manifest f - #12
Merged
Conversation
… report --seen-field; production verifier leak_map.py - references JSON may list `lineage_manifests` (the training manifests of every generation in a model's weight lineage) instead of a candidate-pool directory; rows are read by path, track id and origin_* fields, yielding SONICS stems, FMA ids and MTG-Jamendo ids. - `--model-key KEY` writes seen_in_training_KEY / seen_evidence_KEY / ... and leak/leak_report_KEY.json, leaving the unkeyed annotation of an earlier model untouched. report.py - `--seen-field` splits seen/unseen by any annotation field. The field is joined from the runtime manifest by track id when the eval manifest lacks it, so the eval manifest a run consumed is never rewritten (its sha256 is recorded in that run's run_manifest.json). verify.py - new `production` stage (part of `all`): raw score lines cover the eval cohort once each and are unchanged since import (scores_sha256), scored + failed == cohort, every score carries a boolean decision_ai, the keyed leak annotation is present/coherent and its report current, and the saved report equals a rebuild split by the keyed field. Assumption: which run directory and model key the production stage checks is read from <data root>/results/production_run.json, so no internal model name is committed to this public repo.
… as size_unknown, not size_differs A lineage manifest can name a SONICS file that is on an unmounted drive (here the v10_training symlinks into an offloaded disk). The match is still a name match and still marks the row seen; only the size confirmation is unavailable, which the report now says instead of 'size differs'.
…cored runs and verify.py production
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone of an auto-claude run.
Check:
systemd-run --user --scope -q -p MemoryMax=6G ~/RTX_ENV/bin/python -m pytest -m "not slow" -q -p no:cacheprovider && ~/RTX_ENV/bin/python scripts/check_docs.py && ~/RTX_ENV/bin/python -I scripts/musebench/verify.py production && ~/RTX_ENV/bin/python -I scripts/musebench/verify.py allPlan: .claude/auto-plan.md (local to the run).