Repository navigation
On a branch cut from origin/main, the six goal/10 commits (190b735 cf421ca fea7158 225ad8d cc7d3d4 3ad174b: report.py, e - #11
Merged
Conversation
… --device report.py computes human FPR, FMA-OOD accuracy, per-generator / co-created / per-transform FNR and AUROC vs Jamendo at the run threshold, for all / seen / unseen rows, with Wilson intervals and failure accounting. select.py eval-manifest writes the eval rows (with runtime_path) because the runner's --protocol-split has no 'eval' choice. audit_artifactnet_nonfinite.py: optional --device (default cpu, unchanged), needed because 20% of tracks hit non-finite chunks and a CPU re-run of ~6.6k tracks would take ~8 h.
…; add run verifier The declared ArtifactBench v2 ArtifactNet policy (median of finite scores, at least 4 of 7 chunks) is the primary result when finite_chunk/ exists; the strict any-non-finite result stays as a sensitivity table. verify.py run checks the full-cohort selection, pinned ONNX revision, that every eval track is scored or a recorded failure in both results, and that the saved report equals a rebuild from raw scores.
…heckout audit_artifactnet_nonfinite.py adds the repo root to sys.path so the documented command works without pip install -e (check_docs verifies it).
9,738 of 33,000 strict-run tracks hit a non-finite chunk; the single audit process ran at ~1 track/s (decode-bound, GPU 65%). merge_audits.py splits the failure list round-robin and merges shard audits back into the audit script's schema, rejecting missing, duplicate or extra tracks.
…isions import_scores.py maps a JSONL of per-file scores (path, probability, optional decision field) onto the eval cohort and writes the run layout report.py reads; unknown/unreliable verdicts, errors and missing lines are recorded failures. report.py uses a per-track decision_ai when present (pipelines whose verdict includes a rescue gate), AUROC still from the probability. Needed to report a non-ONNX detector on the same cohort.
build_artifactnet_finite_chunk_result.py copied only audited (inference) failures into the finite-chunk result, so a strict decode failure vanished from both scores and failures (MuseBench run: 32,999 of 33,000 tracks accounted for). Non-audited strict failures are now carried over. MuseBench v9.4 baseline complete: 30,767 of 33,000 eval rows scored under the declared finite-chunk policy (7,506 recovered), 2,233 failures; verify.py all passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone of an auto-claude run.
Check:
systemd-run --user --scope -q -p MemoryMax=6G ~/RTX_ENV/bin/python -m pytest -m "not slow" -q -p no:cacheprovider && ~/RTX_ENV/bin/python scripts/check_docs.py && ~/RTX_ENV/bin/python -I scripts/musebench/verify.py allPlan: .claude/auto-plan.md (local to the run).