Skip to content

Audit residual training overlap by shingle containment - #54

Merged
professor3333 merged 1 commit into
mainfrom
data/containment-audit
Oct 19, 2026
Merged

professor3333 merged 1 commit into
mainfrom
data/containment-audit

Conversation

@professor3333

Copy link
Copy Markdown
Owner

Dedup used 5-word-shingle Jaccard, which stays low when a short document sits inside a longer one. scripts/leakage_audit.py measures containment (share of a document's shingles in its nearest training document) for dev, test and the new annotation sets.

  • Dev: 20 of 112 documents (15.8% of gold relations) at containment ≥ 0.5; end-to-end relation F1 rises with overlap for every system (SciBERT 0.617 → 0.714).
  • Test: 3 of 111 documents (1.2% of gold relations). No test predictions are read.
  • Later-domain and challenge sets: max containment 0.070 against the whole corpus.
  • Eight documents above 0.5 were not recorded as template siblings (seven dev kit variants, one short test excerpt).

The frozen splits are unchanged. Report: reports/leakage/train-overlap.md; dataset card, model card and README limitations updated.

Jaccard misses short excerpts and kit variants of longer training
protocols. Containment finds 20 dev documents (15.8% of dev gold
relations) and 3 test documents (1.2%) with at least half their text in
one training document; the new annotation sets have none. Dev scores
rise with overlap for every system. The audit reads no test predictions
and leaves the frozen splits unchanged.
@professor3333
professor3333 merged commit d289991 into main Oct 19, 2026
1 check passed
@professor3333
professor3333 deleted the data/containment-audit branch October 19, 2026 04:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant