Repository navigation
docs(results): close the v0.2 human review - #48
Merged
Merged
Conversation
… second judge, and a §9 erratum Parts A and C of the v0.2 review sheet gain drafted reviewer notes with the document evidence (vals-16 not answerable from its 10-Q; vals-08 a gold that reads one of two defensible figures; the judge lenient or inconsistent on vals-37, vals-14, vals-33, vals-40 and vals-22). Verdicts stay open for the reviewer. A closing section records an unregistered second-judge pass (claude-opus-5-5): 97.3% and 98.2% agreement, no score changed. PROTOCOL §9 records that the v0.2.1 README's 16,500-character describe_filing figure is wrong (median 22,642, mean 25,474 in the published transcripts).
…n mixed cells vals-25 (gold wider than the question), vals-43 (three points the question does not ask for, and a sign error on freight), vals-46 (the revenue-mix premise does not match the FY2024 10-K), vals-11 on 7c (judge wrong on two runs) and vals-27 on 5c (model variance). Verdicts stay open for the reviewer.
Verdicts entered on Parts A, B' and C; the sheet's status and the README's status line read closed, with a summary of what the review found: six golds with problems (four of them the questions no row answered) and 14 judge verdicts wrong in the cells reviewed, 9 lenient and 5 strict. No score changes.
… next question-set version
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the human review PROTOCOL §4 requires for v0.2. The review sheet gains the reviewer's notes and verdicts on the four questions no row answered, the vals-17 case, the twelve mixed cells, and all 205 rows of the random sample. No score changes:
judgments.jsonlandsummary.jsonare the judge's output, untouched.What it found. Six golds have problems, four of them the questions no row answered: vals-16 asks for prospectus terms its 10-Q does not carry; vals-25's gold includes two cybersecurity risks under a question about regulatory ones; vals-43's gold has the sign of freight growth wrong and requires segment rates the question does not ask for; vals-46's gold says two revenue lines declined that the filing shows rising in 2024; vals-17 needs a year its filing does not report; vals-08's gold takes one of two figures its sentence supports. In the cells reviewed, 14 judge verdicts are wrong, 9 lenient and 5 strict, moving no row by more than two records of 114 net; 203 of 205 stand in the random sample.
Changes
results/v0.2-sonnet-5-5/review.md: notes and verdicts with the document evidence under each Part A question, vals-17, and the mixed cells; the verdict column of Appendix D filled; a "What the review found" summary; a closing section recording an unregistered second-judge pass (claude-opus-5-5, 97.3% agreement on v0.2 and 98.2% on v0.2.1), labelled exploratory.results/v0.2-sonnet-5-5/README.md: the status line reads closed, with the summary in one paragraph.PROTOCOL.md§9: a dated erratum. The v0.2.1 README's "16,500-characterdescribe_filing" is wrong; the published transcripts give a median of 22,642 and a mean of 25,474 characters. The README sentence is left as published.Protocol impact
HARNESS. Nothing a model sees or a score depends on changes. The gold findings are QUESTIONS matters for the next version that changes the question set; they are recorded in the review sheet with their document evidence and change nothing here.
Testing
just test-allgreen (90 passed; ruff and basedpyright clean). Documentation only. Evidence was read from the materialized rung 2 text of accessions 0001404912-25-000022 (KKR), 0000950170-25-067220 (TKO), 0001437107-25-000031 (WBD), 0001591698-24-000151 (PCTY), 0001543151-25-000008 (UBER), 0001617640-25-000016 (ZG) and 0001559720-25-000010 (ABNB).