Repository navigation
results(v0.2): the Sonnet 5.5 column's nine rows — the anchor did not move; the forms that are ours sit with the document controls; 4c matches 7a - #45
Merged
Conversation
… move, the forms that are ours sit with the document controls, 4c matches 7a Run once 2026-11-09 under protocol-v0.2: 1,026 records, $106.29 at list against the pre-registered $118–130, no transport error, 15 turn caps. Judged by claude-sonnet-5 as pre-registered; 1,026 judgments, none in error, and a second pass over the 205-record review sample agreed on 205 of 205. Rung 2, the anchor, scored 93% / 63% against Sonnet 5's 93% / 61%, question for question identical on lookups. The forms that are ours land at 90–93% on lookups and 59–63% on derived questions, inside one another's error bars, at $0.06–0.08 per correct lookup; 4c (xbrlkit serve over the package) matches 7a (the hosted sec graph) on both tiers at the same cost. No paired change from a Sonnet 5 row is distinguishable at 95%. The cross-filing rule §11 pre-registered for the hosted 7a is applied: six of 114 records read Airbnb's FY2023 or FY2022 10-K for a figure the FY2024 10-K does not carry; the five the judge marked correct are scored as confident-wrong of the form (7a derived 70% → 61%), recorded record by record in form-scoring.json; judgments.jsonl and summary.json stay the judge's output. review.md opens the human review; the root README's status paragraph moves to "has run".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Publishes the v0.2 run: the Sonnet 5.5 column's nine rows (4c · 7a · 5c+doc · 7b+doc · 7c+doc · 5c · 7b · 7c · 2), run once on 2026-11-09 under tag
protocol-v0.2, judged byclaude-sonnet-5as pre-registered, with the cross-filing rule §11 pre-registered for the hosted 7a applied to the five records it caught. The model moved and the reading did not: rung 2, the anchor, scored 93% / 63% against Sonnet 5's 93% / 61%, question for question identical on lookups; the forms that are ours land at 90–93% on lookups and 59–63% on derived questions, inside one another's error bars, at $0.06–0.08 per correct lookup; the open-source package (4c) matches the hosted platform (7a) on both tiers at the same cost. $106.29 against the pre-registered $118–130.Changes
results/v0.2-sonnet-5-5/README.md— the run, the result, accuracy and cost tables with the Sonnet 5 rows beside, every row of both columns ranked (the thirteen open rows marked), question-level error bars and paired changes, the cross-filing rule applied, the model-against-the-tools findings (4c's argument names; 7b's regex loop), the open rows, what the run did against the estimate, the seven pre-registered expectations scored, what this does and does not say, files.results/v0.2-sonnet-5-5/review.md— the human-review sheet (PROTOCOL §4): the four questions no row answered, the vals-17 case, the twelve mixed cells, the 205-record 20% sample (seed 20261109) with the judge's repeatability on it. Open; the reviewer's verdicts are added when it closes and change no score.results/v0.2-sonnet-5-5/form-scoring.json— the six 7a records whose tool results carry an accession other than the question's filing's, the accessions read, the five re-scored as confident-wrong of the form.judgments.jsonlandsummary.jsonare the judge's output, unchanged.results/v0.2-sonnet-5-5/also gainsjudgments.jsonl(1,026),run.json,summary.json/summary.md,transcripts.jsonl.gz(SHA-256 in the README), beside the gate, engine-check and artifact-hash files already there.README.md— the status paragraph: v0.2 has run, with the one-paragraph result.Protocol impact
None to the protocol text: PROTOCOL.md is untouched;
protocol-v0.2stands. The cross-filing scoring is the rule §11 pre-registered, applied as written and recorded record by record. Two findings for the next version are recorded in the README and review sheet, not acted on: the golds of vals-17 and vals-22 say the FY2024 10-K carries three years of the metric and it carries two; and 4c'sfact_grid/resolve_elementargument names draw a guessed name from the model one call in three.Testing
No code changed. The run: 1,026 of 1,026 records, no transport error, no 429, 15 turn caps; hosted
secat RoboSystems v1.22.2 at the start and the end of the run. The judge: 1,026 judgments, no judge errors.just test-allgreen on the branch (the pre-commit hook runs it).Next (maintainer): merge, then tag
results-v0.2on main.