Repository navigation
feat(protocol): pre-register v0.2.1 — rows 4c.1 and 7a.1; disclose and fix the v0.2 served-schema defect - #46
Merged
Merged
Conversation
…se with one friction removed Two rows added to the Sonnet 5.5 column under §8's last-number rule; the v0.2 rows carry forward by identity. 4c.1 is 4c on xbrlkit 0.21.5, whose fact_grid takes concepts beside elements and resolve_element phrase beside query (37 of 4c's 44 tool errors were those two names). 7a.1 is 7a with workflow steps 1 and 2 pointed at the shaped tools instead of the Cypher prerequisites (46 of 114 records opened with a 6,718-character example fetch; Cypher followed in 9). §12 pre-registers the rows, the lock delta, six expectations, the cost and the publication. The harness records 4c.1 cannot_attempt on an xbrlkit without the aliases, so the row cannot be taken against 0.21.4 by accident. Claude-Session: https://claude.ai/code/session_01AGZDQcxaocZb3djRwdRK87
…v0.2 handed none
Every tool served from xbrlkit in the v0.2 run — 4c's nine and the search_text /
read_text instrument on the three +doc rows — reached the model as a name, a
description and {"type": "object"}: the capture read the listing under the wire
name inputSchema, which the mcp 2.x Tool object does not carry as an attribute,
and defaulted. 7a's path reads input_schema and was whole. From the published
transcripts: 4c's 44 tool errors in 419 calls are all schema-validation refusals
of names inferred from descriptions; the +doc rows show 6 in 781 served calls. No
score changes. Recorded as a dated §9 erratum and a section in the v0.2 results
README; the capture now reads the attribute by its name and refuses a listing
without a schema, tested against the real server. §12 (v0.2.1) is rewritten so
row 4c.1 is 4c with the schemas handed over, on xbrlkit 0.21.5.
Claude-Session: https://claude.ai/code/session_01AGZDQcxaocZb3djRwdRK87
…ase, for row 4c.1 Claude-Session: https://claude.ai/code/session_01AGZDQcxaocZb3djRwdRK87
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two things, found in this order. (1) A harness defect in the published v0.2 run, disclosed and fixed. Every tool the ladder served from xbrlkit — 4c's nine, and the
search_text/read_textinstrument on 5c+doc, 7b+doc and 7c+doc — reached the model with an empty argument schema: the capture read the listing under the wire nameinputSchema, which the mcp 2.xToolobject does not carry as an attribute, and silently fell back to{"type": "object"}. The 7a path readsinput_schemaand was whole. Measured from the published transcripts: 4c's 44 tool errors in 419 calls are all schema-validation refusals of names the model inferred from descriptions (33conceptsforelements, 4phraseforquery, 7 types and bounds); the three +doc rows show 6 refusals in 781 served calls. No score changes — the judge scores answers — but 4c's error, turn and cost figures carry a harness handicap, and the results README attributed the refusals to the model guessing against a schema it never saw. Recorded as a dated §9 erratum and a section inresults/v0.2-sonnet-5-5/README.md, following the 2026-09-06 precedent; the capture now reads the attribute by its name and refuses a listing without a schema.(2) Protocol v0.2.1 pre-registered (§12): two rows added to the Sonnet 5.5 column beside 4c and 7a under §8's last-number rule, each its base with one friction removed. 4c.1 is 4c with the schemas handed over, on xbrlkit 0.21.5 (RoboFinSystems/xbrlkit#105), whose
fact_gridandresolve_elementalso take the names the model sent. 7a.1 is 7a with workflow steps 1 and 2 pointed at the rung's shaped tools instead of the Cypher prerequisites: 46 of 114 records opened with a 6,718-characterget-example-queriesfetch, and Cypher followed in 9. The v0.2 rows carry forward by identity.Changes
filing_ladder/representations/xbrlkit_serve.py—tool_defs_from(listed)/_schema(tool): the schema underinput_schema(or the wire name), a listing without one raises instead of defaulting;ALIASESandaccepts_aliases(tools)for the 4c.1 guard.PROTOCOL.md— §9 erratum dated 2026-11-09 (the finding, the footprint, no score changes, what it does to the claims, the fix); §12, the pre-registration: the two rows, the frictions measured, the exact change to each (old and new 7a step text quoted; 4c.1's two-part repair and why the row does not separate them), the fairness argument, two lock deltas, six expectations, cost ($15–18 + about $2 of judging; a k = 1 shakedown first, never scored), publication.results/v0.2-sonnet-5-5/README.md— the 4c bullet corrected (the model saw no schema) and a new section "What the served tools' schemas contained (added 2026-11-09)" with the per-row table; nothing else in the directory touched, hashes unchanged.filing_ladder/ladder.py—Rung.XBRLKIT_SERVE_V021(4c.1),Rung.LPG_SHAPED_V021(7a.1) and their specs; both built on their base via_BASES;V0_2_1_RUNGS;parse_rungs("v0.2.1").filing_ladder/prompts.py—LPG_SHAPED_SOURCEshared by 7a and 7a.1;SOURCES[4c.1] is SOURCES[4c];SOURCES[7a.1]with the two new steps.filing_ladder/cli.py— 4c.1 iscannot_attemptwhen the servedfact_gridschema lacksconcepts;--rungshelp listsv0.1.1 | v0.2 | v0.2.1.pyproject.toml,uv.lock—xbrlkit[lpg,mcp]>=0.21.5, locked to 0.21.5.tests/test_xbrlkit_serve.py— the served definitions carry the server's schemas (everySERVE_TOOLSschema has properties;fact_gridhaselementsandconcepts,resolve_elementqueryandphrase,search_textpattern,read_textoffset/length); a listing without a schema is refused.tests/test_v0_2_1.py— the two rows' flags and bases; 4c.1's prompt is 4c's object; 7a.1's prompt equals 7a's outside steps 1 and 2; the alias guard..gitignore—!results/v0.2.1-sonnet-5-5/.README.md— the status sentence on v0.2.1 and the disclosure.Protocol impact
RUNG FIDELITY and PROTOCOL. Rung fidelity: in v0.2, what 4c and the three +doc rows handed the model differed from what §11 described — the served tools' argument schemas were empty. Disclosed in §9 and the results README with the footprint; no published number changes; the token table is unaffected (schemas are not in it). Protocol: §12 adds two rows under §8's last-number rule; 4c and 7a keep their records and hashes; 7a.1's prompt change is on a new row, not on 7a.
Testing
just test-allgreen (format, lint, typecheck, the suite with the new tests). The schema fix and the alias guard were run against the real server over the materialized package0000012927-25-000015on xbrlkit 0.21.5: all nineSERVE_TOOLSschemas carry their properties,accepts_aliasesis true, andfact_gridwithconceptsreturns byte-identical rows toelements. Nothing paid ran: the shakedown and the two rows wait on theprotocol-v0.2.1tag and a go.