Skip to content

feat(protocol): pre-register v0.2.1 — rows 4c.1 and 7a.1; disclose and fix the v0.2 served-schema defect - #46

Merged
jfrench9 merged 3 commits into
mainfrom
feature/v0-2-1-two-rows
Nov 9, 2026
Merged

jfrench9 merged 3 commits into
mainfrom
feature/v0-2-1-two-rows

Conversation

@jfrench9

@jfrench9 jfrench9 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two things, found in this order. (1) A harness defect in the published v0.2 run, disclosed and fixed. Every tool the ladder served from xbrlkit — 4c's nine, and the search_text / read_text instrument on 5c+doc, 7b+doc and 7c+doc — reached the model with an empty argument schema: the capture read the listing under the wire name inputSchema, which the mcp 2.x Tool object does not carry as an attribute, and silently fell back to {"type": "object"}. The 7a path reads input_schema and was whole. Measured from the published transcripts: 4c's 44 tool errors in 419 calls are all schema-validation refusals of names the model inferred from descriptions (33 concepts for elements, 4 phrase for query, 7 types and bounds); the three +doc rows show 6 refusals in 781 served calls. No score changes — the judge scores answers — but 4c's error, turn and cost figures carry a harness handicap, and the results README attributed the refusals to the model guessing against a schema it never saw. Recorded as a dated §9 erratum and a section in results/v0.2-sonnet-5-5/README.md, following the 2026-09-06 precedent; the capture now reads the attribute by its name and refuses a listing without a schema.

(2) Protocol v0.2.1 pre-registered (§12): two rows added to the Sonnet 5.5 column beside 4c and 7a under §8's last-number rule, each its base with one friction removed. 4c.1 is 4c with the schemas handed over, on xbrlkit 0.21.5 (RoboFinSystems/xbrlkit#105), whose fact_grid and resolve_element also take the names the model sent. 7a.1 is 7a with workflow steps 1 and 2 pointed at the rung's shaped tools instead of the Cypher prerequisites: 46 of 114 records opened with a 6,718-character get-example-queries fetch, and Cypher followed in 9. The v0.2 rows carry forward by identity.

Changes

  • filing_ladder/representations/xbrlkit_serve.py — tool_defs_from(listed) / _schema(tool): the schema under input_schema (or the wire name), a listing without one raises instead of defaulting; ALIASES and accepts_aliases(tools) for the 4c.1 guard.
  • PROTOCOL.md — §9 erratum dated 2026-11-09 (the finding, the footprint, no score changes, what it does to the claims, the fix); §12, the pre-registration: the two rows, the frictions measured, the exact change to each (old and new 7a step text quoted; 4c.1's two-part repair and why the row does not separate them), the fairness argument, two lock deltas, six expectations, cost ($15–18 + about $2 of judging; a k = 1 shakedown first, never scored), publication.
  • results/v0.2-sonnet-5-5/README.md — the 4c bullet corrected (the model saw no schema) and a new section "What the served tools' schemas contained (added 2026-11-09)" with the per-row table; nothing else in the directory touched, hashes unchanged.
  • filing_ladder/ladder.py — Rung.XBRLKIT_SERVE_V021 (4c.1), Rung.LPG_SHAPED_V021 (7a.1) and their specs; both built on their base via _BASES; V0_2_1_RUNGS; parse_rungs("v0.2.1").
  • filing_ladder/prompts.py — LPG_SHAPED_SOURCE shared by 7a and 7a.1; SOURCES[4c.1] is SOURCES[4c]; SOURCES[7a.1] with the two new steps.
  • filing_ladder/cli.py — 4c.1 is cannot_attempt when the served fact_grid schema lacks concepts; --rungs help lists v0.1.1 | v0.2 | v0.2.1.
  • pyproject.toml, uv.lock — xbrlkit[lpg,mcp]>=0.21.5, locked to 0.21.5.
  • tests/test_xbrlkit_serve.py — the served definitions carry the server's schemas (every SERVE_TOOLS schema has properties; fact_grid has elements and concepts, resolve_element query and phrase, search_text pattern, read_text offset / length); a listing without a schema is refused. tests/test_v0_2_1.py — the two rows' flags and bases; 4c.1's prompt is 4c's object; 7a.1's prompt equals 7a's outside steps 1 and 2; the alias guard.
  • .gitignore — !results/v0.2.1-sonnet-5-5/. README.md — the status sentence on v0.2.1 and the disclosure.

Protocol impact

RUNG FIDELITY and PROTOCOL. Rung fidelity: in v0.2, what 4c and the three +doc rows handed the model differed from what §11 described — the served tools' argument schemas were empty. Disclosed in §9 and the results README with the footprint; no published number changes; the token table is unaffected (schemas are not in it). Protocol: §12 adds two rows under §8's last-number rule; 4c and 7a keep their records and hashes; 7a.1's prompt change is on a new row, not on 7a.

Testing

just test-all green (format, lint, typecheck, the suite with the new tests). The schema fix and the alias guard were run against the real server over the materialized package 0000012927-25-000015 on xbrlkit 0.21.5: all nine SERVE_TOOLS schemas carry their properties, accepts_aliases is true, and fact_grid with concepts returns byte-identical rows to elements. Nothing paid ran: the shakedown and the two rows wait on the protocol-v0.2.1 tag and a go.

…se with one friction removed

Two rows added to the Sonnet 5.5 column under §8's last-number rule; the v0.2 rows
carry forward by identity. 4c.1 is 4c on xbrlkit 0.21.5, whose fact_grid takes
concepts beside elements and resolve_element phrase beside query (37 of 4c's 44 tool
errors were those two names). 7a.1 is 7a with workflow steps 1 and 2 pointed at the
shaped tools instead of the Cypher prerequisites (46 of 114 records opened with a
6,718-character example fetch; Cypher followed in 9). §12 pre-registers the rows,
the lock delta, six expectations, the cost and the publication. The harness records
4c.1 cannot_attempt on an xbrlkit without the aliases, so the row cannot be taken
against 0.21.4 by accident.

Claude-Session: https://claude.ai/code/session_01AGZDQcxaocZb3djRwdRK87
…v0.2 handed none

Every tool served from xbrlkit in the v0.2 run — 4c's nine and the search_text /
read_text instrument on the three +doc rows — reached the model as a name, a
description and {"type": "object"}: the capture read the listing under the wire
name inputSchema, which the mcp 2.x Tool object does not carry as an attribute,
and defaulted. 7a's path reads input_schema and was whole. From the published
transcripts: 4c's 44 tool errors in 419 calls are all schema-validation refusals
of names inferred from descriptions; the +doc rows show 6 in 781 served calls. No
score changes. Recorded as a dated §9 erratum and a section in the v0.2 results
README; the capture now reads the attribute by its name and refuses a listing
without a schema, tested against the real server. §12 (v0.2.1) is rewritten so
row 4c.1 is 4c with the schemas handed over, on xbrlkit 0.21.5.

Claude-Session: https://claude.ai/code/session_01AGZDQcxaocZb3djRwdRK87
@jfrench9 jfrench9 changed the title feat(protocol): pre-register v0.2.1 — rows 4c.1 and 7a.1, each its base with one friction removed feat(protocol): pre-register v0.2.1 — rows 4c.1 and 7a.1; disclose and fix the v0.2 served-schema defect Nov 9, 2026
@jfrench9
jfrench9 merged commit 13312db into main Nov 9, 2026
2 checks passed
@jfrench9
jfrench9 deleted the feature/v0-2-1-two-rows branch November 9, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant