Repository navigation
Blinded replay: named vs blind A/B (#140) - #147
Merged
Merged
Conversation
A replay option `blind` (service, API, MCP, CLI) renders every prompt the run sends (decide, desk analysts, critic, the Confirm, the exit-review block) without the ticker (per-run alias ASSET_A), calendar dates (relative bar labels) or dollar values (price index with the run's first bar = 100; positions, P&L and sizes as % of the Sleeve). The response contract is unchanged: notional_usd is read as % of the Sleeve and the harness maps it back to dollars after the desk and before the crude book, so the stop, min hold, no-trade band and costs run unchanged. Free-text Sources (Macro View, briefs) are dropped, not scrubbed; FRED, X sentiment (score only) and the held position re-render from numbers. The run summary records the dropped Sources. A prompt variant that names the symbol is refused for a blind run. Each step now records the intent the book acted on. scripts/replay_blind_compare.py runs each Bot's symbols named and blind over one window (optionally a named-control arm with the same Sources dropped) and scores net P&L, hit rate, opens and the gap per symbol and per Bot. Live prompts are unchanged. Claude-Session: https://claude.ai/code/session_01UDNLNu854ULHbHSL59UJKu
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #140 (PRD #145).
replay --blind(API/MCP/CLI): ticker → ASSET_A, dates → bar t-k, prices indexed to 100, sizes/P&L as % of Sleeve; the model's % notional is mapped back to dollars before the book, so RiskGuard, stop, band and costs run unchanged. Live prompts untouched.scripts/replay_blind_compare.py run|score, with a--controlarm (named, same Sources dropped) so memory = control − blind, separate from the lost Sources.https://claude.ai/code/session_01UDNLNu854ULHbHSL59UJKu