Skip to content

🔖 fix: Add a Second Bedrock Message Cache Point So Rebuilt Turns Reach the Previous Turn's Cache - #631

Open
Haarris wants to merge 2 commits into
LibreChat-AI:mainfrom
Haarris:fix/bedrock-second-message-cache-point
Open

Haarris wants to merge 2 commits into
LibreChat-AI:mainfrom
Haarris:fix/bedrock-second-message-cache-point

Conversation

@Haarris

@Haarris Haarris commented Oct 21, 2026 •

Copy link
Copy Markdown

Addresses the lookback part of #617.

Bedrock only finds an earlier cache entry if it is within about 20 positions of a cachePoint. Re-measured on Nov 4 (Bedrock changed how it counts at the end of October): a run of tool calls counts as one position, a run of tool results as one, each text block as one, and the block that carries the cachePoint always counts on its own; 21 positions still hit, 22 miss. So parallel tool calls barely use up the lookback, but a sequential tool loop adds two positions per round.

With the single tail checkpoint from addBedrockTailCacheControl, a turn whose history is rebuilt differently from what was sent live (today: reasoning that is not replayed) can only match the prefix that ends at the previous turn's opening human message. After about 10 sequential tool rounds that prefix is out of reach, so the first call of the next turn writes most of the conversation to the cache again.

addBedrockTailCacheControl takes an optional maxCachePoints (default 1, so existing callers such as summarization are unchanged). With 2, the same backward pass also anchors the latest real human message before the tail (never a tool result, synthetic message or the tail itself). getBedrockMessageCachePointBudget returns 2 only when that fits under Bedrock's limit of 4 next to the system checkpoints in messages[0] and one reserved for the tool list, so it stays valid if #613 adds a second system checkpoint. The graph's serving path passes it.

Live on Bedrock (us.anthropic.claude-sonnet-4-6, us-west-2), Nov 4, two runs each (both runs matched to within 0.001), cost in full-price input tokens per prompt token (reads 0.1, writes 1.25):

Scenario tail only (main) this PR
Turn 1 sent live (assistant steps carry an extra block), turn 2 rebuilt without it, 11 sequential tool rounds each: first call of turn 2 read 41.7%, 0.771 read 84.2%, 0.281
Same scenario, all of turn 2 0.166 0.128
Same scenario, whole conversation 0.184 0.163
Unchanged history, 2 turns x 11 sequential tool rounds 0.156 0.156
Rebuilt history, turn 1 = one step with 11 parallel tool calls 0.202 0.202

With an unchanged history the tail alone already reaches the previous turn (each tool round leaves an entry close to the next turn), and with parallel calls the whole round fits in two positions, so in those cases the second checkpoint changes nothing and costs nothing. It pays off when the rebuilt history differs and the previous turn ran tools one after another. Once signed reasoning is replayed (#619) that case gets rarer, but the second checkpoint still covers hosts that don't persist signatures. The first version of this PR (Oct 21) also ran 4 turns x 2 tool rounds and a 6-question chat, with no difference between main and this PR; those were not re-run.

Validation: 14 new tests in cache.tail.test.ts, including a Converse-payload lookback check for rebuilt and unchanged histories (8 fail on main). 37/37 tail tests, 1,241 message/graph tests, and 205 tracing tests pass; tsc --noEmit, ESLint (no new warnings) and the build pass.

…h the Previous Turn's Cache

Bedrock only looks back about 20 content blocks from a cachePoint. With a single tail checkpoint, a turn whose history is rebuilt differently from what was sent live (e.g. reasoning that is not replayed) can only match the prefix ending at the previous turn's opening human message, which is out of reach after a tool-heavy turn, so the whole conversation is written to the cache again.

addBedrockTailCacheControl takes an optional maxCachePoints (default 1, unchanged for existing callers). With 2, the same backward pass also anchors the latest real human message before the tail. getBedrockMessageCachePointBudget returns 2 only when that fits under Bedrock's limit of 4 alongside the system checkpoints in messages[0] and one reserved for the tool list; the graph's serving path passes it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant