Repository navigation
Conversation
The OpenAI/Google and Anthropic artifact projections charged an image_url block's base64 length against maxToolResultChars, the budget for tool result text (default min(30% of the context window x 4, 400,000)). Any image larger than that, such as a browser screenshot from an MCP tool (roughly 300 KB and up), was replaced with an "[omitted 1 oversized atomic tool-content block: image_url]" notice, so the model never saw it. An image's context cost follows its pixel size, which the serving payload measure already estimates (estimateImageBlockTokens) before the graph accepts an artifact projection, and it retries without the artifact when the payload does not fit. - The projection accumulator keeps a separate image budget of HARD_MAX_TOTAL_TOOL_OUTPUT_SIZE characters, the inline image ceiling that provider image URLs and computer-use screenshots already use. A well-formed image_url block within it is kept without consuming the text budget; anything else takes the existing bounded path. - getToolContentImageUrl() validates an image_url block with the same own-data-property checks as the computer screenshot reader, which now shares the helper. - Text artifacts are still truncated to maxToolResultChars, and an image above the ceiling or the remaining image budget is still omitted with the existing notice.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tool artifact images larger than
maxToolResultCharswere replaced with[omitted 1 oversized atomic tool-content block: image_url]before the model call, because both artifact projections charged an image's base64 length against the tool-result text budget. With the default budget, that dropped browser screenshots from MCP tools (roughly 300 KB and up) for every provider. The user still saw the image in the chat. Fixes #676.An image's context cost follows its pixel size, which the serving payload measure already estimates (
estimateImageBlockTokens) before the graph accepts an artifact projection. When the payload doesn't fit, the graph retries without the artifact. This PR leaves context fit to that measure and gives images their own size limit:HARD_MAX_TOTAL_TOOL_OUTPUT_SIZEcharacters, the inline image ceiling thatisProviderImageUrl()and computer-use screenshots already use. A well-formedimage_urlblock within it is appended as is and does not consume the text budget. Anything else takes the existing bounded path unchanged.getToolContentImageUrl()validates animage_urlblock with the own-data-property checks the computer screenshot reader uses. That reader now calls the same helper.Unchanged: text artifacts are truncated to
maxToolResultChars, and an image above the ceiling or the remaining image budget is omitted with the existing notice. The image budget is shared per projection, like the text budget: across the tool batch for OpenAI/Google, per tool message for Anthropic. Cross-turn tool-image handling, history formatting and pruning are untouched.Verification
Head: 6b56c65.
src/messages/artifactImages.test.ts(both projections): a 1,250,000-character PNG (real 1280×720 header) under a 400,000-character budget is kept; a small image under a 200-character budget is kept; an image above the ceiling is omitted with the notice; text beside an image is still truncated to the budget; two 3,000,000-character images in one OpenAI/Google batch keep the first and omit the second.Graph.contextOverflow.test.ts: a screenshot-sized artifact goes throughRun.processStreamwith the real token counter (claudeando200k_base), 200k context and the default budget, and the image reaches the model call for Anthropic and OpenAI.core.ts/toolContent.tschange, 9 of the 11 new tests fail. The two "above the ceiling" cases pass either way.npx tsc --noEmit,npm run test:circular-deps, ESLint on the touched files: passed.src/messages,src/utilsand the Graph overflow suite: 442 passed.npx jestlocally: 8,229 passed. Failures are the credential-gated Vertex/Anthropic/Google suites and two macOS-specific tests (decisions/node.test.tsshell redaction,localToolResultCaps.test.tsgrep fallback), which fail the same way onmain.Not run against live providers.
🤖 Generated with Claude Code