Skip to content

🖼️ fix: Budget Tool Artifact Images Apart from Tool Result Text - #677

Open
Odrec wants to merge 1 commit into
LibreChat-AI:mainfrom
Odrec:fix/tool-artifact-image-budget
Open

Odrec wants to merge 1 commit into
LibreChat-AI:mainfrom
Odrec:fix/tool-artifact-image-budget

Conversation

@Odrec

@Odrec Odrec commented Nov 12, 2026

Copy link
Copy Markdown

Summary

Tool artifact images larger than maxToolResultChars were replaced with [omitted 1 oversized atomic tool-content block: image_url] before the model call, because both artifact projections charged an image's base64 length against the tool-result text budget. With the default budget, that dropped browser screenshots from MCP tools (roughly 300 KB and up) for every provider. The user still saw the image in the chat. Fixes #676.

An image's context cost follows its pixel size, which the serving payload measure already estimates (estimateImageBlockTokens) before the graph accepts an artifact projection. When the payload doesn't fit, the graph retries without the artifact. This PR leaves context fit to that measure and gives images their own size limit:

  • The projection accumulator gets a separate image budget of HARD_MAX_TOTAL_TOOL_OUTPUT_SIZE characters, the inline image ceiling that isProviderImageUrl() and computer-use screenshots already use. A well-formed image_url block within it is appended as is and does not consume the text budget. Anything else takes the existing bounded path unchanged.
  • getToolContentImageUrl() validates an image_url block with the own-data-property checks the computer screenshot reader uses. That reader now calls the same helper.
  • The URL length is read directly, so a data URL over the 1,000,000-character structured traversal allowance is no longer misjudged as oversized. This covers the case from Large MCP image artifacts omitted despite sufficient maxToolResultChars #602, without raising any limit for other content.

Unchanged: text artifacts are truncated to maxToolResultChars, and an image above the ceiling or the remaining image budget is omitted with the existing notice. The image budget is shared per projection, like the text budget: across the tool batch for OpenAI/Google, per tool message for Anthropic. Cross-turn tool-image handling, history formatting and pruning are untouched.

Verification

Head: 6b56c65.

  • New src/messages/artifactImages.test.ts (both projections): a 1,250,000-character PNG (real 1280×720 header) under a 400,000-character budget is kept; a small image under a 200-character budget is kept; an image above the ceiling is omitted with the notice; text beside an image is still truncated to the budget; two 3,000,000-character images in one OpenAI/Google batch keep the first and omit the second.
  • Graph.contextOverflow.test.ts: a screenshot-sized artifact goes through Run.processStream with the real token counter (claude and o200k_base), 200k context and the default budget, and the image reaches the model call for Anthropic and OpenAI.
  • Negative control: without the core.ts/toolContent.ts change, 9 of the 11 new tests fail. The two "above the ceiling" cases pass either way.
  • npx tsc --noEmit, npm run test:circular-deps, ESLint on the touched files: passed. src/messages, src/utils and the Graph overflow suite: 442 passed.
  • Full npx jest locally: 8,229 passed. Failures are the credential-gated Vertex/Anthropic/Google suites and two macOS-specific tests (decisions/node.test.ts shell redaction, localToolResultCaps.test.ts grep fallback), which fail the same way on main.

Not run against live providers.

🤖 Generated with Claude Code

The OpenAI/Google and Anthropic artifact projections charged an image_url
block's base64 length against maxToolResultChars, the budget for tool
result text (default min(30% of the context window x 4, 400,000)). Any
image larger than that, such as a browser screenshot from an MCP tool
(roughly 300 KB and up), was replaced with an "[omitted 1 oversized
atomic tool-content block: image_url]" notice, so the model never saw it.

An image's context cost follows its pixel size, which the serving payload
measure already estimates (estimateImageBlockTokens) before the graph
accepts an artifact projection, and it retries without the artifact when
the payload does not fit.

- The projection accumulator keeps a separate image budget of
  HARD_MAX_TOTAL_TOOL_OUTPUT_SIZE characters, the inline image ceiling
  that provider image URLs and computer-use screenshots already use.
  A well-formed image_url block within it is kept without consuming the
  text budget; anything else takes the existing bounded path.
- getToolContentImageUrl() validates an image_url block with the same
  own-data-property checks as the computer screenshot reader, which now
  shares the helper.
- Text artifacts are still truncated to maxToolResultChars, and an image
  above the ceiling or the remaining image budget is still omitted with
  the existing notice.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tool artifact images larger than maxToolResultChars never reach the model

1 participant