Skip to content

📊 feat: Preserve Response Accounting and History - #17599

Merged
dustinhealy merged 10 commits into
feat/execution-ownershipfrom
feat/response-accounting-history
Nov 11, 2026
Merged

dustinhealy merged 10 commits into
feat/execution-ownershipfrom
feat/response-accounting-history

Conversation

@dustinhealy

@dustinhealy dustinhealy commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Stored foreground Responses could not be continued or retrieved by their returned response IDs, and saving discarded tool results and message ancestry. Completion also omitted accounting and reported store: false even after successful writes. Connect the foreground controller to a typed history, persistence and terminal lifecycle so JSON, SSE and retrieved responses carry the saved turn and its accounting.

How it works

The foreground controller resolves an owned previous reply or conversation, preserves message ancestry and completed tool results, and records usage/accounting with the saved turn. JSON, SSE and retrieved Responses agree on the persisted result. Optional save failures preserve the generated answer with store: false; history lookup failures do not silently start a new conversation.

Type of change

  • Feature
  • Bug fix

Testing

Tested environments/configuration: Local macOS with CI Node.js 24.22.1, MongoDB replica-set fixtures and the actual execution host, SDK history formatter and Responses event handlers.

Automated tests:

  • 248 Responses and execution lifecycle tests pass, including real MongoDB persistence, owner/tenant isolation, item identity, role-preserving history actual SDK tool replay and a real MongoDB account-erasure race.
  • 199 controller, callback, routing and capability regression tests pass. Foreground composition uses the real execution host, Responses handlers, billing contextualizer and SDK artifact handler with production cleanup enabled for JSON/SSE success, each data writer under cleanup ownership, resolved save failures, cancellation and outcome-readback outages.
  • API typechecking, explicit import sorting/checks on all 21 changed source/test files and scoped static checks pass. Static checks retain four configured skips; the owning typecheck runs separately.
  • Ordered dependency/frontend builds and Lighthouse with injected database latency pass. The final API rebuild and suites use the committed typed completion service.

Screenshots / recordings

No standalone UI change.

Risk / compatibility

Terminal publication waits for provider drain, trailing writes and requested storage. Legacy conversation-ID continuations remain supported. Caller-executed tools retain their explicit replay contract. Optional accounting never fabricates a complete amount from partial coverage, and historical calls retain their recorded quotes rather than being repriced.

Checklist

  • I reviewed my own changes
  • Relevant tests have been added or updated
  • Existing relevant tests pass
  • The change does not introduce new warnings or errors
  • User-facing or complex behavior is documented where necessary
  • Required external dependency changes: N/A
  • Required documentation PR: N/A

@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 014c38b to 374b07c Compare October 29, 2026 13:04
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 374b07c to b8ba493 Compare October 29, 2026 13:09
@dustinhealy
dustinhealy added this pull request to stack #17619 October 29, 2026 13:13
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from b8ba493 to 2f35174 Compare October 29, 2026 13:38
Copilot AI balanced review requested due to automatic review settings October 29, 2026 13:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Missing production integration leaves retrieval, continuation, tool results, and cumulative accounting incomplete, while accounting merges can incorrectly report partially covered costs as complete.

6 open issues
What changed in this PR

Adds Responses API helpers intended to preserve stored output history and cumulative usage across continuations, while distinguishing complete accounting from unavailable data.

Changes:

  • Adds accounting collection, billing quotes, and segment-composition helpers.
  • Adds response-reference resolution, history selection, and message-projection utilities.
  • Extends terminal response builders and tool-completion handlers, with focused regression tests.
File Description
packages/​api/​src/​agents/​responses/​service.ts Adds completion handling, accounting parameters, and background-request validation.
packages/​api/​src/​agents/​responses/​segments.ts Adds saved-prefix composition and cumulative usage merging.
packages/​api/​src/​agents/​responses/​segments.spec.ts Tests segment history, identity, usage, and accounting preservation.
packages/​api/​src/​agents/​responses/​persistence.ts Adds response resolution, ancestry selection, and stored-message projection.
packages/​api/​src/​agents/​responses/​persistence.spec.ts Tests persistence references, history branches, and tool-result projection.
packages/​api/​src/​agents/​responses/​index.ts Exports persistence and accounting helpers.
packages/​api/​src/​agents/​responses/​handlers.ts Adds terminal accounting metadata and stored-response flags.
packages/​api/​src/​agents/​responses/​clientTools.ts Updates comments describing stored tool-history behavior.
packages/​api/​src/​agents/​responses/​accounting.ts Adds usage provenance, billing quotes, and coverage tracking.
packages/​api/​src/​agents/​responses/​accounting.spec.ts Tests pricing, provenance, deduplication, and unavailable accounting.
packages/​api/​src/​agents/​responses/​__tests__/​service.test.ts Tests consistent terminal accounting metadata in JSON and SSE.

Review effort: Balanced

@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 2f35174 to 0c67e1e Compare October 29, 2026 14:21
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 0c67e1e to 5f82166 Compare October 29, 2026 14:56
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 5f82166 to 252cc5f Compare October 29, 2026 15:16
@github-actions

Copy link
Copy Markdown
Contributor

Lighthouse CI failed. The last 80 log lines contain the measured budgets and assertion failures.

│ 23      │ 'http://localhost:3080/api/permissions/mcpServer/effective/all'                                                 │ 3276.3589999999967 │ 4045.475000000006  │ 200    │
│ 24      │ 'http://localhost:3080/api/prompts/groups?limit=10'                                                             │ 3276.6940000000177 │ 4556.694000000018  │ 200    │
│ 25      │ 'http://localhost:3080/api/keys?name=openAI'                                                                    │ 3276.8830000000016 │ 3796.1149999999907 │ 200    │
│ 26      │ 'http://localhost:3080/api/presets'                                                                             │ 3277.845000000001  │ 3796.460000000021  │ 200    │
│ 27      │ 'http://localhost:3080/api/tags'                                                                                │ 3280.5230000000156 │ 3879.44200000001   │ 200    │
│ 28      │ 'http://localhost:3080/api/share/link/16390000-0000-4000-8000-000000000001'                                     │ 3282.103000000003  │ 4306.168000000005  │ 200    │
│ 29      │ 'http://localhost:3080/api/messages/16390000-0000-4000-8000-000000000001?toolPreviews=1'                        │ 3283.67300000001   │ 4388.986000000004  │ 200    │
│ 30      │ 'http://localhost:3080/api/files/config'                                                                        │ 3283.842000000004  │ 4053.8000000000175 │ 200    │
│ 31      │ 'http://localhost:3080/api/user/settings/favorites/tools'                                                       │ 3284.0350000000035 │ 4557.418999999994  │ 200    │
│ 32      │ 'http://localhost:3080/api/endpoints/token-config'                                                              │ 3284.213000000018  │ 4315.981           │ 200    │
│ 33      │ 'http://localhost:3080/api/user/settings/skills/active'                                                         │ 3284.363000000012  │ 4823.143000000011  │ 200    │
│ 34      │ 'http://localhost:3080/api/agents/tools/web_search/auth'                                                        │ 3284.5350000000035 │ 7345.192999999999  │ 200    │
│ 35      │ 'http://localhost:3080/api/agents/tools/calls?conversationId=16390000-0000-4000-8000-000000000001'              │ 3286.1449999999895 │ 4897.638000000006  │ 200    │
│ 36      │ 'http://localhost:3080/api/agents/chat/status/16390000-0000-4000-8000-000000000001?generationProtocolVersion=2' │ 4510.3490000000165 │ 4814.174999999988  │ 200    │
└─────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────┴────────────────────┴────────┘

Inspect .lighthouse HTML/JSON and e2e/lighthouse/README.md. Reuse loaded user/config data; overlap independent reads without bypassing authorization.

┌─────────┬────────────────────────────┬─────────────────────┬───────┐
│ (index) │ audit                      │ median              │ limit │
├─────────┼────────────────────────────┼─────────────────────┼───────┤
│ 0       │ 'largest-contentful-paint' │ 4525.377            │ 4500  │
│ 1       │ 'cumulative-layout-shift'  │ 0.01790116740361323 │ 0.1   │
│ 2       │ 'total-blocking-time'      │ 409.63599999999997  │ 500   │
└─────────┴────────────────────────────┴─────────────────────┴───────┘

  1) [chromium] › e2e/lighthouse/load.spec.ts:10:5 › serial database latency stays within web-vitals budgets 

    Error: Median largest-contentful-paint must stay within 4500

    expect(received).toBeLessThanOrEqual(expected)

    Expected: <= 4500
    Received:    4525.377

       at audit.ts:127

      125 |   console.table(measured);
      126 |   for (const { audit, median, limit } of measured) {
    > 127 |     expect(median, `Median ${audit} must stay within ${limit}`).toBeLessThanOrEqual(limit);
          |                                                                 ^
      128 |   }
      129 |   return results;
      130 | }
        at auditPage (/home/runner/work/LibreChat/LibreChat/e2e/lighthouse/audit.ts:127:65)
        at /home/runner/work/LibreChat/LibreChat/e2e/lighthouse/load.spec.ts:33:19

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    e2e/lighthouse/.test-results/load-serial-database-latency-stays-within-web-vitals-budgets-chromium/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    Error Context: e2e/lighthouse/.test-results/load-serial-database-latency-stays-within-web-vitals-budgets-chromium/error-context.md

    attachment #3: trace (application/zip) ─────────────────────────────────────────────────────────
    e2e/lighthouse/.test-results/load-serial-database-latency-stays-within-web-vitals-budgets-chromium/trace.zip
    Usage:

        npx playwright show-trace e2e/lighthouse/.test-results/load-serial-database-latency-stays-within-web-vitals-budgets-chromium/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────


🤖: global teardown has been started
2026-10-29 15:22:55 �[32minfo�[39m: �[32mMongo Connection options�[39m
2026-10-29 15:22:55 �[32minfo�[39m: �[32m{�[39m
�[32m  "bufferCommands": false�[39m
�[32m}�[39m
🤖:  ✅  Connected to Database
🤖:  ✅  Found user in Database
🤖:  ✅  Deleted 1 convos & 2 messages
🤖:  ✅  Deleted user from Database
🤖: global teardown has been started
2026-10-29 15:22:56 �[32minfo�[39m: �[32mMongo Connection options�[39m
2026-10-29 15:22:56 �[32minfo�[39m: �[32m{�[39m
�[32m  "bufferCommands": false�[39m
�[32m}�[39m
🤖:  ✅  Connected to Database
🤖:  ⚠️  User not found in Database
  1 failed
    [chromium] › e2e/lighthouse/load.spec.ts:10:5 › serial database latency stays within web-vitals budgets 

Open the full run

@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch 2 times, most recently from 7ebf027 to db50d58 Compare October 29, 2026 16:38
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from db50d58 to 6a664b2 Compare October 30, 2026 05:12

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T06:59:26.575012Z 857dbbb Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a664b2c18

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 46a4184dec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 857dbbb89c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@dustinhealy
dustinhealy marked this pull request as ready for review October 30, 2026 11:00
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 4c1af23 to 207d9bc Compare November 5, 2026 06:53
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 207d9bc to 8075c9d Compare November 10, 2026 09:18
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 8075c9d to 8764d82 Compare November 10, 2026 22:53
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 8764d82 to cf36b38 Compare November 10, 2026 23:09
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from cf36b38 to 4067667 Compare November 10, 2026 23:41
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 4067667 to 672ed05 Compare November 10, 2026 23:44
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 672ed05 to 6f6763b Compare November 11, 2026 00:04
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 6f6763b to 1444f4a Compare November 11, 2026 01:46
@dustinhealy
dustinhealy force-pushed the feat/response-accounting-history branch from 1444f4a to 4c26053 Compare November 11, 2026 04:13
@dustinhealy
dustinhealy merged commit 7e51963 into dev Nov 11, 2026
44 checks passed
@dustinhealy
dustinhealy deleted the feat/response-accounting-history branch November 11, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants