Repository navigation
feat(analyst): answer_commit renders the final answer deterministically - #1449
anandgupta42 wants to merge 31 commits into
Conversation
…ally The analyst's final message was free prose shaped by a prompt paragraph, so the same computed values could be presented differently from run to run. `answer_commit` is an ordinary tool, available only to the analyst agent, that takes the single chosen value(s) (`fields`), an N-row table (`rows`) or `unresolved` with a reason, plus an optional two-sentence basis and `answer_only` flag, validates the payload structurally (one value per field, no empty values, rectangular rows, precision applied, at most two basis sentences, strict-schema fillers treated as absent) and returns the exact final text to reply with. An advisory `readings` ledger (clause, reading, result summary, decision) is kept in tool metadata and never rendered. - Session mechanics, message rendering, loop termination and tracing are unchanged; the tool only returns text. - Registered on the analyst-only gated candidate list, so builder, plan, data-explorer and custom agents cannot see it and `tool_lookup` does not list it; `ALTIMATE_ANSWER_COMMIT=0` removes it. - The analyst prompt's final paragraph now asks to finish by calling `answer_commit` and replying with exactly the returned text, with the previous prose contract kept as the fallback when the tool is unavailable. - Not covered: automatic removal in user JSON-schema mode (no hook exists where `StructuredOutput` is injected); documented in the tool description. Tests: 48 renderer/validation/tool cases, 4 registration and scoping cases, prompt-contract assertion updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… and sentence counting Review follow-ups: - The tool description says when to call it (last tool call, once the answer is decided), that the returned text is the verbatim final reply, what `answer_only` means, and that a user output schema takes precedence. - The prompt's unavailable-tool fallback restores the answer-only rule. - `null` and whitespace-only values for optional fields are treated as absent; whitespace-only `value`/`name` is the existing empty-value error. - Table cells escape `|` and a preceding backslash; newlines become spaces. - The two-sentence basis cap ignores common abbreviations and decimals. - Tests cover the above, a custom agent named `analyst` (receives the tool by design) and a same-id plugin tool. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ac8ff6196
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
…counting and precedence test tightened Independent-review follow-ups: - The tool output is exactly the rendered answer; the "reply with the returned text" instruction lives in the tool description, so the prompt and the tool no longer disagree about what to echo. - The basis sentence counter treats a boundary as `.`/`?`/`!` followed by whitespace (including newlines) and any non-space character, still ignoring common abbreviations and decimals; lowercase/newline cases tested. - The plugin-precedence test runs the real tool resolver instead of re-implementing its selection logic. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e84cb9d1f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
View guided diff | Turn on auto-fix | Re-trigger cubic
Code Review SummaryStatus: 14 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)CRITICAL
WARNING
SUGGESTION
Files Reviewed (17 files)
Changed files were inspected for this incremental review; findings in unchanged files were carried forward by targeted verification. No tests were executed in read-only mode. Previous Review Summaries (8 snapshots, latest commit 09e9581)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 09e9581)Status: 14 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)CRITICAL
WARNING
SUGGESTION
Files Reviewed (15 files)
Targeted carry-forward verification also covered Previous review (commit ca9d90a)Status: 16 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)CRITICAL
WARNING
SUGGESTION
Files Reviewed (10 files)
Previous review (commit fd0bc41)Status: 19 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)CRITICAL
WARNING
SUGGESTION
Files Reviewed (12 files)
Previous review (commit cc3dc50)Status: 12 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (6 files)
Previous review (commit c2f62d8)Status: 12 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (7 files)
Previous review (commit 83edc69)Status: 9 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (7 files)
Previous review (commit 2f93220)Status: 9 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (16 files)
Previous review (commit e84cb9d)Status: 7 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)CRITICAL| File | Line | I [Snapshot truncated.] Additional previous summary content was truncated to keep this comment within platform limits. Reviewed by gpt-6-sol · Input: 138 · Output: 63.9K · Cached: 9.3M Review guidance: REVIEW.md from base branch |
…step limit, uses native Effect Review-bot findings of the highest severity tier plus their siblings: - Numeric field values and table cells above `Number.MAX_SAFE_INTEGER` are rejected (JSON parsing would have rounded them); exact strings are accepted instead. - At the final allotted generation step, an analyst turn whose previous step completed `answer_commit` is asked to reply with exactly the returned text, with tools disabled and a user output schema still taking precedence, instead of being asked for a summary that would discard the committed answer. - The tool exposes the namespace projection the registry consumes, runs as a native Effect definition with native permission handling, and its tests use scoped Effect fixtures. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…' into feat/analyst-v10-commit
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d888f8144
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…mit root `$schema` for Anthropic SDKs A live probe against Bedrock Claude rejected the analyst tool list: `tools.45.custom.input_schema: JSON schema is invalid. It must match JSON Schema draft 2020-12`. The `answer_commit` optional-filler helper used `z.tuple([])`, which Zod emits as `"prefixItems": []`, invalid under 2020-12 (`prefixItems` needs at least one item). OpenAI accepted the same schema, so only Anthropic-family providers were affected. - Replace the empty tuple with `z.array(z.never()).length(0)`, which still accepts exactly an empty array and keeps union error selection. - Omit the root `$schema` declaration in the provider transform for the Anthropic, Bedrock and Vertex Anthropic SDKs. - Add `tool-schema-anthropic.test.ts`: every registered tool schema for the analyst and builder agents is validated against the draft 2020-12 meta-schema (ajv, dev dependency) plus Anthropic's restrictions, so an invalid tool schema fails in CI instead of at the provider. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…keep the committed text exact through compaction Two review findings of the highest severity tier, each with red-first tests: - The native `answer_commit` id is reserved: an MCP, plugin or file tool with the same name (including Windows case-equivalent spellings, and with the toggle off) cannot take the analyst's finalizer permission. - The committed answer text survives automatic compaction and provider retries unchanged; continuation ids are tracked so a newer request is not overwritten by an older echo. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…back for `answer_commit` Two contract regressions raised in review: - The analyst prompt again says a top-N with a tie at the boundary returns exactly N concrete rows with a brief tie note; the tool description and the `rows` schema say the same, with `rows.note` as the place for the note. - The unavailable-tool fallback requires exactly the requested values or table in the requested format, with no label and no basis, for answer-only or specific-format requests; `answer_only` rendering stays label-free. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8bc77c568
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…a basis, and explains attribution Findings from live Claude sessions on Bedrock: - Every call failed validation because the model sent `answer_only` as the string `"true"`. Boolean fields now accept case-insensitive `"true"`/`"false"` and `1`/`0` at runtime; the published schema type stays boolean. - Sessions that dropped the previous "state your basis" sentence bound a policy clause about a related object to the requested object's attribution. `basis` is now required unless `answer_only` is set and must name population, grain and the source object/column that supplies the requested entity or value; the tool description and the analyst finishing paragraph say so and separate policy eligibility from metric and entity attribution. - Validation failures return a one-line error plus a minimal valid example payload so the next call can repair instead of repeating the same input. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
3 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 270a65e123
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… is named Answer A single field named Answer, Result or Value (or left blank) renders as the plain answer line instead of repeating the label. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 1ded858c-7ec0-4d83-b5d1-b35ba29a72ec) |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
2 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
…rt when the summary already streamed A compaction summariser that streams partial text and then fails recoverably had already sent deltas for its text part; reusing that part id published the committed answer as another delta on it, and clients that do not treat a blank update as a reset showed the partial summary concatenated with the answer. - `finishCommittedAnswer` reuses a part id only for buffered (`deferText`) echo text; when the stored text was streamed it removes every stored text part and publishes under a fresh id - `run --output`: the collector tracks part ids and drops removed parts from the reply, the previous reply and the fallback; `run.ts` forwards `message.part.removed` - ACP: compaction-summary message text is no longer forwarded as agent chunks (ACP cannot retract chunks); clearing the summary flag on recovery lets the committed delta through exactly once; subscriptions that join mid-stream pick the flag up from the fetched message - tests: streamed partial summary + 503, and two empty summaries after streamed text, each deliver exactly the committed text to ACP and run output; collector removal cases; ordinary buffered echo still reuses its id and publishes once Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: f2e93ca2-2472-4dd4-8e60-92a1d7577e2f) |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
2 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09e9581d57
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… never finalize onto a completed compaction summary - `message-v2.ts`: an explicit policy code (`guardrail_intervened` incl. nested orchestration codes, `content_filter`, `content_policy*`, `policy_violation`, `safety_*`, `moderation_*`) keeps the error fatal even when the message text looks like a context overflow; explicit overflow codes (`context_length_exceeded`, `content_length_exceeded`) and HTTP 413 stay overflows; policy metadata survives response truncation - `prompt.ts`: when the message that would carry a recovered committed answer is a finished, error-free, non-empty compaction summary (a user request was queued during compaction), the summary stays intact as the compaction boundary and the committed text is published once on a new assistant message with the committed request's parent, agent, mode, model, variant and path; failed/empty summariser recovery still converts its own summary message - tests: 503 guardrail with overflow text during echo stays fatal without publishing or compacting; queued user after successful compaction keeps the summary and gets tools after the new committed-answer message; 413 `content_length_exceeded` still compacts and resumes; 21 `fromError` cases for code vs text precedence Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 0e108438-9a94-4db5-8d38-85d0d69d75ef) |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
1 similar comment
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 39e618a057
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… queued during compaction The round-19 path that publishes a recovered committed answer on a new assistant message timestamped it with the current time, which sorted it after a user request queued during compaction, so later prompts replayed it as the reply to that newer request. The new message now takes the earliest newer user request's persisted `time_created` minus one millisecond (compaction markers and synthetic continuations excluded), completion never precedes creation, and the loop reloads history after the recovery so the queued request sees the answer in its original place. Test: persisted order is summary, recovered answer, queued user; the queued request's model history shows the committed text before its own turn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: e7492d6f-67bd-4d0e-bd0f-ac58cf698f7f) |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
…erflow The overflow check searched every nested `code` in the response body, so a diagnostic field such as `details.request.code: context_length_exceeded` could mask a top-level `policy_violation` and let the echo guard publish a committed answer against a provider denial. Overflow precedence now reads only `error.code` or the root `code` (plus HTTP 413); policy codes are still matched anywhere in the body. Tests: a nested diagnostic overflow code never masks a top-level policy denial; the explicit top-level overflow cases are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 11926dbc-c43c-43b9-ae17-e5931ddf2f39) |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
2 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 669946ca8a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| recovered = await finishCommittedAnswer( | ||
| exiting.info, | ||
| [], | ||
| loopFailure ? MessageV2.fromError(loopFailure.error, { providerID: exiting.info.providerID }) : exiting.info.error, | ||
| ) |
There was a problem hiding this comment.
Continue queued requests after compaction recovery
When a new user request is queued while a post-commit compaction is running, a recoverable summarizer failure makes the compaction branch break out of the loop; this exit guard publishes the previous request's committed answer but then proceeds to resolve every callback in the generation with that answer. The queued caller therefore receives the stale answer and its persisted request is never processed by this loop. After recovering the commit, recheck for newer user requests and continue the loop rather than exiting.
Useful? React with 👍 / 👎.
Issue for this PR
Closes #
Type of change
What does this PR do?
Why. The analyst's final message was free prose shaped by a prompt paragraph, so the same computed values could be presented differently from run to run: sometimes with the requested fields only, sometimes with extra commentary, excluded candidates or alternative readings. Users and downstream consumers get a consistent final answer only if its rendering is code, not prose.
What changed. A new tool,
answer_commit, available only to theanalystagent. The model calls it last with the single chosen value(s) (fields, one field per item the request asks for, in the order asked, so a request for "the repository and the most copied file" carries two fields and a compound request is never collapsed into one), an N-row table (rows) orunresolvedwith a reason, plus a one-or-two-sentencebasis(population, grain, and the source object/column that supplies the requested entity or value) and ananswer_onlyflag.basisis required on every payload, includinganswer_onlyandunresolved; it is rendered before the answer line in the normal case and never rendered when the request asks for only the answer. It is the model's reasoning input, not output: live sessions showed that when the basis was skipped, a rule about a related object displaced the requested object's attribution, and that sessions which wrote the basis kept the requested object. Boolean fields accept the string forms"true"/"false"and1/0, because live Claude sessions on Bedrock send them that way; a validation failure returns a one-line error plus a minimal valid example. The tool validates the payload structurally (exactly one of the three shapes; one value per field; no empty values; rectangular rows;null,"",[]and whitespace fillers from strict-schema clients treated as absent) and returns exactly the rendered final text. Numbers with more than 15 significant digits or outside the safe-integer range are rejected with instructions to pass the exact value as a string, because JSON parsing has already changed them (1234567890123456.7arrives as…456.8); digits lost before the tool runs (0.10000000000000001arrives as0.1) cannot be detected, so the description states the contract: numbers up to 15 significant digits, exact decimals as strings. The complete rendered text is capped atMAX_RENDERED_ANSWER_CHARACTERS(60,000); over the cap the model is told to aggregate, narrow, or write the full table to a file and commit the summary, so an echo can never be truncated at the output limit.basisis bounded by length (400 characters) rather than a punctuation-based sentence count, and the precision range (0 to 10) is stated in the description and the validation message.rowsaccepts an optional per-columnprecisionmap so numeric cells render with fixed decimals (5→5.00), after live sessions showed a numeric cell rendered differently from the equivalent string; every precision key must name a column exactly and column labels must be unique when precision is used.rows.formatselects the table's text form when the request names one:markdown(default),csv,tsv(header row included, RFC 4180 quoting with embedded quotes doubled, null cells empty, precision honoured) orlines(one row per line, cells separated by one space, no header); withanswer_onlythe reply is exactly the formatted table, so a request for CSV is not answered with a Markdown table that the terminal tool cannot repair. Rendering is a pure function (renderAnswer) with golden tests; markdown table cells are escaped; a single field literally named "Answer"/"Result"/"Value" is not double-labelled. An advisoryreadingsledger (clause, reading, result summary, decision) is kept in tool metadata and never rendered.Scoping: the tool is added on the analyst-only gated candidate list, not the global catalogue, so builder, plan, data-explorer and custom agents cannot see it and
tool_lookupdoes not list it; execution also rejects non-analyst callers; a custom agent namedanalystreceives it by design. The native id is reserved, so an MCP, plugin or file tool with the same name (including case-equivalent spellings on Windows) cannot take the finalizer permission.ALTIMATE_ANSWER_COMMIT=0removes it. The analyst prompt's final paragraph asks to finish by callinganswer_commitwith one field per requested item and replying with exactly the returned text, always supplyingbasis; the previous prose contract (answer-only rule and top-N tie rule included) remains the fallback when the tool is unavailable.Session loop, scoped to the analyst and marked in
session/prompt.tsandsession/compaction.ts: (1) once the lastanswer_commitattempt for the current request has completed, the next generation runs tool-free with the instruction (sent as the final user turn) to reply with exactly the returned text, whether or not a step limit or compaction intervened; a failed attempt, including a failed retry after a success, keeps the tools for repair; a user request that arrives before the echo is sent, or while it streams, clears the commit and is answered normally; the echo generation's text is buffered rather than streamed, and after astoporlengthfinish the loop publishes one authoritative text part (start, delta, completion) holding the stored committed output, sorun,--outputand ACP each receive the committed text exactly once and the final text never depends on the model's output budget; one guard finalizes from the stored text whenever the request would otherwise end after a completed commit for any reason other than an abort or a content filter (a failing compaction summariser, a summariser that returns no text twice, a context overflow on the echo, a thrown preparation error, a transient transport failure), recording the cause as assistant-message metadata (answer_commit_echo_error); transport recovery is structural, not phrase-based: only 5xx, 429, 408 or statusless network failures recover, while any 4xx, explicit policy codes (policy_violation,content_filter,content_policy*,guardrail_intervenedincluding nested orchestration codes,safety_*,moderation_*), auth errors and local pre-request errors stay fatal; an explicit structured policy code wins over overflow-looking message text and stays fatal, while only the provider's own error code (error.codeor the rootcode, never a nested diagnostic field) or HTTP 413 classifies a context overflow, so a provider 413 orcontent_length_exceededstill compacts and apolicy_violationwrapped around overflow text does not; a recoverable echo or compaction-summariser failure after a commit never persists a transient fatal error orsession.error; the committed text is persisted before its message is marked finished; a recovered answer reuses a text part only when that part was buffered (never streamed), otherwise every streamed part is removed and the answer is published under a fresh part id; when the message that would carry it is a completed, error-free compaction summary (a request was queued during compaction) the summary stays intact as the compaction boundary and the answer is published on a new assistant message that is persisted before the queued request (one millisecond before its creation time) so later history still shows it as the reply to the original request, therun --outputcollector drops removed parts from the reply and its fallbacks, and ACP (which cannot retract chunks) no longer forwards compaction-summary text as agent chunks, letting the committed delta through exactly once when the summary flag is cleared; a request that arrives during the echo still gets the earlier request's committed answer first and is then processed with tools; "newer request" is decided by the persisted creation time and insertion sequence, never by comparing caller-supplied message ids; an echo that itself triggers reactive compaction is finalized from the committed text instead of retried; an error-freeotherfinish also delivers the committed text but keepsotheras its finish reason, and an unmapped (unknown) finish delivers it and ends asstop; aborted andcontent-filterechoes keep their text and finish reason; the echo instruction never carries a copy of the committed output, so a compacted context cannot overflow on it;run --max-turns Ntracks a completedanswer_commitfor the active request from the stream (no message fetch on the step-start path) and exempts its echo, including a retry after a reactive overflow, until any new real user request or a failed later commit attempt clears it; real request persistence emitssession.request.startedso a request from another attached client also clears it; the complete rendered text is also kept in the tool result's metadata, which the per-model tool-output cap does not touch, so a long answer survives even where the model-facing output is capped; (2) the committed text is preserved verbatim through automatic compaction and provider retries, with continuation ids tracked so a newer request is not overwritten by an older echo; (3) requests that supply an output JSON schema never seeanswer_commit: the tool is removed whenStructuredOutputis injected, echo mode is never entered for them, and earlier commit tool parts are passed to the model as masked text so a historical stub cannot re-advertise the tool; a text question that merely mentions a JSON schema is unaffected. Loop termination,--outputcollection and tracing are otherwise unchanged.Also in this PR: a repository-wide test (
tool-schema-anthropic.test.ts) that validates every registered tool's JSON schema for the analyst and builder agents against JSON Schema draft 2020-12 plus Anthropic's restrictions, after a live probe showed Bedrock rejecting the first version of this tool's schema (z.tuple([])emitsprefixItems: []); the provider transform now omits the root$schemafor Anthropic-family SDKs.ajvis added as a dev dependency for that test.Base. Stacked on
feat/analyst-harness-dab-2026-10(#1441); the diff shown here is only this feature.Behaviour deltas (was → now). Analyst text requests: final message produced by a prompt paragraph → produced by calling
answer_commitand echoing its text on a tool-free next step (fallback to the old paragraph when the tool is off/unavailable); at the step limit, summary request → echo of the committed answer; compaction may rewrite the final text → committed text preserved. Analyst JSON-schema requests:answer_commitoffered alongsideStructuredOutput→ not offered. Payloads: any JSON number accepted → more than 15 significant digits rejected; any size → rendered text over 60,000 characters rejected. Tables: always Markdown → Markdown, CSV, TSV or plain lines as requested. ACP clients: compaction-summary text streamed as agent chunks → not forwarded (all agents; the summary is internal compaction machinery and a later recovery may replace it). Other agents: otherwise unchanged. Registry: one analyst-gated candidate added, its id reserved against external tools. Permissions: one analyst allow entry. Anthropic-family requests: root$schemano longer sent.How did you verify your code works?
Unit/integration, run outside any sandbox, after an independent adversarial review (approve-with-changes) and an Engineer independent review, with every bot P0/P1 through three rounds fixed:
test/altimate/tools/answer-commit.test.ts: renderer goldens for every shape, validation errors (empty value, ragged rows, two-of-three shapes, missing basis incl. answer-only, basis over 400 characters, precision outside 0 to 10 naming the range, parsed precision-loss literals, rendered-size cap at 59,999 / 60,000 / over with escaped cells and basis counted), filler tolerance, boolean-string coercion, label dedupe, per-column row precision (integers to fixed decimals, strings untouched, independent per column), row formats (CSV with a comma and a quote in one cell, TSV with a tab, lines, precision in CSV, Markdown default, size cap after quoting), pipe/backslash escaping, empty-tuple regression, tool-description contract incl. the one-field-per-requested-item wording on the description and thefieldsschema.test/agent/answer-commit.test.ts: analyst sees the tool; builder does not;tool_lookupdoes not list it; toggle off removes it; customanalystagent receives it; real-resolver plugin precedence; MCP/plugin/file id reservation incl. Windows case; echo right after a successful commit without a step limit and with a sibling tool; failed commit and success-then-failed-retry keep tools; new user message resets, including one persisted after the snapshot or during the echo; length-limited or mismatched echo replaced with the stored text and delivered exactly once to therunoutput collector and ACP deltas;stopecho without text yields a completed timed part; error-freeotherfinish delivers the committed text once and keeps its finish; non-retryable echo transport failure or two empty compaction summaries publish the committed output once and record the cause in metadata (the no-commit empty-summary control keeps its error), while aborted, content-filtered and HTTP-policy-rejected echoes keep their fatal error; commit under a 100-token tool-output cap still yields the full table; step-limit echo; compaction/retry preservation with no full-answer copy in the request; JSON-schema requests omit the tool, never echo, and keep masked commit history; a streamed partial compaction summary followed by a 503, and two empty summaries after streamed text, each deliver exactly the committed text to the real ACP handler and the run collector on a fresh part id with every streamed part removed; collector removal cases for the reply, previous reply and fallback; a 503 guardrail denial with overflow text during the echo stays fatal without publishing or compacting; a queued user after successful compaction keeps the summary, receives the committed answer on a new message ordered before it, and is then processed with tools; 23fromErrorcases for policy-code versus overflow-code precedence, including a nested diagnostic overflow code that must not mask a top-level denial.test/altimate/tools/tool-schema-anthropic.test.ts: all registered tool schemas validate (analyst and builder, with and without search tools).test/session/prompt.test.ts(structured-output case updated to assertanswer_commitis absent),test/mcp/lifecycle.test.ts, prompt-contract assertion intest/altimate/sql-validation-e2e.test.ts.test/cli/run/budget-finalization.test.ts:--max-turns 1run publishes the committed output on its echo turn; a run without a commit still aborts at N+1. Help snapshot updated for the option text.Results at the current head: 142 pass / 0 fail across
test/session/message-v2.test.tsand the answer_commit agent suite, and 182 pass / 0 fail across the agent suite,test/session/prompt.test.tsand the run budget tests one commit earlier; the ACP and run-output suites ran green (204 pass) at the head that introduced them; the previous head ran 335 pass / 0 fail on the answer_commit tool suite, the Anthropic schema test andtest/altimate/sql-validation-e2e.test.ts(prompt contract); the previous head ran 377 pass / 0 fail withtest/session/prompt.test.ts, the run budget tests and the help snapshots included (the wholetest/sessiondirectory last ran green at 1,425 pass two commits earlier);bun run typecheckclean; every edit in upstream-shared files (tool/registry.ts,agent/agent.ts,session/prompt.ts,session/processor.ts,session/compaction.ts,mcp/index.ts,provider/transform.ts) insidealtimate_changemarkers.Live model evidence (Bedrock, Claude Opus 5.5, analyst agent, this branch): the first live session on the round-3 build committed successfully and then failed on the tool-free echo request with HTTP 400 "conversation must end with a user message", because the echo instruction was appended as an assistant turn; the instruction is now the final user turn, and the repeated live session committed once with no validation error, stopped normally, and produced a final text byte-identical to the tool's rendered output (9.4 s). The mocked-provider tests passed on both builds, so the agent test now also asserts the echo turn's role. Earlier, 65 live sessions on the previous build called the tool exactly once each with zero validation errors. Not run in CI: any live model session (no network).
Screenshots / recordings
Not a UI change.
Checklist
🤖 Generated with Claude Code