Skip to content

answer_commit drops requested elements that appear only in basis, and loses trailing zeros in numeric cells #1481

Open

Description

@anandgupta42

Problem

Two independent defects in answer_commit (packages/opencode/src/altimate/tools/answer-commit.ts, present on the analyst-commit branches, not yet on main) cause a correct computation to be reported as an incomplete or differently formatted final answer.

  1. Compound requests lose elements. With answer_only: true the basis is never rendered; only fields / rows reach the final output. When a question asks for two things ("the repository and the most frequently copied file"), models commit one field and place the second item only in basis. The second item is silently dropped.
  2. Trailing zeros are lost. Cell values are z.union([z.string(), number, z.null()]) and rendered with String(value). A JSON number 5.00 reaches the tool as JS 5 and renders as 5, next to sibling cells such as 4.18. precision is opt-in, so nothing restores the scale.
  3. Valid numbers are rejected. Numbers with more than 15 significant digits (for example a float32 artefact 18.440000534057617 printed by sql_execute) are rejected until the model re-sends them as strings.

Mechanism

  • Numeric validation: the 15-significant-digit check at about answer-commit.ts:31; rows schema union at about :56-61; precision optional at :37-46 (line numbers from branch feat/analyst-v10-commit).
  • answer_only rendering skips basis, so a requested element that exists only there never reaches the output.

Evidence

Recorded analyst sessions:

  • Compound request: 5 of 5 sessions that used answer_commit committed only the repository field and returned an 18-byte answer, while the same model without the tool answered with both items in all 5 sessions. 4 of the 5 committed the same repository as the correct answers.
  • Trailing zero: in 5 sessions on a question whose answer has averages like 4.18 and 5.00, 3 sessions passed the number 5 and returned an incorrect answer; the 2 that passed the string "5.00" were correct. The source SQL had printed 5.000000 and 5.0.
  • 9 further calls were rejected for passing 15+ significant digit doubles or float32 artefacts.

Impact

Answer-losing: about 8 incorrect answers in the recorded sessions that were otherwise computed correctly. No time or cost effect.

Proposed fix

  • Before rendering with answer_only, compare the number of requested outputs (parse "X and Y", "for each ...") with the committed fields; on mismatch reject with the list of missing targets, or fall back to rendering the basis.
  • Render each numeric column with the maximum decimal scale present in that column (or the source result's scale).
  • Accept numbers with more than 15 significant digits and round to precision instead of rejecting.

Acceptance check

  • Unit test: rows with cells 5 and 4.18 renders 5.00 and 4.18 when the column scale is 2.
  • Unit test: a two-target question committed with one field is rejected naming the missing target (or renders the basis).
  • Unit test: a float32 artefact value is accepted and rounded.

Applies to the answer_commit tool introduced in #1449.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions