Skip to content

perf(insights): present structured evidence before prose - #859

Merged
izadoesdev merged 1 commit into
stagingfrom
codex/insights-structured-evidence
Sep 27, 2026
Merged

izadoesdev merged 1 commit into
stagingfrom
codex/insights-structured-evidence

Conversation

@izadoesdev

@izadoesdev izadoesdev commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Revenue investigations can spend their remaining turns submitting prose after reading native monetary evidence, which requires a structured claim. Put the existing revenue and retention object alternatives before prose in both finish schemas so the model sees those forms first. This is schema presentation, not a parser fix; valid structured claims already parsed correctly.

The change moves four lines in one file. It does not change accepted claim forms, native rendering, validation, publication rules, or the investigation budget.

Validation:

  • Root lint, type checks (33 tasks), and tests (29 tasks) pass; Insights has 692 passing tests and 6 skipped live tests. Focused review covered 67 claim/revenue and 41 retention checks, plus actual-schema valid/invalid objects.
  • Eight frozen synthetic trials reused two known cases with two repeats per arm. Accepted outcomes were 3/4 candidate versus 2/4 baseline; finish rejections were 3 versus 9; model turns were 17 versus 21. Full-window monetary comparisons were correct in 2/2 candidate outcomes versus 1/2 baseline.
  • Observed total time was 61.437s versus 86.940s with mocked tool I/O. This small confirmation does not establish production latency or broad quality improvement. Clipped-window handling remained inconsistent at 1/2 accepted in each arm; one candidate summary added an unsupported implication, and one accepted clipped result cited only the signal. Retention and prose behavior have compatibility tests, not live behavioral parity evidence.

The separate first-party live comparison is pending authorization and has not run. This remains a draft.

Based directly on current staging. No feature dependencies. Known shared-file overlaps: #849, #850, #851, and #751 modify Insights agent behavior; this slice only reorders the two claim unions and does not include those changes.

AI assistance: Codex implemented and reviewed the change and ran the checks and synthetic evaluations. Human review remains pending.


Summary by cubic

Reorders evidence claim schemas so structured revenue and retention objects appear before the prose variant in both finish schemas.

  • Affects only schema union ordering in apps/insights/src/agent.ts; no changes to accepted claim forms, validation, native rendering, or investigation budget.
  • Intended to reduce wasted turns where the model submits prose after seeing native monetary evidence that requires a structured claim.

Written for commit 4323a51. Summary will update on new commits.

Review in cubic

@vercel

vercel Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
databuddy-status Ready Ready Preview Sep 27, 2026 4:01pm UTC
2 Skipped Deployments
Project Deployment Actions Updated
dashboard Skipped Skipped Sep 27, 2026 4:01pm UTC
documentation Skipped Skipped Sep 27, 2026 4:01pm UTC

@coderabbitai

coderabbitai Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 2cfee080-6088-4180-978b-a31d5f775347

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@izadoesdev

Copy link
Copy Markdown
Member Author

@greptile review

Please review the current head de7cc01efd0116d3f8e674d3938c504032785807 for correctness, security, and regressions as part of the requested review of all open PRs. This PR should remain a draft; this request is for review only.

@greptile-apps

greptile-apps Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Reorders schema validation union types in insights agent.

The PR appears safe to merge based on the reviewed schema changes.

Summary

This PR puts structured revenue and retention claims before prose in two Insights finish-schema unions. It changes schema presentation without changing the accepted claim forms or their rendering.

Reviews (2) · Last reviewed commit: "perf(insights): present structured evide..."

@unkey-deploy

unkey-deploy Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Unkey Deploy

Name Status Preview Inspect Updated (UTC)
links (preview) Ready Visit Preview Inspect Sep 27, 2026 4:00pm

@izadoesdev
izadoesdev marked this pull request as ready for review September 27, 2026 16:01
@izadoesdev
izadoesdev merged commit 193bed2 into staging Sep 27, 2026
21 checks passed
@izadoesdev
izadoesdev deleted the codex/insights-structured-evidence branch September 27, 2026 16:07

This branch was successfully deployed

2 active and 2 inactive deployments
Preview – databuddy-status — 4323a510 Deployed Sep 27, 2026 by vercel[bot]
links - preview — 4323a510 Deployed Sep 27, 2026 by unkey-deploy[bot]
Preview – documentation — 4323a510 Deployed Sep 27, 2026 by vercel[bot]
Preview – dashboard — 4323a510 Deployed Sep 27, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant