Skip to content

fix(insights): keep conflicting measurements private - #851

Merged
izadoesdev merged 2 commits into
stagingfrom
codex/insights-measurement-publication
Sep 27, 2026
Merged

izadoesdev merged 2 commits into
stagingfrom
codex/insights-measurement-publication

Conversation

@izadoesdev

@izadoesdev izadoesdev commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Problem and behavior

A native goal or funnel read can cover fewer days than requested. The completion gate marks the investigation incomplete, but the agent can still publish the original full-period comparison, including by omitting the contradictory read from its citations.

Reject published product outcomes when successful original-population native reads, or cited same-subject diagnostic reads, contradict requested measurement dates, website, definition, or the evaluated definition across periods. The agent can return a private limitation using actual coverage. Independently inspected definition repairs remain allowed. This is an explicit metadata/definition contradiction veto, not a general numerical claim verifier.

Validation

  • Focused regression: 53 passed; Insights: 743 passed, 6 skipped.
  • Root lint, all 33 typecheck tasks, and all 29 test tasks passed. Excluded test-file typecheck: 46 existing baseline diagnostics, zero additions.
  • Frozen synthetic native comparison: 8/8 attempts returned, 25 Gateway responses, zero transport failures. Correctly private clipped outcomes improved from 0/2 to 2/2; full-window publication retained 2/2 in both arms. Candidate kept exact actual Sep1–4 versus requested Aug29–Sep4.
  • Baseline used 12 turns / 20 reads / 40.061s; candidate used 13 turns / 18 reads / 77.188s across four investigations each. Both negative controls needed one rejected finish. No speed gain is demonstrated; one positive candidate repeat omitted an optional source follow-up, and provider timing varies.
  • Independent local review found no actionable issue. Real production data is absent from tests and evaluation fixtures.

Scope and overlaps

Independent slice from current staging (020c04133), with no dependency on another feature branch. Overlaps #850 in apps/insights/src/agent.ts and SPEC.md, and #751 in the same agent/spec surface. #850 fixes funnel-step completion; this PR fixes publication validation. #849 changes selection separately. No model/prompt, completion, action/check, persistence, billing, schema or deployment change.

AI-assisted implementation and review; draft for maintainer review.

Current CI: all 19 contexts finished (18 successful, one neutral); no actionable inline threads or submitted reviews. Configured code review is incomplete: CodeRabbit skipped automatic review and cubic reached its monthly quota. The PR remains draft; no merge or production deployment.

@vercel

vercel Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
dashboard Ready Ready Preview Sep 27, 2026 4:08pm UTC
databuddy-status Ready Ready Preview Sep 27, 2026 4:08pm UTC
documentation Ready Ready Preview Sep 27, 2026 4:08pm UTC

@coderabbitai

coderabbitai Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ffdb0d8f-0fba-427a-85b8-0b5e80573a7f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@izadoesdev

Copy link
Copy Markdown
Member Author

@greptile review

Please review the current head c52b89936f4510da20788a8f26f6bd6343099426 for correctness, security, and regressions as part of the requested review of all open PRs. This PR should remain a draft; this request is for review only.

@greptile-apps

greptile-apps Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium risk] Adds validation logic for measurement conflicts in insights.

The PR is not ready to merge: the traffic-collapse exception can publish a coverage finding without evidence of a coverage problem, and explicit repository style requirements remain unmet.

Findings

  1. P1 Cohort ordering bypasses comparison ▶
  2. P2 Another nested ternary ▶
  3. P2 Test fixtures use unknown ▶
  4. P2 Nested ternary violates requirements ▶

Summary

The PR adds a publication veto for contradictory goal and funnel measurement metadata, normalizes filter ordering for comparisons, and expands investigation regressions. Later changes also adjust publication rules, evidence formatting, and definition-question handling.

  • The cohort-ordering issue from the previous review is fixed.
  • The earlier nested-ternary finding remains, and the new test adds a second occurrence.
  • The new traffic-collapse exception can publish an unsupported coverage finding.

Reviews (2) · Last reviewed commit: "fix(insights): compare reordered cohort ..."

Comment thread apps/insights/src/agent.ts
Comment thread apps/insights/src/investigation-flow.test.ts
@unkey-deploy

unkey-deploy Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Unkey Deploy

Name Status Preview Inspect Updated (UTC)
links (preview) Ready Visit Preview Inspect Sep 27, 2026 4:06pm

Comment thread apps/insights/src/investigation-flow.test.ts
Comment thread apps/insights/src/investigation-flow.test.ts
@greptile-apps

greptile-apps Bot commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

  • P1 Decline mistaken for coverage loss apps/insights/src/agent.ts:1938 ▶

    A week-over-week website traffic drop of at least 90% can now publish as a measurement-coverage finding with only the signal cited. The percentage shows a decline, not missing tracking or a missing population. A real decline can therefore be published as a coverage problem without collection or implementation evidence.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

  • P2 Comparison dates lose their year apps/insights/src/agent.ts:143 ▶

    This formatter omits the year from published revenue comparisons and saved-check evidence. For a comparison across years, readers lose the year that previously identified each measured window; dates with the same month and day in different years can even appear identical.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@izadoesdev
izadoesdev merged commit f9c2029 into staging Sep 27, 2026
21 checks passed
@izadoesdev
izadoesdev deleted the codex/insights-measurement-publication branch September 27, 2026 16:18
izadoesdev added a commit that referenced this pull request Sep 27, 2026
…review-gaps

test(insights): close review gaps from #864 and #851

This branch was successfully deployed

4 active deployments
Preview – dashboard — b92c8906 Deployed Sep 27, 2026 by vercel[bot]
Preview – documentation — b92c8906 Deployed Sep 27, 2026 by vercel[bot]
Preview – databuddy-status — b92c8906 Deployed Sep 27, 2026 by vercel[bot]
links - preview — b92c8906 Deployed Sep 27, 2026 by unkey-deploy[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant