chore: add script comparing new evaluator prompt against historical decisions (CM-1470)

[ulemons]

Summary

Adds a one-off script to measure how the new TS/Bedrock evaluator prompt (CM-1470) agrees with the decisions the old external Python evaluator agent already made in production, without needing to re-invoke that old agent.

Changes

• backend/src/bin/scripts/compare-evaluator-accuracy.ts: reads projectCatalog rows evaluated before the CM-1470 deploy cutoff (2026-09-18), samples them per decision category (action + evaluationReason) so a small run still covers every category, re-runs each through POST /v1/project-evaluation, and reports agreement/accuracy — flagging "already onboarded" / "already part of LF" rows separately, since the new prompt has no CDP lookup tool and can't structurally reach those two reasons.
• backend/package.json: adds script:compare-evaluator-accuracy and script:compare-evaluator-accuracy:local pnpm entries, following the existing script:* convention.

[cursor]

PR Summary

Low Risk
Adds manual script and pnpm entries only; no changes to runtime evaluation logic, though running it hits the DB and authenticated evaluation API (LLM cost).

Overview
Adds a one-off operational script to benchmark the new TS/Bedrock project evaluator against historical projectCatalog decisions from before the CM-1470 deploy cutoff (2026-09-18), without re-running the old Python agent.

The script stratified-samples pre-cutoff onboard/skip rows (by action + evaluationReason), re-calls POST /v1/project-evaluation with configurable concurrency, and writes a JSON report with agreement metrics plus token/latency cost summaries. Rows whose baseline reason was “already onboarded” / “already part of LF” are flagged separately because the new prompt cannot produce those CDP-only reasons.

backend/package.json gains script:compare-evaluator-accuracy and a :local variant (and fixes a trailing comma on the mailing-list local script).

^{Reviewed by Cursor Bugbot for commit 37deec8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 734af43. Configure here.}

[Copilot]

🔵 Needs a closer look

Mutable pipeline actions currently corrupt the historical baseline and can produce misleading accuracy results.

Pull request overview

Adds a one-off evaluator comparison tool for measuring the new Bedrock prompt against historical decisions.

Changes:

• Adds stratified baseline sampling and concurrent endpoint evaluation.
• Generates JSON accuracy reports.
• Adds production and local pnpm commands.

File summaries

File	Description
backend/src/bin/scripts/compare-evaluator-accuracy.ts	Implements sampling, comparison, and reporting.
backend/package.json	Registers script commands.

Review details

• Files reviewed: 1/2 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

Invalid numeric arguments can silently produce a misleading empty comparison report.

Review details

• Files reviewed: 1/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

The baseline derives verdicts from mutable workflow actions, excluding successfully onboarded projects and potentially including non-agent skips.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 1/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Mutable pipeline actions currently corrupt the historical baseline, while argument and response validation gaps can produce misleading reports.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 1/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Mutable catalog actions skew the baseline, and malformed responses can corrupt accuracy results.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 1/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced