Skip to content

fix(evolution): preserve failed evidence for retry - #258

Open
Issacboy123 wants to merge 3 commits into
fix/together-deepseek-v4-pro-idfrom
fix/retryable-evidence-processing
Open

Issacboy123 wants to merge 3 commits into
fix/together-deepseek-v4-pro-idfrom
fix/retryable-evidence-processing

Conversation

@Issacboy123

Copy link
Copy Markdown
Contributor

Summary

Evolution could silently skip unreadable trace evidence by advancing its processed cursor past a failed materialization. This change stops processing at the first failure, keeps that trace eligible for retry, and exposes a generic retry notice while still allowing a valid earlier prefix to be processed.

This PR is stacked on the Together model-ID correction.

Reasoning

Representing the first materialization failure at the evidence-catalog boundary gives the control plane a precise contiguous range to commit. This prevents manual and automatic jobs from consuming later evidence out of order. Successful materializations remain reusable while pending, then are discarded after their cursor is committed, avoiding indefinite retention without caching failures.

Changes

  • Replaced the successes-only evidence query with an ordered scan that returns the successful prefix and generic metadata for the first failed trace.
  • Prevented manual jobs, automatic eligibility, and excluded-only cursor commits from advancing beyond the first failed trace.
  • Retried failed materializations on later scans and discarded successful cached entries after their cursors were committed.
  • Added the retry notice to the evolution overview and interactive menu without exposing exception text.
  • Added regressions for cache reuse and cleanup, retry behavior, safe cursor ranges, threshold truncation, exclusions, overview counts, and menu rendering.

Testing

  • node ../../node_modules/vitest/dist/cli.js --run test/evolution-runtime.test.ts test/evolution-menu.test.ts
  • node ../../node_modules/vitest/dist/cli.js --run test/invocation-tracing.test.ts
  • node ../../node_modules/vitest/dist/cli.js --run test/control-plane.test.ts test/control-service.test.ts
  • npm run check

@Pepps233 Pepps233 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a maximum retry amount / retry counter per unreadable / broken trace. Currently, if a trace is broken due to permission issues such as file permissions, it will retry forever. Maybe add a maximum retry amount of 2?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants