Skip to content

fix: stabilize catalog and native acceptance checks - #255

Open
jefft72 wants to merge 2 commits into
mainfrom
codex/token-pr-companion-00-catalog-fixtures
Open

jefft72 wants to merge 2 commits into
mainfrom
codex/token-pr-companion-00-catalog-fixtures

Conversation

@jefft72

@jefft72 jefft72 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Review size: 457 meaningful changed lines, plus 0 generated lines and 0 documentation lines, counted as additions plus deletions.
Base: main at 6a4091d; head: codex/token-pr-companion-00-catalog-fixtures.
Fixes the typecheck and runtime failures caused by tests assuming deepseek-ai/DeepSeek-V4-Pro always remains in Together's live catalog.
Runs the full existing CI workflow for every pull request, including requests based on prerequisite branches.
Fixes native acceptance fixtures reporting ECONNRESET instead of checksum failure, or cancelling an update before npm reaches the real archive download.

Reasoning

Model availability changes outside the repository, so generator behavior is tested using fixed provider responses while normal builds continue to refresh public catalogs.
The fixtures run the real strict generator in temporary copies, preserving every existing Together metadata, reasoning-control, and compatibility assertion and verifying that absent, deprecated, or tool-incompatible models stay excluded.
This small prerequisite isolates catalog determinism from the dependency security changes in the following PR.
Removing the pull-request base-branch filter verifies each stacked head with the existing build and test shards, retaining all jobs and permissions.

The owned registry fixture closes each connection and declares response sizes to avoid pooled-socket races.
Cancellation has separate bounded startup and shutdown deadlines, observes the real archive request before signalling, and always cleans up the owned process group.
The integrity, installation-preservation, pinned-repair, and archive-download assertions remain strict.

Changes

  • Extracts the existing isolated generator test harness with bounded subprocess execution, guaranteed temporary-directory cleanup, and rejection of unexpected fetch URLs.
  • Uses fixed Together catalog inputs for all existing Kimi, GPT-OSS, DeepSeek, and MiniMax metadata and reasoning-control assertions.
  • Adds regression cases for omitted, deprecated, and non-tool-capable Together models.
  • Preserves the existing strict generator failure-before-mutation checks and environment-key coverage.
  • Removes only the main base-branch filter from the CI pull-request trigger.
  • Uses explicit response lengths and closed connections for the local registry, with separate bounded startup and cancellation deadlines.
  • Records actual child completion and requested cancellation, and cleans up the detached fixture group on every exit path.

Testing

  • Reproduced the original TypeScript and runtime failures after npm run build using captured public catalog responses with Together's DeepSeek-V4-Pro omitted.
  • npm ci --ignore-scripts using this PR's exact main-branch lockfile.
  • npm run build, with a Node preload supplying the captured catalog responses after the fix.
  • npm run check, including after catalog regeneration.
  • From packages/ai: node ../../node_modules/vitest/dist/cli.js --run test/together-models.test.ts test/generate-models-strict.test.ts: 7 tests passed.
  • ./test.sh, using the repository's isolated test runner.
  • Parsed the CI YAML and verified the only semantic change is removing the pull-request base-branch filter.
  • Reproduced the baseline corrupt-download socket race with an 8-second real npm request stall and the premature cancellation with a 35-second real npm startup delay.
  • node scripts/npm-lifecycle.mjs REVIEWED_TARBALL EVIDENCE_DIRECTORY passed full npm/Bun lifecycle checks on Linux ARM64 with Node 22.19.0, npm 11.16.0, and Bun 1.3.11 under the controlled stalls, including observed SIGTERM at the archive boundary, strict integrity rejection, prefix preservation, pinned repair, retained data, and cleanup.
  • The local lifecycle reproduction used immutable reviewed artifact bytes from the original failed CI run; their embedded runtime source commit is preserved.
    The hosted workflow separately rebuilds the current pull-request artifact.
  • Completed an independent review of the complete catalog, CI-trigger, and npm lifecycle fixture diff.

@jefft72 jefft72 added the bug fix label Oct 3, 2026
@jefft72 jefft72 self-assigned this Oct 3, 2026
@jefft72 jefft72 changed the title fix: stabilize catalog tests and cover stacked PRs fix: stabilize catalog and native acceptance checks Oct 3, 2026
@Pepps233
Pepps233 self-requested a review October 4, 2026 18:02
@Pepps233
Pepps233 added this pull request to stack #256 October 4, 2026 18:34
Use fixed provider inputs in isolated copies of the real model generator so Together catalog removals cannot invalidate metadata regression tests, while retaining strict failure and compatibility coverage. Run the existing CI workflow for pull requests against prerequisite branches so every stacked head receives the full build and test checks.
Prevent real npm integrity checks from failing on idle pooled sockets by explicitly closing and framing owned registry responses. Start bounded cancellation escalation only after the real archive request, retain a finite startup deadline, record observed child completion, and clean up the detached fixture group while preserving strict checksum, installation, and repair assertions.
@Pepps233
Pepps233 force-pushed the codex/token-pr-companion-00-catalog-fixtures branch from a1f2755 to 18e8620 Compare October 4, 2026 18:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants