Skip to content

fix: deliver full caption evidence to answers and cite moments - #203

Merged
devhims merged 1 commit into
mainfrom
fix/session-evidence-quality
Oct 15, 2026
Merged

devhims merged 1 commit into
mainfrom
fix/session-evidence-quality

Conversation

@devhims

@devhims devhims commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Problem / Motivation

Production sessions lost evidence the session already held before the finalizer wrote its answer:

  • Coffee (7c283520): session search found the 922-character window containing "a sixteen to one water to coffee brew ratio", but the finalizer projection cut excerpts at 800 characters, just before it (character 835). The follow-up retracted Rob's stated 16:1 ratio.
  • Runway (21a77b1e): the correction follow-up read the saved transcript from caption 300 onward, an offset the research model chose. The wind passage at about 4:52 never reached the answer. Research routes do not inherit earlier-turn transcripts.

Inline citations also linked to the whole video, because compact answers collapse each citation group to a per-video source.

What changed

  • Transcript excerpts reach the finalizer at 2,000 characters, the analysis window size, instead of 800. Raw transcript pages and session search hits are affected. Analyzed packets still pass findings rather than original caption text.
  • get_video_transcript snaps any offset to the start of its 5,000-caption page. A model-chosen offset returns the whole page; long transcripts page as before. Caching and continuation follow the page; traces keep the requested offset.
  • Inspect guidance asks a follow-up that checks, corrects or extends an earlier claim to read the saved transcript again, without an offset. This is prompt guidance, not enforced. Rephrasing still needs no read.
  • Compact answers name the earliest cited moment per video, as in [1 · 4:52]. Sources stay one per video. The dashboard links each timed reference to that moment.

The transcript checker on main is unchanged. Earlier revisions added quote verification, fraction and ratio inference and quote anchors; review found repeated boundary defects, so that work is removed from this PR and will be redesigned separately.

Scope and limits

This fixes evidence delivery and improves reference presentation. It does not:

  • fix photography's existing false "unverified" flags (Jason Vong's spoken shutter speeds);
  • provide exact quotes or caption-level timestamps: the caption-ending session still links window starts (08:32, 00:01);
  • enforce transcript rereading;
  • regenerate analyses already saved in production.

Tests

Build passed. Unit suite: 1,818 passed. Agent runtime, session and memory integration suites: 354 passed. Web tests: 110 passed. Docs check passed. The 12 new or updated tests fail on main's source: offset snapping and page continuation, the captured coffee search packet reaching the finalizer whole, and timed references across the backend and dashboard. One integration test now uses a genuine page boundary instead of offset 1.

Live replays with the exact user messages and production models (Fireworks GLM 5.3 Flash), three trials each on this commit:

Session Result
Coffee follow-up, captured search packets Keeps "about a sixteen to one water to coffee brew ratio" in 3/3 (main retracted in 5/5)
Runway follow-up, live research route Reads the full 536-caption transcript in 3/3 and answers runway 27 with timed references (main read none in 5/5)

Compact answer text changes from [1] to [1 · m:ss] for timed references; the OpenAPI description is updated. Deploy the Worker and web app after merge. Production verification remains pending.

@vercel

vercel Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
video2ctx-web Ready Ready Preview Oct 15, 2026 5:44pm UTC

devhims commented Oct 15, 2026

Copy link
Copy Markdown
Owner Author

Changes needed — reviewed 6a5ead0.

  • P2: Incorrect quotes pass as verified. transcript-grounding.ts:60–64 accepts “safe to fly” from “unsafe to fly”, and joins nonadjacent/reversed windows into invented quotes. Both survive the analyst/finalizer path without warnings. Require complete-word matches in the original contiguous transcript order.
  • P2: Fraction grounding overrides explicit units. transcript-grounding.ts:107–108 accepts “1/2 minute” as 0.5 seconds, also accepting conflicting milliseconds. Base rejects these; head forwards them without uncertainty. Only infer seconds when the source has no conflicting explicit unit.

Coverage: This addresses the coffee excerpt truncation and runway page loss, but does not fully resolve the observed sessions. Caption-window anchors still reproduce 512s instead of 529s and 1s instead of 16s. Photography’s camera-shake/motion-blur error and missing-source coverage are unchanged.

Verified: 1,835 platform tests, 238 runtime integrations, 110 web tests, both builds and docs passed; 37 reviewer controls passed and 8 assertions reproduced the defects above. All eight Verify jobs passed. Synthetic local testing only; all four original flows still need live retesting after deployment.

devhims commented Oct 15, 2026

Copy link
Copy Markdown
Owner Author

Changes needed — reviewed 47931f0.

  • P2: Valid quotes can fail the entire analysis. transcript-analyst.ts:404–406 restores whole captions into an unbounded anchor. A 22-character quote spanning two captions produces 2,223 characters and fails the tool’s 2,000-character excerpt validation. Base and previous head succeed. Bound anchor text while preserving its timestamp.
  • P2: Explicit units still become seconds. transcript-grounding.ts:120–123 accepts 1/2-minute and 1/2 a minute as 0.5 seconds without uncertainty; hyphenated milliseconds also bypass the guard. Parse the complete attached unit before inferring seconds.
  • P2: Numeric quote fragments remain “verified.” transcript-grounding.ts:55–56 accepts “5 seconds” from “0.5 seconds” and “250 seconds” from “1,250 seconds” through the analyst/tool path. Preserve numeric boundaries during normalization.
  • P2: Quotes can anchor to uncited captions. transcript-analyst.ts:401–406 includes the next window’s first caption. Selecting windows [0,2] can link repeated wording at 60s instead of its selected occurrence at 120s. Resolve anchors from actual catalog membership.

The previous unsafe/safe and disconnected/reversed-window cases are fixed; ending fixtures now correctly link 529s and 16s. Older saved analyses are not automatically regenerated.

Verified: 1,844 platform, 87 runtime integration and 110 web tests passed; platform build/docs passed. Independent controls: 48 passed, 13 failures reproducing these findings. All eight Verify jobs passed. Local mock testing only; the four observed flows still need live retesting, including unchanged visual-classification and missing-source behavior.

devhims commented Oct 15, 2026

Copy link
Copy Markdown
Owner Author

Changes needed — reviewed c508916. All 61 unchanged regression assertions pass, including the four previous reproductions. Additional boundary tests found:

  • P2: Quote anchors overwrite each other. transcript-analyst.ts:411–412 gives different quotes from one long caption the same timestamp-only ID. The second replaces the first’s evidence; validation/finalization accept the wrong quoted excerpt. Make IDs distinguish differing anchor text.
  • P2: Split captions still select uncited text. transcript-analyst.ts:406–407 reconstructs entire captions from chunk membership. Selecting windows [0,2] can find repeated words in uncited chunk 1 and link 0s instead of 120s. Restrict matching to the selected text spans.
  • P2: Numeric quotes still change meaning. transcript-grounding.ts:55–57 verifies “250 seconds” from “1/250 seconds” and positive 5 from −5. It newly accepts “12 samples” from “B12 samples” and “125 seconds” from “1,25 seconds.” Preserve complete numeric/alphanumeric tokens and validate comma grouping.
  • P2: Explicit units still become seconds. source-numbers.ts:43–46 and the conflicting-unit guard accept 1/2‑minute (U+2011) as 0.5s and 1/250 microseconds/µs as 0.004s without uncertainty. Handle Unicode separators and explicit non-second units.

Verified: 1,853 platform, 151 integration and 110 web tests passed; platform build/docs passed. Additional controls: 18 passed, 15 failures reproducing these boundaries. All eight Verify jobs passed. Ending links remain correct at 529s/16s. Mock testing only; live QA behavior and automatic repair of older saved analyses remain unverified/out of scope.

devhims commented Oct 15, 2026

Copy link
Copy Markdown
Owner Author

Changes needed — reviewed 02de1f5. The four last-reported reproductions are fixed, but these material gaps remain:

  • P2: New percentage-to-mass regression. transcript-grounding.ts:159–170 accepts 5 g from source 5% of 200 grams using the altered quote 5 of 200 grams. Dropping % makes the quote match and bypasses unit-mismatch detection; c508916 rejected it. Preserve significant symbols when validating numerical support.
  • P2: Chunking changes verified wording. transcript-analyst.ts:292–296 splits 'a '.repeat(999) + 'unsafe to fly' into un / safe to fly. Selecting either the second chunk or both verifies “safe to fly”; the genuine quotation is dropped. Match against original-text word boundaries.
  • P2: Numeric fragments still change magnitude. transcript-grounding.ts:58–64 accepts “250 seconds” from 1 / 250 seconds, “5 seconds” from .5 seconds, and positive 5 from − 5. The first two also survive as structured quantities without uncertainty.
  • P2: Parenthesized units become seconds. transcript-grounding.ts:131–133 accepts 1/2 (minute) as 0.5s and 1/250 (milliseconds) as 0.004s because attached-unit parsing skips parentheses. Reject conflicting explicit units before inferring seconds.

Verified: 1,860 platform, 151 integration and 110 web tests passed; platform build/docs passed. Independent controls: 103 passed, 15 failing assertions reproduced these defects. All eight Verify jobs passed. Ending links remain correct at 529s/16s. Local mock testing only; live QA remains unverified, and older analyses are not automatically regenerated.

Production sessions lost evidence before the answer: a session search hit was
cut before the matching caption, and a follow-up read only part of a saved
transcript. Inline citations also opened the whole video.

- Transcript excerpts reach the finalizer at 2,000 characters, the analysis
  window size, instead of 800 (run 63ece937 lost "a sixteen to one" at
  character 835). Analysis packets still pass findings, not caption text.
- get_video_transcript snaps a model-chosen offset to the start of its
  5,000-caption page, so a read never drops earlier captions (run 0bb4aadb).
  Caching and continuation follow the page; traces keep the requested offset.
- Inspect guidance asks correction follow-ups to read the saved transcript
  again, since research routes do not inherit earlier transcripts. This is
  prompt guidance and is not enforced.
- Compact answers name the earliest cited moment per video ([1 · 4:52]); the
  dashboard links each timed reference to that moment.
@devhims
devhims force-pushed the fix/session-evidence-quality branch from 02de1f5 to 5c4b53c Compare October 15, 2026 17:43
@devhims devhims changed the title fix: keep caption evidence intact through answers and follow-ups fix: deliver full caption evidence to answers and cite moments Oct 15, 2026

devhims commented Oct 15, 2026

Copy link
Copy Markdown
Owner Author

Approve — reviewed 5c4b53c for the narrowed evidence-flow scope. No material new findings.

Independent checks confirm the coffee passage survives projection, paging preserves cache keys and requested-offset traces, and timed references resolve safely in the dashboard. Checker behavior matches baseline; the earlier quote/fraction additions were removed.

Verified: 31 independent controls, 1,818 unit tests, 151 integrations and 110 web tests passed; platform build/docs passed. All eight Verify jobs passed.

Limits: Mock/local testing; live correctness was not independently verified. Exact quotes, caption-level ending links (still 8:32/0:01), photography false flags and mandatory rereading remain explicitly deferred.

@devhims
devhims merged commit 2e2e5c8 into main Oct 15, 2026
11 checks passed
@devhims
devhims deleted the fix/session-evidence-quality branch October 15, 2026 17:55

This branch was successfully deployed

1 active deployment
Preview — 5c4b53ca Deployed Oct 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant