Skip to content

feat: show AssemblyAI speaker diarization in transcripts - #2243

Open
pavzagor wants to merge 2 commits into
CapSoftware:mainfrom
pavzagor:codex/assemblyai-diarization
Open

feat: show AssemblyAI speaker diarization in transcripts#2243
pavzagor wants to merge 2 commits into
CapSoftware:mainfrom
pavzagor:codex/assemblyai-diarization

Conversation

@pavzagor

@pavzagor pavzagor commented Sep 8, 2026

Copy link
Copy Markdown

Cap already stores AssemblyAI word speakers, but transcription did not request them and captions/UI discarded them. This enables diarization for full recordings and editable-transcript backfills, shows Speaker A/B labels in the transcript, editor and player captions, and preserves labels through transcript edits, video cuts, copying, VTT/text downloads and agent API round-trips. Existing transcripts without labels continue to render normally.

Live chunks remain provisional and use no speaker labels: AssemblyAI identities are scoped to a transcription request. On recording completion, Cap queues a full-recording transcription instead of promoting independent chunks into a misleading final transcript. This adds a full transcription pass for recordings previously eligible for live promotion; the final labels appear when that pass completes. Queue failures propagate for workflow retry.

Validation:

  • 2,749 web tests passed; 27 existing opt-in tests skipped. Added speaker-boundary, storage/edit/export round-trip, unknown-speaker, escaping, agent API, live handoff and translation validation tests.
  • pnpm typecheck and pnpm exec biome ci . --linter-enabled=false passed; scoped Biome checks passed.
  • Real AssemblyAI run through the local app, MySQL, MinIO and media server: 33.5-second synthetic two-speaker recording, four alternating turns, correctly identified A/B; transcript ID b7ffdd4a-2d21-4425-b8bd-bffa3f28159f.
  • Browser verified transcript seeking, speaker captions, caption edit → save → reload, editor labels, word deletion → render → reload with remapped timestamps, and VTT download preserving voice tags. Mobile checked at 360px and 393px without horizontal overflow.
  • Independent review completed; agent API metadata/entity findings resolved. Greptile translation finding fixed: both provider responses and cached translations must preserve cue IDs, timings and voice tags.

Also corrected the existing Slack-manifest test's stale expected brand color to match the current manifest, so the full web suite passes. No database migration or new environment variable is required; uses the existing ASSEMBLY_API_KEY.

Upstream validation on 2766dc0: CI and Recording Reliability passed. Greptile re-reviewed 24 files and added no new comments; security checks passed. Vercel preview remains blocked on Cap Software team authorization.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; the previous translation-validation issue is fully fixed and no new changes or outstanding findings remain.

Summary

  • Requests speaker labels for full-recording transcription while keeping live chunks unlabeled.
  • Preserves speaker metadata through editing, cuts, translation, API round-trips, captions, and exports.
  • Replaces live transcript promotion with a final full-recording transcription pass.
  • Validates translated and cached VTT cue structure, timings, IDs, and voice tags before use.

5. Preserve all newlines and formatting
6. Do not add any explanations or comments
7. Return ONLY the translated VTT content
7. Preserve all <v Speaker ...> and </v> voice tags exactly, including speaker labels; translate only the spoken text inside them

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Voice tags are not validated

The prompt asks the model to preserve speaker voice tags, but the response is accepted whenever it merely contains WEBVTT. If a provider drops or changes a <v Speaker ...> tag while translating, the response is cached unchanged, so subsequent translated captions lose or misattribute that speaker. Validate the returned cue structure and speaker annotations against the source before caching it.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/web/actions/videos/translate-transcript.ts
Line: 154

Comment:
**Voice tags are not validated**

The prompt asks the model to preserve speaker voice tags, but the response is accepted whenever it merely contains `WEBVTT`. If a provider drops or changes a `<v Speaker ...>` tag while translating, the response is cached unchanged, so subsequent translated captions lose or misattribute that speaker. Validate the returned cue structure and speaker annotations against the source before caching it.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

@richiemcilroy

Copy link
Copy Markdown
Member

@greptileai please review the PR

@pavzagor

pavzagor commented Sep 9, 2026

Copy link
Copy Markdown
Author

@richiemcilroy anything else needed from my side to merge this?

@richiemcilroy

Copy link
Copy Markdown
Member

Hey @pavzagor if you can get both of these to 5/5 we're good to go!

@richiemcilroy

Copy link
Copy Markdown
Member

hey @greptileai please re-review the PR

@pavzagor

Copy link
Copy Markdown
Author

@richiemcilroy this is 5/5 lfg!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants