From 13cccf2a1c0f1198aca2dc81382b256a1782eeb5 Mon Sep 17 00:00:00 2001 From: bluebird42 <2034043+bluebird42@users.noreply.github.com> Date: Sun, 1 Nov 2026 16:44:39 +0100 Subject: [PATCH 1/2] Add opt-in audio description for videos Spaces can turn on audio description. New uploads then get, in English and Dutch, a full-mix MP3 of the video's audio with spoken descriptions of what is on screen, and a descriptions WebVTT. Descriptions play in the pauses of the speech, or over it with the original audio ducked to 20%, and stay with the scene they describe. Five optional background flow steps detect scenes, describe the video with labelled stills, translate, speak and mix. They skip themselves unless the run asks for audio description, and publication never waits for them. The MP3 is published as an HLS alternate audio rendition marked describes-video and listed under a new audio_descriptions manifest key. Hosts decide who may use it through the optional UsageLimits.can_use_audio_description?/1 callback. --- CONTRIBUTING.md | 1 + README.md | 4 +- docs/audio_description.md | 140 ++++++ docs/flow_engine.md | 8 + lib/mave_core/assets.ex | 36 +- lib/mave_core/assets/audio_description.ex | 48 ++ lib/mave_core/assets/video.ex | 3 +- lib/mave_core/audio_description.ex | 79 ++++ lib/mave_core/audio_description/cues.ex | 177 ++++++++ .../audio_description/mistral_speech.ex | 95 ++++ lib/mave_core/audio_description/mix.ex | 85 ++++ lib/mave_core/audio_description/prompts.ex | 165 +++++++ lib/mave_core/audio_description/scaleway.ex | 108 +++++ lib/mave_core/audio_description/timeline.ex | 360 +++++++++++++++ lib/mave_core/audio_description/wav.ex | 169 +++++++ lib/mave_core/embeds.ex | 26 ++ lib/mave_core/embeds/manifest_publisher.ex | 49 +- lib/mave_core/flow.ex | 47 +- lib/mave_core/flow/presets.ex | 77 +++- lib/mave_core/flow/step_registry.ex | 48 ++ .../flow/steps/ai_describe_video_step.ex | 241 ++++++++++ .../flow/steps/ai_speak_descriptions_step.ex | 84 ++++ .../steps/ai_translate_descriptions_step.ex | 57 +++ .../flow/steps/audio_description_support.ex | 169 +++++++ .../flow/steps/media_build_hls_master_step.ex | 30 +- .../flow/steps/media_detect_scenes_step.ex | 206 +++++++++ .../steps/media_mix_audio_description_step.ex | 256 +++++++++++ lib/mave_core/public_api.ex | 33 +- lib/mave_core/spaces.ex | 40 +- lib/mave_core/spaces/space.ex | 3 +- lib/mave_core/uploads.ex | 4 +- lib/mave_core/usage_limits.ex | 14 +- .../live/dashboard/settings/general_tab.ex | 99 +++- .../videos/video_settings_component.ex | 43 ++ .../20261101120000_add_audio_descriptions.exs | 25 + .../audio_description_publication_test.exs | 117 +++++ .../audio_description/media_test.exs | 177 ++++++++ .../audio_description/timeline_test.exs | 281 ++++++++++++ test/mave_core/audio_description_test.exs | 71 +++ .../steps/audio_description_steps_test.exs | 427 ++++++++++++++++++ .../audio_description_setting_test.exs | 80 ++++ 41 files changed, 4143 insertions(+), 39 deletions(-) create mode 100644 docs/audio_description.md create mode 100644 lib/mave_core/assets/audio_description.ex create mode 100644 lib/mave_core/audio_description.ex create mode 100644 lib/mave_core/audio_description/cues.ex create mode 100644 lib/mave_core/audio_description/mistral_speech.ex create mode 100644 lib/mave_core/audio_description/mix.ex create mode 100644 lib/mave_core/audio_description/prompts.ex create mode 100644 lib/mave_core/audio_description/scaleway.ex create mode 100644 lib/mave_core/audio_description/timeline.ex create mode 100644 lib/mave_core/audio_description/wav.ex create mode 100644 lib/mave_core/flow/steps/ai_describe_video_step.ex create mode 100644 lib/mave_core/flow/steps/ai_speak_descriptions_step.ex create mode 100644 lib/mave_core/flow/steps/ai_translate_descriptions_step.ex create mode 100644 lib/mave_core/flow/steps/audio_description_support.ex create mode 100644 lib/mave_core/flow/steps/media_detect_scenes_step.ex create mode 100644 lib/mave_core/flow/steps/media_mix_audio_description_step.ex create mode 100644 priv/repo/migrations/20261101120000_add_audio_descriptions.exs create mode 100644 test/mave_core/assets/audio_description_publication_test.exs create mode 100644 test/mave_core/audio_description/media_test.exs create mode 100644 test/mave_core/audio_description/timeline_test.exs create mode 100644 test/mave_core/audio_description_test.exs create mode 100644 test/mave_core/flow/steps/audio_description_steps_test.exs create mode 100644 test/mave_core_web/live/dashboard/settings/audio_description_setting_test.exs diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 11f6d26..1357ebb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -191,6 +191,7 @@ published manifests/media and send playback events to Core. Technical reference, as needed: - [Flow engine](docs/flow_engine.md): templates, execution, retries and debugging. +- [Audio description](docs/audio_description.md): steps, outputs, providers and host policy. - [Prepared analytics](docs/analytics_sessions.md): setup and data lifecycle. - [Webhook delivery](docs/webhook_delivery_outbox.md): signed lifecycle events. - [Encoding boosters](deploy/encoding-booster/README.md): optional remote compute; diff --git a/README.md b/README.md index 1329327..b67fc01 100644 --- a/README.md +++ b/README.md @@ -20,8 +20,8 @@ and publish videos — with your media in S3-compatible storage that you control - **Manage:** organize videos into folders, invite your team, and replace a video without changing its public embed. -- **Publish:** resumable uploads, adaptive HLS, posters, subtitles, and - customizable player settings. +- **Publish:** resumable uploads, adaptive HLS, posters, subtitles, + [audio description](docs/audio_description.md), and customizable player settings. - **Integrate:** embed with web components, iframe, React, or Vue; automate with the video API and signed webhooks. - **Understand:** see views, drop-off, and viewing sources in your dashboard, diff --git a/docs/audio_description.md b/docs/audio_description.md new file mode 100644 index 0000000..34ea2fc --- /dev/null +++ b/docs/audio_description.md @@ -0,0 +1,140 @@ +# Audio description + +Audio description makes a video followable for blind and low-vision viewers: a +voice describes what happens on screen. A space opts in under **Settings → +Features → Audio description**. From then on, every new upload and reprocessing +run of a video of at most 10 minutes gets, in English and Dutch: + +- `audio_description_.mp3`: a full mix of the video's own audio with the + spoken descriptions. It is also an HLS alternate audio rendition, so players + can switch to it. +- `descriptions_.vtt`: a WebVTT with `Kind: descriptions`, containing exactly + the spoken descriptions, timed to when they are spoken. + +Existing videos only get audio descriptions when they are processed again. + +## How descriptions are placed + +- **English first.** The video is described in English from what is on screen. + Every other language is a translation of that English text, never a separate + description. +- **In the pauses, otherwise ducked.** Where speech is, is measured with + speech-to-text (the `ai.transcribe_audio` segments, or a transcription made for + this purpose), never guessed by the description model. A description that fits a + pause near its moment (up to 1 s earlier or 2 s later) plays there and leaves the + original audio untouched. When none fits, it plays at its moment and the original + audio fades to 20% underneath it (0.3 s fades). Descriptions are not shortened to + make room. +- **In sync with the picture.** Cuts are detected with FFmpeg `scdet` and grouped + into scenes of at least 3 s, so a fast montage is still described. A description + starts while its scene is on screen and at least half of it is spoken during the + scene; a pause after the cut never counts as room for it. One that can no longer + start in its scene, because earlier descriptions ran long, is left out. +- **Labelled stills.** The image model gets one 360p still per ~3 s per shot + (at least one per cut), labelled `S at