[bluebird42]

What

Spaces can turn on audio description (Settings → Features). Every new upload or reprocessing run of a video up to 10 minutes then gets, in English and Dutch:

• audio_description_<lang>.mp3: a full mix of the video's own audio with spoken descriptions of what happens on screen. It is also published as an HLS alternate audio rendition with CHARACTERISTICS="public.accessibility.describes-video", so players can switch to it.
• descriptions_<lang>.vtt: a WebVTT with Kind: descriptions, containing exactly the spoken descriptions, timed to when they are spoken.

How descriptions are placed

• English is described first, from the picture. Other languages are translations of that text.
• Where speech is, is measured with speech-to-text (the existing transcription segments), never guessed by the model. A description that fits a nearby pause plays there. Otherwise it plays at its moment and the original audio fades to 20% underneath it (0.3 s fades).
• Cuts are detected with FFmpeg scdet and grouped into scenes of at least 3 s. A description starts while its scene is on screen and at least half of it is spoken during the scene.

Flow steps

Five optional background steps in all publishing presets: media.detect_scenes (FLAME), ai.describe_video, ai.translate_descriptions, ai.speak_descriptions and media.mix_audio_description (FLAME).

• They skip themselves unless the run input has audio_description, so other spaces pay nothing.
• Publication never waits for them. When a mix finishes, Core stores the result, rebuilds the HLS master and republishes the manifest.
• Transient provider errors are retried; other failures make the step unavailable.

Contracts

• Database: migration adds spaces.audio_description_enabled and an audio_descriptions table (one row per video version and language).
• manifest.json: a new audio_descriptions key (omitted when empty). The existing audio_tracks and subtitles are unchanged, so current players are unaffected.
• API: the detailed video payload has the same audio_descriptions list.
• Dashboard: the video page links the MP3 and VTT files.
• Host policy: an optional UsageLimits.can_use_audio_description?/1 callback decides who may turn the feature on. Without a policy, every space may. Turning it off is always allowed. The check is repeated per run.

Providers

Scaleway Generative APIs (Gemma 4) describe and translate, and Mistral (Voxtral) speaks, both in the EU.

• New secret: SCALEWAY_API_KEY. MISTRAL_API_KEY is reused.
• Optional settings: MAVE_AUDIO_DESCRIPTION_* (model, voices per language, max length).
• Everything is documented in docs/audio_description.md.

Tests

• Unit: timeline placement, cue cleanup, VTT, WAV decoding and the mix filter.
• Integration: an ffmpeg run of the whole pipeline with stub providers, covering scene detection, the ducked mix, the VTT and the HLS rendition.
• Publication: the HLS master CHARACTERISTICS line and the manifest key.
• Opt-in: the host policy and the settings toggle.
• mix credo --strict and mix security are clean.
• Locally, three existing tests fail on main as well: two analytics tests, and a WebP poster test because that ffmpeg has no libwebp.