Teams calls: opus-rs, jitter buffer, muted join, and speaker tracking

Cargo.toml

@@ -127,8 +127,6 @@ str0m = { version = "0.24.1", default-features = false, features = ["openssl"] }
 # The certificate Chime's media server shows, read for the log only
 # (str0m-openssl's own OpenSSL, so no crate is added).
 openssl = "0.10.78"
-# Opus in pure Rust, without unsafe code or a C library to find.
-opus-decoder = "0.1.1"
 # The TURN client's long-term credential: HMAC-SHA1 (sha1 is above) keyed
 # with MD5.
 hmac = "0.13"
@@ -141,14 +139,16 @@ rustls-native-certs = "0.8"
 tracing = { version = "0.1.44", default-features = false, features = ["log", "std"] }
 
 # Talking in a huddle: the microphone through cpal (rodio's), WebRTC's
-# audio processing and an Opus encoder, all pure Rust.
-# opus-rs: libopus 1.6 ported (BSD-3-Clause, no dependencies); its unsafe
-# is SIMD and unchecked indexing. sonora: WebRTC's AEC3, noise
+# audio processing and Opus, all pure Rust.
+# opus-rs: libopus 1.6 ported (BSD-3-Clause, no dependencies), encoding
+# what we send and decoding what we hear (0.1.39 plays a mono packet in a
+# stereo decoder; 0.1.40 decodes in-band FEC and conceals loss like
+# libopus); its unsafe is SIMD and unchecked indexing. sonora: WebRTC's AEC3, noise
 # suppression, AGC2 and high-pass filter, M145, ported (BSD-3-Clause);
 # its unsafe is SIMD behind runtime CPU detection. sonora-agc2's RNN
 # voice detector decides what DTX leaves out; sonora-common-audio's sinc
 # resampler brings a 44.1 kHz microphone to 48. Both come with sonora.
-opus-rs = "0.1.34"
+opus-rs = "0.1.40"
 sonora = "0.2.0"
 sonora-agc2 = "0.2.0"
 sonora-common-audio = "0.2.0"
@@ -369,11 +369,6 @@ opt-level = 2
 debug = false
 
 [patch.crates-io]
-# opus-decoder 0.1.1 with two fixes (vendor/opus-decoder/VENDORED.md): a
-# hybrid packet naming more redundancy bytes than it holds panicked (in
-# release too, which aborts the app), and frames with 16 short blocks
-# overflowed a shift in debug builds. Drop once upstream has both.
-opus-decoder = { path = "vendor/opus-decoder" }
 # crmne/egui apps-0.36, shared with ZapFast and Spotifast: egui 0.36.1 with
 # bidi shaping, egui's busy-loop fix and Wayland frame pacing. Every egui
 # crate comes from the same revision so they share one emath and epaint.

TODO.md

@@ -808,7 +808,9 @@ engineering, as for the rest of the session sign-in.
         decided it.
       - Opus: **`opus-decoder`** (pure Rust, `forbid(unsafe_code)`, no C,
         MIT/Apache, passes the 12 RFC 8251 vectors by its own account,
-        but young: 0.1, March 2026). libopus through the `opus` crate
+        but young: 0.1, March 2026). Since 2026-10-23 playback is `opus-rs`
+        instead, which also encodes: one dependency, and its SILK
+        concealment measured close to libopus where opus-decoder's was not. libopus through the `opus` crate
         builds its bundled C with CMake on Windows and macOS (or needs
         the system's library on Linux, which the Flatpak runtime has);
         it is the fallback if the decoder sounds wrong. It has not.
@@ -865,19 +867,10 @@ engineering, as for the rest of the session sign-in.
         440 Hz tone instead of the microphone, logging what was sent and
         Chime's RTCP receiver reports every 5 s. The loopback test has the
         pretend Chime decode our Opus and hear the mute.
-      - In-band FEC is off, an upstream `opus-rs` bug to report: with
-        0.1.34, `OpusEncoder::new(48000, 1, Application::Voip)`,
-        `use_inband_fec = true` and `packet_loss_perc = 10`, one second of
-        a 0.3 sine at 440 Hz decodes to 3.9 times the energy put in (5.2
-        times for a 120 Hz harmonic "vowel"; 2.7 to 3.3 with CBR; 1.8 to
-        4.6 across 16–40 kbit/s), frame levels swinging from -1 to -50
-        dBFS where -14 went in. `opus-decoder` and `opus-rs`'s own
-        decoder agree sample for sample, so the encoder's LBRR is at
-        fault. With FEC off: 0.95 to 0.98, as it should be; white noise
-        is fine either way. Turn it on when fixed;
-        `encoded_speech_decodes_back_with_both_decoders` shows it.
-        Playing stays on `opus-decoder`: `opus-rs`'s decoder conceals a
-        loss but cannot decode FEC.
+      - In-band FEC is on since `opus-rs` 0.1.36 fixed its LBRR (0.1.34
+        decoded voiced sound at up to 5.2 times the energy put in);
+        `encoded_speech_decodes_back` guards it, and the speaker plays the
+        far end's copies (`a_lost_frame_plays_from_the_copy_in_the_next`).
       - Cost: `opus-rs` (no dependencies; its unsafe is SIMD and
         unchecked indexing), `sonora` and its six crates (unsafe only in
         SIMD behind runtime CPU detection), `derive_more`; all
@@ -935,119 +928,11 @@ engineering, as for the rest of the session sign-in.
             devices instead.
       - [ ] Cameras of one model: `/dev/v4l/by-id/` ids from the
             helper would tell them apart across replugging.
-- [ ] **File upstream: opus-decoder's collapse mask overflows.** A real
-      huddle stopped playing in a debug build: opus-decoder panicked on
-      the sound device's thread. Fixed in our patched copy
-      (`vendor/opus-decoder`, see its VENDORED.md; still unfixed on
-      Rusopus main `ecb22cf` as of 2026-10-07); drop the copy once a
-      fixed release is out. To file at
-      <https://github.com/TadeuszWolfGang/Rusopus/issues>:
-
-      > **`extract_collapse_mask` overflows its `u8` with 16 short blocks
-      > (attempt to shift left with overflow, celt/vq.rs:118)**
-      >
-      > opus-decoder 0.1.1, `src/celt/vq.rs`:
-      >
-      > ```rust
-      > fn extract_collapse_mask(iy: &[i32], n: usize, b: usize) -> u8 {
-      >     ...
-      >     let mut mask = 0u8;
-      >     for i in 0..b {
-      >         ...
-      >         if nonzero != 0 {
-      >             mask |= 1 << i; // line 118
-      > ```
-      >
-      > `b` can be 16: a transient 20 ms CELT frame has 8 short blocks,
-      > and a negative `tf_change` doubles them once more (the
-      > `time_divide` step before `quant_partition_mono`, undone later by
-      > `post_tf_collapse_mask`). With overflow checks on
-      > (any debug build) `1 << i` panics for `i >= 8`:
-      >
-      > ```
-      > thread 'cpal_alsa_out' panicked at
-      > opus-decoder-0.1.1/src/celt/vq.rs:118:21:
-      > attempt to shift left with overflow
-      > ```
-      >
-      > libopus keeps this mask in an `unsigned` (`celt/vq.c`,
-      > `static unsigned extract_collapse_mask(int *iy, int N, int B)`,
-      > and `alg_unquant` returns `unsigned`); the bits above 8 are folded
-      > down by `quant_band`'s `cm |= cm >> B` after the time-divide
-      > Haar steps, and only the final mask is stored as `unsigned char`.
-      > This crate's `post_tf_collapse_mask` already works on a `u32`, so
-      > the fix is to make `AlgUnquantResult::collapse_mask` and
-      > `extract_collapse_mask` `u32` (the `as u32` at bands.rs:1301 then
-      > goes).
-      >
-      > Reproduce: decode this 55-byte packet with
-      > `OpusDecoder::new(48_000, 2)` and `decode_float(.., false)` in a
-      > debug build:
-      >
-      > ```
-      > f8 75 d5 48 6a 8c cf b8 7d b1 d2 3b 43 7d 6b 6b 17 c1 14 fe 7d
-      > a5 ae 93 56 58 c4 69 d1 30 da 3f 75 ab 8e ab 1c 2c 0e f2 e7 e0
-      > 6b f5 08 84 7d 51 67 f9 16 30 90 1d 29
-      > ```
-      >
-      > About 1.4% of random 20 ms CELT packets hit it. In release builds
-      > the shift wraps to bit `i % 8`; after the fold that gives the same
-      > final mask, so the output matched a `u32` build bit for bit over
-      > 20,000 random packets. The panic is the only harm, but it takes
-      > down the audio thread of whatever is decoding.
-- [ ] **File upstream: a malformed hybrid packet panics opus-decoder in
-      release builds too.** Fixed in our patched copy
-      (`vendor/opus-decoder`, libopus's `len*8 < ec_tell` check; still
-      unfixed on Rusopus main `ecb22cf`). Found by
-      fuzzing while fixing the above: about 1 in 80,000 random packets.
-      Our release builds abort on a panic, so any huddle participant (or
-      Chime) sending such a packet closes NoSlacking for everyone
-      listening; SRTP rules out corruption on the way, not a hostile
-      sender. Debug builds catch it now (`audio::guard`). Until upstream
-      fixes it we carry the copy. To file:
-
-      > **Hybrid redundancy longer than the frame: `range start index
-      > out of range` (lib.rs:676 and :724)**
-      >
-      > In `OpusMode::Hybrid`, `redundancy_bytes = ec.dec_uint(256) + 2`
-      > is read from the packet and then used as
-      > `&frame[frame.len() - redundancy_bytes..]` without checking it
-      > against `frame.len()`. A frame shorter than the redundancy it
-      > claims underflows the subtraction and panics ("range start index
-      > 18446744073709551534 out of range for slice of length 76" in a
-      > release build).
-      >
-      > libopus (`src/opus_decoder.c`, `opus_decode_frame`) checks
-      > right after reading it:
-      >
-      > ```c
-      > len -= redundancy_bytes;
-      > /* This is a sanity check. It should never happen for a valid
-      >    packet, so the exact behaviour is not normative. */
-      > if (len*8 < ec_tell(&dec))
-      > {
-      >    len = 0;
-      >    redundancy_bytes = 0;
-      >    redundancy = 0;
-      > }
-      > ```
-      >
-      > Reproduce with a fresh `OpusDecoder::new(48_000, 1)` and
-      > `decode_float(.., false)`, release or debug, on this 153-byte
-      > packet (TOC 0x69, two 76-byte hybrid frames):
-      >
-      > ```
-      > 69 af 0e e6 85 96 57 1b b5 8c c1 a2 21 35 a5 94 3d 0d 19 d1 c8 e7
-      > 7c d0 63 03 bd c4 31 df 4e 76 64 f8 27 93 e6 c1 2e 44 06 6c 2e a8
-      > df a2 5d 8f b0 e1 a8 8f ce 1f d7 8a 47 af 68 f8 71 37 f5 9e 65 a3
-      > 2a 18 28 26 82 e1 88 a7 b8 27 ad 60 fa 63 9d 18 42 b4 b7 92 e3 60
-      > 34 5e 40 7e 7c ee 8b 98 8f 1c de 63 ad 44 ce 75 0b 2f 15 f7 be 4f
-      > 2d a3 9e 57 bb a8 dd bc f9 0a 3a 14 c2 73 f7 34 98 bb 54 28 c4 db
-      > fe 4d 7f 97 0a 58 92 0b 47 59 37 15 85 30 e0 b1 78 d8 9f 26 72
-      > ```
-      >
-      > A decoder fed by the network should turn this into an error (or
-      > libopus's behaviour), never a panic.
+- [ ] **opus-rs from crates.io again.** Playing uses our fork
+      (Yannicked/opus-rs, pinned in `[patch.crates-io]`) for in-band FEC
+      decoding (restsend/opus-rs#67) and libopus's concealment (#68). Drop
+      the patch and the fork's line in deny.toml once a release has both.
+      The Rusopus PRs (#5–#7) stay open for opus-decoder's sake.
 
 ## Research notes
 

deny.toml

@@ -69,8 +69,8 @@ allow-wildcard-paths = true
 unknown-registry = "deny"
 unknown-git = "deny"
 allow-registry = ["https://github.com/rust-lang/crates.io-index"]
-# The shared fastframe crates and the egui and winit forks they build on
-# (see [patch.crates-io] in Cargo.toml). Nothing else may come from git.
+# The shared fastframe crates and the egui and winit forks they build on.
+# Nothing else may come from git.
 allow-git = [
     "https://github.com/crmne/fastframe",
     "https://github.com/crmne/egui",

src/backend/teams_call.rs

@@ -145,14 +145,23 @@ struct Wanted {
 
 /// A call's controls: for the caller to keep, and for the call's task
 /// (hang-up and mute for the call, and what its devices should do).
+/// A call starts unmuted; a meeting (`muted`) muted, as a huddle does,
+/// so joining one is never heard before you choose to speak.
 fn controls(
     chosen: &crate::devices::Chosen,
+    muted: bool,
 ) -> (Running, mpsc::UnboundedReceiver<Control>, Wanted) {
     let (control, controls) = mpsc::unbounded_channel();
     let (devices, chosen) = watch::channel(chosen.clone());
-    let (muted, microphone) = watch::channel(true);
-    // Unmuted from the start: the microphone task opens it on this.
-    let _ = muted.send(false);
+    let (mute, microphone) = watch::channel(true);
+    if muted {
+        // The call mutes its sound and tells the meeting as soon as it
+        // takes its controls, as when you mute.
+        let _ = control.send(Control::Mute(true));
+    } else {
+        // Unmuted from the start: the microphone task opens it on this.
+        let _ = mute.send(false);
+    }
     // The camera starts off.
     #[cfg(feature = "huddle-camera")]
     let (camera_sender, camera) = watch::channel(false);
@@ -165,7 +174,7 @@ fn controls(
         devices,
         call_id: None,
         control,
-        muted,
+        muted: mute,
         #[cfg(feature = "huddle-camera")]
         camera: camera_sender,
         #[cfg(feature = "huddle-share")]
@@ -194,7 +203,7 @@ impl Caller {
         sink: Sink,
     ) {
         self.stop();
-        let (mut running, controls, wanted) = controls(&self.chosen);
+        let (mut running, controls, wanted) = controls(&self.chosen, false);
         running.team.clone_from(&team);
         tokio::spawn(run(
             client,
@@ -226,7 +235,7 @@ impl Caller {
         if let Some(old) = self.ringing.take() {
             let _ = old.decide.send(Decision::Decline);
         }
-        let (mut running, controls, wanted) = controls(&self.chosen);
+        let (mut running, controls, wanted) = controls(&self.chosen, false);
         running.team.clone_from(&team);
         let (decide, decisions) = oneshot::channel();
         let call_id = call.call_id.clone();
@@ -242,7 +251,7 @@ impl Caller {
     /// ending the last call first.
     pub fn join_meeting(&mut self, client: TeamsClient, team: String, join: Join, sink: Sink) {
         self.stop();
-        let (mut running, controls, wanted) = controls(&self.chosen);
+        let (mut running, controls, wanted) = controls(&self.chosen, true);
         running.team.clone_from(&team);
         tokio::spawn(run_meeting(client, team, join, controls, wanted, sink));
         self.running = Some(running);
@@ -413,7 +422,9 @@ fn roster(callee: &str, far: FarEnd) -> Roster {
 /// the interface knows people by, their names for any it does not), and
 /// you. The meeting mixes everyone's sound, so whoever it says spoke
 /// last (`speaker`, by MRI) is speaking while that sound is loud
-/// (`far.speaking`); you as `far.me_speaking` says.
+/// (`far.speaking`), unless muted since: the speaker is often only the
+/// one from before we joined, and a muted one cannot be the sound; you
+/// as `far.me_speaking` says.
 fn meeting_roster(people: &[Attendee], speaker: Option<&str>, far: FarEnd) -> Roster {
     let mut everyone: Vec<Person> = people
         .iter()
@@ -425,6 +436,7 @@ fn meeting_roster(people: &[Attendee], speaker: Option<&str>, far: FarEnd) -> Ro
             me: false,
             muted: attendee.muted,
             speaking: far.speaking
+                && !attendee.muted
                 && speaker.is_some_and(|s| s.eq_ignore_ascii_case(&attendee.mri)),
             name: attendee.name.clone(),
             waiting: attendee.waiting,
@@ -998,7 +1010,7 @@ impl<T: Fn(Listen)> Shown<'_, T> {
                 (self.tell)(Listen::Roster(self.roster()));
             }
             CallEvent::Speaker(mri) => {
-                self.speaker = Some(mri);
+                self.speaker = mri;
                 (self.tell)(Listen::Roster(self.roster()));
             }
             CallEvent::Ended { result, .. } => self.ended = Some(result),
@@ -1284,6 +1296,16 @@ mod tests {
         // No one named yet: no one.
         let roster = meeting_roster(&people, None, loud);
         assert!(roster.people.iter().all(|p| !p.speaking));
+        // The last one named has muted since: someone else is the sound.
+        let people = [
+            attendee("8:orgid:ana"),
+            Attendee {
+                muted: true,
+                ..attendee("8:orgid:bo")
+            },
+        ];
+        let roster = meeting_roster(&people, Some("8:orgid:bo"), loud);
+        assert!(roster.people.iter().all(|p| !p.speaking));
     }
 
     #[test]

src/huddle_audio.rs

@@ -51,6 +51,7 @@ pub mod bitstream;
 #[cfg(feature = "huddle-camera")]
 pub mod camera_send;
 pub mod cameras;
+pub mod capture;
 pub mod chime;
 #[cfg(feature = "huddle-video")]
 pub mod decode;