diff --git a/docs/guide/build.md b/docs/guide/build.md index 29a6b514f..6c75071a3 100644 --- a/docs/guide/build.md +++ b/docs/guide/build.md @@ -26,6 +26,17 @@ On the first run, the WebRTC source tree will be loaded into the `//w | webrtc.src.dir | The absolute checkout path for the WebRTC source tree. | /\/webrtc | | webrtc.install.dir | The install path for the compiled WebRTC library. Is also used to link against a pre-compiled WebRTC library to reduce build time. | /\/webrtc/build | +## WebRTC patches and cached builds + +The native build applies the patches in +`webrtc-jni/src/main/cpp/dependencies/webrtc/patches/common/` on every platform, +followed by the platform-specific patches. Common safety patches are required: +configuration stops if one cannot be applied. + +A compiled WebRTC build records its GN arguments, extra targets, and patch +fingerprint. Changes to patch content invalidate both build-tree and installed +caches, so a native rebuild also includes the updated patches. + [build-linux-ubuntu]: https://chromium.googlesource.com/chromium/src/+/master/docs/linux/build_instructions.md#system-requirements [build-linux-other]: https://chromium.googlesource.com/chromium/src/+/master/docs/linux/build_instructions.md#Notes-for-other-distros [build-windows]: https://chromium.googlesource.com/chromium/src/+/master/docs/windows_build_instructions.md#visual-studio diff --git a/docs/tools/audio/voice-activity-detector.md b/docs/tools/audio/voice-activity-detector.md index ea7edf905..118b2a84f 100644 --- a/docs/tools/audio/voice-activity-detector.md +++ b/docs/tools/audio/voice-activity-detector.md @@ -1,104 +1,153 @@ -# Voice Activity Detector +# Voice Activity Detection -The Voice Activity Detector (VAD) helps you determine when speech is present in an audio stream. It analyzes short chunks of PCM audio and returns the probability that the chunk contains voice. +Two WebRTC detectors are available. Choose according to the result your +application needs; neither is guaranteed to classify every recording better. +Both analyze mono signed 16-bit PCM and maintain state across audio frames. -This can be used to: -- Drive push-to-talk or auto-mute logic -- Skip encoding/sending silence to save bandwidth -- Trigger UI indicators when the user is speaking +| API | Native implementation | Result | Input frames | +| --- | --- | --- | --- | +| `VoiceActivityDetector` | Combined standalone and pitch-based detector | Estimated probability from 0 to 1 | 10 ms | +| `StandaloneVoiceActivityDetector` | Standalone WebRTC VAD directly | Boolean speech decision | 10, 20 or 30 ms | -API: `dev.onvoid.webrtc.media.audio.VoiceActivityDetector` +The existing `VoiceActivityDetector.process()` and `getLastVoiceProbability()` +methods remain available with the original probability estimator. No migration +is required. The standalone detector is an additional class, with its own state. -## Overview +## Probability estimator -`VoiceActivityDetector` exposes a minimal API: -- `process(byte[] audio, int samplesPerChannel, int sampleRate)`: Analyze one audio frame. -- `getLastVoiceProbability()`: Retrieve the probability (0.0..1.0) that the last processed frame contained voice. -- `dispose()`: Release native resources. Always call this when done. - -Internally, VAD uses a native implementation optimized for real-time analysis. The class itself does not perform resampling or channel mixing, so provide audio matching the given `sampleRate` and expected format. - -## Audio format expectations - -- PCM signed 16-bit little-endian (typical Java byte[] from microphone capture via this library) -- Mono is recommended. If you have stereo, downmix to mono before calling `process` or pass samples-per-channel accordingly -- Frame size: commonly 10 ms per call (e.g., 160 samples at 16 kHz for 10 ms) -- Supported sample rates: 8 kHz, 16 kHz, 32 kHz, 48 kHz (use one of these for best results) - -## Basic usage +`dev.onvoid.webrtc.media.audio.VoiceActivityDetector` combines the standalone VAD +with pitch and spectral features. Your application chooses a threshold on the +returned probability. The result is delayed by a few chunks and is not updated +on every call. WebRTC initially returns `1.0` until enough audio has arrived; +do not interpret that initial value as observed speech. ```java import dev.onvoid.webrtc.media.audio.VoiceActivityDetector; -// Create the detector VoiceActivityDetector vad = new VoiceActivityDetector(); - try { - // Example parameters - int sampleRate = 16000; // 16 kHz - int frameMs = 10; // 10 ms frames - int samplesPerChannel = sampleRate * frameMs / 1000; // 160 samples + byte[] frame = new byte[320]; // 10 ms, mono signed 16-bit PCM at 16 kHz + // In your audio loop, fill frame from the source, then: + vad.process(frame, 160, 16000); + float probability = vad.getLastVoiceProbability(); + // After the initial buffering period, apply your application's threshold. + boolean speech = probability > 0.7f; // Example threshold, not a universal setting. +} +finally { + vad.dispose(); +} +``` - // audioFrame must contain 16-bit PCM data for one frame (mono) - byte[] audioFrame = new byte[samplesPerChannel * 2]; // 2 bytes per sample +Pass exactly `sampleRate / 100` samples per call. The byte buffer must contain at +least twice that many bytes; trailing bytes are ignored. Samples are mono signed +16-bit PCM in native byte order, matching the library's audio callbacks. Supported +sample rates are 8000, 16000, 32000 and 48000 Hz; this detector resamples to 16 kHz +internally. Downmix stereo before calling `process()`. - // Fill audioFrame from your audio source here - // ... +Keep one instance per continuous stream and create a new instance when beginning +a new stream or changing sample rates. The standalone detector's `Mode` setting +does not configure this estimator. Its probability is a model estimate, not a +guarantee that a given percentage of frames will be classified correctly. - // Analyze frame - vad.process(audioFrame, samplesPerChannel, sampleRate); +## Standalone detector - // Query probability of voice in the last frame - float prob = vad.getLastVoiceProbability(); // 0.0 .. 1.0 +`dev.onvoid.webrtc.media.audio.StandaloneVoiceActivityDetector` provides direct +boolean decisions and four aggressiveness modes. It does not expose a probability +score, perform channel mixing, or resample input. - boolean isSpeaking = prob >= 0.5f; // choose a threshold that works for your app +```java +import dev.onvoid.webrtc.media.audio.StandaloneVoiceActivityDetector; +StandaloneVoiceActivityDetector vad = new StandaloneVoiceActivityDetector( + StandaloneVoiceActivityDetector.Mode.AGGRESSIVE); +try { + short[] frame = new short[160]; // 10 ms of mono audio at 16 kHz + // Fill frame with PCM samples from your audio source. + boolean speech = vad.isSpeech(frame, 16000); } finally { - // Always release resources vad.dispose(); } ``` -## Continuous processing loop +Provide one mono PCM frame in a `short[]` with exactly one of these lengths: + +| Sample rate | 10 ms | 20 ms | 30 ms | +| --- | --- | --- | --- | +| 8000 Hz | 80 | 160 | 240 | +| 16000 Hz | 160 | 320 | 480 | +| 32000 Hz | 320 | 640 | 960 | +| 48000 Hz | 480 | 960 | 1440 | + +Lengths are sample counts, not bytes. Numeric `short` values have no byte order; +when decoding a PCM `byte[]`, use the source's byte order. + +Modes are `QUALITY` (the default), `LOW_BITRATE`, `AGGRESSIVE`, and +`VERY_AGGRESSIVE`. Higher aggressiveness rejects more noise and can miss more +speech. `setMode(mode)` changes aggressiveness without clearing history. + +Keep one detector per independent stream and feed frames in order. Do not reset +between frames: the detector uses adaptive state and built-in hangover, so a few +frames after speech ends can still be classified as speech. Call `reset()` when +starting a new stream or changing sample rates; it preserves the selected mode. + +Downmix stereo and resample unsupported rates before detection. The +[Audio Converter](/tools/audio/audio-converter) accepts 10 ms frames and can do +both operations. Split larger callbacks or buffer partial callbacks into complete +frames. + +## Lifecycle and concurrency + +Both constructors load the platform native library. A `PeerConnectionFactory` is +not required. Include the matching native classifier jar as usual. + +Calls, including `dispose()`, are serialized on each detector. Callers must still +preserve stream order and avoid modifying an input array during detection. +Disposal is idempotent. Processing or querying a disposed detector throws +`IllegalStateException`, as do standalone configuration/reset calls after disposal. +Invalid frame lengths/rates throw `IllegalArgumentException`; null input buffers +or standalone modes throw `NullPointerException`. + +The standalone detector reports failure to allocate its native instance as +`OutOfMemoryError`. The probability estimator retains WebRTC's native +allocation-failure behavior; catching Java exceptions does not guarantee recovery +from native memory exhaustion. + +A probability detector and a standalone detector do not share history, even when +they receive the same audio. Feeding, configuring, resetting, or disposing one +does not update the other. Neither API automatically changes encoding, DTX, mute +state, or transmitted audio. + +## Audio track integration + +The examples module includes `VoiceActivityExample`, an `AudioTrackSink` adapter +using `StandaloneVoiceActivityDetector`. It buffers callback data into 10 ms +frames, converts mono or stereo input to 16 kHz mono with `AudioConverter`, and +reports each boolean decision. It also supports 44.1 kHz input. ```java -VoiceActivityDetector vad = new VoiceActivityDetector(); +import dev.onvoid.webrtc.examples.VoiceActivityExample; +VoiceActivityExample sink = new VoiceActivityExample(speech -> { + // Enqueue the decision for your application/UI; return promptly. +}); +track.addSink(sink); try { - int sampleRate = 16000; - int frameMs = 10; - int samplesPerChannel = sampleRate * frameMs / 1000; // 160 samples - byte[] audioFrame = new byte[samplesPerChannel * 2]; - - while (running) { - // Read PCM frame from your capture pipeline into audioFrame - // ... - - vad.process(audioFrame, samplesPerChannel, sampleRate); - float prob = vad.getLastVoiceProbability(); - - if (prob > 0.8f) { - // High confidence of speech - // e.g., enable VU meter, unmute, or mark active speaker - } - else { - // Likely silence or noise - } - } + // Keep the sink attached while receiving audio. } finally { - vad.dispose(); + track.removeSink(sink); + sink.close(); } ``` -## Tips and best practices - -- Threshold selection: Start with 0.5–0.8 and tune for your environment. -- Frame size consistency: Use a consistent frame duration and sample rate. -- Resource management: VAD holds native resources; ensure `dispose()` is called. -- Preprocessing: Consider using `AudioProcessing` (noise suppression, gain control) before VAD for improved robustness in noisy environments. See the Audio Processing guide. +The adapter invokes the consumer on the audio callback thread. Keep work there +short. Apply application-level timing to produce stable speech-start/speech-end +events, rather than toggling a UI or recording for every frame. Audio format +changes clear the adapter's partial frame and reset its detector. ## Related guides -- [Audio Processing](/guide/audio/audio-processing) \ No newline at end of file +- [Audio Processing](/guide/audio/audio-processing) +- [Audio Converter](/tools/audio/audio-converter) +- [Audio Recorder](/tools/audio/audio-recorder) diff --git a/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/VoiceActivityExample.java b/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/VoiceActivityExample.java new file mode 100644 index 000000000..73c2ead6d --- /dev/null +++ b/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/VoiceActivityExample.java @@ -0,0 +1,101 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc.examples; + +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.function.Consumer; +import java.util.Objects; + +import dev.onvoid.webrtc.media.audio.AudioConverter; +import dev.onvoid.webrtc.media.audio.AudioTrackSink; +import dev.onvoid.webrtc.media.audio.StandaloneVoiceActivityDetector; + +/** + * Attach to an AudioTrack with addSink(), then removeSink() before close(). + * Buffers arbitrary callback lengths into 10 ms frames and converts interleaved + * signed 16-bit PCM to 16 kHz mono, including input at 44.1 kHz. The consumer runs + * on the audio callback thread and should return promptly. It receives raw frame + * decisions; a UI should apply its own speaking-state timing off this thread. + */ +public final class VoiceActivityExample implements AudioTrackSink, AutoCloseable { + + private final Consumer onDecision; + private final StandaloneVoiceActivityDetector detector; + private AudioConverter converter; + private int rate; + private int channels; + private byte[] pending; + private int buffered; + private final byte[] converted = new byte[320]; + private final short[] mono = new short[160]; + private final ByteBuffer convertedSamples = ByteBuffer.wrap(converted).order(ByteOrder.nativeOrder()); + private boolean closed; + + public VoiceActivityExample(Consumer onDecision) { + this.onDecision = Objects.requireNonNull(onDecision, "onDecision"); + detector = new StandaloneVoiceActivityDetector(StandaloneVoiceActivityDetector.Mode.AGGRESSIVE); + } + + @Override + public synchronized void onData(byte[] data, int bitsPerSample, int sampleRate, int channels, int frames) { + if (closed) { + return; + } + if (bitsPerSample != 16 || sampleRate < 8000 || sampleRate > 96000 || sampleRate % 100 != 0 + || (channels != 1 && channels != 2) || frames < 0 + || (long) frames * channels * 2 != data.length) { + throw new IllegalArgumentException("Expected interleaved 16-bit mono/stereo PCM at a whole 10 ms rate"); + } + if (rate != sampleRate || this.channels != channels) { + AudioConverter replacement = new AudioConverter(sampleRate, channels, 16000, 1); + if (converter != null) { + converter.dispose(); + } + converter = replacement; + rate = sampleRate; + this.channels = channels; + pending = new byte[sampleRate / 100 * channels * 2]; + buffered = 0; + detector.reset(); + } + int offset = 0; + while (offset < data.length && !closed) { + int count = Math.min(data.length - offset, pending.length - buffered); + System.arraycopy(data, offset, pending, buffered, count); + buffered += count; + offset += count; + if (buffered == pending.length) { + converter.convert(pending, converted); + convertedSamples.asShortBuffer().get(mono); + buffered = 0; + onDecision.accept(detector.isSpeech(mono, 16000)); + } + } + } + + @Override + public synchronized void close() { + if (!closed) { + closed = true; + if (converter != null) { + converter.dispose(); + } + detector.dispose(); + } + } +} diff --git a/webrtc-jni/src/main/cpp/dependencies/jni-voithos/include/JavaUtils.h b/webrtc-jni/src/main/cpp/dependencies/jni-voithos/include/JavaUtils.h index dd273ef55..aead881df 100644 --- a/webrtc-jni/src/main/cpp/dependencies/jni-voithos/include/JavaUtils.h +++ b/webrtc-jni/src/main/cpp/dependencies/jni-voithos/include/JavaUtils.h @@ -42,13 +42,18 @@ } \ } while(0) +// Throws the requested Java exception without replacing an already pending one. +// Uses JNI directly, so reporting an allocation failure needs no C++ allocation. +void ThrowJavaException(JNIEnv * env, const char * className, const char * message) noexcept; void ThrowCxxJavaException(JNIEnv * env); JNIEnv * AttachCurrentThread(); bool ExceptionCheck(JNIEnv * env); jclass FindClass(JNIEnv * env, const char * name); jmethodID GetMethod(JNIEnv * env, jclass cls, const char * name, const char * sig); jmethodID GetStaticMethod(JNIEnv * env, jclass cls, const char * name, const char * sig); -jfieldID GetHandleField(JNIEnv * env, jobject obj, const std::string & fieldName); +// Specify the declaring class when subclasses may hide a handle field. +jfieldID GetHandleField(JNIEnv * env, jobject obj, const std::string & fieldName, + const char * declaringClass = nullptr); jfieldID GetFieldID(JNIEnv * env, jobject obj, const std::string & fieldName, const char * type); jfieldID GetFieldID(JNIEnv * env, jclass cls, const std::string & fieldName, const char * type); @@ -78,9 +83,10 @@ jlong GetHandleLong(JNIEnv * env, jobject obj) } template -T * GetHandle(JNIEnv * env, jobject obj, const std::string & handleName) +T * GetHandle(JNIEnv * env, jobject obj, const std::string & handleName, + const char * declaringClass = nullptr) { - jfieldID field = GetHandleField(env, obj, handleName); + jfieldID field = GetHandleField(env, obj, handleName, declaringClass); if (!field) { ExceptionCheck(env); @@ -97,9 +103,10 @@ T * GetHandle(JNIEnv * env, jobject obj) } template -void SetHandle(JNIEnv * env, jobject obj, const std::string & handleName, T * t) +void SetHandle(JNIEnv * env, jobject obj, const std::string & handleName, T * t, + const char * declaringClass = nullptr) { - jfieldID field = GetHandleField(env, obj, handleName); + jfieldID field = GetHandleField(env, obj, handleName, declaringClass); if (!field) { ExceptionCheck(env); diff --git a/webrtc-jni/src/main/cpp/dependencies/jni-voithos/src/JavaUtils.cpp b/webrtc-jni/src/main/cpp/dependencies/jni-voithos/src/JavaUtils.cpp index e3b3a9407..835c3eb63 100644 --- a/webrtc-jni/src/main/cpp/dependencies/jni-voithos/src/JavaUtils.cpp +++ b/webrtc-jni/src/main/cpp/dependencies/jni-voithos/src/JavaUtils.cpp @@ -36,6 +36,19 @@ bool ExceptionCheck(JNIEnv * env) return false; } +void ThrowJavaException(JNIEnv * env, const char * className, const char * message) noexcept +{ + if (env->ExceptionCheck()) { + return; + } + + jni::JavaLocalRef cls(env, env->FindClass(className)); + + if (cls.get() != nullptr) { + env->ThrowNew(cls.get(), message); + } +} + void ThrowCxxJavaException(JNIEnv * env) { std::exception_ptr eptr = std::current_exception(); @@ -105,9 +118,21 @@ jmethodID GetStaticMethod(JNIEnv * env, jclass cls, const char * name, const cha return method; } -jfieldID GetHandleField(JNIEnv * env, jobject obj, const std::string & fieldName) +jfieldID GetHandleField(JNIEnv * env, jobject obj, const std::string & fieldName, + const char * declaringClass) { - return GetFieldID(env, obj, fieldName, "J"); + if (declaringClass == nullptr) { + return GetFieldID(env, obj, fieldName, "J"); + } + + auto cls = jni::ClassLoaderGetClass(env, declaringClass); + + if (cls.get() == nullptr) { + ExceptionCheck(env); + return nullptr; + } + + return GetFieldID(env, cls.get(), fieldName, "J"); } jfieldID GetFieldID(JNIEnv * env, jobject obj, const std::string & fieldName, const char * type) diff --git a/webrtc-jni/src/main/cpp/dependencies/webrtc/CMakeLists.txt b/webrtc-jni/src/main/cpp/dependencies/webrtc/CMakeLists.txt index 467693a55..c9c24d056 100644 --- a/webrtc-jni/src/main/cpp/dependencies/webrtc/CMakeLists.txt +++ b/webrtc-jni/src/main/cpp/dependencies/webrtc/CMakeLists.txt @@ -193,7 +193,20 @@ set(WEBRTC_EXTRA_TARGETS # the JNI code is compiled with below. Changing an arg or a target here thus # rebuilds WebRTC on its own; CI's cache key covers this file. set(WEBRTC_ARGS_FILE "webrtc-java-gn-args.txt") -set(WEBRTC_BUILD_RECORD "${COMPILE_ARGS} extra_targets=\"${WEBRTC_EXTRA_TARGETS}\"") +# Patch content is part of a compiled library's identity, including local +# installs outside CI. Otherwise an installed archive bypasses patch application. +file(GLOB WEBRTC_PATCHES CONFIGURE_DEPENDS + "${CMAKE_CURRENT_LIST_DIR}/patches/common/*.patch" + "${CMAKE_CURRENT_LIST_DIR}/patches/${SOURCE_TARGET}/*.patch") +list(SORT WEBRTC_PATCHES) +set(WEBRTC_PATCH_RECORD "") +foreach(PATCH IN LISTS WEBRTC_PATCHES) + file(SHA256 "${PATCH}" PATCH_HASH) + file(RELATIVE_PATH PATCH_NAME "${CMAKE_CURRENT_LIST_DIR}/patches" "${PATCH}") + string(APPEND WEBRTC_PATCH_RECORD "${PATCH_NAME}:${PATCH_HASH}\n") +endforeach() +string(SHA256 WEBRTC_PATCH_HASH "${WEBRTC_PATCH_RECORD}") +set(WEBRTC_BUILD_RECORD "${COMPILE_ARGS} extra_targets=\"${WEBRTC_EXTRA_TARGETS}\" patches=\"${WEBRTC_PATCH_HASH}\"") function(webrtc_args_match DIRECTORY RESULT) set(MATCHES FALSE) @@ -420,7 +433,7 @@ if(LINUX) sysroot_install() endif() -file(GLOB PATCHES "${CMAKE_CURRENT_LIST_DIR}/patches/${SOURCE_TARGET}/*.patch") +set(PATCHES ${WEBRTC_PATCHES}) if (PATCHES) message(STATUS "WebRTC: patch") @@ -437,6 +450,8 @@ if (PATCHES) if (RESULT EQUAL 0) message(STATUS "Patch applied: ${PATCH}") + elseif(PATCH MATCHES "/patches/common/") + message(FATAL_ERROR "Required WebRTC patch could not be applied: ${PATCH}") else() message(STATUS "Warning: Patch ${PATCH} not applied ....") endif() diff --git a/webrtc-jni/src/main/cpp/dependencies/webrtc/patches/common/.gitattributes b/webrtc-jni/src/main/cpp/dependencies/webrtc/patches/common/.gitattributes new file mode 100644 index 000000000..9812ceb1f --- /dev/null +++ b/webrtc-jni/src/main/cpp/dependencies/webrtc/patches/common/.gitattributes @@ -0,0 +1 @@ +*.patch text eol=lf diff --git a/webrtc-jni/src/main/cpp/dependencies/webrtc/patches/common/vad_allocation_failure.patch b/webrtc-jni/src/main/cpp/dependencies/webrtc/patches/common/vad_allocation_failure.patch new file mode 100644 index 000000000..e843401d7 --- /dev/null +++ b/webrtc-jni/src/main/cpp/dependencies/webrtc/patches/common/vad_allocation_failure.patch @@ -0,0 +1,11 @@ +--- a/common_audio/vad/webrtc_vad.c ++++ b/common_audio/vad/webrtc_vad.c +@@ -23,5 +23,8 @@ + + VadInst* WebRtcVad_Create(void) { + VadInstT* self = (VadInstT*)malloc(sizeof(VadInstT)); ++ if (self == NULL) { ++ return NULL; ++ } + + self->init_flag = 0; diff --git a/webrtc-jni/src/main/cpp/include/JNI_StandaloneVoiceActivityDetector.h b/webrtc-jni/src/main/cpp/include/JNI_StandaloneVoiceActivityDetector.h new file mode 100644 index 000000000..50e7209e6 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/JNI_StandaloneVoiceActivityDetector.h @@ -0,0 +1,18 @@ +/* JNI declarations for dev.onvoid.webrtc.media.audio.StandaloneVoiceActivityDetector. */ +#ifndef JNI_STANDALONE_VOICE_ACTIVITY_DETECTOR_H_ +#define JNI_STANDALONE_VOICE_ACTIVITY_DETECTOR_H_ + +#include + +#ifdef __cplusplus +extern "C" { +#endif +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_initialize(JNIEnv*, jobject, jint); +JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_processInternal(JNIEnv*, jobject, jshortArray, jint); +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_setModeInternal(JNIEnv*, jobject, jint); +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_resetInternal(JNIEnv*, jobject, jint); +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_dispose(JNIEnv*, jobject); +#ifdef __cplusplus +} +#endif +#endif diff --git a/webrtc-jni/src/main/cpp/src/JNI_StandaloneVoiceActivityDetector.cpp b/webrtc-jni/src/main/cpp/src/JNI_StandaloneVoiceActivityDetector.cpp new file mode 100644 index 000000000..26700a175 --- /dev/null +++ b/webrtc-jni/src/main/cpp/src/JNI_StandaloneVoiceActivityDetector.cpp @@ -0,0 +1,176 @@ +/* + * Copyright 2021 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "JNI_StandaloneVoiceActivityDetector.h" +#include "JNI_WebRTC.h" +#include "JavaUtils.h" + +#include "common_audio/vad/include/webrtc_vad.h" + +#include +#include + +namespace +{ + +VadInst * GetDetector(JNIEnv * env, jobject caller) +{ + auto * detector = GetHandle(env, caller, "nativeHandle", PKG_INTERNAL "NativeObject"); + if (env->ExceptionCheck()) { + return nullptr; + } + if (!detector) { + ThrowJavaException(env, "java/lang/IllegalStateException", "StandaloneVoiceActivityDetector has been disposed"); + } + return detector; +} + +bool ValidMode(JNIEnv * env, jint mode) +{ + if (mode < 0 || mode > 3) { + ThrowJavaException(env, "java/lang/IllegalArgumentException", "Invalid VAD mode"); + return false; + } + return true; +} + +} // namespace + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_initialize +(JNIEnv * env, jobject caller, jint mode) +{ + try { + if (!ValidMode(env, mode)) { + return; + } + auto * existing = GetHandle(env, caller, "nativeHandle", PKG_INTERNAL "NativeObject"); + if (env->ExceptionCheck()) { + return; + } + if (existing) { + ThrowJavaException(env, "java/lang/IllegalStateException", "Detector is already initialized"); + return; + } + std::unique_ptr detector(WebRtcVad_Create(), WebRtcVad_Free); + if (!detector) { + ThrowJavaException(env, "java/lang/OutOfMemoryError", "Cannot allocate VAD"); + return; + } + if (WebRtcVad_Init(detector.get()) != 0 || WebRtcVad_set_mode(detector.get(), mode) != 0) { + ThrowJavaException(env, "java/lang/IllegalStateException", "Cannot initialize VAD"); + return; + } + SetHandle(env, caller, "nativeHandle", detector.get(), PKG_INTERNAL "NativeObject"); + if (!env->ExceptionCheck()) { + detector.release(); + } + } + catch (...) { + ThrowCxxJavaException(env); + } +} + +JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_processInternal +(JNIEnv * env, jobject caller, jshortArray samples, jint sampleRate) +{ + try { + VadInst * detector = GetDetector(env, caller); + if (!detector) { + return JNI_FALSE; + } + if (!samples) { + ThrowJavaException(env, "java/lang/NullPointerException", "samples"); + return JNI_FALSE; + } + jsize length = env->GetArrayLength(samples); + if (env->ExceptionCheck()) { + return JNI_FALSE; + } + // Bound the copy independently of the Java-side validation. + std::array frame; + if (length <= 0 || static_cast(length) > frame.size() || + WebRtcVad_ValidRateAndFrameLength(sampleRate, length) != 0) { + ThrowJavaException(env, "java/lang/IllegalArgumentException", "Unsupported VAD rate or frame length"); + return JNI_FALSE; + } + env->GetShortArrayRegion(samples, 0, length, frame.data()); + if (env->ExceptionCheck()) { + return JNI_FALSE; + } + int result = WebRtcVad_Process(detector, sampleRate, frame.data(), length); + if (result < 0) { + ThrowJavaException(env, "java/lang/IllegalStateException", "VAD processing failed"); + return JNI_FALSE; + } + return result == 1 ? JNI_TRUE : JNI_FALSE; + } + catch (...) { + ThrowCxxJavaException(env); + return JNI_FALSE; + } +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_setModeInternal +(JNIEnv * env, jobject caller, jint mode) +{ + try { + VadInst * detector = GetDetector(env, caller); + if (!detector || !ValidMode(env, mode)) { + return; + } + if (WebRtcVad_set_mode(detector, mode) != 0) { + ThrowJavaException(env, "java/lang/IllegalStateException", "Cannot set VAD mode"); + } + } + catch (...) { + ThrowCxxJavaException(env); + } +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_resetInternal +(JNIEnv * env, jobject caller, jint mode) +{ + try { + VadInst * detector = GetDetector(env, caller); + if (!detector || !ValidMode(env, mode)) { + return; + } + if (WebRtcVad_Init(detector) != 0 || WebRtcVad_set_mode(detector, mode) != 0) { + ThrowJavaException(env, "java/lang/IllegalStateException", "Cannot reset VAD"); + } + } + catch (...) { + ThrowCxxJavaException(env); + } +} + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_StandaloneVoiceActivityDetector_dispose +(JNIEnv * env, jobject caller) +{ + try { + auto * detector = GetHandle(env, caller, "nativeHandle", PKG_INTERNAL "NativeObject"); + if (env->ExceptionCheck()) { + return; + } + SetHandle(env, caller, "nativeHandle", nullptr, PKG_INTERNAL "NativeObject"); + if (!env->ExceptionCheck()) { + WebRtcVad_Free(detector); + } + } + catch (...) { + ThrowCxxJavaException(env); + } +} diff --git a/webrtc-jni/src/main/cpp/src/JNI_VoiceActivityDetector.cpp b/webrtc-jni/src/main/cpp/src/JNI_VoiceActivityDetector.cpp index c62b594cc..6f93a8c34 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_VoiceActivityDetector.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_VoiceActivityDetector.cpp @@ -15,50 +15,122 @@ */ #include "JNI_VoiceActivityDetector.h" - -#include "JavaObject.h" +#include "JNI_WebRTC.h" #include "JavaUtils.h" #include "modules/audio_processing/vad/voice_activity_detector.h" -JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_VoiceActivityDetector_process -(JNIEnv * env, jobject caller, jbyteArray data, jint sampleCount, jint sampleRate) -{ - webrtc::VoiceActivityDetector * activityDetector = GetHandle(env, caller); - CHECK_HANDLE(activityDetector); +#include +#include - jbyte * dataPtr = env->GetByteArrayElements(data, nullptr); - const int16_t * data16Ptr = reinterpret_cast(dataPtr); - - activityDetector->ProcessChunk(data16Ptr, sampleCount, sampleRate); +namespace +{ - env->ReleaseByteArrayElements(data, dataPtr, JNI_ABORT); +webrtc::VoiceActivityDetector * GetDetector(JNIEnv * env, jobject caller) +{ + auto * detector = GetHandle(env, caller, "nativeHandle", PKG_INTERNAL "NativeObject"); + if (env->ExceptionCheck()) { + return nullptr; + } + if (!detector) { + ThrowJavaException(env, "java/lang/IllegalStateException", "VoiceActivityDetector has been disposed"); + } + return detector; } -JNIEXPORT jfloat JNICALL Java_dev_onvoid_webrtc_media_audio_VoiceActivityDetector_getLastVoiceProbability +} // namespace + +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_VoiceActivityDetector_initialize (JNIEnv * env, jobject caller) { - webrtc::VoiceActivityDetector * activityDetector = GetHandle(env, caller); - CHECK_HANDLEV(activityDetector, 0); + try { + auto * existing = GetHandle(env, caller, "nativeHandle", PKG_INTERNAL "NativeObject"); + if (env->ExceptionCheck()) { + return; + } + if (existing) { + ThrowJavaException(env, "java/lang/IllegalStateException", "Detector is already initialized"); + return; + } + // Keep WebRTC's native allocation-failure policy. Its constructors are + // built without exception cleanup; JNI cannot promise OOM recovery here. + auto detector = std::make_unique(); + SetHandle(env, caller, "nativeHandle", detector.get(), PKG_INTERNAL "NativeObject"); + if (!env->ExceptionCheck()) { + detector.release(); + } + } + catch (...) { + ThrowCxxJavaException(env); + } +} - return activityDetector->last_voice_probability(); +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_VoiceActivityDetector_process +(JNIEnv * env, jobject caller, jbyteArray audio, jint samplesPerChannel, jint sampleRate) +{ + try { + auto * detector = GetDetector(env, caller); + if (!detector) { + return; + } + if (!audio) { + ThrowJavaException(env, "java/lang/NullPointerException", "audio"); + return; + } + // Validate before calling upstream code: invalid formats can trigger RTC_CHECK + // rather than returning an error. The copy must also fit the aligned buffer. + if ((sampleRate != 8000 && sampleRate != 16000 && sampleRate != 32000 && sampleRate != 48000) || + samplesPerChannel != sampleRate / 100) { + ThrowJavaException(env, "java/lang/IllegalArgumentException", "Expected 10 ms mono PCM at 8, 16, 32 or 48 kHz"); + return; + } + jsize length = env->GetArrayLength(audio); + if (env->ExceptionCheck()) { + return; + } + if (length < samplesPerChannel * 2) { + ThrowJavaException(env, "java/lang/IllegalArgumentException", "Insufficient PCM input buffer"); + return; + } + std::array frame; + env->GetByteArrayRegion(audio, 0, samplesPerChannel * 2, reinterpret_cast(frame.data())); + if (env->ExceptionCheck()) { + return; + } + detector->ProcessChunk(frame.data(), samplesPerChannel, sampleRate); + } + catch (...) { + ThrowCxxJavaException(env); + } } -JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_VoiceActivityDetector_dispose +JNIEXPORT jfloat JNICALL Java_dev_onvoid_webrtc_media_audio_VoiceActivityDetector_getLastVoiceProbability (JNIEnv * env, jobject caller) { - webrtc::VoiceActivityDetector * activityDetector = GetHandle(env, caller); - CHECK_HANDLE(activityDetector); - - SetHandle(env, caller, nullptr); - - delete activityDetector; + try { + auto * detector = GetDetector(env, caller); + return detector ? detector->last_voice_probability() : 0; + } + catch (...) { + ThrowCxxJavaException(env); + return 0; + } } -JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_VoiceActivityDetector_initialize +JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_VoiceActivityDetector_dispose (JNIEnv * env, jobject caller) { - webrtc::VoiceActivityDetector * activityDetector = new webrtc::VoiceActivityDetector(); - - SetHandle(env, caller, activityDetector); -} \ No newline at end of file + try { + auto * detector = GetHandle(env, caller, "nativeHandle", PKG_INTERNAL "NativeObject"); + if (env->ExceptionCheck()) { + return; + } + SetHandle(env, caller, "nativeHandle", nullptr, PKG_INTERNAL "NativeObject"); + if (!env->ExceptionCheck()) { + delete detector; + } + } + catch (...) { + ThrowCxxJavaException(env); + } +} diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/StandaloneVoiceActivityDetector.java b/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/StandaloneVoiceActivityDetector.java new file mode 100644 index 000000000..b8a3b1c7a --- /dev/null +++ b/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/StandaloneVoiceActivityDetector.java @@ -0,0 +1,129 @@ +/* + * Copyright 2021 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc.media.audio; + +import java.util.Objects; + +import dev.onvoid.webrtc.internal.DisposableNativeObject; +import dev.onvoid.webrtc.internal.NativeLoader; + +/** + * Stateful WebRTC voice activity detector for mono signed 16-bit PCM samples. + * Accepts 10, 20 or 30 ms frames at 8000, 16000, 32000 or 48000 Hz. Each instance + * belongs to one continuous audio stream. Calls are serialized, including disposal; + * callers must still supply frames in stream order and avoid modifying samples + * during a call. Changing the sample rate or starting a new stream requires reset. + * Call {@link #dispose()} when finished. No peer connection factory is required. + * For the combined pitch-based probability estimator, use {@link VoiceActivityDetector}. + * Each class maintains its own stream history. + */ +public final class StandaloneVoiceActivityDetector extends DisposableNativeObject { + + static { + try { + NativeLoader.loadLibrary("webrtc-java"); + } + catch (Exception e) { + throw new RuntimeException("Load library 'webrtc-java' failed", e); + } + } + + /** Higher aggressiveness rejects more noise but can miss more speech. */ + public enum Mode { + /** Default quality mode. */ + QUALITY(0), + /** Low bitrate mode. */ + LOW_BITRATE(1), + /** Aggressive mode. */ + AGGRESSIVE(2), + /** Most aggressive mode. */ + VERY_AGGRESSIVE(3); + + private final int value; + + Mode(int value) { + this.value = value; + } + } + + private Mode mode; + + /** Creates a detector in {@link Mode#QUALITY} mode. */ + public StandaloneVoiceActivityDetector() { + this(Mode.QUALITY); + } + + /** + * Creates a detector with the given aggressiveness. + * @param mode detection mode, never null. + */ + public StandaloneVoiceActivityDetector(Mode mode) { + this.mode = Objects.requireNonNull(mode, "mode"); + initialize(mode.value); + } + + /** + * Analyzes one frame in stream order. Samples are numeric PCM values, so no + * byte order applies. This method performs no mixing or resampling. + * @param samples one mono frame of signed 16-bit PCM. + * @param sampleRate sample rate in Hz. + * @return true if WebRTC classifies the frame as speech; no confidence score + * is available. Decisions include WebRTC's built-in hangover. + * @throws NullPointerException if samples is null. + * @throws IllegalArgumentException if the rate or frame length is unsupported. + * @throws IllegalStateException if the detector has been disposed. + */ + public synchronized boolean isSpeech(short[] samples, int sampleRate) { + Objects.requireNonNull(samples, "samples"); + if (sampleRate != 8000 && sampleRate != 16000 && sampleRate != 32000 && sampleRate != 48000) { + throw new IllegalArgumentException("Unsupported sample rate: " + sampleRate); + } + int tenMs = sampleRate / 100; + if (samples.length != tenMs && samples.length != tenMs * 2 && samples.length != tenMs * 3) { + throw new IllegalArgumentException("Expected a mono frame of 10, 20 or 30 ms"); + } + return processInternal(samples, sampleRate); + } + + /** + * Changes aggressiveness without clearing stream history. + * @param mode detection mode, never null. + * @throws IllegalStateException if disposed. + */ + public synchronized void setMode(Mode mode) { + Objects.requireNonNull(mode, "mode"); + setModeInternal(mode.value); + this.mode = mode; + } + + /** + * Clears stream history while preserving the selected mode. + * @throws IllegalStateException if disposed. + */ + public synchronized void reset() { + resetInternal(mode.value); + } + + /** Frees native resources. Repeated calls are harmless. */ + @Override + public synchronized native void dispose(); + + private native void initialize(int mode); + private native boolean processInternal(short[] samples, int sampleRate); + private native void setModeInternal(int mode); + private native void resetInternal(int mode); +} diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/VoiceActivityDetector.java b/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/VoiceActivityDetector.java index 08dfe68d0..fa82eca87 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/VoiceActivityDetector.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/media/audio/VoiceActivityDetector.java @@ -17,17 +17,29 @@ package dev.onvoid.webrtc.media.audio; import dev.onvoid.webrtc.internal.DisposableNativeObject; +import dev.onvoid.webrtc.internal.NativeLoader; /** * A voice activity detector that analyzes audio data chunks to determine the probability of voice presence in the * audio stream. *

- * This class provides native methods to process audio and retrieve the probability of voice activity. + * Each instance belongs to one continuous mono signed 16-bit PCM stream. Calls, + * including disposal, are serialized; callers must still preserve stream order. + * No peer connection factory is required. * * @author Alex Andres */ public class VoiceActivityDetector extends DisposableNativeObject { + static { + try { + NativeLoader.loadLibrary("webrtc-java"); + } + catch (Exception e) { + throw new RuntimeException("Load library 'webrtc-java' failed", e); + } + } + /** * Constructs a new {@code VoiceActivityDetector} instance. */ @@ -36,26 +48,27 @@ public VoiceActivityDetector() { } /** - * Processes audio data to detect voice activity. + * Processes one 10 ms frame of mono signed 16-bit PCM in native byte order. + * Supported sample rates are 8000, 16000, 32000 and 48000 Hz. * * @param audio The audio data to process. * @param samplesPerChannel The number of samples in the audio chunk. * @param sampleRate The sample rate of the audio data in Hz. */ - public native void process(byte[] audio, int samplesPerChannel, int sampleRate); + public synchronized native void process(byte[] audio, int samplesPerChannel, int sampleRate); /** * Gets the probability of voice presence from the last processed audio data. * * @return A value between 0.0 and 1.0 representing the probability of voice presence. */ - public native float getLastVoiceProbability(); + public synchronized native float getLastVoiceProbability(); /** - * Releases native resources held by this object. + * Releases native resources held by this object. Repeated calls are harmless. */ @Override - public native void dispose(); + public synchronized native void dispose(); /** * Initializes the native resources for the voice activity detector. This method is called during construction and diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/StandaloneVoiceActivityDetectorTest.java b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/StandaloneVoiceActivityDetectorTest.java new file mode 100644 index 000000000..861916f1b --- /dev/null +++ b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/StandaloneVoiceActivityDetectorTest.java @@ -0,0 +1,251 @@ +/* + * Copyright 2021 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc.media.audio; + +import static org.junit.jupiter.api.Assertions.*; + +import java.io.InputStream; +import java.io.ByteArrayOutputStream; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.Arrays; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import dev.onvoid.webrtc.TestBase; +import dev.onvoid.webrtc.media.audio.StandaloneVoiceActivityDetector.Mode; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +class StandaloneVoiceActivityDetectorTest extends TestBase { + + private StandaloneVoiceActivityDetector detector; + + @BeforeEach + void setUp() { + detector = new StandaloneVoiceActivityDetector(); + } + + @AfterEach + void tearDown() { + detector.dispose(); + } + + @Test + void acceptsSupportedFrames() { + for (Mode mode : Mode.values()) { + detector.setMode(mode); + for (int rate : new int[] {8000, 16000, 32000, 48000}) { + for (int ms : new int[] {10, 20, 30}) { + detector.reset(); + assertFalse(detector.isSpeech(new short[rate / 1000 * ms], rate), + mode + " " + rate + " Hz / " + ms + " ms silence"); + } + } + } + } + + @Test + void detectsSpeech() throws Exception { + short[] speech = speech(); + for (Mode mode : Mode.values()) { + detector.setMode(mode); + detector.reset(); + int active = 0; + for (int offset = 0; offset < speech.length; offset += 160) { + if (detector.isSpeech(Arrays.copyOfRange(speech, offset, offset + 160), 16000)) { + active++; + } + } + assertTrue(active > 50, "Expected most synthetic speech frames to be active in " + mode); + // Hangover can keep a few silent frames active; sustained silence must clear it. + boolean last = true; + for (int i = 0; i < 100; i++) { + last = detector.isSpeech(new short[160], 16000); + } + assertFalse(last); + } + } + + @Test + void rejectsInvalidInput() { + assertThrows(NullPointerException.class, () -> detector.isSpeech(null, 16000)); + assertThrows(NullPointerException.class, () -> new StandaloneVoiceActivityDetector(null)); + assertThrows(NullPointerException.class, () -> detector.setMode(null)); + for (int rate : new int[] {-16000, 0, 44100, Integer.MAX_VALUE}) { + assertThrows(IllegalArgumentException.class, () -> detector.isSpeech(new short[160], rate)); + } + for (int size : new int[] {0, 1, 159, 161, 640, 1441}) { + assertThrows(IllegalArgumentException.class, () -> detector.isSpeech(new short[size], 16000)); + } + } + + @Test + void nativeBoundaryRejectsInvalidFrames() throws Exception { + // Bypass Java validation to exercise the bounds check protecting the JNI copy. + Method process = StandaloneVoiceActivityDetector.class.getDeclaredMethod("processInternal", short[].class, int.class); + process.setAccessible(true); + for (short[] frame : new short[][] {new short[0], new short[159], new short[1441]}) { + InvocationTargetException error = assertThrows(InvocationTargetException.class, + () -> process.invoke(detector, frame, 16000)); + assertTrue(error.getCause() instanceof IllegalArgumentException); + } + InvocationTargetException rateError = assertThrows(InvocationTargetException.class, + () -> process.invoke(detector, new short[160], 44100)); + assertTrue(rateError.getCause() instanceof IllegalArgumentException); + InvocationTargetException nullError = assertThrows(InvocationTargetException.class, + () -> process.invoke(detector, null, 16000)); + assertTrue(nullError.getCause() instanceof NullPointerException); + assertFalse(detector.isSpeech(new short[160], 16000)); + } + + @Test + void resetPreservesMode() throws Exception { + short[] speech = speech(); + for (Mode mode : Mode.values()) { + detector.setMode(mode); + detector.reset(); + boolean[] expected = decisions(detector, speech); + detector.reset(); + assertArrayEquals(expected, decisions(detector, speech)); + StandaloneVoiceActivityDetector fresh = new StandaloneVoiceActivityDetector(mode); + try { + assertArrayEquals(decisions(fresh, speech), expected); + } + finally { + fresh.dispose(); + } + } + } + + @Test + void instancesKeepSeparateHistory() throws Exception { + StandaloneVoiceActivityDetector other = new StandaloneVoiceActivityDetector(); + try { + decisions(detector, speech()); + assertFalse(other.isSpeech(new short[160], 16000)); + detector.dispose(); + assertFalse(other.isSpeech(new short[160], 16000)); + } + finally { + other.dispose(); + } + } + + @Test + void repeatedLifecycles() { + short[] frame = new short[1440]; + for (int i = 0; i < 1000; i++) { + StandaloneVoiceActivityDetector vad = new StandaloneVoiceActivityDetector(Mode.values()[i % 4]); + try { + assertFalse(vad.isSpeech(frame, 48000)); + vad.reset(); + assertFalse(vad.isSpeech(frame, 48000)); + } + finally { + vad.dispose(); + vad.dispose(); + } + } + } + + @Test + void fullScaleAudioDoesNotCrash() { + short[] frame = new short[1440]; + for (int i = 0; i < frame.length; i++) { + frame[i] = i % 2 == 0 ? Short.MIN_VALUE : Short.MAX_VALUE; + } + for (int i = 0; i < 1000; i++) { + detector.setMode(Mode.values()[i % 4]); + detector.isSpeech(frame, 48000); + } + } + + @Test + void disposalIsIdempotent() { + detector.dispose(); + detector.dispose(); + assertThrows(IllegalStateException.class, () -> detector.isSpeech(new short[160], 16000)); + assertThrows(IllegalStateException.class, () -> detector.setMode(Mode.AGGRESSIVE)); + assertThrows(IllegalStateException.class, () -> detector.reset()); + } + + @Test + void disposalCanRaceProcessing() throws Exception { + ExecutorService executor = Executors.newFixedThreadPool(2); + CountDownLatch start = new CountDownLatch(1); + try { + Future processing = executor.submit(() -> { + start.await(); + for (int i = 0; i < 1000; i++) { + try { + detector.isSpeech(new short[160], 16000); + detector.setMode(Mode.AGGRESSIVE); + detector.reset(); + } + catch (IllegalStateException disposed) { + break; + } + } + return null; + }); + Future disposing = executor.submit(() -> { + start.await(); + detector.dispose(); + return null; + }); + start.countDown(); + processing.get(10, TimeUnit.SECONDS); + disposing.get(10, TimeUnit.SECONDS); + assertThrows(IllegalStateException.class, () -> detector.reset()); + } + finally { + executor.shutdownNow(); + } + } + + private static boolean[] decisions(StandaloneVoiceActivityDetector vad, short[] samples) { + boolean[] result = new boolean[samples.length / 160]; + for (int i = 0; i < result.length; i++) { + result[i] = vad.isSpeech(Arrays.copyOfRange(samples, i * 160, (i + 1) * 160), 16000); + } + return result; + } + + private static short[] speech() throws Exception { + try (InputStream input = StandaloneVoiceActivityDetectorTest.class.getResourceAsStream("/audio/vad-speech-16k.pcm")) { + assertNotNull(input); + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + byte[] chunk = new byte[4096]; + int count; + while ((count = input.read(chunk)) != -1) { + bytes.write(chunk, 0, count); + } + ByteBuffer buffer = ByteBuffer.wrap(bytes.toByteArray()).order(ByteOrder.LITTLE_ENDIAN); + short[] samples = new short[buffer.remaining() / 2]; + buffer.asShortBuffer().get(samples); + return samples; + } + } +} diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/VoiceActivityDetectorTest.java b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/VoiceActivityDetectorTest.java index 3a869ba10..96f4b78ba 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/VoiceActivityDetectorTest.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/VoiceActivityDetectorTest.java @@ -18,62 +18,243 @@ import static org.junit.jupiter.api.Assertions.*; +import java.io.ByteArrayOutputStream; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.Arrays; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import dev.onvoid.webrtc.TestBase; + import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; -public class VoiceActivityDetectorTest { +class VoiceActivityDetectorTest extends TestBase { + + private VoiceActivityDetector detector; + + @BeforeEach + void setUp() { + detector = new VoiceActivityDetector(); + } - private VoiceActivityDetector detector; + @AfterEach + void tearDown() { + detector.dispose(); + } + @Test + void subclassFieldsDoNotHideHandle() { + // NativeObject's private pointer must not bind to a subclass's unrelated field. + // Nonzero values used to be interpreted as pointers by process()/dispose(). + for (long value : new long[] {0, 7, Long.MIN_VALUE, Long.MAX_VALUE}) { + ShadowedDetector vad = new ShadowedDetector(value); + try { + vad.process(new byte[320], 160, 16000); + assertEquals(1.0f, vad.getLastVoiceProbability()); + assertEquals(value, vad.nativeHandle); + } + finally { + vad.dispose(); + } + } + } - @BeforeEach - void setUp() { - detector = new VoiceActivityDetector(); - } + private static class ShadowedDetector extends VoiceActivityDetector { + private long nativeHandle; - @AfterEach - void tearDown() { - if (detector != null) { - detector.dispose(); - } - } + private ShadowedDetector(long value) { + nativeHandle = value; + } + } - @Test - void testInitialization() { - assertNotNull(detector, "Detector should be initialized"); - } + @Test + void preservesProbabilityDelay() { + // Upstream starts at 1 and updates after collecting three 10 ms chunks. + // This also guards against replacing this estimator with binary decisions. + assertEquals(1.0f, detector.getLastVoiceProbability()); + detector.process(new byte[320], 160, 16000); + detector.process(new byte[320], 160, 16000); + assertEquals(1.0f, detector.getLastVoiceProbability()); + detector.process(new byte[320], 160, 16000); + assertEquals(0.01f, detector.getLastVoiceProbability(), 0.000001f); + } - @Test - void testProcessAndGetProbability() { - // Create sample audio data (silence - all zeros). - byte[] silenceData = new byte[320]; // 10ms at 16kHz mono with 16-bit samples - int samplesPerChannel = 160; - int sampleRate = 16000; + @Test + void acceptsSupportedRates() { + for (int rate : new int[] {8000, 16000, 32000, 48000}) { + VoiceActivityDetector vad = new VoiceActivityDetector(); + try { + for (int i = 0; i < 6; i++) { + vad.process(new byte[rate / 100 * 2], rate / 100, rate); + } + assertEquals(0.01f, vad.getLastVoiceProbability(), 0.000001f); + } + finally { + vad.dispose(); + } + } + } - // Process silent audio. - detector.process(silenceData, samplesPerChannel, sampleRate); + @Test + void detectsSpeechProbability() throws Exception { + byte[] speech = speech(); + float maximum = 0; + for (int offset = 0; offset < speech.length; offset += 320) { + detector.process(Arrays.copyOfRange(speech, offset, offset + 320), 160, 16000); + float probability = detector.getLastVoiceProbability(); + assertTrue(probability >= 0 && probability <= 1); + if (offset >= 640) { + assertTrue(probability > 0 && probability < 1, "Expected a non-binary probability"); + } + // Ignore the initial value of 1 before the first probability update. + if (offset >= 640) { + maximum = Math.max(maximum, probability); + } + } + assertTrue(maximum > 0.5f, "Expected speech in the synthetic speech fixture"); + for (int i = 0; i < 100; i++) { + detector.process(new byte[320], 160, 16000); + } + assertEquals(0.01f, detector.getLastVoiceProbability(), 0.000001f); + } - float probability = detector.getLastVoiceProbability(); + @Test + void rejectsInvalidInput() { + assertThrows(NullPointerException.class, () -> detector.process(null, 160, 16000)); + for (int count : new int[] {-1, 0, 159, 161, 320, Integer.MAX_VALUE}) { + assertThrows(IllegalArgumentException.class, () -> detector.process(new byte[320], count, 16000)); + } + for (int rate : new int[] {-16000, 0, 44100, Integer.MAX_VALUE}) { + assertThrows(IllegalArgumentException.class, () -> detector.process(new byte[320], 160, rate)); + } + assertThrows(IllegalArgumentException.class, () -> detector.process(new byte[319], 160, 16000)); + // Existing callers may supply a larger buffer; only the requested samples are read. + detector.process(new byte[640], 160, 16000); + } - assertTrue(probability >= 0.0f && probability <= 1.0f, - "Probability should be between 0.0 and 1.0"); - } + @Test + void historiesAreIndependent() throws Exception { + StandaloneVoiceActivityDetector standalone = new StandaloneVoiceActivityDetector(); + VoiceActivityDetector other = new VoiceActivityDetector(); + try { + byte[] speech = speech(); + for (int offset = 0; offset < speech.length; offset += 320) { + short[] frame = new short[160]; + ByteBuffer.wrap(speech, offset, 320).order(ByteOrder.nativeOrder()).asShortBuffer().get(frame); + standalone.isSpeech(frame, 16000); + other.process(Arrays.copyOfRange(speech, offset, offset + 320), 160, 16000); + } + assertEquals(1.0f, detector.getLastVoiceProbability()); + for (int i = 0; i < 3; i++) { + detector.process(new byte[320], 160, 16000); + } + assertEquals(0.01f, detector.getLastVoiceProbability(), 0.000001f); + standalone.reset(); + standalone.setMode(StandaloneVoiceActivityDetector.Mode.VERY_AGGRESSIVE); + other.dispose(); + assertEquals(0.01f, detector.getLastVoiceProbability(), 0.000001f); + } + finally { + standalone.dispose(); + other.dispose(); + } + } - @Test - void testDispose() { - // The First operation should work. - detector.process(new byte[320], 160, 16000); + @Test + void repeatedLifecycles() { + byte[] frame = new byte[320]; + for (int i = 0; i < 1000; i++) { + VoiceActivityDetector vad = new VoiceActivityDetector(); + try { + for (int j = 0; j < 3; j++) { + vad.process(frame, 160, 16000); + } + assertEquals(0.01f, vad.getLastVoiceProbability(), 0.000001f); + } + finally { + vad.dispose(); + vad.dispose(); + } + } + } - // Dispose resources. - detector.dispose(); + @Test + void fullScaleAudioDoesNotCrash() { + byte[] frame = new byte[960]; + ByteBuffer pcm = ByteBuffer.wrap(frame).order(ByteOrder.nativeOrder()); + for (int i = 0; i < 480; i++) { + pcm.putShort(i % 2 == 0 ? Short.MIN_VALUE : Short.MAX_VALUE); + } + for (int i = 0; i < 1000; i++) { + detector.process(frame, 480, 48000); + float probability = detector.getLastVoiceProbability(); + assertTrue(probability >= 0 && probability <= 1); + } + } - // After disposal, operations should either throw an exception or be safe to call. - assertThrows(Exception.class, () -> { - detector.process(new byte[320], 160, 16000); - }); + @Test + void disposalIsIdempotent() { + detector.dispose(); + detector.dispose(); + assertThrows(IllegalStateException.class, () -> detector.process(new byte[320], 160, 16000)); + assertThrows(IllegalStateException.class, () -> detector.getLastVoiceProbability()); + } - detector = null; - } + @Test + void disposalCanRaceProcessing() throws Exception { + ExecutorService executor = Executors.newFixedThreadPool(2); + CountDownLatch start = new CountDownLatch(1); + try { + Future processing = executor.submit(() -> { + start.await(); + for (int i = 0; i < 1000; i++) { + try { + detector.process(new byte[320], 160, 16000); + detector.getLastVoiceProbability(); + } + catch (IllegalStateException disposed) { + break; + } + } + return null; + }); + Future disposing = executor.submit(() -> { + start.await(); + detector.dispose(); + return null; + }); + start.countDown(); + processing.get(10, TimeUnit.SECONDS); + disposing.get(10, TimeUnit.SECONDS); + } + finally { + executor.shutdownNow(); + } + } + private static byte[] speech() throws Exception { + try (InputStream input = VoiceActivityDetectorTest.class.getResourceAsStream("/audio/vad-speech-16k.pcm")) { + assertNotNull(input); + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + byte[] chunk = new byte[4096]; + int count; + while ((count = input.read(chunk)) != -1) { + bytes.write(chunk, 0, count); + } + byte[] pcm = bytes.toByteArray(); + ByteBuffer source = ByteBuffer.wrap(pcm).order(ByteOrder.LITTLE_ENDIAN); + ByteBuffer target = ByteBuffer.allocate(pcm.length).order(ByteOrder.nativeOrder()); + while (source.hasRemaining()) { + target.putShort(source.getShort()); + } + return target.array(); + } + } } diff --git a/webrtc/src/test/resources/audio/README.md b/webrtc/src/test/resources/audio/README.md new file mode 100644 index 000000000..525aa2a45 --- /dev/null +++ b/webrtc/src/test/resources/audio/README.md @@ -0,0 +1,10 @@ +# VAD fixture + +`vad-speech-16k.pcm` is one second of mono signed 16-bit little-endian PCM at +16000 Hz. It is synthetic speech from the phrase "The quick brown fox jumps +over the lazy dog.", generated with eSpeak NG 1.51, voice `en-us`, default +settings. The first 200 ms were skipped; the next second was retained after +resampling from 22050 Hz to 16000 Hz with linear interpolation in Python. + +This generated audio fixture is provided under the repository's Apache-2.0 +license. No speech engine or external download is needed to run the tests. diff --git a/webrtc/src/test/resources/audio/vad-speech-16k.pcm b/webrtc/src/test/resources/audio/vad-speech-16k.pcm new file mode 100644 index 000000000..77ad513ef Binary files /dev/null and b/webrtc/src/test/resources/audio/vad-speech-16k.pcm differ