Stream history: in-memory H.264+AAC ring via ffmpeg (opt-in CMake toggle)

[Gin890]

Implements the in-memory ffmpeg stream-history ring discussed in #Stream (option 1): video + audio are encoded in the background into an in-memory ring, and nothing touches storage until save() is called, when the ring is muxed into a single MP4/MKV file.

Toggle (compile-time)

-DPA_STREAM_HISTORY_FFMPEG=ON

• OFF (default) — every existing build/CI path is byte-for-byte unaffected (the new .cpp isn't even compiled; StreamHistorySession.cpp keeps the old #if shape).
• ON, Windows — headers from 3rdParty/ffmpeg/ (the ffmpeg include/ tree) + import libs avcodec/lib avformat/lib avutil/lib swscale/lib swresample/lib in 3rdPartyBinaries/ (mirrors the OpenCV import-lib precedent). The existing THIRD_PARTY_DLLS glob auto-deploys any ffmpeg DLLs placed in 3rdPartyBinaries/.
• ON, macOS/Linux — pkg-config (avcodec avformat avutil swscale swresample); Ubuntu: sudo apt install libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libswresample-dev, macOS: brew install ffmpeg.
• ON but ffmpeg missing / too old — FATAL_ERROR at configure time with an explicit message (ffmpeg ≥ 5.1 / libavcodec 59+ required for the channel-layout API).

What the tracker does (Recording/StreamHistoryTracker_Ring.{h,cpp})

• Video — same worker pattern as the JPEG tracker (bounded queue of 10, drop-oldest when lagging, AsyncTask on unlimited_normal()). Replaces JPEG with the ffmpeg C API: hardware H.264 encoders first (h264_qsv/h264_nvenc/h264_amf on Windows, h264_videotoolbox on macOS, h264_v4l2m2m on Linux) with libx264 veryfast fallback; no B-frames, forced keyframe every 2 s, NV12 fast path straight into sws_scale with a Qt toImage() RGB888 fallback for other pixel formats. Honors the existing StreamHistoryOption settings (RESOLUTION / VIDEO_FPS / ENCODING_MODE / VIDEO_QUALITY / VIDEO_BITRATE).
• Audio — on_samples → swresample to 48 kHz → native aac encoder, 1024-sample frames, inline (sub-millisecond). Includes a deinterleave for DUAL_44100, which the session declares as 1ch but the pipeline actually delivers as interleaved stereo.
• Ring — (WallClock, encoded packet) deques evicted by the HISTORY_SECONDS window plus a hard ~90 MB byte cap (≈23 min at the default 512 kbps). Zero I/O until prompted.
• save(filename) — snapshot the ring (shared packets, no deep copy), rewind to the first keyframe in the ring (oldest decodable start; loses at most one ~2 s keyframe interval from the head of the window), trim audio to the same start point, mux to MP4 (or MKV when the filename ends .mkv) with libavformat using copy-only semantics. ErrorReports saves to Video.mp4, so MP4 is the default target.

One deviation from the discussion worth calling out: the plan said Opus, but I used AAC — the save target is Video.mp4 (Discord attachment) and H.264+AAC MP4 plays everywhere, whereas Opus-in-MP4 has spotty player support. The ring design (compressed audio packets, 48 kHz, ~20 ms frames) is otherwise as discussed; swapping to libopus is a ~30-line change if you'd rather have it.

Failure behavior

Encoder init, scaler, resampler, or open failures log and disable just that stream (video or audio) — the bot never crashes, matching the existing tracker's failure style. save() with an empty ring logs and returns false.

Verification

No ffmpeg/Qt toolchain in this sandbox, so this is not a full production build. What was done instead:

• Compiled the real StreamHistoryTracker_Ring.cpp (unmodified, straight from the repo) with g++ 13 -std=c++23 -Wall -Wextra against signature-faithful ffmpeg ≥5.1 API stubs + a minimal Qt QVideoFrame/QImage model, and ran it behaviorally: full video+audio pipeline → MP4 mux (packet counts, keyframe cadence, µs timestamps, A/V alignment), HISTORY_SECONDS eviction, DUAL_44100 deinterleave/resample (44100 → 48000, exact frame counts), empty-ring save, .mkv muxer selection, and x264 veryfast/zerolatency/bframes=0 options. All pass, repeated runs included.
• The stub x264 encoder deliberately models a hardware-encoder-style one-deep output queue (every send_frame hits EAGAIN until drained), which exercises the EAGAIN/retry path.
• ASan+UBSan build of the same harness passes all tests.

The stub review caught and fixed (commit 7cd8981), against real ffmpeg 5.1 semantics:

• save() rewound to the last keyframe → a 30 s window saved only the final ~2 s. Now rewinds to the first keyframe.
• Video mux timestamps off by a factor of 1e6 (us * den / num on a 1/1e6 timebase) → unplayable files. Now av_rescale_q from 1/1000000.
• Muxed packets never set stream_index → real libavformat would route audio into the video stream.
• swr_alloc_set_opts2(&swr, ...) — 9-arg hybrid that does not exist in ffmpeg ≥5.1 (8 params, out layout first); would not compile against real headers.
• avcodec_send_frame EAGAIN (full encoder input queue — realistic on QSV/NVENC/AMF) permanently clogged the encoder instead of draining + retrying.
• Unlocked video_enabled writes racing on_frame; ffmpeg libs linked PRIVATE on POSIX; explicit libavutil/error.h/rational.h includes.

Please still do a real build with the toggle ON before relying on it — in particular the hardware-encoder paths (QSV/NVENC/AMF) are the part most likely to need a first-pass tweak.

Files

• SerialPrograms/Source/CommonFramework/Recording/StreamHistoryTracker_Ring.{h,cpp} — new tracker
• SerialPrograms/Source/CommonFramework/Recording/StreamHistorySession.cpp — 4-line #if selection
• SerialPrograms/CMakeLists.txt — option, per-platform link blocks, compile definition
• SerialPrograms/cmake/SourceFiles.cmake — new header in the list; new .cpp appended only when the option is ON

[Gin890]

MP4 blank screen: malformed avcC fixed (commit 93253e8)

Symptom: saved MP4 had the correct duration but decoded to a blank screen, while the same ring saved as MKV played fine.

Root cause: h264_avcc_from_annexb_extradata() (added in b6e2221) built a malformed avcC record for the MP4 stream header:

1. a stray 5th byte (sps[4], the first byte of the SPS RBSP) was written before the reserved/lengthSizeMinusOne byte, and
2. the numOfSequenceParameterSets byte was 0xE0 (= 0 SPS) instead of 0xE1 (= 1 SPS).

MP4 players take the SPS from the stream header (avcC); with a zero SPS count they get none, so every sample fails to decode -> blank screen, while the container metadata (duration, size) is still correct. MKV is unaffected because it stores the Annex B SPS/PPS in-band/in CodecPrivate.

The record now follows the ISO/IEC 14496-15 avcC layout 01 <profile> <compat> <level> FF E1 <spsLen16> <SPS> 01 <ppsLen16> <PPS> — byte-for-byte the same layout ffmpeg's own mov muxer writes from the same SPS/PPS. Verified by building the record with this code, muxing it into an MP4, and decoding (clean, non-blank output). Linux build + full command-line test suite pass (1962/1962).

[Gin890]

MP4 save now fails with "non monotonically increasing dts" — fixed (commit 4d40e92)

Root cause (separate from the avcC bug): ring video packets are stamped with the wall-clock time of the frame they were drained from, and a hardware encoder can emit several packets for one frame (or flush two or more at once) — so consecutive ring packets share a timestamp. The MP4 muxer requires strictly increasing dts and fails the whole save:

[mp4 @ ...] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 596 >= 596

mux_to_file() now clamps a non-increasing video dts to last + 1 (1 ms at the 1/1000 video timebase), and skips empty ring packets instead of passing an empty sample to the muxer (which would also break the Annex B → AVCC packet conversion).

Verified with a standalone lavf harness: two consecutive H.264 packets with equal dts → AVERROR(EINVAL) from av_interleaved_write_frame() (the reported error); with the clamp the same sequence muxes and decodes cleanly (60/60 frames). Linux build + full command-line test suite pass (1962/1962).

On keeping the Annex B → AVCC conversion for MP4: yes, it is still needed.

• The h264_amf wrapper emits Annex B (SPS/PPS extradata and 00 00 00 01-coded packets) even with AV_CODEC_FLAG_GLOBAL_HEADER — established empirically on AMD iGPUs, where AMF is the first encoder that works.
• The vendored ffmpeg 8.x MP4 muxer crashes (access violation in avformat_write_header) when handed H.264 in that form — that crash is exactly why the conversion was added in b6e2221.
• With the avcC record fixed in 93253e8 the conversion now produces a spec-conformant AVCC MP4 (verified by muxing and decoding its output).
• The only way to drop it would be an encoder that natively emits AVCC (nvenc/qsv do, but AMF does not, and ffmpeg's AMF wrapper exposes no option to force it).

[Gin890]

Audio never opened: AAC encoder missing sample_fmt — fixed (commit aa6809c)

"Stream history: Failed to open AAC encoder. Disabling audio." was not a Windows/vendored-ffmpeg issue — it reproduces on any ffmpeg build. init_audio_encoder() set rate, timebase, bitrate, channel layout and flags, but never actx->sample_fmt, so avcodec_open2() failed with EINVAL ("Invalid audio sample format: -1") every time. Audio has been silently disabled on all platforms since the ring tracker was added.

The native AAC encoder only accepts AV_SAMPLE_FMT_FLTP (planar float), so the fix:

• init_audio_encoder() now sets actx->sample_fmt = AV_SAMPLE_FMT_FLTP (and frame_size = 1024) before opening,
• on_samples() now builds each 1024-sample encode frame as FLTP by de-interleaving the accumulated packed resampler output into the frame's per-channel planes (previously it memcpy'd packed data into data[0] of a frame declared as AV_SAMPLE_FMT_FLT, which the encoder would have rejected anyway),
• the open-failure log is split so a channel-layout failure and an avcodec_open2() failure are reported separately, the latter with the ffmpeg error string.

Verified with a standalone harness replicating the full pipeline (resample 44.1/48 kHz → packed FLT → de-interleave → FLTP frames → AAC encode with AV_CODEC_FLAG_GLOBAL_HEADER): encoder opens, produces packets, and exposes the 5-byte AudioSpecificConfig extradata that the MP4/MKV muxers need; AAC-with-ASC muxes and decodes cleanly in both MP4 and MKV. Linux build + full command-line test suite pass (1962/1962).

[Gin890]

Audio resampler never initialized — fixed (commit 886a710)

The AAC encoder now opens (the sample_fmt/FLTP fix is working — see the audio = AAC 1ch @ 48 kHz line), but make_audio_resampler() only called swr_alloc_set_opts2(), which allocates the SwrContext and fills in the options — it does not initialize it. The header is explicit: "Once all values have been set, it must be initialized with swr_init()."

So every swr_convert() call logged [SWR] Context has not been initialized and failed, no samples were ever accumulated, and saved files had a video track but no audio.

Fix: call swr_init(swr) after allocation in make_audio_resampler(), and swr_free() it if initialization fails (the existing "Failed to create audio resampler. Disabling audio." path now reports real failures).

Verified the exact case from the log (48 kHz mono → 48 kHz mono, packed FLT in/out) in a standalone harness: swr_init = 0, swr_convert returns the full 1024 samples with clean data. Linux build + full command-line test suite pass (1962/1962).

[Gin890]

MKV audio/duration: Matroska needs millisecond timestamps — fixed (commit e49393d)

"MKV: no audio, plays 30 s then freezes on the last frame for >20 min" traced to a Matroska-specific timestamp issue, reproduced with a standalone harness muxing real H.264+AAC packets through the same avformat calls:

The matroska muxer writes packet timestamps into the file verbatim. Its write_header() rewrites every stream's timebase to 1/1000 (Matroska's native TimestampScale is milliseconds) but does not rescale the dts/pts the application submits. We submitted audio with a 1/48000 timebase and sample-count timestamps, so:

• every audio SimpleBlock landed 48× too late on the file timeline (a 30 s clip's 1024-sample audio blocks were spaced 1.024 s apart, each carrying only 21 ms of sound) → the player had no continuous audio to play during the video → "no audio"
• the segment Duration element became 1441813 ms ≈ 24 min for a 30 s clip → after the video ends, the player waits out the remaining ~19 min on the last frame → "freeze on the last frame"

MP4 was unaffected because the mov muxer rescales from the stream timebase itself (hence MP4 audio worked).

Fix in mux_to_file():

1. For .mkv only: use a 1/1000 stream timebase for both streams and rescale the packet dts into it (video: µs → ms; audio: samples → ms). .mp4 keeps the encoder's video timebase and the 1/48000 audio timebase it already handled correctly.
2. A strictly-increasing dts guard for audio packets, mirroring the video one.
3. Interleave the video/audio writes in timestamp order instead of writing all video then all audio, which laid the Matroska clusters out of time order (all audio data physically after all video data).

Harness verification: old behavior → MKV Duration element 1441813 (24 min) for a 30 s clip; new behavior → duration ≈ 30 s, audio packets decode at 0…30 s, MP4 path unchanged. Linux build + full command-line test suite pass (1962/1962).

[pa-devbot]

Follow-up to the encoder-format audit (nvenc/qsv/amf/videotoolbox/v4l2m2m wrappers reviewed against the ffmpeg 8.1 sources): no new encoder-specific conversion code is needed - every candidate emits its extradata and packets in the same form. This commit adds one log line at mux time: the first 4 bytes of the first video ring packet, printed next to the existing extradata hex. If a driver ever reported one H.264 form (AVCC extradata) while emitting the other (Annex B packets), the MP4 save would otherwise be a silent blank file; this makes the mismatch immediately visible in the log. Log-only, no behavior change. (aedba6c)

[pa-devbot]

Added h264_mf (the OS-native Media Foundation H.264 encoder) to the Windows candidate list, after h264_amf and before mpeg4. It is compiled into the vendored LGPL build (verified in avcodec-63.dll) and present on essentially all Windows 8+: when none of the GPU vendors' encoders (qsv/nvenc/amf) is available, the session now falls back to real, universally playable H.264 instead of MPEG-4 Part 2. Device-backed like the hardware wrappers, so a runtime failure follows the existing leak-and-poison path; rotation/poison/fallback logic needed no changes. The mux-time log lines report its extradata/packet form on first use. (9508c51, rebased onto dc8129c)

[Mysticial]

This is GPL safe right? We're using the LGPL part of ffmpeg because we're only including headers and their pre-compiled binaries?

[jw098]

I believe so. I specifically used a LGPL build of ffmpeg. So, I think it should be LGPL?