confucius4_r2t2: make streaming chunks incremental instead of O(utterance)

[scriptease]

Confucius4-R2T2 streaming re-decodes the accumulated audio on every LSP chunk. Each chunk re-ran the mel frontend, the full audio tower and a full thinker prefill over all audio so far, so per-chunk cost grew with the utterance. Once a chunk took longer than its 320 ms of audio, the live route fell behind the socket and drained the queued chunks after the speaker stopped. On longer dictations this shows up as a finalize of 10 s or more at full GPU. Two users of a TypeWhisper integration have hit it.

This PR keeps the reference algorithm and its output, and only removes the recomputation.

Changes

• Audio tower, model-local (confucius4_r2t2/session.cpp): the tower is block-diagonal per n_window_infer window with per-chunk convolutions and positions, so embeddings of completed windows do not change. They are cached, and only the trailing partial window is re-encoded.
• Thinker prefix reuse, additive and opt-in framework change: QwenCausalDecodeRuntime::prefill_embeddings_into_cache gains an optional keep_prefix_steps, default 0. It retains the K/V rows of an unchanged prompt prefix and falls back to a full prefill whenever the cache was rebuilt. retainable_prefix_steps reports what would be kept. GreedyQwenDecoderRuntime::generate_incremental exposes this with 512-step capacity buckets. Existing callers are unchanged.
• R2T2 keeps the template head and the cached audio tokens, and prefills only the tail audio, the template close and the stable text prefix.
• There are no server, route or streaming-event changes. Deltas and transcript.text.done are produced exactly as before.

Validation

Apple Silicon, Metal, r2t2-q8_0.gguf, 320 ms chunks.

	before	after
60 s speech via live route in real time: last audio byte to transcript.text.done	24.9 s	0.37 s
60 s file, audiocpp_cli --mode streaming, wall	87 s	39 s
14 s sample_16k.wav, wall	13.6 s	8.8 s
streaming transcripts, 14 s and 60 s		identical

• qwen_chunked_prefill_test is extended with kept-prefix parity against the reference prefill, including decode after a kept prefix, block-size changes and the fallback paths. It passes.
• test_confucius4_r2t2_graph_reuse --backend metal and test_confucius4_r2t2_transcription --backend metal pass.
• tests/confucius4_r2t2/compare.py against golden_sample16k.json gives the same result as before this change, including the same two pre-existing internal chunk differences.
• Memory was not measured. The larger capacity bucket can hold up to 384 more decode-cache steps than before, and the cached audio embeddings grow with the stream.
• CUDA and Vulkan were not tested.

Known limits

• The stable text prefix is still re-prefilled every chunk, by design of LSP. It runs at about 0.88x real time at 150 s. Very long uninterrupted streams are better served by the endpointing in feat(confucius4_r2t2): add bounded VAD endpointing for long streaming ASR #608, which this change complements.
• Crossing a 512-step cache bucket costs one full prefill.
• Peak normalization and the log-mel floor span all accumulated audio, so a louder segment rewrites earlier windows' features. Cached windows are reused only while their log-mel features are bit-identical; otherwise the encoder cache and the thinker rows of those audio tokens are dropped, and that chunk costs a full re-encode and prefill. Added after review; see the thread below.

Credit

The R2T2 port is by @davidxifeng. His branch r2t2_acc independently takes the same approach, including memcmp checks that fall back whenever reuse would not be bit-identical; this PR now does the same for the audio cache. I asked him first in davidxifeng#1. If you or he prefer his version, I'm happy to close this.

🤖 Generated with Claude Code

[0xShug0]

@scriptease I'm not sure whether this is only a documentation issue or indicates a more subtle problem. The streaming behavior appears to have changed.

Cached audio windows are not always immutable. R2T2 recomputes a global log-mel floor from all accumulated audio. Later, louder audio can therefore change features belonging to earlier windows, but this PR permanently reuses their cached embeddings.

I reproduced this with a realistic quiet-then-normal stream:

• Normal sample_16k.wav: baseline and PR output identical.
• First section reduced by 36 dB: final transcript identical, but one chunk’s decoded/fixed text changed.
• First section reduced by 80 dB: final transcript remained identical, but many intermediate results and emitted delta boundaries changed. For example, baseline emitted "'ve been here for over" as one delta; the PR emitted "'ve been", " here for", " over" separately.

[scriptease]

Thanks for testing this, you're right. Cached windows kept stale features once louder audio moved the peak normalization and log-mel floor. I reproduced it with the first 8.5 s of sample_16k.wav at -36 dB and -80 dB: the PR diverged from baseline in 8 and 18 chunks, and the final text changed as well.

The new commit keeps the log-mel features of cached windows and compares them bitwise on every chunk. On any difference it drops the encoder cache and the thinker rows of those audio tokens, so that chunk recomputes fully. With it, every chunk's committed and decoded text matches baseline exactly on those clips and on the unmodified sample. The cost on normal speech is unchanged: 36 s against 80 s for baseline on a 60 s clip.

I also rebased onto main, including 955c872, and thanks for merging that.

🤖 Generated with Claude Code