Repository navigation
Sync gfx11 with master (2026-11-03) - #176
Merged
Merged
Conversation
) * server : support typed content input for rerank endpoint Assisted-by: Opencode/Qwen3.8 27B * server : address rerank multimodal review findings - reject multimodal rerank input when the model has no rerank chat template (images were silently dropped) - translate handle_media download/base64 failures to 400 instead of 500 - reject negative top_n with 400 instead of letting resize throw - fix comment typos, redundant comments, indentation and trailing whitespace - README: document that TEI input accepts the same shape as Jina; drop stray blank line Assisted-by: opencode * Trimmed README --------- Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
…e-type (ggml-org#23343) * nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type * gguf-py : write Metaspace prepend_scheme as GGUF metadata * vocab : parse metaspace prepend_scheme into enum, rename key * vocab : use metaspace prepend_scheme in detokenize, add TODOs for other tokenizers * vocab : use constant for metaspace escaped space --------- Co-authored-by: Martin Cimmino <martin.cimmino@domyn.com>
…patibility (ggml-org#30828) std::aligned_alloc is unavailable on Android bionic below API 28 (Termux targets API 24); use posix_memalign like ggml.c does.
* Multi-column on Intel XMX matrix engines refactored Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0). * Fixed regression bug real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4 * Removed GGML_TYPE_Q8_0 activation * rewritten xmx_mul_mat 64-byte alignment and 16-byte pitch check on the activations * Implemented fused gate/up kernel One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types. * Removed the temporary XMX_NOGLU environment switch * Q2_K min-sum optimization Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group * Added dispatch check and GRF sweep tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%). Q6_K is unchanged. MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361. Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without. * Renamed variables * Added 2 environment variables GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it. * Update esimd.hpp 1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces. 2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once. The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
qwen4exp published the hc-wide residual [n_embd, hc, n_tokens] as t_layer_inp, but the drafter reads layer inputs as [n_embd, n_tokens] rows: llama_context sizes each embd_layer_inp buffer as n_embd * n_batch and the DFlash encoder input is n_extract * n_embd wide. Evaluating a DFlash draft over a qwen4exp target therefore aborts on the first request in llama_context::extract_layer_inputs. Collapse the streams with a plain mean before exposing them, as deepseek4 already does for its hyper-connection residual, and build the mean only for the layers a drafter actually requested. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* glm5 : avoid sel_mask_2d input * cont : avoid new_pool_flat input
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
* opencl: add Adreno depthwise conv2d Assisted-by: OpenCode * opencl: use local program for depthwise conv2d Assisted-by: OpenCode * opencl: use singed type for args --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
…l-org#30994) * server: realtime API: send complete after committed audio drains * server: realtime API: complete a commit at its end row Completion is triggered when the stream runs dry, which is a timing proxy: audio appended while the tail drains leaks into the committed item, a final commit before the drain merges both items, and back to back commits collapse into one. mtmd_audio_stream_flush() records the row count at the end of the flushed audio (mtmd_audio_stream_get_n_rows_flushed), next to the rows produced so far (mtmd_audio_stream_get_n_rows). The slot keeps a FIFO of these end rows and completes each commit once every row up to it is decoded. --------- Co-authored-by: Pascal <admin@serveurperso.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
…l-org#30684) Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
…INARY_NX` (back-to-back ADDs, etc) (ggml-org#30843) * hex-fusion: support for GELU+MUL, REPEAT+GET_ROWS and BINARY_NX (back-to-back ADD/MUL/...) * hex-fusion: factor out fusion logic into separate .h and .cpp and update all related code * hex-fusion: allow for RMS_NORM+MUL fusions for non-contig tensors * hex-fusion: streamline fusion eligibility checks and cover missing corner cases * hex-fusion: streamline fusion flow * hex-opbatch: make n_tensors limit more scalable * hex-bufs: adjust the trace buff sizing * hex-glu: cleanup glu-ops interfaces and add kernel_params * hex-kparams: clean up kparams handling in unary, ssm-conv, and other ops * hex-kparams: update the rest of the ops PAD,DIAG,... to use kparams * hex-unary: add support for collapsing dims * hex-ops: cleanup redundant runtime checks * hex-ops: consistent use of kernel_type in all kparams * hex-fusion: remove redundant check in gdn+cpy * hex-fusion: further streamlining of fusion rules and code * hex-fusion: cleanup naming * hex-opfmt: cleanup op formatting * hex-fusion: bump fusion lookahead to 64 * hex-fusion: fuse SSM_CONV+SILU * hex-fusion: fix hvx register spills in fused ssm_conv+silu * hex-fusion: add more GLU fusions * hex-fusion: remove duplicate checks and use ggml-common helpers * hex-kparams: refactor kparams into htp-kparams.h/cpp * hex-fusion: fix binary-nx fusion regression * hex-kparams: another pass at cleaning up kparams (removing unused, etc) * hex-kparams: streamling nameing and move _check_ wrappers into .h * hex-kparams: templetize htp_kparams_check_<op> to reduce code bloat * hex-ops: de-duplicate op dispatch switches * hex-kparams: trim down exposed kparams calls * hex-kparams: simplify type casting of kparams structs * hex-fence: de-dup fence tensor construction * hex-opbatch: simplify tensor shape init/checks * hex-mm: cleanup leftovers of the original QK defines * hex-fa: fix stack overflow on older npu firmware * hex-ctx: move dma_caches into ops_ctx
* vocab : fix Nemotron-3.5-Embed tokenization * Update src/llama-vocab.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * rename * Update src/llama-vocab.cpp Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
* spec : add probabilistic sampling for DFlash2 drafts * spec : apply review suggestions --------- Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
* remove python from tests * add behavior and backend ctest labels * add behavior check workflow [no ci]
* fix time overflow * use double
Run npm audit fix, then bump the remaining vulnerable packages within their current majors: storybook 10.6.2, vitest 4.1.12, vite 7.3.7, @modelcontextprotocol/sdk 1.32.1, dompurify 3.4.17, mermaid 11.17.2, @vite-pwa/assets-generator 1.0.4 and @tailwindcss/typography 0.5.20. Override katex to 0.18.11 and postcss-selector-parser to 7.1.6 since their dependents still pin vulnerable ranges, bump the sharp override to 0.35.5, and pin @storybook/svelte so npm does not resolve the 11.0 alpha through the addon-svelte-csf peer range. Storybook stays on 10.6.2 as 10.6.3 is newer than min-release-age. Assisted by: Claude (Opus 5.5 Medium)
* hex-ceil: add CEIL(f32) op * hex-ceil: add f16 version * hex-ops: update supported ops * hex-ops: update docs with supported ops
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* qwen4exp : build the QSA selection without the kq_mask * clean * lang * qwen4exp : reshape the fused QSA selection to the kq_mask streams ggml_top_k_mask with a single keep row returns [n_kv, n_tokens], while build_attn_qsa expects the kq_mask layout [n_kv, n_tokens/n_stream, 1, n_stream]. With several streams the shape assert failed at graph reserve. * test the QSA optimization * reword * update build_fused_top_k_mask instead of handwiring * simplify * keep il at the end for consistency
…31053) * CUDA: fix build after chunked CUDA graphs (ggml-org#30804) * Apply suggestion from @JohannesGaessler Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Fix newlines in common.cuh --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
The router only removes models it took from the Hugging Face cache, but the models manager offered "Delete from disk" for every model, so a model from --models-preset or --models-dir failed with a "not removable" toast. Read the source and can_remove fields of /models and show the delete action disabled for those models, with a tooltip (a hint line in the mobile drawer) saying how to remove them instead. Assisted-by: Claude (Opus 5.5 Medium)
* sycl: fold q8_0 reorder dmmv into the generic template * sycl: template the dmmv non-K launchers * sycl: template the dmmv K-quant launchers
…#30956) * Update ggml-sycl.cpp Increases -sm tensor prompt eval t/s by +19% with zero PPL change. * Update ggml-sycl.cpp Replace first pair of wait() calls with ext_oneapi_submit_barrier(). The wait() calls was discovered to cause MTP degradation. * Update ggml-sycl.cpp Adding comm_exchange function pointer to switch between 2 and 4 buffer ggml_sycl_comm_exchange versions @ initialization time (versus hot path) controlled by SYCL_ENABLE_DEFAULT_CONTEXTS=1 setting. * Update ggml-sycl.cpp adding back 2nd half of 4-buffer free. * Update ggml-sycl.cpp Removing extra whitespace. * Update ggml-sycl.cpp Excluding Windows until it is validated on Windows.
* vulkan: avoid redundant scans when zeroing skipped rows * vulkan: remove unreachable zero-expert fallback
…g#30146) * vulkan: improve small-n split-K selection on RDNA3 and RDNA4 * Clean up test cases by removing commented code Removed tests as requested
* vulkan: add tensor-parallel all-reduce communication * vulkan: require shared semaphores for tensor communication Assisted-by: Codex * vulkan: reduce directly from shared host buffers Assisted-by: Codex * vulkan: fix ring all-reduce races with 3+ devices Two hazards in the tensor-parallel ring all-reduce made -sm tensor output non-deterministic with three GPUs: - With n = 3, step t's host-slot region starts at t * cels * 2 bytes, which is not a multiple of 32, so consecutive steps' transfer copies share a 32-byte sector of the imported host memory. On NVIDIA the later copy's leading bytes (up to 10 F16 values at the start of every chunk) end up stale in host memory, even when the copies are ordered by a semaphore. Give every step its own 256-byte-aligned region. n = 2 and n = 4 already had aligned regions for the usual sizes, so their layout is unchanged. - A single-shot all-reduce leaves every device's host slot being read by all peers, but the next ring round only ordered its first transfer after nextd, and only from round 2 on. With n >= 3 a device could overwrite its slot while prevd was still summing it. Wait for all peers on the first ring round after a single-shot all-reduce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vulkan: fold the tensor-parallel all-reduce into the compute submissions With -sm tensor the meta backend drives every device from one thread and calls graph_compute once per subgraph (two per layer), so decode was bound by the CPU cost of submitting, not by the GPUs: ~1500 vkQueueSubmit per token for a 64-layer model on two GPUs, with the GPUs idle ~40% of the time. - Don't split subgraphs by flops while a communicator is active. The heuristic was tuned for whole graphs and turned each small subgraph into ~5 submits. - Fold the small (single-shot) all-reduce into the compute submissions: the last command buffer of a subgraph stays open, the all-reduce appends the gather copy to it and submits it, then opens the next command buffer with the peer waits and the reduce, which the next subgraph is recorded into. Each subgraph now costs one submit per device. The reserved progress value is attached by whichever path submits the open context. - Alternate between two host slots, placed outside the ring's regions, so the gather no longer needs a WAR wait on the peers' previous reduce. - On synchronize, submit the deferred work of every device in the group first, so the tail of the graph runs on all devices in parallel. - Wait for the transfer queue as well as the compute queue before freeing or reallocating communication buffers, and don't leave a stale cap behind when a reallocation fails. qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 32.2 -> 49.7 t/s, 3 GPUs 20.9 -> 40.0, 4 GPUs 15.0 -> 27.6; pp512 on 2 GPUs 2138 -> 2320. Greedy output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * - Two-phase all-reduce over shared host memory: write own slot + signal, wait peers + add from their slots - One exported timeline per device, no backend hooks; comm work recorded in the backend compute ctx - F16 wire above 1 MiB, F32 below; small reductions ride with the next graph submit - Double-buffered, grow-only host slots; zero-size splits zeroed - Remove ring, transfer-queue leg, device staging, shrink, Windows shims - comm_init returns nullptr when unsupported; add GGML_VK_COMM_DISABLE - No flops-based submit splitting for graphs below 1 GFLOP [rebased onto the ring all-reduce fix and the folded all-reduce, which this design replaces; their backend hooks are removed here] * add minimal submit split size * replace opaque_fd with sync_fd for spec compliance * add precision check * add semaphore pool * rename env var, add logging if requirements not met * reduce comments * code improvements * vulkan: use posix_memalign for the tensor-parallel host buffers std::aligned_alloc is unavailable on Android bionic below API 28, which upstream switched away from in ggml-org#30828; do the same for the communicator's host slots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vulkan: synchronize the tensor-parallel all-reduce through the host Exchanging SYNC_FD semaphores costs a kernel wake-up on every all-reduce: on NVIDIA the waiting GPU starts ~35 us after its peer has finished, which is most of the idle time in a decode step. OPAQUE_FD semaphores avoid that but must not be imported into another physical device. Use only device-local timeline semaphores instead. Each device signals done[i] once its partial sum is in host memory (after a barrier that makes it available to the host), and a bridge thread signals go[i] from the host once all peers of i are done. The waiting side makes the host memory visible with a host-write barrier, since its submit predates the data. The thread spins for up to 200 us on each wait (GGML_VK_ALLREDUCE_SPIN_US) and then blocks, so it is idle when there is no work. This also takes the per-round fd export, dup and import calls off the main thread and drops the VK_KHR_external_semaphore_fd requirement and the binary semaphore pool. Graphs below 1 GFLOP, such as tensor-parallel decode slices, are no longer split into several submits. qwen3.5 27B Q8_0 on RTX 4090s, tg128: 2 GPUs 37.2 -> 45.4 t/s, 3 GPUs 24.1 -> 35.5, 4 GPUs 16.5 -> 27.1; pp512 unchanged. Greedy output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * use sync_fd on mesa drivers * refactor for readability * move sync_fd semaphore handling to thread, trigger mesa submit threads * move allreduce functions to new cpp file * clean comments * smaller fixes, code improvements --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
…tforms (ggml-org#31437) * ggml : use `vm_allocate` instead of `posix_memalign` on iOS * ggml : use vm_allocate on all Apple platforms, run test-host-buffer-release by default
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
ggml-org#31249) The note under 'Run the inference' still said that the CANN backend only supports FP16/Q4_0/Q8_0 models, while the DataType table in the same file, and ggml-cann.cpp (BF16 is enabled for MUL_MAT, GET_ROWS, CPY and CONT for every non-310P device), say otherwise. Spell out the per-device support and mark BF16 as unsupported on 310P instead of leaving the cell empty. GGML_CANN_NO_PINNED is read by ggml-cann.cpp when allocating host buffers and when advertising the host_buffer device capability, but it was not documented, so the escape hatch for a failing pinned host allocation was not discoverable. Signed-off-by: li-lizhe <147392333@qq.com>
…ggml-org#29510) * llama : use llm_attn_mask and kv_rows for unified attention Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * cont : optimize `get_seq_cells` * llama : build the kv_rows lists from per-sequence cell bitmaps Keep a bitmap over the cells for each sequence (the transpose of `seq`), updated in seq_pos_inc/dec, and build the cell lists by walking its set bits. The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed. Also leave out of the lists the cells that the SWA hides from every token of the slice (standard and chunked SWA). * cont : make get_kv_rows_p_min private * cont : minor metal * llama : use kv_rows for every ubatch of a unified cache A ubatch made of equal-length runs with a single sequence each (f.ex. the drafts of several slots) now splits into per-sequence slices. Any other ubatch (interleaved, uneven or coupled sequences) becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph. * llama : merge get_kv_rows_seq_ids and get_kv_rows_p_min into get_kv_rows_slices Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB) --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* model : add AR-only Clover MoE support Add GGUF conversion, model loading, and causal hybrid attention for Clover, reusing shared attention, recurrent-state, MoE, and LoRA helpers. * Update tests/test-llama-archs.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * convert : filter Clover tensors using filter_tensors * clover : use the standard ssm_a tensor name --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
…#31429) * feat: support PocketTTS voices with no voice cloning The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the backbone KV state after the voice prompt. The converter turns each state back into the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice encoder is left out, and the "voice" field of /tts and --tts-voice select one of them. A speaker reference is rejected when the model has no voice encoder. Assisted-by: Claude Fable 5.1 * convert : pocket-tts voices never fail the conversion, log the drift of the state The voice states shipped by Kyutai are not always the ones the pack's weights give for the prompt: italian, german and spanish match within the bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state comes from other weights), while the reference loads them as is and they still sound like the same voice. The converter now re-prompts the backbone with the recovered embeddings, compares the whole state and logs the distance per voice, warning above 5%. A state that does not belong to the weights at all is skipped, so the packs of kyutai/pocket-tts convert too. Assisted-by: Claude Fable 5.1 * mtmd : list the predefined voices, expose them in GET /props mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the names, with the voices worded as a feature of any audio generation model. The server lists them under "voices" in /props when the model ships some. Assisted-by: Claude Fable 5.1 * server : an unknown voice is a bad request A model with predefined voices answers 400 naming the voice instead of 500 "failed to process TTS prompt". A model without them ignores "voice" with a warning, as the other audio generation models do. Assisted-by: Claude Fable 5.1 * refactor api * fix conversion * small renaming * fix lint * tts : a model with predefined voices speaks with the first one by default Without a voice prompt such a model only produces near silence. When neither a voice nor a speaker reference is given, llama-tts and the server now take the first predefined voice and say so in the log, as the LFM3-Audio template does with its default voice. Assisted-by: Claude Fable 5.1 * mtmd : one comment per voice accessor, converter states the rule only The clip.h comment described clip_gen_get_voice but sat above all three functions; the converter comment quoted the drift measured per pack, which goes stale with the next upload of the weights. Assisted-by: Claude Fable 5.1 * clean up redundant code and comments * mtmd: match the LFM3-Audio voice name case-insensitively The tts chat template lowers the voice name, so Male speaks with the male voice; the voice lookup lowers it the same way and names the bitmap after the canonical voice. --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pascal <admin@serveurperso.com>
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…-1102 Sync master with upstream (2026-11-02)
Co-Authored-By: Claude Code <noreply@anthropic.com>
Reviewed resultPR #176 has a reproducible Flash Attention prefill penalty and five prompt-specific MTP decode losses. No VLM slowdown was confirmed. The large ordinary 27B decode percentage swings remain confounded by sustained drift reproduced on two boards.
The two Gemma penalties are below the 2% screening threshold after repetition; their attention code change and diagnostic recovery are still documented. For MTP, verification cycles are faster while fewer useful tokens survive each cycle. These losses are reproducible for the catalog prompts, with changed outputs/acceptance; they are not evidence that every prompt or each GPU kernel slows down. The first-pass Qwen3.6-27B decode loss of 14.94% is not a stable branch effect: reversed pairs are +1.15% on halo41 and -0.97% on halo06. Both builds show substantial sustained drift. Original and aggregate losses remain visible and are not treated as clean passes. MTP source probes recover the old output and acceptance for Qwen3.5-2B Q4_K_M code (old RMS or old Q6) and Qwen3.8-27B Q4_K_M prose (old Q3/Q6 together; old Q6 alone does not). Other affected prompts retain unresolved source interactions despite a confirmed accepted-token-yield explanation. The clearest kernel fix is to specialize contiguous Flash Attention so it does not compile the unused indexed-KV path. An isolated diagnostic removes the measured penalty for the affected shapes. MTP needs numerical/acceptance validation before choosing a production change; reverting a faster reduction is not automatically the best fix. No production patch, remote push, PR comment, or baseline-record update was made as part of this test. Confirmed prefill regression: contiguous Flash Attention pays for optional KV indexingQwen3-1.7B Q4_K_M at 3968 input tokens is 2.21% slower in two independent stock-build pairs. Gemma-2B Q4_K_M has smaller repeated increases: 1.73% at 128 tokens and 1.84% at 8000 tokens. Decode is essentially unchanged for these cases. Commit The Qwen kernel Compiled-object evidence supports the cause:
Both selected kernels use 256 VGPRs before and after the change. The PR needs more temporary storage, so it spills more values to scratch memory. Qwen's static assembly contains 307 A separate diagnostic removes only the optional non-sparse KV-row lookup in these two attention instantiations and asserts that indexed rows are absent. It restores the baseline spill/branch counts. Two new paired runs per workload produced:
All 12 diagnostic arms passed sanity, input parity, and ten prefill/ten decode sample checks. Runtime symbol binding confirms the intended overlay was used. The first diagnostic pair per workload enabled binding logs; the reversed pair did not. Qwen and long-input Gemma recover in both orders. Short-input Gemma varies by submillisecond amounts, so its exact end-to-end improvement is less certain. Gemma's long GPU traces also contain unrelated outliers; their complete family means should not be interpreted as a clean causal breakdown. A production fix should compile separate contiguous and indexed KV paths, retaining the feature while allowing the compiler to remove indexing from the ordinary path. The diagnostic is limited to two shapes and is not a production patch. Stock comparison binaries and source worktrees were preserved. Evidence: paired traces, compiled resources, assembly comparison, launch shapes, diagnostic measurements and runtime bindings, and diagnostic diff. MTP throughput: accepted-token yield and numerical changesMTP predicts a few tokens, then verifies them with the target model. Useful throughput depends on both the time for each draft/verify cycle and the number of output tokens that survive verification. A faster cycle can still produce fewer useful tokens per second when acceptance falls. Two small-model losses reproduce in both build orders:
The cycle-time metric includes drafting, verification, sampling, and rollback work in server eval time; it is not one kernel's latency. Verification-cycle counts are inferred only where the server's rounded mean accepted length permits a unique integer. Acceptance counters, output lengths, and hashes are retained for every measured request. The Q4_0 prose response changes from 116 to 125 tokens and mean draft acceptance falls from 38.9937% to 32.6944%. The Q4_K_M code response remains capped at 200 tokens, but the text changes and acceptance falls from 73.5135% to 66.3317%. Both changes affect useful work per cycle. These are reproducible losses for the catalog prompts; they do not establish a universal loss on arbitrary prompts. Both builds have fast/unsafe math disabled. Source-level numerical changes remain possible with these flags: grouping a floating-point reduction differently changes its rounding. In this PR, commit For Q4_K_M code, three independent diagnostics—scalar RMS, old Q6_K reduction, and both together—each recover all five baseline repeat output hashes and the exact five-request acceptance signatures. The affected behavior depends on the interaction of numerical changes and token-selection boundaries; it cannot be assigned uniquely to RMS alone. The associated diagnostic throughputs are separate measurements and are not pooled into the stock-build repeat estimate. For Q4_0 prose, restoring old Q6_K reduction alone leaves the stock PR output and acceptance unchanged. Scalar RMS, with or without the old Q6 reduction, produces a third 95-token answer with 39.5349% acceptance. That improves yield, but it does not restore the original baseline answer. The evidence establishes sensitivity to RMS numerics without isolating the complete baseline-to-PR output difference. The separate MRoPE diagnostics refine this result below. Runtime symbol bindings were checked for the diagnostic overlays. The tested source worktrees and stock binaries remain unchanged. Recovering a previous answer does not prove it has better quality, and reverting a faster reduction is not automatically the best production fix. Broader fixed-prompt accuracy and acceptance checks are needed before choosing between numerical stability and kernel-speed changes. Evidence: repeat comparison, first repeat cycle decomposition, reverse repeat cycle decomposition, numerical diagnostics, and source/dispatch scope. The MRoPE isolation adds a concrete second source of output sensitivity for Q4_0 prose. Disabling only the new MRoPE/SET_ROWS fusion produces a 128-token answer and 34.0764% acceptance; restoring the old rotary implementation as well gives the exact same pattern. Combining scalar RMS with the old rotary path produces a fourth answer, 114 tokens with 34.4125% acceptance. None of those restores the 116-token gfx11 answer. Actual fusion rejection and symbol bindings were recorded in both serving modes. Offline assembly inspection explains why a fusion with the same source formula can alter numerical behavior. The new fused half-output kernel uses Qwen3.6-27B Q4_K_M also reproduces an acceptance-driven MTP loss: prose -5.74% and code -2.33% TPS. Useful-token yield falls 7.32% and 3.80%, while cycle time improves approximately 1.65–1.72% and 1.44–1.54%, respectively. Prose acceptance changes 59.4595% → 52.6667%, and code 96.0506% → 92.2222%; output lengths change 104 → 130 and 164 → 113. Both repeat orders agree. Completed numerical diagnostics are summarized below. The combined scalar RMS + old Q6 + old MRoPE diagnostic also produces the same 114-token / 34.4125% pattern as RMS + old MRoPE. It does not restore the original gfx11 answer. The remaining exact source interaction for Q4_0 prose is unresolved; the report therefore attributes its measured throughput loss to accepted-token yield and identifies demonstrated numerical sensitivities without asserting a single complete source cause. Qwen3.8-27B Q4_K_M prose reproduces a -7.18% MTP throughput change in both orders (-7.15% / -7.20%). Useful-token yield falls 8.89%, while cycle time improves 1.87% / 1.82%. Acceptance changes 67.4603% → 58.5185%, and response length 127 → 124 tokens. Its code/list timing changes disagree by order and show sustained drift; those timing magnitudes are inconclusive as branch effects. The baseline identity failure is reviewed separately. The completed 27B numerical probes separate two cases:
All nine diagnostic variants completed: 18 variant/task combinations, 21 MTP prompt rows plus their controls. The diagnostic audit verified five requests per row, input parity, required symbol bindings, and execution of the MRoPE fusion gate. See the complete numerical probe summary, which reports output and acceptance recovery separately and excludes unpaired diagnostic throughput from stock performance estimates. |
Full regression catalog: first passAll 166 rows below ran the original catalog checks; see the audit and reviewed findings for any failed checks and input-parity details. Input and output lengths are actual measured token counts. Workload changes include generated length and saved MTP output text changes; they are not silently excluded from the comparison. The CSV contains all raw metrics, hashes, repeated metrics, and findings. LLM TTFT is prefill-derived, VLM TTFT is streaming latency, and MTP TTFT is server prompt time.
|
liangliangchang
approved these changes
Nov 4, 2026
Annieren
approved these changes
Nov 4, 2026
Annieren
left a comment
Annieren
left a comment
There was a problem hiding this comment.
MTP regression will address separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merge fork master through
2c7fa78ffinto gfx11.Resolve upstream FP8, fused matmul, and DirectIO changes while retaining gfx11 MMQ/MoE tuning, ROCmFP4, row padding, and channels-major SSM convolution.
Validation
Unsupported cases were skipped. gfx1150/gfx1153, full performance parity, and vision inference were not tested.
Generated with Claude Code