Repository navigation
ggml : support -sm tensor for Hadamard-folded qwen35 (Bonsai 2) - #363
Merged
Merged
Conversation
Tensor split aborted on qwen35 models with folded Hadamard rotations at ggml-backend-meta.cpp:1086 (split_state.ne[j] % div == 0): the rotation reshape tiles the activation feature axis, and the meta backend had no split state for it. - llama_mul_mat_hadamard reshapes to 4D [n, ne0/n, ne1, ne2*ne3] so the split axis survives the rotation, and asserts ne0 % n == 0. - Split blocks for Hadamard-rotated weights use lcm(type block, rotation block) so shards never cut a rotation block. - ssm_out uses a value_dim split when GDN V is grouped, and views of the GATED_DELTA_NET output carry the matching axis-1 split. - Elementwise MUL/ADD/SUB/DIV of a feature-sharded activation with a full-width mirrored operand slices the operand per shard (ADD_ID is excluded). Split rules by Benjamin Brown (@KodeMunkie), posted on ggml-org#29600; ADD_ID guard added here. Tested on 2x H200: Bonsai 2 27B PQ2_0 and Q2_0 now run under -sm tensor with mean KLD 2.5e-4 vs single GPU (top-1 99.3%). Single-GPU CUDA output and Qwen3-4B / Qwen3.5-4B results (single and tensor) are bit-identical to the base commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tensor split aborts on Bonsai 2 at
ggml-backend-meta.cpp:1086(GGML_ASSERT(split_state.ne[j] % div == 0)), because the Hadamard reshape tiles the activation feature axis and the meta backend has no split state for it. Same bug as #351, with a much smaller fix: 3 files, +50/-4, no new ggml.h hints and no gather path.llama_mul_mat_hadamardreshapes to 4D[n, ne0/n, ne1, ne2*ne3]so the split axis survives the rotation.ssm_outgets a value_dim split when GDN V is grouped, and views of the GATED_DELTA_NET output carry the matching split.The split rules are Benjamin Brown's (@KodeMunkie), from ggml-org#29600; the ADD_ID guard is new here.
Testing
2x H200, CUDA 12.8, against
prisme3de715. KLD on wikitext-2, c512, 16 chunks, reference = base build on one GPU.-sm tensor-sm tensor-sm tensor-sm tensorGreedy output under
-sm tensoris byte-identical to base on both Qwen models, so the change is neutral off the Hadamard path. 27B PQ2_0 speed with-ts 1/1: tg128 about 112 vs 120 tok/s on one GPU, pp512 +24%. It's a capacity feature, not a speedup.Not covered: more than 2 devices, and a real MoE split (the ADD_ID guard is untested there).