Skip to content

cuda: exclude sm_75 from the short-query in-place quantized-K/V MMA FA - #361

Open
umbertov wants to merge 1 commit into
PrismML-Eng:prismfrom
umbertov:fix/348-fattn-mma-f16
Open

umbertov wants to merge 1 commit into
PrismML-Eng:prismfrom
umbertov:fix/348-fattn-mma-f16

Conversation

@umbertov

Copy link
Copy Markdown

Overview

Fixes the Turing GPU failure reported in #348.

In ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1, skip the F16 MMA path on Turing GPUs (sm_75). That kernel does not support these GPUs, and using it can make prompt processing fail with the following message:

ERROR: CUDA kernel flash_attn_ext_f16 has no device code compatible with CUDA arch 750. ggml-cuda.cu was compiled for: 750,800,860,890,900,1200,1210

On sm_75, this keeps dispatch on the existing compatible path for this case.

Additional information

This change does not add a new Flash Attention kernel or enable the MMA path on Turing.

I have tested this change on my machine with RTX 2080 ti with the following command line invocation:

./build/bin/llama-server \
  -hf prism-ml/Ternary-Bonsai-2-27B-gguf:TQ1_0 \
  --jinja \
  -ngl 99 \
  -c 131072 \
  --ubatch-size 512 \
  -fa on \
  -ctk q4_0 \
  -ctv q4_0 \
  --port 8080 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00

Requirements

  • I have read and agree with the contributing guidelines: yes
  • AI usage disclosure: yes. AI supported the codebase exploration which led to the code change. I reviewed the change and am responsible for it. This PR was not automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant