Skip to content

fix: pass backend="cpu" when requesting CPU gemm_4bit_forward kernel - #2090

Open
SIDDARTHAREDDY8 wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
SIDDARTHAREDDY8:bhai-blitz-8-bitsandbytes-1
Open

SIDDARTHAREDDY8 wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
SIDDARTHAREDDY8:bhai-blitz-8-bitsandbytes-1

Conversation

@SIDDARTHAREDDY8

@SIDDARTHAREDDY8 SIDDARTHAREDDY8 commented Sep 20, 2026

Copy link
Copy Markdown

Fixes #2064.

Summary
bitsandbytes/backends/cpu/ops.py requested the fused CPU 4-bit GEMM kernel from kernels-community/quantization-bitsandbytes via get_kernel(...) without a backend argument. The kernels package infers the backend from the installed torch build, so on any CUDA torch build the inferred backend is CUDA, every CPU-only build variant is rejected, and get_kernel raises at every process start on hosts with AVX512-BF16 (logged as a warning by the best-effort fallback handler). This one-line change passes backend="cpu" explicitly so the CPU build variant is selected regardless of the torch build.

Test notes

  • Verified get_kernel accepts a backend kwarg against the kernels library source (huggingface/kernels, load.py): "The backend to load the kernel for. Can only be cpu or the backend that Torch is compiled for."
  • Executed the patched module-level loading block in isolation with a stubbed kernels.get_kernel: it now calls get_kernel(repo, version=1, backend="cpu").
  • The issue reporter's own repro shows get_kernel(REPO, version=1, backend="cpu") loads the CPU build variant cleanly where the inferred call fails.
  • Full E2E not possible here: this box has CPU-only torch (where the inferred backend is already cpu), and the kernels package is not installed. py_compile passes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CPU gemm_4bit_forward kernel is requested without backend="cpu", so it never loads on a CUDA torch build

1 participant