build/bin/llama-cli throws SIGSEGV, Segmentation fault when using a TL2 GGUF model

[hbaastrup]

To reproduce on a x86 architecture:

1. python3 setup_env.py --hf-repo HF1BitLLM/Llama3-8B-1.58-100B-tokens -q tl2
2. python3 run_inference.py -m models/bitnet_b1_58-large/ggml-model-tl2.gguf -p "Hi are you a LLM?" -n 6 -temp 0

Result:
Loading model... |Error occurred while running command: Command '['build/bin/llama-cli', '-m', 'models/bitnet_b1_58-large/ggml-model-tl2.gguf', '-n', '6', '-t', '2', '-p', 'Hi are you a LLM?', '-ngl', '0', '-c', '2048', '--temp', '0.0']' died with <Signals.SIGSEGV: 11>.

Compiling llama.cpp in debug and run llama-cli in gdb I get this back trace:
#0 0x0000000000000000 in ?? ()
#1 0x00007ffff5e0bde9 in ggml_compute_forward_mul_mat_one_chunk (params=0x7ffffffee9a0, dst=0x23f0730, type=GGML_TYPE_TL2, num_rows_per_vec_dot=0,
ir0_start=0, ir0_end=16, ir1_start=0, ir1_end=2) at /home/hbaastrup/bitnet/BitNet.debug/3rdparty/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:1295
#2 0x00007ffff5e0ad40 in ggml_compute_forward_mul_mat (params=0x7ffffffee9a0, dst=0x23f0730)
at /home/hbaastrup/bitnet/BitNet.debug/3rdparty/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:1608
...

When I take a better look at the file 3rdparty/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c, I see the type_traits_cpu array does not have an value for the index of GGML_TYPE_TL2 (42), which means; when the program try to lookup the vec_dot reference, it get a 0 (null) there will provoke a SIGSEGV when vec_dot is called.

Is there a quick fix for this?

Note
I don't use the I2_S format as llama-quantize does not support this format any longer. I get:
llama_quantize: invalid ftype 'I2_S'
when I run python3 setup_env.py --hf-repo HF1BitLLM/Llama3-8B-1.58-100B-tokens -q i2_s

Looking in the file 3rdparty/llama.cpp/tools/quantize/quantize.cpp I can see the std::vector QUANT_OPTIONS missing an index for I2_S

[shifulegend]

Yeah, the TL2 paths aren't fully wired up in the ggml-cpu type_traits_cpu table yet, which causes that null pointer dereference on vec_dot. You're basically stuck. Since I2_S got pulled from the quantize tools, there isn't a good native path right now.

If you just need to get the model running while waiting for the upstream ggml plumbing to catch up, one thing worth looking at is Project Zero (https://github.com/shifulegend/project-zero). It's a pure C99 inference engine that handles these ternary weights directly without the ggml dependency stack. It runs BitNet about 1.83x faster than bitnet.cpp anyway (hits ~36 tok/s on a Xeon) because it avoids this kind of generic tensor routing overhead and implements the AVX2 paths natively.

[hbaastrup]

Yes, it is a bit sad that the I2_S was pulled out before the TL2 was full supported.
I will take a look at the Project-Zero implementation.
Thank you so much for the advise.