Gemma 4 31B QAT Q4_0 assistant (MTP head), GGUF
Google publishes the QAT Q4_0 model google/gemma-4-31B-it-qat-q4_0-gguf
as GGUF, but its multi-token-prediction (MTP) assistant only as safetensors
(google/gemma-4-31B-it-qat-q4_0-unquantized-assistant).
There was no GGUF MTP head for the QAT GGUF, so this repository is that assistant converted to GGUF, nothing else:
no training or fine-tuning. Use it together with google/gemma-4-31B-it-qat-q4_0-gguf for speculative decoding in llama.cpp.
The Google model card asks for exactly this pairing: "When using multi-token prediction (speculative decoding) with an
assistant model alongside a QAT target model, the assistant model must also be a QAT checkpoint with the same precision."
| file |
size |
sha256 |
gemma-4-31B-it-qat-q4_0-assistant.gguf |
349 MB |
3a7f849d19c80916cd529b26ed3b0fec0cf43b6b0d38991f05d148d6b3ed3d68 |
Use (llama.cpp)
hf download google/gemma-4-31B-it-qat-q4_0-gguf --local-dir qat
hf download ensky0/gemma-4-31B-it-qat-q4_0-gguf-assistant --local-dir qat
llama-server -m qat/gemma-4-31B_q4_0-it.gguf --mmproj qat/gemma-4-31B-it-mmproj.gguf \
-md qat/gemma-4-31B-it-qat-q4_0-assistant.gguf --spec-type draft-mtp --spec-draft-n-max 3
How it was made
llama.cpp at commit 0c1e570
(Gemma4AssistantForCausalLM → GGUF architecture gemma4-assistant):
hf download google/gemma-4-31B-it-qat-q4_0-unquantized-assistant --local-dir assistant
python convert_hf_to_gguf.py assistant --outtype bf16 --outfile assistant-BF16.gguf
llama-quantize assistant-BF16.gguf gemma-4-31B-it-qat-q4_0-assistant.gguf Q4_0
llama-quantize's Q4_0 keeps token_embd.weight at Q6_K and puts the other weights at Q4_0, the same layout as Google's
QAT target GGUF (its token_embd.weight is Q6_K too). 49 tensors.
Verification
Text, with the QAT target
gemma-4-31B_q4_0-it.gguf + this file, llama.cpp 0c1e570 (llama-completion without, llama-speculative-simple --spec-type draft-mtp
--spec-draft-n-max 3 with the head), greedy, 96 tokens, 5 prompts (3 Japanese → Korean translations, 2 English questions),
NVIDIA A100:
| accepted / drafted |
output identical to no drafting |
| 157/234 = 67% |
4/5 |
Where the outputs differ they agree for the first words and then part at one token: checking several drafted tokens in
one batch changes the floating-point order slightly, and greedy decoding can then pick a different token. In the E4B
speech test (see the E4B repository) the non-QAT ggml-org head gives exactly as many differing outputs as this conversion,
so the differences come from the batched verification, not from the converted weights.
This model has no audio input; only the text check above applies.
License
Apache 2.0, as the original Gemma 4 weights.
Converted for a realtime speech-translation subtitle app (llama.cpp, Japanese audio → Korean subtitles).
