Gemma 4 E2B QAT Q4_0 assistant (MTP head), GGUF
Google publishes the QAT Q4_0 model google/gemma-4-E2B-it-qat-q4_0-gguf
as GGUF, but its multi-token-prediction (MTP) assistant only as safetensors
(google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant).
There was no GGUF MTP head for the QAT GGUF, so this repository is that assistant converted to GGUF, nothing else:
no training or fine-tuning. Use it together with google/gemma-4-E2B-it-qat-q4_0-gguf for speculative decoding in llama.cpp.
The Google model card asks for exactly this pairing: "When using multi-token prediction (speculative decoding) with an
assistant model alongside a QAT target model, the assistant model must also be a QAT checkpoint with the same precision."
| file |
size |
sha256 |
gemma-4-E2B-it-qat-q4_0-assistant.gguf |
77 MB |
1e6281cb762a906b3f0940d81092a01496a1f7fbeb1589b7520658fd34974181 |
Use (llama.cpp)
hf download google/gemma-4-E2B-it-qat-q4_0-gguf --local-dir qat
hf download ensky0/gemma-4-E2B-it-qat-q4_0-gguf-assistant --local-dir qat
llama-server -m qat/gemma-4-E2B_q4_0-it.gguf --mmproj qat/gemma-4-E2B-it-mmproj.gguf \
-md qat/gemma-4-E2B-it-qat-q4_0-assistant.gguf --spec-type draft-mtp --spec-draft-n-max 3
How it was made
llama.cpp at commit 0c1e570
(Gemma4AssistantForCausalLM → GGUF architecture gemma4-assistant):
hf download google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant --local-dir assistant
python convert_hf_to_gguf.py assistant --outtype bf16 --outfile assistant-BF16.gguf
llama-quantize assistant-BF16.gguf gemma-4-E2B-it-qat-q4_0-assistant.gguf Q4_0
llama-quantize's Q4_0 keeps token_embd.weight at Q6_K and puts the other weights at Q4_0, the same layout as Google's
QAT target GGUF (its token_embd.weight is Q6_K too). 49 tensors.
Verification
Text, with the QAT target
gemma-4-E2B_q4_0-it.gguf + this file, llama.cpp 0c1e570 (llama-completion without, llama-speculative-simple --spec-type draft-mtp
--spec-draft-n-max 3 with the head), greedy, 96 tokens, 5 prompts (3 Japanese → Korean translations, 2 English questions),
NVIDIA A100:
| accepted / drafted |
output identical to no drafting |
| 132/308 = 43% |
3/5 |
Where the outputs differ they agree for the first words and then part at one token: checking several drafted tokens in
one batch changes the floating-point order slightly, and greedy decoding can then pick a different token. In the E4B
speech test (see the E4B repository) the non-QAT ggml-org head gives exactly as many differing outputs as this conversion,
so the differences come from the batched verification, not from the converted weights.
License
Apache 2.0, as the original Gemma 4 weights.
Converted for a realtime speech-translation subtitle app (llama.cpp, Japanese audio → Korean subtitles).
