Gemma 4 12B QAT Q4_0 assistant (MTP head), GGUF
Google publishes the QAT Q4_0 model google/gemma-4-12B-it-qat-q4_0-gguf
as GGUF, but its multi-token-prediction (MTP) assistant only as safetensors
(google/gemma-4-12B-it-qat-q4_0-unquantized-assistant).
There was no GGUF MTP head for the QAT GGUF, so this repository is that assistant converted to GGUF, nothing else:
no training or fine-tuning. Use it together with google/gemma-4-12B-it-qat-q4_0-gguf for speculative decoding in llama.cpp.
The Google model card asks for exactly this pairing: "When using multi-token prediction (speculative decoding) with an
assistant model alongside a QAT target model, the assistant model must also be a QAT checkpoint with the same precision."
| file |
size |
sha256 |
gemma-4-12B-it-qat-q4_0-assistant.gguf |
323 MB |
25f143b4c15b20cd04216e35e99bd7a56afc6f65e7a4e090a3e20091bb590cbb |
Use (llama.cpp)
hf download google/gemma-4-12B-it-qat-q4_0-gguf --local-dir qat
hf download ensky0/gemma-4-12B-it-qat-q4_0-gguf-assistant --local-dir qat
llama-server -m qat/gemma-4-12b-it-qat-q4_0.gguf --mmproj qat/mmproj-gemma-4-12b-it-qat-q4_0.gguf \
-md qat/gemma-4-12B-it-qat-q4_0-assistant.gguf --spec-type draft-mtp --spec-draft-n-max 3
How it was made
llama.cpp at commit 0c1e570
(Gemma4UnifiedAssistantForCausalLM → GGUF architecture gemma4-assistant):
hf download google/gemma-4-12B-it-qat-q4_0-unquantized-assistant --local-dir assistant
python convert_hf_to_gguf.py assistant --outtype bf16 --outfile assistant-BF16.gguf
llama-quantize assistant-BF16.gguf gemma-4-12B-it-qat-q4_0-assistant.gguf Q4_0
llama-quantize's Q4_0 keeps token_embd.weight at Q6_K and puts the other weights at Q4_0, the same layout as Google's
QAT target GGUF (its token_embd.weight is Q6_K too). 49 tensors.
Verification
Text, with the QAT target
gemma-4-12b-it-qat-q4_0.gguf + this file, llama.cpp 0c1e570 (llama-completion without, llama-speculative-simple --spec-type draft-mtp
--spec-draft-n-max 3 with the head), greedy, 96 tokens, 5 prompts (3 Japanese → Korean translations, 2 English questions),
NVIDIA A100:
| accepted / drafted |
output identical to no drafting |
| 153/235 = 65% |
3/5 |
Where the outputs differ they agree for the first words and then part at one token: checking several drafted tokens in
one batch changes the floating-point order slightly, and greedy decoding can then pick a different token. In the E4B
speech test (see the E4B repository) the non-QAT ggml-org head gives exactly as many differing outputs as this conversion,
so the differences come from the batched verification, not from the converted weights.
Speech translation (Japanese audio → Korean)
Target gemma-4-12b-it-qat-q4_0.gguf + mmproj-gemma-4-12b-it-qat-q4_0.gguf, Google's speech-translation prompt, greedy,
model turn opened with <|channel>thought\n<channel|> as the 12B chat template does with thinking off, NVIDIA A100:
| set |
MTP head |
drafted tokens |
identical output |
accepted / drafted |
first Korean character |
done |
chrF |
| conversation, HTH preview (68 clips) |
none |
- |
- |
- |
0.281 s |
0.418 s |
38.6 |
| conversation |
this file |
2 |
42/68 |
1158/1978 = 59% |
0.230 s |
0.331 s |
37.6 |
| conversation |
this file |
3 |
42/68 |
1241/2640 = 47% |
0.237 s |
0.346 s |
37.7 |
| read speech, Common Voice 8.0 ja (80 clips) |
none |
- |
- |
- |
0.270 s |
0.405 s |
51.6 |
| read speech |
this file |
2 |
56/80 |
1364/2148 = 64% |
0.221 s |
0.313 s |
52.2 |
| read speech |
this file |
3 |
55/80 |
1499/2796 = 54% |
0.223 s |
0.317 s |
52.2 |
Gemma 4 12B QAT vs E4B QAT for speech translation (the targets, without MTP)
Japanese audio → Korean, Google's speech-translation prompt, greedy, NVIDIA A100. chrF against Korean references that Gemini
wrote from the audio, checked in two verification rounds. 12B is better on clean or conversational speech and much worse on short, band-limited live-recording segments:
| set |
clips (speech) |
median length |
12B QAT |
E4B QAT |
| read speech: Common Voice 8.0 ja test (first 150 of 200 random utterances, seed 0) |
150 |
4.8 s |
53.5 |
52.1 |
| conversation: HTH japanese-casual-conversational-speech-golden-dataset-preview |
68 |
2.6 s |
38.6 |
35.6 |
| live recording (our own, not published; cut by a VAD into segments of at most 3 s) |
107 |
1.6 s |
5.8 |
14.9 |
| first Korean character / done (live recording) |
0.244 s / 0.345 s |
0.149 s / 0.203 s |
On the live-recording set 12B answers 43 of 110 segments with a refusal ("I'm sorry, I can't help with that") instead of a
transcription. Google's original google/gemma-4-12B-it (BF16, transformers 5.18) and llama.cpp's llama-mtmd-cli fail on
the same segments the same way, so it is the model, not the GGUF or llama.cpp; clean synthetic speech is transcribed
correctly by both models. What makes it fail (12B QAT, refusals on the conversation set made live-recording-like):
| change |
refusals |
chrF |
| as is |
6/68 |
38.6 |
| low-pass 4 kHz |
9/68 |
36.4 |
| low-pass 2 kHz |
16/68 |
36.1 |
| first 1.5 s only |
18/68 |
| live-recording set as is / two neighbouring segments joined |
43/110 / 18/55 |
Short segments matter most, missing high frequencies next (energy above 4 kHz: live recording 0.3%, conversation 1.7%, read
speech 0.9%). 12B is encoder-free (raw 40 ms waveform frames projected straight into the language model), E4B has a
dedicated ~300M audio encoder; for low-latency subtitles cut into short segments, E4B is the better choice.
License
Apache 2.0, as the original Gemma 4 weights.
Converted for a realtime speech-translation subtitle app (llama.cpp, Japanese audio → Korean subtitles).
