gemma-4-31B-it · NVFP4A16 (W4A16, FP4 weights)
4-bit quantization of google/gemma-4-31B-it (commit 842da37), produced for efficient vLLM serving. Quantized by axteam.
|
|
| Base model |
google/gemma-4-31B-it (commit 842da37) |
| Method |
NVFP4A16 (W4A16, FP4 weights) |
| Format |
compressed-tensors nvfp4-pack-quantized |
| Weights / activations |
W4A16 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), BF16 activations |
| Architecture |
Gemma 4 · Dense |
| Size on disk |
~19.0 GB |
| Calibration |
256 samples (Korean-heavy; -it includes thinking-mode answers) |
| License |
Gemma 4 / Apache-2.0 |
Quantization details
- Scheme — W4A16 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), BF16 activations, stored as compressed-tensors
nvfp4-pack-quantized.
- Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms. These are left unquantized to preserve quality (a small fraction of total parameters).
- Tokenizer /
chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.
Evaluation
Quantization fidelity was checked against the BF16 original using perplexity and top-1 next-token agreement on an internal mixed English/Korean set. In our runs the 8-bit variants (AWQ INT8 / FP8) stay closest to the original, while the 4-bit variants (AWQ INT4 / NVFP4) trade a little quality for roughly half the footprint.
Usage (vLLM)
vllm serve axteam/gemma-4-31B-it-NVFP4A16
from vllm import LLM, SamplingParams
llm = LLM(model="axteam/gemma-4-31B-it-NVFP4A16")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.chat(
[{"role": "user", "content": "Write a short joke about saving RAM."}],
params,
)
print(out[0].outputs[0].text)
Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64.
Thinking mode: add the <|think|> token at the start of the system prompt to enable step-by-step reasoning.
Reducing KV-cache memory (optional, serve-time)
The KV cache is unquantized (BF16) here — the standard for quantized checkpoints
(RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice,
independent of the weight quantization above. For long context or large batches you can
roughly halve KV-cache memory with FP8:
vllm serve axteam/gemma-4-31B-it-NVFP4A16 --kv-cache-dtype fp8
--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2.
For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.
License
This is a quantized derivative of Google's Gemma 4. Use is governed by the
Gemma 4 license. All original
weights, tokenizer, and chat template belong to Google DeepMind.
한국어 요약
google/gemma-4-31B-it 를 NVFP4A16 (W4A16, FP4 weights) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.
- 방식 — FP4(E2M1) 가중치 16묶음 + 활성화는 BF16(W4A16). W4A4 NVFP4보다 품질이 크게 좋음(활성화 손실 없음), Blackwell 활성화 속도만 포기. 4비트 중 품질 최상급. 형식은 compressed-tensors
nvfp4-pack-quantized.
- BF16 유지 — 비전 타워·비전 투영·토큰 임베딩(lm_head와 묶임)·MoE 라우터·노름 (전체의 극히 일부).
- 토크나이저·chat_template·processor·generation_config 는 원본과 바이트까지 동일 (구글이 2026-07에 고친 chat_template 그대로).
config.json 은 원본 + quantization_config.
- 품질 — BF16 원본 대비 perplexity·top-1 일치도로 검증(영/한 혼합셋). 8비트(AWQ INT8·FP8)가 원본에 가장 가깝고, 4비트(AWQ INT4·NVFP4)는 용량을 절반으로 줄이는 대신 품질을 조금 양보.
- 사용 —
vllm serve axteam/gemma-4-31B-it-NVFP4A16. 권장 샘플링 temperature=1.0, top_p=0.95, top_k=64.
- KV 캐시 절감(선택, 서빙 시) — 기본은 BF16 KV(양자화 체크포인트 표준, RedHat·NVIDIA 동일). 가중치 양자화와 무관한 런타임 선택. 긴 문맥·대배치에선
vllm serve … --kv-cache-dtype fp8 로 KV 메모리 절반. (GGUF라면 llama.cpp --cache-type-k q8_0 --cache-type-v q8_0)
- 라이선스 — 구글 Gemma 4 파생물. Gemma 4 라이선스 적용. 원본 가중치·토크나이저·chat_template 권리는 Google DeepMind.
