INT8 weight-only quantization of google/gemma-4-E4B-it in the
compressed-tensors format, ready for vLLM.
Text, image and audio inputs are supported as in the original: only the text decoder is quantized.
|
Original (bf16) |
This model (W8A16) |
| Checkpoint size |
16.0 GB |
12.1 GB |
| Text decoder Linear weights |
bf16 |
INT8 |
| Vision / audio encoders |
bf16 |
bf16 |
| Activations |
bf16 |
bf16 |
The "Model size" shown by the Hub counts tensor elements, and INT8 weights are packed 4 per int32, so it shows ~5B. The real parameter count is the same as the original (~8B with Per-Layer Embeddings).
Quantization details
- Method: round-to-nearest (RTN) with llm-compressor, no calibration data, so no language bias from a calibration set.
- Scheme:
W8A16, symmetric, per-channel INT8 weights, 16-bit activations.
- Quantized: all
Linear layers of the text decoder (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), 258 layers in total.
- Kept in bf16:
- Per-Layer Embeddings (
embed_tokens_per_layer, per_layer_input_gate, per_layer_projection, per_layer_model_projection)
- Vision tower and audio tower
- Multimodal projections (
embed_vision, embed_audio)
- Token embeddings and
lm_head
A large part of the E4B checkpoint is Per-Layer Embeddings, which stay in bf16, so the size reduction is smaller than the "8-bit" label suggests.
The full recipe is in recipe.yaml.
Evaluation
Perplexity and top-1 next-token agreement against the bf16 original on Wikipedia
(wikimedia/wikipedia, dump 20231101), 16,384 tokens per language, window 1024 tokens.
| Language |
PPL bf16 |
PPL W8A16 |
Δ PPL |
Top-1 agreement |
Russian (ru) |
20.27 |
20.26 |
-0.07% |
97.8% |
Kyrgyz (ky) |
24.44 |
24.41 |
-0.12% |
97.9% |
English (en) |
48.91 |
48.87 |
-0.08% |
97.6% |
Perplexity differences are within noise for this sample size: no measurable degradation on any language,
including low-resource Kyrgyz. Top-1 agreement of ~98% means the remaining disagreements are mostly near-ties
in the original model's distribution.
Only the text path was evaluated. Vision and audio encoder weights are identical to the original.
Usage
vLLM
vllm serve Bekten/gemma-4-E4B-it-W8A16 --max-model-len 32768
No --quantization flag is needed: vLLM reads the compressed-tensors config from config.json.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="Bekten/gemma-4-E4B-it-W8A16",
messages=[{"role": "user", "content": "Кыргызстандын борбору кайсы шаар?"}],
)
print(resp.choices[0].message.content)
Transformers
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "Bekten/gemma-4-E4B-it-W8A16"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": [{"type": "text", "text": "Привет! Расскажи о себе."}]}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Transformers decompresses the weights to bf16 on load, so memory savings apply to vLLM, not to Transformers.
On small GPUs, load with device_map="cpu", run one forward pass to trigger decompression, then move the model to GPU.
License
Same as the base model: apache-2.0. Use of this model is subject to the terms of
google/gemma-4-E4B-it.
