Gemma4-E2B-it-W4A16

Load path (important)

These weights are compressed-tensors (pack-quantized / int-quantized).

# Preferred
vllm serve 88plug/<ModelName> --trust-remote-code

Do not deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors.

Gemma4-E2B-W4A16

INT4 post-training quantization of google/gemma-4-e2b-it — Google's 2B-active multimodal MoE with 128 experts. The smallest capable multimodal MoE. Runs on any 8 GB GPU.

Quantized with datafree RTN (QuantizationModifier) — AutoRound (blocked — see KNOWN-FAILURES) wrapper blocked by Gemma4 q_proj.linear children (see KNOWN-FAILURES). Temporary path until upstream fixed. The LLM backbone is W4A16; vision tower, projector, and PLE layers remain BF16.

At a Glance

PLE layers kept at BF16

embed_tokens_per_layer and per_layer_model_projection implement Per-Layer Embeddings — ablations show catastrophic output degradation if quantized. Always excluded.

Memory Requirements

The smallest capable multimodal MoE: W4A16 fits the model on any modern 8 GB GPU with room for KV cache.

Quick Start

Tested with vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are in compressed-tensors format — vLLM detects and loads quantization automatically. No --quantization flag needed.

vLLM

docker run --gpus device=0 -p 8080:8080 \
  vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \
  88plug/Gemma4-E2B-it-W4A16 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Weights are in compressed-tensors format — no --quantization flag needed.

Python client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="x")

response = client.chat.completions.create(
    model="88plug/Gemma4-E2B-it-W4A16",
    messages=[{"role": "user", "content": "Explain sparse mixture-of-experts in two sentences."}],
    max_tokens=256,
)
print(response.choices[0].message.content)

Quantization Design

The recipe targets all Linear modules in the LLM backbone with W4A16 (4-bit symmetric weight quantization, activations remain BF16). The following are excluded and kept at BF16:

All self_attn.{q,k,v,o}_proj and mlp.{gate,up,down}_proj layers across all transformer blocks are quantized to W4A16.

Competitor Comparables

First-to-market note: No compressed-tensors or vLLM-native quant found for gemma-4-e2b-it at release time. This is the first W4A16 for Gemma4 E2B.

Benchmarks

Results from 88plug benchmark ladder — vLLM v0.21.0, lm-evaluation-harness. No fabricated numbers.

Quality Targets

SGLang Note

SGLang does not natively support compressed-tensors weights. To use SGLang, run the BF16 base model (google/gemma-4-e2b-it) directly:

docker run --gpus device=0 -p 30000:30000 \
  lmsysorg/sglang:v0.5.8-cu129 python -m sglang.launch_server \
  --model-path google/gemma-4-e2b-it \
  --tp 1 \
  --mem-fraction-static 0.85 \
  --port 30000

SGLang benchmark results above reflect BF16 baseline throughput, not this quant.

llama.cpp / GGUF

Fits entirely on an 8 GB GPU with Q4 quantization. Convert from the BF16 base checkpoint — not from compressed-tensors weights. VLM requires a separate mmproj GGUF for image input.

python convert_hf_to_gguf.py google/gemma-4-e2b-it \
  --outfile Gemma4-E2B-BF16.gguf
python convert_hf_to_gguf.py google/gemma-4-e2b-it \
  --mmproj --outfile Gemma4-E2B-mmproj.gguf

llama-quantize Gemma4-E2B-BF16.gguf Gemma4-E2B-Q8_0.gguf Q8_0
llama-quantize --imatrix calibration_datav3.txt \
  Gemma4-E2B-BF16.gguf Gemma4-E2B-IQ4_XS.gguf IQ4_XS

llama-server \
  --model Gemma4-E2B-Q8_0.gguf \
  --mmproj Gemma4-E2B-mmproj.gguf \
  --n-gpu-layers 999 \
  --ctx-size 32768 \
  --port 8081

Citation

@misc{gemma4report,
  title  = {Gemma 4 Technical Report},
  author = {Google DeepMind},
  year   = {2025},
  url    = {https://huggingface.co/google/gemma-4-e2b-it}
}

About

88plug AI Lab ships compressed-tensors quantizations for native vLLM v0.21.0+ deployment.

This release: Provisional tier — datafree RTN (weight-only rounding, no calibration corpus). A gold AutoRound re-quant is scheduled; 88plug architecture forbids new provisional W4A16 uploads.

Browse all releases → huggingface.co/88plug