gemma-4-26B-A4B-MXFP8-FP8KV-CT-RTN-AutoRound

This checkpoint is an AutoRound model-free MXFP8 RTN quantization of google/gemma-4-26B-A4B, exported in llm_compressor / compressed-tensors format. Routed experts and the self-attention projections present in the source are stored as F8_E4M3; sensitive/shared and multimodal weights remain BF16. Static FP8 KV scales were calibrated with AutoRound using the text dataset NeelNanda/pile-10k; the vision tower was not quantized.

On the paired repository lm_eval protocol, the four primary metrics were non-decreasing relative to the BF16 baseline using the same vLLM FP8 KV cache. The AQA gate was GO. This is a result for those tasks and settings only, not a claim of lossless quantization or general quality improvement.

Read this first: max_model_len=131072 was configured, but the benchmark prompts were not a 128K long-context stress test. This run evaluated text tasks only; vision quality, multimodal prompts, thinking-on, throughput and latency were not measured. The tokenizer configuration used by the run had no chat_template, so GSM8K used the repository's bare-prompt path.

1. Model summary

The model configuration sets tie_word_embeddings: true. The nominal 26.544B model count and the index-derived unique serialized element count are different counting bases; the size and dtype accounting below uses the latter. The A4B designation is the upstream model's active-parameter naming, not a measurement made by this quantization run.

Serving was validated through the vLLM 0.29.0 backend on three RTX 5090 GPUs with TP=1 / PP=3. That is the tested topology, not a claim that other hardware or parallel layouts are equivalent.

2. Precision plan

Percentages below are relative to the 25,805,936,206 unique stored model-weight elements. Scale tensors are metadata and are not included in this denominator.

Shares are rounded independently; the unrounded category shares sum to 100%. The exported weight configuration is Linear, 8-bit float, symmetric, group-wise, group size 32, static weights, with U8 scales and memoryless_minmax observer. The config also records dynamic 8-bit float input-activation quantization metadata. Static KV is 8-bit float, tensor-wise and dynamic=false.

The exported checkpoint also has 11,635 U8 block-scale tensors and 114 F32 static KV scales. Of the latter, the 60 text K/V scales (30 layers × K/V) are finite and nonzero; the 54 vision scales are zero because calibration and evaluation were text-only.

Ignore groups

Norms and other non-target tensors remain BF16 by the model-free defaults. The checkpoint contains no multi_modal_projector key, so it is not included in the ignore list.

2.1 Quantization plan and artifact deviations

There was no unplanned weight-precision change. The embed_tokens warning reflects that embeddings are not Linear quantization targets; the key remains present in BF16.

2.2 Size accounting

The ratio is 51,611,872,412 / 28,412,087,908 = 1.8165×. Effective bits per unique serialized model-weight element are 28,412,087,908 × 8 / 25,805,936,206 = 8.8079; this includes block/KV scale storage represented in the exported index and is not the nominal MXFP8 weight bit width.

3. Evaluation results

Measured 2026-09-28 with lm_eval 0.4.13 using its vLLM backend (vLLM 0.29.0), TP=1 / PP=3, seed 42, BF16 compute dtype and FP8 KV cache for both baseline and candidate. Values are means ± lm_eval standard error (SE); N is the evaluated sample count. The final column is the score difference divided by sqrt(SE_baseline² + SE_MXFP8²); it is an approximate independent-SE scale, not a paired significance test.

The arithmetic mean of the four primary scores is 0.7291 (baseline) vs 0.7350 (MXFP8), a +0.58 pp difference; this mean is descriptive and has no combined SE. PIQA acc_norm was 0.8395 ± 0.0086 vs 0.8341 ± 0.0087 (−0.54 pp); HellaSwag acc_norm was 0.8305 ± 0.0037 vs 0.8317 ± 0.0037 (+0.12 pp). The AQA gate uses the primary metrics above and reports GO (max_accuracy_drop=0.02). The benchmark differences are within one approximate combined SE; this run does not establish statistical superiority.

3.2 What was held fixed

Both measured runs used max_model_len=131072, kv_cache_dtype=fp8, tensor_parallel_size=1, pipeline_parallel_size=3, gpu_memory_utilization=0.85, dtype=bfloat16, attention_backend=FLASHINFER, language_model_only=true, max_num_seqs=1, trust_remote_code=true, add_bos_token=true, prefix caching disabled, thinking disabled and seed 42. PIQA/MMLU/HellaSwag used the repository task group with batch_size=auto; GSM8K used 5-shot and batch_size=64, with a 2,048-token generation budget. Inference-engine internal scheduling details not exposed by lm_eval are not claimed as controlled measurements.

3.3 Versus baseline

All four AQA primary scores are non-decreasing in this paired run. The differences are small relative to their reported task-level uncertainty, so the result supports the stated gate decision but does not prove that MXFP8 improves accuracy. An earlier failed run had a different repeated BF16 GSM8K strict-match score (0.7286 vs 0.7180); its cause is unknown and it is not mixed into this gate comparison.

3.4 Functional evidence and provenance

Direct evidence: the exported checkpoint loaded in vLLM 0.29.0 with FP8 KV cache and completed all four listed lm_eval tasks. Its quantization_config.json records compressed-tensors MXFP8 weights and a static 8-bit float KV scheme. The validation run ID is gemma-4-26B-A4B_MXFP8_20260927-220842_a0c394.

Not measured: full 128K prompts, multimodal/vision quality, thinking-on, agentic tasks, throughput or latency. Environment support probes and sibling model runs are not substituted for this checkpoint's measured evidence.

4. Usage

4.1 vLLM through the validated lm_eval path

The run validated vLLM loading/evaluation via lm_eval. After the repository is available, the following uses its Hub ID in place of the original local checkpoint path:

MODEL_ID=INCModel4/gemma-4-26B-A4B-MXFP8-FP8KV-CT-RTN-AutoRound
MODEL_ARGS="pretrained=${MODEL_ID},tensor_parallel_size=1,pipeline_parallel_size=3,max_model_len=131072,gpu_memory_utilization=0.85,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,attention_backend=FLASHINFER,language_model_only=True,kv_cache_dtype=fp8,max_num_seqs=1,enable_thinking=False"

# Select three idle GPUs before running; this example assumes physical GPUs 1-3 are idle.
export CUDA_VISIBLE_DEVICES=1,2,3
lm_eval --model vllm --model_args "$MODEL_ARGS" \
  --tasks piqa,mmlu,hellaswag --batch_size auto --log_samples --seed 42
lm_eval --model vllm --model_args "$MODEL_ARGS" \
  --tasks gsm8k --batch_size 64 --log_samples --seed 42 --num_fewshot 5

Hard constraints and evidence:

For text inference, supply a plain prompt compatible with the task rather than assuming a chat template. A direct production vllm serve deployment was not separately benchmarked by this run.

4.2 Transformers inspection path

The following reads model configuration only; it is not a weight-loading or serving validation path:

from transformers import AutoConfig

model_id = "INCModel4/gemma-4-26B-A4B-MXFP8-FP8KV-CT-RTN-AutoRound"
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
print(config.model_type)

The checkpoint contains about 28.41 GB of indexed tensors. Loading full weights through an inspection-oriented Transformers path can use substantially more host/GPU memory; use the tested vLLM backend for inference.

5. Reproducibility

5.1 Hardware / OS

• GPUs: 3 × NVIDIA GeForce RTX 5090 used (compute capability 12.0; physical devices 1, 2 and 3); 32,607 MiB total memory reported per device.
• Host probe on 2026-09-28: NVIDIA driver 610.57.04; Intel Xeon 6767P (256 logical CPUs); 188.4 GiB RAM.
• OS: Ubuntu 24.04.4 LTS, kernel 6.8.0-139-generic, x86_64.
• CUDA: PyTorch build CUDA 13.0; runtime 13.0.88.
• Storage: model and run were on local filesystem paths; filesystem type was not recorded.

5.2 Environment

Versions are from the run manifest / frozen Conda environment, not a statement of minimum compatible versions.

5.3 Version selection

No packages were installed or upgraded for this run. These exact versions were used for the successful quantization and evaluation. Compatibility with other AutoRound, Transformers, vLLM or FlashInfer versions was not established here.

5.4 Artifact-level dependency warning

AutoRound 0.16.0.dev193 with Transformers 5.17.0 initially failed when its fused-MoE meta skeleton could not materialize the new self_attn.k_scale and self_attn.v_scale parameters. The successful run set AR_DISABLE_META_LOAD=1, which disabled that meta-load path and allowed CPU loading/initialization. Peak host RAM was 64.14 GiB. No source-code patch was applied.

5.5 Environment variables

For quantization, set the tested workaround and map three verified idle physical devices:

export AR_DISABLE_META_LOAD=1
export CUDA_VISIBLE_DEVICES=<three-idle-physical-GPU-IDs>

AR_DISABLE_META_LOAD is a quantization workaround; it is not required for inference. Do not put Hub tokens in this README or in a run YAML.

6. Reproduce the artifact

The original run was orchestrated with AQA; do not bypass it with a manually assembled AutoRound or lm_eval pipeline. Use fresh local paths and select three idle physical GPUs before starting.

6.1 Quantization YAML

The following is the core of the validated run.yaml; adapt only model, environment and output paths:

model: /path/to/gemma-4-26B-A4B

quant:
  scheme: MXFP8
  method: model_free
  export_format: llm_compressor
  ignore_layers: "router,vision_tower,embed_vision,embed_tokens,mlp"
  static_kv_fp8: true
  device_map: "0,1,2"

eval:
  backend: vllm
  harness: lm_eval
  benchmarks: [piqa, mmlu, hellaswag, gsm8k]
  baseline: true
  thinking: "off"
  gen_max_toks: 2048
  lm_eval:
    group: true
    seed: 42
    apply_chat_template: false
    fewshot_as_multiturn: false
    model_args:
      attention_backend: FLASHINFER
      language_model_only: true
      kv_cache_dtype: fp8
      max_model_len: 131072
      max_num_seqs: 1
      gpu_memory_utilization: 0.85
      tensor_parallel_size: 1
      pipeline_parallel_size: 3
      enable_thinking: false
      enable_prefix_caching: false

gate:
  max_accuracy_drop: 0.02

model_free uses the model-free RTN weight path, but static_kv_fp8 still loads the checkpoint and performs calibration forwards for KV scales. The successful log used the standard text calibration dataset NeelNanda/pile-10k; the vision tower was not quantized.

6.2 Exact quantization arguments

These are the exact AutoRound flags recorded from the successful AQA builder run; only output_dir is parameterized here. Reproduce the run through AQA as shown below:

auto-round --model_name /models/gemma-4-26B-A4B --scheme MXFP8 --model_free \
  --static_kv_dtype fp8 \
  --ignore_layers router,vision_tower,embed_vision,embed_tokens,mlp \
  --device_map 0,1,2 --format llm_compressor \
  --output_dir <fresh-run-dir>/quantized

6.3 Validate and run

export AR_DISABLE_META_LOAD=1
export CUDA_VISIBLE_DEVICES=<three-idle-physical-GPU-IDs>
aqa validate run.yaml
aqa run run.yaml --dry-run
aqa run run.yaml

The tested run completed the full AQA baseline → quantization → evaluation pipeline. Its output had six safetensors shards and an index size of 28,412,087,908 bytes.

6.4 Evaluate

The measured commands used the same vLLM model arguments shown in §4.1. The GSM8K task used five few-shot examples, seed 42 and a 2,048-token generation budget, with chat-template and multiturn formatting disabled because of the tokenizer configuration observed in this run.

7. Known issues and caveats

• Do not infer long-context behavior from max_model_len=131072; no 128K prompt was measured.
• Do not infer vision/multimodal quality from the text-only results. The vision weights remain BF16, but their quality was not evaluated.
• PP=2 at the same length hit a CUDA graph warmup OOM on this host; the tested topology is TP=1 / PP=3 on three RTX 5090 GPUs.
• FlashInfer XQA did not support this model's 512 head dimension in the tested path; vLLM fell back to native FlashInfer decode. FP8 KV evaluation still completed.
• The token embedding warning is benign: it is not an AutoRound Linear target, its key is present, and it remains BF16.
• A previous BF16 GSM8K baseline repeat scored 0.7286 vs 0.7180 in the successful run. The cause is unknown; the reported gate compares baseline and MXFP8 only within the successful run.
• PIQA acc_norm decreased by 0.54 pp even though the AQA primary acc increased. Consult both metric and uncertainty tables; do not call the quantization lossless.

8. License and attribution

The base model metadata identifies Apache 2.0 and links to the Gemma 4 license information. Users must comply with the base-model license and preserve its attribution. This quantized checkpoint is not an official Google release.

Quantization: Intel AutoRound. Inference validation: vLLM and FlashInfer. Evaluation: lm-evaluation-harness. Text calibration used NeelNanda/pile-10k; calibration samples are not included in this repository. This repository contains the quantized weights, tokenizer/processor configuration and model configuration; it does not redistribute benchmark datasets.