granite-4.0-h-small-w4a16-llmcompressor

Model Overview

• Model Architecture: GraniteMoeHybridForCausalLM
• Input: Text
• Output: Text
• Source Model: granite-4.0-h-small
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.29.0
• Quantization Framework: LLM Compressor v0.13.0
• Quantization Method: 4-bit Weight-Only Quantization (W4A16)
• Compatible Stack:
• ZenDNN v6.1.0
• ZenTorch v2.13.0.0
• PyTorch v2.13.0.0
• LLM Compressor v0.13.0
• vLLM v0.29.0
• Published with: LLM Compressor v0.13.0

This is a quantized version of granite-4.0-h-small created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from granite-4.0-h-small using LLM Compressor with the GPTQ algorithm. This reduces the model weights from 60.0 GiB to 16.1 GiB on disk (~73% reduction).

• Method: 4-bit Weight-Only Quantization (W4A16)
• Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128, actorder=static
• Weights: INT4 (4-bit integer, symmetric, group-wise), stored in pack-quantized format
• Activations: BF16 (unquantized)
• Group Size: 128
• Calibration: 128 samples from HuggingFaceH4/ultrachat_200k, sequence length 2048

granite-4.0-h-small is a hybrid Mamba-MoE model: of its 40 layers, 4 are full-attention blocks and the other 36 are Mamba (linear-attention) blocks, and every layer carries a 72-expert MoE block (top-10 routing) alongside a shared MLP.

• Quantized: all 72 routed experts in every layer (block_sparse_moe.experts.*.{gate,up,down}_proj), the shared MLP (shared_mlp.{input,output}_linear), the Mamba projections (mamba.{in,out}_proj), and self_attn.{q,k,v,o}_proj in the 4 full-attention layers.
• Kept in BF16: the MoE routers (block_sparse_moe.router), the Mamba state-space internals that are not Linear layers (conv1d, A_log, D, dt_bias, and the gated mamba.norm), lm_head, embed_tokens, and the layer norms.

Two details make this work. The model is loaded inside load_context(), which is the LLM Compressor v0.13 MoE linearization path: it exposes the fused expert tensors as individual Linear submodules so GPTQ can build a Hessian per expert, with no manual module swap. And the router is skipped because it is a tiny Linear whose logits decide expert assignment, where 4-bit rounding error can flip the top-k selection and change which experts run.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization.gptq import GPTQModifier
from llmcompressor.utils import load_context

model_id = "ibm-granite/granite-4.0-h-small"
output_dir = "./granite-4.0-h-small-w4a16-llmcompressor"
CALIB_SIZE = 128
MAX_SEQ_LENGTH = 2048

# Step 1: Load the BF16 model inside load_context(), which linearizes the MoE
# experts so GPTQ can target them as ordinary Linear modules.
with load_context(AutoModelForCausalLM):
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        torch_dtype=torch.bfloat16,
        device_map="cpu",
        trust_remote_code=True,
    )
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Build the GPTQ calibration set.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")
ds = ds.map(
    lambda ex: {"text": "\n".join(m["content"] for m in ex["messages"] if m.get("content"))},
    remove_columns=ds.column_names,
)
if not getattr(tokenizer, "pad_token", None):
    tokenizer.pad_token = tokenizer.eos_token
calib_ds = ds.map(
    lambda ex: tokenizer(
        ex["text"], truncation=True, max_length=MAX_SEQ_LENGTH, add_special_tokens=False
    ),
    remove_columns=["text"],
)

# Step 3: Define the W4A16 GPTQ recipe. Experts, shared MLP, Mamba projections
# and attention are all quantized; only lm_head and the MoE router are skipped.
recipe = GPTQModifier(
    scheme="W4A16",
    targets=["Linear"],
    ignore=["lm_head", "re:.*block_sparse_moe.router"],
)

# Step 4: One-shot quantize with calibration data and save in
# compressed-tensors format.
oneshot(
    model=model,
    dataset=calib_ds,
    recipe=recipe,
    max_seq_length=MAX_SEQ_LENGTH,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/granite-4.0-h-small-w4a16-llmcompressor",
    dtype="bfloat16",
    trust_remote_code=True,
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.13.0.0
zentorch==2.13.0.0
vllm==0.29.0
llmcompressor==0.13.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Evaluation Command

lm_eval \
    --model vllm \
    --model_args pretrained=amd/granite-4.0-h-small-w4a16-llmcompressor,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

• Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0.0 / PyTorch v2.13.0.0. It may not load correctly on other versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
• Hybrid Architecture: The Mamba state-space internals (conv1d, A_log, D, dt_bias, gated norms) are not Linear layers and stay in BF16, so the INT4 saving applies to the projections, experts and attention rather than to the full recurrent path.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.