AMD

Model · Text generation

granite-4.0-h-tiny-w8a8-llmcompressor

AMD

ZenDNN v6.1.0 - ZenTorch v2.13.0.0 - PyTorch v2.13.0.0 - LLM Compressor v0.13.0 - vLLM v0.29.0 This is a quantized version of granite-4.0-h-tiny created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference. The model was quantized from granite-4.0-h-tiny using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 12.9 GiB to 6.6 GiB on disk (~49% reduction). granite-4.0-h-tiny is a hybrid Mamba-MoE model: of its 40 layers, 4 are full-attention blocks and the other 36 are Mamba (linear-attention) blocks, and every layer carries a 64-expert MoE block alongside a shared MLP. The recipe only needs two ignore entries. lmhead…

Model · Text generation

granite-4.0-h-small-w8a8-llmcompressor

AMD

ZenDNN v6.1.0 - ZenTorch v2.13.0.0 - PyTorch v2.13.0.0 - LLM Compressor v0.13.0 - vLLM v0.29.0 This is a quantized version of granite-4.0-h-small created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference. The model was quantized from granite-4.0-h-small using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 60.0 GiB to 30.4 GiB on disk (~49% reduction). granite-4.0-h-small is a hybrid Mamba-MoE model: of its 40 layers, 4 are full-attention blocks and the other 36 are Mamba (linear-attention) blocks, and every layer carries a 72-expert MoE block (top-10 routing) alongside a shared MLP. The recipe only needs two…

Model · Text generation

granite-4.0-h-small-w4a16-llmcompressor

AMD

ZenDNN v6.1.0 - ZenTorch v2.13.0.0 - PyTorch v2.13.0.0 - LLM Compressor v0.13.0 - vLLM v0.29.0 This is a quantized version of granite-4.0-h-small created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference. The model was quantized from granite-4.0-h-small using LLM Compressor with the GPTQ algorithm. This reduces the model weights from 60.0 GiB to 16.1 GiB on disk (~73% reduction). granite-4.0-h-small is a hybrid Mamba-MoE model: of its 40 layers, 4 are full-attention blocks and the other 36 are Mamba (linear-attention) blocks, and every layer carries a 72-expert MoE block (top-10 routing) alongside a shared MLP. Two details make this work. The model is…