Search public pages, research tools, and SAVRN solutions.
Open-weight model · Image and text to text
by Red Hat AI RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic
FP8-dynamic variant of gemma-4-26B-A4B-it.
What it takes to serve gemma-4-26B-A4B-it-FP8-dynamic (26.5B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 53.1 GB | 63.7 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 26.5 GB | 31.9 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 13.3 GB | 15.9 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.
By Red Hat AI, published under apache-2.0, revision ed35d7abe5d9.
FP8-dynamic variant of gemma-4-26B-A4B-it.
This model is a quantized version of google/gemma-4-26B-A4B-it. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.
This model was obtained by quantizing the weights and activations of google/gemma-4-26B-A4B-it to FP8 data type using dynamic per-token quantization, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Weights are quantized statically using per-channel FP8 scaling, and activations are quantized dynamically at inference time using per-token scaling. Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor. Vision tower, embedding, output head, and MoE router layers are kept in their original precision.
This model can be deployed using vLLM. For detailed instructions including multi-GPU deployment, multimodal inference, thinking mode, function calling, and benchmarking, see the Gemma 4 26B-A4B vLLM usage guide.
vllm serve RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
Tip: For text-only workloads, pass
--limit-mm-per-prompt '{"image": 0, "audio": 0}'to skip vision encoder memory allocation and free up GPU memory for a longer context window.
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
model = "RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic"
messages = [
{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]
outputs = client.chat.completions.create(
model=model,
messages=messages,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
generated_text = outputs.choices[0].message.content
print(generated_text)
This model was created by applying data-free FP8 dynamic quantization with LLM Compressor, as presented in the code snippet below.
from llmcompressor import model_free_ptq
MODEL_ID = "google/gemma-4-26B-A4B-it"
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-dynamic"
model_free_ptq(
model_stub=MODEL_ID,
save_directory=SAVE_DIR,
scheme="FP8_DYNAMIC",
ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*"],
)
This model was evaluated on GSM8K Platinum, MMLU-Pro, IFEval, MATH-500, AIME 2025, GPQA Diamond, LiveCodeBench v6, and BFCLv4 (function calling) using lm-evaluation-harness, lighteval, and BFCL — all served with vLLM (OpenAI-compatible API). Accuracy results are reported both without and with thinking enabled; BFCLv4 was evaluated with thinking enabled.
| Category | Benchmark | google/gemma-4-26B-A4B-it | RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic | Recovery |
|---|---|---|---|---|
| Instruction Following | IFEval (0-shot, prompt-level strict) | 89.96 | 89.34 | 99.3% |
| IFEval (0-shot, inst-level strict) | 93.21 | 92.69 | 99.4% | |
| Reasoning | GSM8K Platinum (0-shot, strict-match) | 95.43 | 95.37 | 99.9% |
| MMLU-Pro (0-shot, custom-extract) | 83.47 | 83.26 | 99.7% | |
| MATH-500 (0-shot, pass@1) | 84.80 | 85.93 | 101.3% | |
| AIME 2025 (0-shot, pass@1) | 80.00 | 80.00 | 100.0% | |
| GPQA Diamond (0-shot, pass@1) | 73.20 | 74.75 | 102.1% | |
| Coding | LiveCodeBench v6 (0-shot, pass@1) | 74.48 | 73.90 | 99.2% |
| Category | Benchmark | google/gemma-4-26B-A4B-it | RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic | Recovery |
|---|---|---|---|---|
| Instruction Following | IFEval (0-shot, prompt-level strict) | 94.39 | 95.07 | 100.7% |
| IFEval (0-shot, inst-level strict) | 96.16 | 96.60 | 100.5% | |
| Reasoning | GSM8K Platinum (0-shot, strict-match) | 95.95 | 95.64 | 99.7% |
| MMLU-Pro (0-shot, custom-extract) | 85.19 | 85.07 | 99.9% | |
| MATH-500 (0-shot, pass@1) | 85.87 | 85.93 | 100.1% | |
| AIME 2025 (0-shot, pass@1) | 88.75 | 91.67 | 103.3% | |
| GPQA Diamond (0-shot, pass@1) | 80.81 | 80.13 | 99.2% | |
| Coding | LiveCodeBench v6 (0-shot, pass@1) | 77.90 | 76.00 | 97.6% |
| Tool Calling | BFCLv4 Overall | 67.62% | 68.31% | 101.0% |
| BFCLv4 Single Turn | 83.85% | 83.04% | 99.0% | |
| BFCLv4 Multi-Turn | 62.13% | 64.50% | 103.8% | |
| BFCLv4 Agentic | 62.16% | 61.91% | 99.6% |
The results were obtained using the following commands:
vllm serve RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic \
--tensor-parallel-size 2 \
--max-model-len 69632 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image":0,"audio":0}' \
--async-scheduling \
--default-chat-template-kwargs '{"enable_thinking": true}'
lm_eval --model local-chat-completions \
--tasks gsm8k_platinum_cot_llama \
--model_args "model=RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \
--num_fewshot 0 \
--apply_chat_template \
--output_path results_gsm8k_platinum.json \
--seed 1234 \
--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
lm_eval --model local-chat-completions \
--tasks mmlu_pro_chat \
--model_args "model=RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \
--num_fewshot 0 \
--apply_chat_template \
--output_path results_mmlu_pro.json \
--seed 1234 \
--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
lm_eval --model local-chat-completions \
--tasks ifeval \
--model_args "model=RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \
--num_fewshot 0 \
--apply_chat_template \
--output_path results_ifeval.json \
--seed 1234 \
--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"
model_parameters:
provider: hosted_vllm
model_name: hosted_vllm/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic
base_url: http://0.0.0.0:8000/v1
api_key: ''
timeout: 3600
concurrent_requests: 32
generation_parameters:
temperature: 1.0
max_new_tokens: 65536
top_p: 0.95
top_k: 64
seed: 1234
lighteval endpoint litellm litellm_config.yaml 'math_500|0' \
--output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'aime25|0' \
--output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \
--output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'lcb:codegeneration_v6|0' \
--output-dir results/ --save-details
"gemma-4-26b-a4b-it-FP8-dynamic": ModelConfig(
model_name="gemma-4-26b-a4b-it-FP8-dynamic",
display_name="Gemma-4-26B-A4B-it-FP8-dynamic (FC)",
url="https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic",
org="Google",
license="Apache 2.0",
model_handler=OpenAICompletionsHandler,
input_price=None,
output_price=None,
is_fc_model=True,
underscore_to_dot=True,
),
bfcl generate --model gemma-4-26b-a4b-it-FP8-dynamic --test-category all
bfcl evaluate --model gemma-4-26b-a4b-it-FP8-dynamic --test-category all
17 files, 28.7 GB in total. The weights are 1 file totalling 28.6 GB in safetensors.
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model.safetensors | Weights | 28.6 GB | d2988fae7675 |
| config.json | Configuration | 19.6 KB | — |
| every_eval_ever/aime25.json | Configuration | 3.7 KB | — |
| every_eval_ever/gpqa_diamond.json | Configuration | 2.4 KB | — |
| every_eval_ever/gsm8k_platinum.json | Configuration | 4.0 KB | — |
| every_eval_ever/ifeval.json | Configuration | 6.2 KB | — |
| every_eval_ever/lcb_codegeneration_v6.json | Configuration | 2.4 KB | — |
| every_eval_ever/math_500.json | Configuration | 2.2 KB | — |
| every_eval_ever/mmlu_pro.json | Configuration | 1.9 KB | — |
| generation_config.json | Configuration | 203 B | — |
| processor_config.json | Configuration | 1.7 KB | — |
| recipe.yaml | Configuration | 230 B | — |
| README.md | Documentation | 13.2 KB | — |
| chat_template.jinja | Other | 18.7 KB | — |
| .gitattributes | Repository | 1.6 KB | — |
| tokenizer.json | Tokenizer | 32.2 MB | cc8d3a0ce364 |
| tokenizer_config.json | Tokenizer | 2.1 KB | — |
Released by Red Hat AI through its official repository on Hugging Face. Read the license.
| Precision | Weights in memory |
|---|---|
| As published | 28.6 GB |
| 16-bit | 53.1 GB |
| 8-bit | 26.5 GB |
| 4-bit | 13.3 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
About 63.7 GB at 16-bit and 15.9 GB at 4-bit: the weights (26.5B parameters) plus a working margin. A long context needs more.
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Yes. gemma-4-26B-A4B-it-FP8-dynamic is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
262,144 tokens, from the maximum position embeddings in its published configuration.
This is a merge of pre-trained language models created using mergekit. This model was merged using the aura merge method. Aura is an experimental method with a live heatmap visualizer. This model took 10 hours to merge using graphv18.py The following models were included in the merge: - TheDrummer/Orion-26B-A4B-v1.1 - Gryphe/Pantheon-Reasoning-26B-A4B-1.1-V2 - electroglyph/gemma4-26b-fiction-bf16 The following YAML configuration was used to produce this model
Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages. Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning. The models are available in five distinct sizes: E2B, E4B, 12B, 26B A4B, and 31B. Their diverse sizes make them deployable in environments ranging from…
Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on small models) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages. Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning. The models are available in four distinct sizes: E2B, E4B, 26B A4B, and 31B. Their diverse sizes make them deployable in environments ranging from high-end phones…
LM Studio Community models highlights program. Highlighting new & noteworthy models by the community. Join the conversation on Discord. 4-bit quantized version of Qwen3.8-27B using MLX, optimized for Apple Silicon. Special thanks to the Apple Machine Learning Research team for creating MLX. LM Studio is not the creator, originator, or owner of any Model featured in the Community Model Program. Each Community Model is created and provided by third parties. LM Studio does not endorse, support, represent or guarantee the completeness, truthfulness, accuracy, or reliability of any Community Model. You understand that Community Models can produce content that might be offensive, harmful…
LM Studio Community models highlights program. Highlighting new & noteworthy models by the community. Join the conversation on Discord. 8-bit quantized version of Qwen3.8-27B using MLX, optimized for Apple Silicon. Special thanks to the Apple Machine Learning Research team for creating MLX. LM Studio is not the creator, originator, or owner of any Model featured in the Community Model Program. Each Community Model is created and provided by third parties. LM Studio does not endorse, support, represent or guarantee the completeness, truthfulness, accuracy, or reliability of any Community Model. You understand that Community Models can produce content that might be offensive, harmful…
LM Studio Community models highlights program. Highlighting new & noteworthy models by the community. Join the conversation on Discord. 6-bit quantized version of Qwen3.8-27B using MLX, optimized for Apple Silicon. Special thanks to the Apple Machine Learning Research team for creating MLX. LM Studio is not the creator, originator, or owner of any Model featured in the Community Model Program. Each Community Model is created and provided by third parties. LM Studio does not endorse, support, represent or guarantee the completeness, truthfulness, accuracy, or reliability of any Community Model. You understand that Community Models can produce content that might be offensive, harmful…