Model Overview
- Model Architecture: Qwen3_5ForConditionalGeneration
- Input: Text / Image
- Output: Text
- Model Optimizations:
- Weight quantization: FP4 and FP8
- Activation quantization: FP4 and FP8
- Release Date: 2026-09-21
- Version: 2.0
- Model Developers: RedHatAI
This model is an updated quantized version of Qwen/Qwen3.8-27B, using a mixed-precision FP4/FP8 scheme with an unquantized language-model head and updated quantization scales. See Evaluation for accuracy results.
Model Optimizations
This model was produced by applying mixed-precision quantization to Qwen/Qwen3.8-27B. MLP projections are quantized to FP4, attention projections and the final MLP layers are quantized to FP8, and the KV cache is quantized to FP8, while the language-model head is kept in full precision to preserve output quality. The quantization scales were updated by calibrating on a 512-sample subset of the perfectblend dataset with a recipe that combines AWQ and GPTQ.
Only the weights and activations of the linear operators within the transformer blocks are quantized using LLM Compressor. The checkpoint is ~24.7 GB on disk (versus ~54 GB in BF16), reducing disk size and GPU memory requirements by roughly 70%.
Deployment
vLLM Serving
vllm serve RedHatAI/Qwen3.8-27B-NVFP4 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
For optimal peformance, consider using the DSpark draft model RedHatAI/Qwen3.8-27B-speculator.dspark for speculative decoding, shown below.
vllm serve RedHatAI/Qwen3.8-27B-NVFP4 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--speculative-config '{"model":"RedHatAI/Qwen3.8-27B-speculator.dspark","num_speculative_tokens":8,"method":"dspark"}'
See Performance Evaluation below for further details.
Creation
This model was created by applying LLM Compressor with calibration samples from perfectblend, as presented in the code snippet below.
from compressed_tensors.quantization.quant_scheme import (
FP8_DYNAMIC,
NVFP4,
QuantizationScheme,
)
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier
from llmcompressor.utils import load_context
MODEL_ID = "Qwen/Qwen3.8-27B"
# Load model.
with load_context(Qwen3_5ForConditionalGeneration):
model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = [
AWQModifier(duo_scaling="both"),
GPTQModifier(
config_groups={
"attention": QuantizationScheme(
targets=[
r"re:.*self_attn\.(q|k|v|o)_proj$",
r"re:.*linear_attn\.(in_proj_qkv|in_proj_z|out_proj)$",
r"re:.*layers\.(56|57|58|59|60|61|62|63)\.mlp\..*(gate|up|down)_proj$",
],
**FP8_DYNAMIC,
),
"mlp": QuantizationScheme(
targets=[r"re:.*mlp\..*(gate|up|down)_proj$"],
**NVFP4,
),
},
ignore=[
"re:visual.*",
"re:model.visual.*",
"re:.*lm_head",
],
kv_cache_scheme={
"num_bits": 8,
"type": "float",
"symmetric": True,
"strategy": "tensor",
"dynamic": False,
"observer": "static_minmax",
},
),
]
# Apply quantization.
oneshot(
model=model,
processor=processor,
recipe=recipe,
dataset="perfectblend",
splits="train[:512]",
max_seq_length=4096,
num_calibration_samples=512,
moe_calibrate_all_experts=True,
)
# Save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-NVFP4-GPTQ-AWQ"
model.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)
Evaluation
This model was evaluated on GSM8K Platinum, MATH-500, AIME 2025, GPQA Diamond, and IFEval using lm-evaluation-harness and lighteval, and on SWE Bench using Inspect AI, served with vLLM (OpenAI-compatible API). Evaluations were run on 1x B200 GPU.
Accuracy
Recovery vs. BF16 baseline
| Category |
Benchmark |
Qwen/Qwen3.8-27B |
RedHatAI/Qwen3.8-27B-NVFP4 |
Recovery |
| Reasoning |
GSM8K Platinum |
96.25% |
96.72% |
100.49% |
| MATH-500 |
83.67% |
84.27% |
100.72% |
| AIME 2025 |
96.67% |
95.00% |
98.27% |
| GPQA Diamond |
89.56% |
89.22% |
99.62% |
| Instruction Following |
IFEval |
91.19% |
91.99% |
100.88% |
| Agentic - Coding |
SWE Bench |
78.8% |
78.0% |
98.98% |
NVFP4 build comparison
| Category |
Benchmark |
RedHatAI/Qwen3.8-27B-NVFP4 |
unsloth/Qwen3.8-27B-NVFP4 |
Inferact/Qwen3.8-27B-NVFP4 |
| Reasoning |
GSM8K Platinum |
96.72% |
95.42% |
93.77% |
| MATH-500 |
84.27% |
85.67% |
82.47% |
| AIME 2025 |
95.00% |
93.75% |
91.66% |
| GPQA Diamond |
89.22% |
89.39% |
87.04% |
| Instruction Following |
IFEval |
91.99% |
91.81% |
91.50% |
Reproduction
The results were obtained using the following commands. Each benchmark was run multiple times with different random seeds — 3 repetitions for GSM8K Platinum, MATH-500, GPQA Diamond, and IFEval, and 8 repetitions for AIME 2025 — and the reported score is the mean across seeds.
GSM8K Platinum & IFEval (lm-eval, 0-shot)
Run once per seed:
lm_eval --model local-chat-completions \
--tasks gsm8k_platinum_cot_llama \
--model_args "model=RedHatAI/Qwen3.8-27B-NVFP4,max_length=69632,base_url=http://127.0.0.1:3235/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
--num_fewshot 0 \
--apply_chat_template \
--output_path results_gsm8k_platinum.json \
--seed 1234 \
--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,max_gen_toks=32000,seed=1234"
lm_eval --model local-chat-completions \
--tasks ifeval \
--model_args "model=RedHatAI/Qwen3.8-27B-NVFP4,max_length=69632,base_url=http://127.0.0.1:3235/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
--num_fewshot 0 \
--apply_chat_template \
--output_path results_ifeval.json \
--seed 1234 \
--gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,max_gen_toks=32000,seed=1234"
MATH-500, AIME 2025, GPQA Diamond (lighteval, 0-shot)
litellm_config.yaml:
model_parameters:
provider: hosted_vllm
model_name: hosted_vllm/RedHatAI/Qwen3.8-27B-NVFP4
base_url: http://127.0.0.1:3235/v1
api_key: ''
timeout: 3600
concurrent_requests: 32
generation_parameters:
temperature: 1.0
max_new_tokens: 65536
top_p: 0.95
top_k: 20
seed: 1234
Run once per seed (changing seed in the config each time):
lighteval endpoint litellm litellm_config.yaml 'math_500@1@3|0' --output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'aime25@1@8|0' --output-dir results/ --save-details
lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond@1@3|0' --output-dir results/ --save-details
Performance Evaluation
Each plot sweeps request load for the math_reasoning and HumanEval benchmark datasets. The x-axis shows per-user interactivity in tokens per second, where higher values mean a snappier response for an individual request. The y-axis shows total server throughput in tokens per second, where higher values mean the system is serving more aggregate load. Each colored line represents a different model and speculator configuration.
The plots demonstrate the benefit of not just quantizing the LLM, but combining it with a trained speculator model, RedHatAI/Qwen3.8-27B-speculator.dspark. Each sweep was done using TP=1,DP=4 on B200s.
