SAVRN
Search Contact SAVRN

Open-weight model · Image and text to text

Qwen3.8-27B-NVFP4-QAD

by Local Inference Lab local-inference-lab/Qwen3.8-27B-NVFP4-QAD

WORK IN PROGRESS A mixed NVFP4/MXFP8 quantization-aware distillation of Qwen3.8-27B, trained for one epoch. The student learns from the original BF16 teacher while its MLP weights are quantized in the forward pass.

Parameters19.2B
Context262,144
Weights22.7 GB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads—

Runs On

What it takes to serve Qwen3.8-27B-NVFP4-QAD (19.2B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 38.5 GB 46.1 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 19.2 GB 23.1 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 9.6 GB 11.5 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Local Inference Lab, published under apache-2.0, revision f40a31cd813a.

WORK IN PROGRESS A mixed NVFP4/MXFP8 quantization-aware distillation of Qwen3.8-27B, trained for one epoch. The student learns from the original BF16 teacher while its MLP weights are quantized in the forward pass. Distillation updates the MLP weights and text normalization weights to account for quantization error. This is a trained distillation checkpoint, not a post-training conversion of the original weights. Attention/GDN projections and the LM head were frozen in their MXFP8 representations during distillation. Packed NVFP4 and MXFP8 weights reconstruct to the same BF16 weight values used by the student during training. The tokenizer, chat template, generation configuration and…

Read Local Inference Lab's full model card

*** WORK IN PROGRESS ***

Qwen3.8-27B-QAD-E1

A mixed NVFP4/MXFP8 quantization-aware distillation of Qwen3.8-27B, trained for one epoch.

The student learns from the original BF16 teacher while its MLP weights are quantized in the forward pass. Distillation updates the MLP weights and text normalization weights to account for quantization error. This is a trained distillation checkpoint, not a post-training conversion of the original weights.

Precision

Component Representation
MLP gate, up and down projections NVFP4, 16-element blocks
Attention and gated-delta-network projections MXFP8, 32-element blocks
LM head MXFP8, 32-element blocks
Trained text normalization weights FP32 masters
Token embeddings, GDN convolutions and dynamics Original BF16
Vision encoder and remaining source tensors Unchanged

Attention/GDN projections and the LM head were frozen in their MXFP8 representations during distillation. Packed NVFP4 and MXFP8 weights reconstruct to the same BF16 weight values used by the student during training. The tokenizer, chat template, generation configuration and multimodal processors are retained from the base model.

Activation calibration

MLP activation scales use the p99.999 token-row maximum from 390,497,191 raw-text and chat tokens. Each token contributes its maximum absolute input value across channels; an exact BF16 histogram records these row maxima. Raw and chat histograms are pooled before selecting the quantile.

Within each of the 64 dense layers, gate (w1) and up (w3) have separate scale tensors containing the same value. Down (w2) has an independent scale. These are dense projections; there is no expert axis.

Calibration uses the trained weight representations with BF16 activations. The deployment configuration specifies calibrated NVFP4 MLP activations and dynamic MXFP8 attention/head activations. Serving kernels therefore introduce activation quantization beyond the training forward pass.

Format

Hugging Face safetensors with ModelOpt mixed-precision quantization metadata. The runtime must support the base model architecture, NVFP4 dense linears and MXFP8 linears. Packed-weight reconstruction and export integrity are checked; downstream evaluation and serving-quality qualification are separate.

License

Apache 2.0, following the base model.

Configuration

Architecture
Qwen3_5ForConditionalGeneration
Context length (tokens)
262,144
Layers
64
Hidden size
5,120
Feed-forward size
17,408
Attention heads
24
Key/value heads
4
Head dimension
256
Vocabulary size
248,320
Model type
qwen3_5
Quantization
modelopt

Identity and Version

Repository
local-inference-lab/Qwen3.8-27B-NVFP4-QAD
Publisher
Local Inference Lab
Task
Image and text to text
Modality
Image and text
Library
transformers
Parameters
19.2B parameters
Languages
Not stated by the source
Revision
f40a31cd813a6746067e7d6446ff2cb708dbb779
First published
2026-09-18
Last updated
2026-09-18

Files and Weights

82 files, 22.7 GB in total. The weights are 67 files totalling 22.7 GB in safetensors.

Weights67 files · 22.7 GB
Configuration7 files · 380.9 KB
Tokenizer4 files · 22.9 MB
Documentation2 files · 14.3 KB
Other1 file · 9.0 KB
Repository1 file · 1.6 KB
Every file
FileTypeSizeSHA-256
source-preserved-000.safetensorsWeights1.0 GB dfc5b9c211ce
source-preserved-001.safetensorsWeights744.5 MB 9ce944d534ea
text-embedding-head.safetensorsWeights3.9 GB c02faa7685a3
text-layer-000.safetensorsWeights270.0 MB 416fc5c17d3c
text-layer-001.safetensorsWeights270.0 MB 5a0d0ee61663
text-layer-002.safetensorsWeights270.0 MB d1978b0f2db0
text-layer-003.safetensorsWeights258.6 MB 9b4076707aaa
text-layer-004.safetensorsWeights270.0 MB 4613eb8c74d7
text-layer-005.safetensorsWeights270.0 MB 644d9e099d90
text-layer-006.safetensorsWeights270.0 MB e322b663b284
text-layer-007.safetensorsWeights258.6 MB 641463acc2f6
text-layer-008.safetensorsWeights270.0 MB e7c3da5815f4
text-layer-009.safetensorsWeights270.0 MB 98616159b0a6
text-layer-010.safetensorsWeights270.0 MB 0c9d861b519f
text-layer-011.safetensorsWeights258.6 MB 36ba812bc1e4
text-layer-012.safetensorsWeights270.0 MB 82f57f8dfebd
text-layer-013.safetensorsWeights270.0 MB 7de91983df09
text-layer-014.safetensorsWeights270.0 MB ec81517060d8
text-layer-015.safetensorsWeights258.6 MB 0cff517505d2
text-layer-016.safetensorsWeights270.0 MB bfd5cfde382a
text-layer-017.safetensorsWeights270.0 MB 788c3757eec4
text-layer-018.safetensorsWeights270.0 MB 99465d9c5b70
text-layer-019.safetensorsWeights258.6 MB 13f14249f499
text-layer-020.safetensorsWeights270.0 MB 1442315cacf0
text-layer-021.safetensorsWeights270.0 MB 5bf546a3fedc
text-layer-022.safetensorsWeights270.0 MB 80f9db1d563c
text-layer-023.safetensorsWeights258.6 MB 1ed100d86e47
text-layer-024.safetensorsWeights270.0 MB a56a434c0f44
text-layer-025.safetensorsWeights270.0 MB 85482313ae8a
text-layer-026.safetensorsWeights270.0 MB c7dd584fc357
text-layer-027.safetensorsWeights258.6 MB 6a4df9a6e1d9
text-layer-028.safetensorsWeights270.0 MB 6f355d0e0d9d
text-layer-029.safetensorsWeights270.0 MB 2679e76d126e
text-layer-030.safetensorsWeights270.0 MB 1ed2ca2cff37
text-layer-031.safetensorsWeights258.6 MB ccf0b3971b84
text-layer-032.safetensorsWeights270.0 MB 48890ea0eb8c
text-layer-033.safetensorsWeights270.0 MB 524ef558fd6b
text-layer-034.safetensorsWeights270.0 MB 2460cc732283
text-layer-035.safetensorsWeights258.6 MB 5d919f95daf2
text-layer-036.safetensorsWeights270.0 MB a6dd1217b471
text-layer-037.safetensorsWeights270.0 MB 89d3efd61b6c
text-layer-038.safetensorsWeights270.0 MB 261a58fcf2e2
text-layer-039.safetensorsWeights258.6 MB a63389fc82d9
text-layer-040.safetensorsWeights270.0 MB 5b44ab38be9f
text-layer-041.safetensorsWeights270.0 MB a5207cdfeec5
text-layer-042.safetensorsWeights270.0 MB ce50b0b1e0f8
text-layer-043.safetensorsWeights258.6 MB 06874448617c
text-layer-044.safetensorsWeights270.0 MB 4363a34c5229
text-layer-045.safetensorsWeights270.0 MB 5e3f11c427e1
text-layer-046.safetensorsWeights270.0 MB 58c56ec28216
text-layer-047.safetensorsWeights258.6 MB d9248ef97c97
text-layer-048.safetensorsWeights270.0 MB 45b963f8004a
text-layer-049.safetensorsWeights270.0 MB be021a9df805
text-layer-050.safetensorsWeights270.0 MB 775c251469c1
text-layer-051.safetensorsWeights258.6 MB 1771caf121ec
text-layer-052.safetensorsWeights270.0 MB 835f98d85b25
text-layer-053.safetensorsWeights270.0 MB 12c093f34a38
text-layer-054.safetensorsWeights270.0 MB bafad84d9fb3
text-layer-055.safetensorsWeights258.6 MB e2f428809c1f
text-layer-056.safetensorsWeights270.0 MB dbeaf81839a3
text-layer-057.safetensorsWeights270.0 MB 999e90a762f9
text-layer-058.safetensorsWeights270.0 MB 8d6a681e1577
text-layer-059.safetensorsWeights258.6 MB 91e906712e9e
text-layer-060.safetensorsWeights270.0 MB 09b9f89f6aab
text-layer-061.safetensorsWeights270.0 MB 982e38efcfff
text-layer-062.safetensorsWeights270.0 MB df2e51ae29d1
text-layer-063.safetensorsWeights258.6 MB fbf632e3ecac
config.jsonConfiguration97.5 KB —
generation_config.jsonConfiguration202 B —
hf_quant_config.jsonConfiguration88.8 KB —
materialization.jsonConfiguration1.1 KB —
model.safetensors.index.jsonConfiguration192.6 KB —
preprocessor_config.jsonConfiguration390 B —
video_preprocessor_config.jsonConfiguration385 B —
LICENSEDocumentation11.5 KB —
README.mdDocumentation2.7 KB —
chat_template.jinjaOther9.0 KB —
.gitattributesRepository1.6 KB —
merges.txtTokenizer3.4 MB —
tokenizer.jsonTokenizer12.8 MB 0997f410c57a
tokenizer_config.jsonTokenizer17.9 KB —
vocab.jsonTokenizer6.7 MB —

License and Download

License
apache-2.0
Access
Open weights, no gate
Download size
22.7 GB
Download from Local Inference Lab

Released by Local Inference Lab through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published22.7 GB
16-bit38.5 GB
8-bit19.2 GB
4-bit9.6 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About Qwen3.8-27B-NVFP4-QAD

How much GPU memory does Qwen3.8-27B-NVFP4-QAD need?

About 46.1 GB at 16-bit and 11.5 GB at 4-bit: the weights (19.2B parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run Qwen3.8-27B-NVFP4-QAD on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use Qwen3.8-27B-NVFP4-QAD commercially?

Yes. Qwen3.8-27B-NVFP4-QAD is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

What is Qwen3.8-27B-NVFP4-QAD's context length?

262,144 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Image and text to text

Qwen3.8-27B-NVFP4

RadixArk

The RadixArk Qwen3.8-27B-NVFP4 model is the quantized version of Qwen/Qwen3.8-27B. The quantization was produced at RadixArk using NVIDIA Model Optimizer, following a mixed NVFP4 W4A4 recipe. Run on SGLang: launch command and per-platform recipes in the Qwen3.8-27B cookbook. This model is not owned or developed by RadixArk. It is a quantized derivative of Qwen's model; see the upstream Qwen3.8-27B model card for the source model's capabilities, training information, limitations, and license. Global Developers looking to deploy an off-the-shelf, pre-quantized model in AI agent systems, chatbots, RAG systems, and other AI-powered applications. Hugging Face 08/14/2026 via…

Open weights apache-2.0 18.2B parameters 262,144 tokens Model Optimizer
View model

Model · Image and text to text

Qwen3.6-27B-NVFP4

Unsloth AI

2.5x faster throughput than other NVFP4 quants. This is an Unsloth NVFP4 quantized checkpoint calibrated on a mixture of our Unsloth dataset + UltraChat dataset. Works on a 24GB VRAM GPU. Benchmarks on 1xB200 128 concurrency. For accuracy benchmarks, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all: Read all benchmarks in our NVFP4 blog To install vLLM in a separate venv: Then to serve the 27B NVFP4 quant: Also do NOT use the Marlin backend since it's 2x slower - use the native vLLM or cute-DSL / CUTLASS / flashinfertrtllm backends! You must use the below or you will get 2x slower inference! This checkpoint…

Open weights apache-2.0 21.2B parameters 262,144 tokens transformers
View model

Model · Image and text to text

Qwen3.8-27B-NVFP4-RTX5090

Gittensor Model Hub

Runs on SparkInfer, SGLang and vLLM unmodified — configs for all three are below. Serving many users at once? See concurrency. SparkInfer × this NVFP4 build × the DSpark v2 drafter — an engine, a checkpoint, and a speculative drafter optimized against each other, compounding to 4.3×. The drafter never changes what the model says: the target verifies every drafted token. GeForce RTX 5090–specific NVFP4 checkpoint of Qwen/Qwen3.8-27B, quantized with NVIDIA Model Optimizer. Serves the full native 262,144-token context on 32 GB. With the DSpark v2 drafter: 264.8 tok/s overall — up to 420 on code — on SparkInfer (its bench harness; the HTTP server is autoregressive-only today) and 161.7 tok/s on…

Open weights apache-2.0 14.6B parameters 262,144 tokens transformers
View model

Model · Image and text to text

Qwen3.6-35B-A3B-NVFP4

Unsloth AI

1.56x faster throughput than other NVFP4 quants. This is an Unsloth NVFP4 quantized checkpoint calibrated on a mixture of our Unsloth dataset + UltraChat dataset. Works on a 32GB VRAM GPU. Benchmarks on 1xB200 128 concurrency. Use the 35B NVFP4 Fast version for 1.79x faster at a little less accuracy For accuracy benchmarks, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all: Read all benchmarks in our NVFP4 blog To install vLLM in a separate venv: Then to serve the 35B variant: You must use the below or you will get 2x slower inference! Also do NOT use the Marlin backend since it's 2x slower - use the native…

Open weights apache-2.0 24.6B parameters 262,144 tokens transformers
View model

Model · Image and text to text

gemma-4-26B-A4B-it

Google

Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages. Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning. The models are available in five distinct sizes: E2B, E4B, 12B, 26B A4B, and 31B. Their diverse sizes make them deployable in environments ranging from…

Open weights apache-2.0 25.8B parameters 262,144 tokens transformers
View model