Qwen3.5-9B-W4A16-AutoRound-NVFP4 by Vishva R: Open Model

Runs On

What it takes to serve Qwen3.5-9B-W4A16-AutoRound-NVFP4 (7.1B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

Precision	Weights	Memory needed	Cheapest setup	Per hour	Also fits
16-bit	14.2 GB	17.0 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
8-bit	7.1 GB	8.5 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
4-bit	3.5 GB	4.3 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 23, 2026.

Qwen3.5-9B-W4A16-AutoRound-NVFP4 on every accelerator the SAVRN Index prices, at every precision

Model Card

By Vishva R, published under apache-2.0, revision a98b33ac2610.

This is an NVFP4 (W4A16: 4-bit weight, 16-bit activation) quantized version of Qwen/Qwen3.5-9B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention. Exported via compressed-tensors (llmcompressor format) with MTP layers preserved. - NVFP4 Quantization — Weights are quantized to FP4 micro-scaled formats using group size 16; activations remain in 16-bit for stable numerical fidelity. - Vision Tower (visual / quantnontextmodule): Kept in BF16 to preserve visual reasoning and OCR precision. - Special Modules (ignorelayers & layerconfig): linearattn, mtp, and mtp.fc are strictly retained in native bfloat16 to ensure…

Read Vishva R's full model card

This is an NVFP4 (W4A16: 4-bit weight, 16-bit activation) quantized version of Qwen/Qwen3.5-9B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention. Exported via compressed-tensors (llm_compressor format) with MTP layers preserved.

Quantization Details

Parameter	Value
Method	AutoRound (NVFP4 Scheme)
Format	`llm_compressor` / `compressed-tensors`
Group Size	16
Symmetric	Yes
Iterations	1000
Calibration Samples	512
Batch Size	8
Sequence Length	2048
Torch Compile	Disabled

Key Notes

NVFP4 Quantization — Weights are quantized to FP4 micro-scaled formats using group size 16; activations remain in 16-bit for stable numerical fidelity.
Vision Tower (visual / quant_nontext_module): Kept in BF16 to preserve visual reasoning and OCR precision.
Special Modules (ignore_layers & layer_config): linear_attn, mtp, and mtp.fc are strictly retained in native bfloat16 to ensure zero regression in speculative decoding and attention mechanisms.

vLLM Serving

Serve the model directly via vLLM with tool-calling, MTP speculative decoding, and multimodal options enabled:

vllm serve Vishva007/Qwen3.5-9B-W4A16-AutoRound-NVFP4 \
  --port 8000 \
  --host 0.0.0.0 \
  --max-model-len 18000 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 18000 \
  --max-num-seqs 128 \
  --trust-remote-code \
  --dtype bfloat16 \
  --mm-processor-kwargs '{"max_pixels": 10485760, "max_length": 65536, "truncation": false}' \
  --mm-processor-cache-type shm \
  --limit-mm-per-prompt '{"image": 50}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}'

MTP / Speculative Decoding

This model supports Multi-Token Prediction (MTP) for accelerated inference throughput using speculative decoding.

When serving with compatible backends like vLLM, enable MTP using:

--speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}'

Notes

Native draft weights for MTP were preserved during quantization to ensure seamless compatibility without drafting degradation.
You can tune num_speculative_tokens (e.g., between 1 and 3) to optimize speed and latency depending on context length and hardware.

Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.14 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.14-runpod`	`d7lxsa4w9m`
PyTorch 2.14 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.14-runpod`	`yk0y6j6rpg`
PyTorch 2.14 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.14-runpod`	`gsp4gwx0nw`

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`	`y3j8xvk4f4`
PyTorch 2.13 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.13-runpod`	`vigpissn5w`

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`	`qjko5yiwzi`
PyTorch 2.12 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.12-runpod`	`ifg6xmye0f`

Configuration

Architecture: Qwen3_5ForConditionalGeneration
Context length (tokens): 262,144
Layers: 32
Hidden size: 4,096
Feed-forward size: 12,288
Attention heads: 16
Key/value heads: 4
Head dimension: 256
Vocabulary size: 248,320
Model type: qwen3_5
Quantization: compressed-tensors

Identity and Version

Repository: Vishva007/Qwen3.5-9B-W4A16-AutoRound-NVFP4
Publisher: Vishva R
Task: Image and text to text
Modality: Image and text
Library: Not stated by the source
Parameters: 7.1B parameters
Languages: Not stated by the source
Revision: a98b33ac26107040d6d3ffb9598efd2dba39da7c
First published: 2026-10-13
Last updated: 2026-10-16

Files and Weights

15 files, 11.7 GB in total. The weights are 4 files totalling 11.7 GB in safetensors.

Weights4 files · 11.7 GB

Configuration6 files · 143.0 KB

Tokenizer2 files · 20.0 MB

Documentation1 file · 6.1 KB

Other1 file · 7.8 KB

Repository1 file · 1.6 KB

Every file

File	Type	Size	SHA-256
model-00001-of-00003.safetensors	Weights	5.4 GB	0983e1b6830a
model-00002-of-00003.safetensors	Weights	3.8 GB	4e3f33e3322d
model-00003-of-00003.safetensors	Weights	2.0 GB	106ec926c62c
model_extra_tensors.safetensors	Weights	486.6 MB	e9f83dae0fed
config.json	Configuration	16.5 KB	—
generation_config.json	Configuration	116 B	—
model.safetensors.index.json	Configuration	111.8 KB	—
preprocessor_config.json	Configuration	390 B	—
processor_config.json	Configuration	1.2 KB	—
quantization_config.json	Configuration	13.0 KB	—
README.md	Documentation	6.1 KB	—
chat_template.jinja	Other	7.8 KB	—
.gitattributes	Repository	1.6 KB	—
tokenizer.json	Tokenizer	20.0 MB	a5cd9732badc
tokenizer_config.json	Tokenizer	1.5 KB	—

License and Download

License: apache-2.0
Access: Open weights, no gate
Download size: 11.7 GB

Download from Vishva R

Released by Vishva R through its official repository on Hugging Face. Read the license.

Built From

Derived from Qwen/Qwen3.5-9B
Quantized from Qwen/Qwen3.5-9B

Memory Requirements

Precision	Weights in memory
As published	11.7 GB
16-bit	14.2 GB
8-bit	7.1 GB
4-bit	3.5 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About Qwen3.5-9B-W4A16-AutoRound-NVFP4

How much GPU memory does Qwen3.5-9B-W4A16-AutoRound-NVFP4 need?

About 17 GB at 16-bit and 4.3 GB at 4-bit: the weights (7.1B parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run Qwen3.5-9B-W4A16-AutoRound-NVFP4 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use Qwen3.5-9B-W4A16-AutoRound-NVFP4 commercially?

Yes. Qwen3.5-9B-W4A16-AutoRound-NVFP4 is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

What is Qwen3.5-9B-W4A16-AutoRound-NVFP4's context length?

262,144 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Image and text to text

llava-1.5-7b-hf

Llava Hugging Face

Below is the model card of Llava model 7b, which is copied from the original Llava model card that you can find here. Check out also the Google Colab demo to run Llava on a free-tier Google Colab instance: Or check out our Spaces demo! LLaVA is an open-source chatbot trained by fine-tuning LLaMA/Vicuna on GPT-generated multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture. LLaVA-v1.5-7B was trained in September 2023. Paper or resources for more information: https://llava-vl.github.io/ First, make sure to have transformers >= 4.35.3. The model supports multi-image and multi-prompt generation. Meaning that you can pass multiple…

Open weights llama2 7.1B parameters 4,096 tokens transformers

View model

Model · Image and text to text

Huihui-Qwen3.8-27B-abliterated-EXL3-3.0bpw

Lee

This repository contains an EXL3 3.0 bpw quantization of The original model and abliteration work are attributed to huihui-ai; this repository contains the EXL3 quantization produced by grimlee. The source model is an abliterated / uncensored derivative of Qwen3.8-27B. For details about the original model modification and its behavior, refer to the source model card. The checkpoint retains the Qwen3.8 multimodal model structure, including the vision component. The optional SM89 runtime work documented below does not change the model format or weights. The checkpoint uses the standard EXL3 format with a target body bitrate of 3.0 bits per weight. The target bitrate applies to the quantized…

Open weights apache-2.0 6.7B parameters 262,144 tokens transformers

View model

Model · Image and text to text

openvla-7b-finetuned-libero-spatial

OpenVLA Collaboration

This model was produced by fine-tuning the OpenVLA 7B model via LoRA (r=32) on the LIBERO-Spatial dataset from the LIBERO simulation benchmark. We made a few modifications to the training dataset to improve final performance (see the OpenVLA paper for details). Below are the hyperparameters we used for all LIBERO experiments: - No gradient accumulation (i.e. gradaccumulationsteps == 1) - shufflebuffersize == 100000 See the OpenVLA GitHub README for instructions on how to run and evaluate this model in the LIBERO simulator.

Open weights mit 7.5B parameters transformers

View model

Model · Image and text to text

openvla-7b-finetuned-libero-object

OpenVLA Collaboration

This model was produced by fine-tuning the OpenVLA 7B model via LoRA (r=32) on the LIBERO-Object dataset from the LIBERO simulation benchmark. We made a few modifications to the training dataset to improve final performance (see the OpenVLA paper for details). Below are the hyperparameters we used for all LIBERO experiments: - No gradient accumulation (i.e. gradaccumulationsteps == 1) - shufflebuffersize == 100000 See the OpenVLA GitHub README for instructions on how to run and evaluate this model in the LIBERO simulator.

Open weights mit 7.5B parameters transformers

View model

Model · Image and text to text

openvla-7b-finetuned-libero-10

OpenVLA Collaboration

This model was produced by fine-tuning the OpenVLA 7B model via LoRA (r=32) on the LIBERO-10 (LIBERO-Long) dataset from the LIBERO simulation benchmark. We made a few modifications to the training dataset to improve final performance (see the OpenVLA paper for details). Below are the hyperparameters we used for all LIBERO experiments: - No gradient accumulation (i.e. gradaccumulationsteps == 1) - shufflebuffersize == 100000 See the OpenVLA GitHub README for instructions on how to run and evaluate this model in the LIBERO simulator.

Open weights mit 7.5B parameters transformers

View model

Model · Image and text to text

openvla-7b-finetuned-libero-goal

OpenVLA Collaboration

This model was produced by fine-tuning the OpenVLA 7B model via LoRA (r=32) on the LIBERO-Goal dataset from the LIBERO simulation benchmark. We made a few modifications to the training dataset to improve final performance (see the OpenVLA paper for details). Below are the hyperparameters we used for all LIBERO experiments: - No gradient accumulation (i.e. gradaccumulationsteps == 1) - shufflebuffersize == 100000 See the OpenVLA GitHub README for instructions on how to run and evaluate this model in the LIBERO simulator.

Open weights mit 7.5B parameters transformers

View model