Vishva007/Qwen3.5-9B-W4A16-AutoRound-NVFP4
This is an NVFP4 (W4A16: 4-bit weight, 16-bit activation) quantized version of Qwen/Qwen3.5-4B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention.
Exported via compressed-tensors (llm_compressor format) with MTP layers preserved.
Quantization Details
| Parameter |
Value |
| Method |
AutoRound (NVFP4 Scheme) |
| Format |
llm_compressor / compressed-tensors |
| Group Size |
16 |
| Symmetric |
Yes |
| Iterations |
1000 |
| Calibration Samples |
512 |
| Batch Size |
4 |
| Sequence Length |
2048 |
| Torch Compile |
Disabled |
Key Notes
- NVFP4 Quantization — Weights are quantized to FP4 micro-scaled formats using group size 16; activations remain in 16-bit for stable numerical fidelity.
- Vision Tower (
visual / quant_nontext_module): Kept in BF16 to preserve visual reasoning and OCR precision.
- Special Modules (
ignore_layers & layer_config): linear_attn, mtp, and mtp.fc are strictly retained in native bfloat16 to ensure zero regression in speculative decoding and attention mechanisms.
vLLM Serving
Serve the model directly via vLLM with tool-calling, MTP speculative decoding, and multimodal options enabled:
vllm serve Vishva007/Qwen3.5-9B-W4A16-AutoRound-NVFP4 \
--port 8000 \
--host 0.0.0.0 \
--max-model-len 18000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.90 \
--enable-prefix-caching \
--enable-chunked-prefill \
--max-num-batched-tokens 18000 \
--max-num-seqs 128 \
--trust-remote-code \
--dtype bfloat16 \
--mm-processor-kwargs '{"max_pixels": 10485760, "max_length": 65536, "truncation": false}' \
--mm-processor-cache-type shm \
--limit-mm-per-prompt '{"image": 50}' \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}'
MTP / Speculative Decoding
This model supports Multi-Token Prediction (MTP) for accelerated inference throughput using speculative decoding.
When serving with compatible backends like vLLM, enable MTP using:
--speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}'
Notes
- Native draft weights for MTP were preserved during quantization to ensure seamless compatibility without drafting degradation.
- You can tune
num_speculative_tokens (e.g., between 1 and 3) to optimize speed and latency depending on context length and hardware.
Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.
PyTorch 2.14
| Template |
CUDA Version |
Docker Image |
Template ID |
Deploy |
| PyTorch 2.14 (CUDA 12.6) |
12.6 |
vishva123/cuda-12.6-pytorch-2.14-runpod |
d7lxsa4w9m |
| PyTorch 2.14 (CUDA 13.0) |
13.0 |
vishva123/cuda-13.0-pytorch-2.14-runpod |
yk0y6j6rpg |
| PyTorch 2.14 (CUDA 13.2) |
13.2 |
vishva123/cuda-13.2-pytorch-2.14-runpod |
gsp4gwx0nw |
PyTorch 2.13
| Template |
CUDA Version |
Docker Image |
Template ID |
Deploy |
| PyTorch 2.13 (CUDA 12.6) |
12.6 |
vishva123/cuda-12.6-pytorch-2.13-runpod |
gmlupxnxfk |
| PyTorch 2.13 (CUDA 13.0) |
13.0 |
vishva123/cuda-13.0-pytorch-2.13-runpod |
y3j8xvk4f4 |
| PyTorch 2.13 (CUDA 13.2) |
13.2 |
vishva123/cuda-13.2-pytorch-2.13-runpod |
vigpissn5w |
PyTorch 2.12
| Template |
CUDA Version |
Docker Image |
Template ID |
Deploy |
| PyTorch 2.12 (CUDA 12.6) |
12.6 |
vishva123/cuda-12.6-pytorch-2.12-runpod |
ctmz86zmf0 |
| PyTorch 2.12 (CUDA 13.0) |
13.0 |
vishva123/cuda-13.0-pytorch-2.12-runpod |
qjko5yiwzi |
| PyTorch 2.12 (CUDA 13.2) |
13.2 |
vishva123/cuda-13.2-pytorch-2.12-runpod |
ifg6xmye0f |
