Skip to page content
SAVRN
Search Plan a project

Open-weight model · Image and text to text

Qwen3.5-4B-W4A16-GPTQ

by Nick Chapman nchapman/Qwen3.5-4B-W4A16-GPTQ

Qwen3.5-4B-W4A16-GPTQ is an open-weight model for image and text to text from Nick Chapman, released under Apache License 2.0. It has 4.5B parameters and a 262,144-token context. At 16-bit it needs about 10.9 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 30 downloads a month.

INT4 weight-only GPTQ quantization of Qwen/Qwen3.5-4B for memory-constrained single-GPU serving. Also the verifier for the (DFlash2 speculative decoding, τ 3.8 at the serving point / 4.1 greedy vs native MTP 2.9 / 3.0 on the same ladder).

Parameters4.5B
Context262,144
Weights4.0 GB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads30

Runs On

What it takes to serve Qwen3.5-4B-W4A16-GPTQ (4.5B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 9.1 GB 10.9 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 4.5 GB 5.4 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 2.3 GB 2.7 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 18, 2026.

Qwen3.5-4B-W4A16-GPTQ on every accelerator the SAVRN Index prices, at every precision

Model Card

By Nick Chapman, published under apache-2.0, revision d5874365f709.

INT4 weight-only GPTQ quantization of Qwen/Qwen3.5-4B for memory-constrained single-GPU serving. Also the verifier for the (DFlash2 speculative decoding, τ 3.8 at the serving point / 4.1 greedy vs native MTP 2.9 / 3.0 on the same ladder). - W4A16 (INT4 weights, BF16 activations), group 128, act-order, compressed-tensors (pack-quantized) — vLLM Marlin kernels - Calibrated with llm-compressor dataset-native proportions from (~2.6M effective tokens). The exact rows are hash-pinned in — that pin is the citable calibration generation. (A -cascade suffix on local paths in the lab repo refers to the same generation; this repo carries only the current one and supersedes the initial pre-cascade…

Read Nick Chapman's full model card

INT4 weight-only GPTQ quantization of Qwen/Qwen3.5-4B for memory-constrained single-GPU serving. Also the verifier for the Qwen3.5-4B-DFlash2 (DFlash2 speculative decoding, τ 3.8 at the serving point / 4.1 greedy vs native MTP 2.9 / 3.0 on the same ladder).

Quantization recipe

  • W4A16 (INT4 weights, BF16 activations), group 128, act-order, compressed-tensors (pack-quantized) — vLLM Marlin kernels
  • Calibrated with llm-compressor
  • Calibration set: calib-cascade-v1 — 999 conversations sampled at dataset-native proportions from nvidia/Nemotron-Cascade-2-SFT-Data (~2.6M effective tokens). The exact rows are hash-pinned in warp-3090-quant-lab-frozen-data — that pin is the citable calibration generation. (A -cascade suffix on local paths in the lab repo refers to the same generation; this repo carries only the current one and supersedes the initial pre-cascade calibration.)

Fidelity vs BF16 base (378-conv multi-domain probe, domain-weighted)

  • weighted KL: 0.109; external OOD guard perplexity delta: +0.76%
  • every chat / math / IF / science / multilingual domain improved 16-40% over the initial calibration

Usage (vLLM)

from vllm import LLM
llm = LLM(model="nchapman/Qwen3.5-4B-W4A16-GPTQ")

Chat template: froggeric v22.5. Sampling defaults follow the Qwen3.5 family recommendation (temperature 1.0, top_p 0.95, top_k 20).

Configuration

Architecture
Qwen3_5ForConditionalGeneration
Context length (tokens)
262,144
Layers
32
Hidden size
2,560
Feed-forward size
9,216
Attention heads
16
Key/value heads
4
Head dimension
256
Vocabulary size
248,320
Model type
qwen3_5
Quantization
compressed-tensors

Identity and Version

Repository
nchapman/Qwen3.5-4B-W4A16-GPTQ
Publisher
Nick Chapman
Task
Image and text to text
Modality
Image and text
Library
transformers
Parameters
4.5B parameters
Languages
Not stated by the source
Revision
d5874365f7093b28b69379722d2f45d660fcbca6
First published
2026-10-08
Last updated
2026-10-14

Files and Weights

12 files, 4.0 GB in total. The weights are 2 files totalling 4.0 GB in safetensors.

Weights2 files · 4.0 GB
Configuration5 files · 112.3 KB
Tokenizer2 files · 20.0 MB
Documentation1 file · 1.9 KB
Other1 file · 28.2 KB
Repository1 file · 1.6 KB
Every file
FileTypeSizeSHA-256
model.safetensorsWeights3.8 GB f7577d368deb
model_mtp.safetensorsWeights241.2 MB 7f993d7b896c
config.jsonConfiguration8.0 KB —
generation_config.jsonConfiguration198 B —
model.safetensors.index.jsonConfiguration102.6 KB —
processor_config.jsonConfiguration1.2 KB —
recipe.yamlConfiguration322 B —
README.mdDocumentation1.9 KB —
chat_template.jinjaOther28.2 KB —
.gitattributesRepository1.6 KB —
tokenizer.jsonTokenizer20.0 MB 67d8113912fa
tokenizer_config.jsonTokenizer1.5 KB —

License and Download

License
apache-2.0
Access
Open weights, no gate
Download size
4.0 GB
Download from Nick Chapman

Released by Nick Chapman through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published4.0 GB
16-bit9.1 GB
8-bit4.5 GB
4-bit2.3 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About Qwen3.5-4B-W4A16-GPTQ

How much GPU memory does Qwen3.5-4B-W4A16-GPTQ need?

About 10.9 GB at 16-bit and 2.7 GB at 4-bit: the weights (4.5B parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run Qwen3.5-4B-W4A16-GPTQ on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use Qwen3.5-4B-W4A16-GPTQ commercially?

Yes. Qwen3.5-4B-W4A16-GPTQ is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

What is Qwen3.5-4B-W4A16-GPTQ's context length?

262,144 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Image and text to text

h2o-lightning-4b

H2O.ai

H2O-Lightning-4B is a 4B-parameter decision model from H2O.ai, built on Qwen/Qwen3.5-4B. It answers typed decision questions about a record (a document, a ticket, a policy, a conversation) and about images that come with the record (photos, screenshots, scanned documents, charts). It returns a probability for every option: - yes/no (noul): gives the probability that a statement is true; It runs on unmodified vLLM 0.30.0 with a small standard-library shim in front (h2olightningshim.py). Each decision is one forward pass and one output token, so the cost is input tokens only. Try it in your browser: live demo on Hugging Face Spaces (text and images; built by the Hugging Face team and running…

Open weights apache-2.0 4.5B parameters 262,144 tokens transformers
View model

Model · Image and text to text

Vinci-Piccolo-1.0

SimpleDirect

Vinci Piccolo is a small, open-weight chat model fine-tuned for character and honesty — the first model in the Vinci family from SimpleDirect. The character you'd want in an AI, open and small enough to run yourself. Try it: chat app — free · ollama run hf.co/simpledirect/Vinci-Piccolo-1.0-GGUF Weight-file size is not a runtime-memory requirement — model loading, KV cache, context length and batching all need memory beyond the weights. See Hardware requirements below for the figures we do give. Most fine-tuning optimizes for capability. Vinci Piccolo is fine-tuned for something else: a consistent character and an honest disposition. It is trained against a written, public Constitution that…

Open weights apache-2.0 4.5B parameters 262,144 tokens transformers
View model

Model · Image and text to text

Omni-Edu-4B

Hao Liang

This model is a fine-tuned version of Qwen/Qwen3.5-4B-Base on the Omni-Edu-70K dataset. The following hyperparameters were used during training: - learningrate: 5e-06 - trainbatchsize: 1 - evalbatchsize: 8 - distributedtype: multi-GPU - numdevices: 8 - gradientaccumulationsteps: 8 - totaltrainbatchsize: 64 - totalevalbatchsize: 64 - lrschedulertype: cosine - lrschedulerwarmupsteps: 0.1 - numepochs: 3.0 - Transformers 5.2.0 - Pytorch 2.10.0 - Datasets 4.0.0 - Tokenizers 0.22.2

Open weights other 4.5B parameters 262,144 tokens transformers
View model

Model · Image and text to text

DN-MOPD-Qwen3.5-4B-baseline-label-160updates

XinLi

The Label baseline of the DN-MOPD paper at Qwen3.5-4B continued to 160 updates (paper Table 5): multi-teacher on-policy distillation with label routing (each prompt is scored by the expert of its domain, every domain multiplier is 1). Released for comparison with DN-MOPD-Qwen3.5-4B; it is not the proposed method. Paper: Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation (arXiv:2609.35347, project page) · Code: github.com/LiXin97/DN-MOPD The full recipe, with the launch scripts for every row of the paper's tables, is in recipes/qwen3.5/ and docs/recipe.md. This model was trained and evaluated with the non-thinking chat format. Pass enablethinking=False to the…

Open weights apache-2.0 4.5B parameters 262,144 tokens transformers
View model

Model · Image and text to text

DN-MOPD-Qwen3.5-4B-160updates

XinLi

A Qwen3.5-4B student trained with DN-MOPD (Domain-Normalized Multi-Teacher On-Policy Distillation) continued to 160 updates (paper Table 5). Three same-size RL experts (math, code, instruction following) teach one student on its own responses; each prompt is scored by the expert of its domain, and DN-MOPD rescales each domain's token-level feedback by its measured spread, wd = clip(σall / σd, 0.25, 4), so that no domain dominates the shared update. Paper: Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation (arXiv:2609.35347, project page) · Code: github.com/LiXin97/DN-MOPD The full recipe, with the launch scripts for every row of the paper's tables, is in…

Open weights apache-2.0 4.5B parameters 262,144 tokens transformers
View model

Model · Image and text to text

ezjev-4b-s2

Everett

A typed-decision model (Jev-style /v1/systemone: choice, noul and score questions answered with a probability per option from one forward pass). LoRA fine-tune of Qwen/Qwen3.5-4B, merged into full weights, in two stages: 1. Stage 1 (ezjev-4b): LoRA r=16 on all language-model linear layers (attention, Gated-DeltaNet, MLP), LR 1e-4, one epoch over ~85k rows / ~106k questions. Loss: cross-entropy + Brier over the option labels, llm2jev chat prompt (thinking off). 2. Stage 2: a second LoRA at LR 5e-5 on ~30k rows targeting weak task families (RAG hallucination, product relevance, code-output selection, select-all-that-apply, clinical NLI, phishing emails, claim verification) with 40% replay.…

Open weights other 4.5B parameters 262,144 tokens transformers
View model