SAVRN
Search Contact SAVRN

Open-weight model · Text generation

dQwen3.5-4B-Base

by IFML UT-IFML/dQwen3.5-4B-Base

A masked diffusion language model adapted from Qwen3.5-4B. The backbone is hybrid: only its attention layers are made bidirectional, and the Gated DeltaNet layers stay causal. This is a base model, with no instruction tuning.

Parameters4.2B
Context262,144
Weights8.4 GB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads395

Runs On

What it takes to serve dQwen3.5-4B-Base (4.2B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 8.4 GB 10.1 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 4.2 GB 5.0 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 2.1 GB 2.5 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By IFML, published under apache-2.0, revision 58a4e7b4dedf.

A masked diffusion language model adapted from Qwen3.5-4B. The backbone is hybrid: only its attention layers are made bidirectional, and the Gated DeltaNet layers stay causal. This is a base model, with no instruction tuning. Paper: dQwen3.5: Hybrid-Attention Diffusion Language Models. Code: https://github.com/AntonXue/dQwen Needs a CUDA GPU and transformers>=5.13 (tested with torch 2.7.1+cu128, flash-linear-attention 0.5.1). generate decodes the whole canvas at once, committing positions above a confidence threshold (tau=0.9); pass blocklength=32 for left-to-right block decoding, or tau=None, stepsperblock=k for a fixed budget. The 50B-token checkpoint from the paper is…

Read IFML's full model card

A masked diffusion language model adapted from Qwen3.5-4B. The backbone is hybrid: only its attention layers are made bidirectional, and the Gated DeltaNet layers stay causal. This is a base model, with no instruction tuning.

Paper: dQwen3.5: Hybrid-Attention Diffusion Language Models. Code: https://github.com/AntonXue/dQwen

The dQwen3.5 family

model total params backbone
dQwen3.5-0.8B-Base 0.75B hybrid
dQwen3.5-2B-Base 1.88B hybrid
dQwen3.5-4B-Base (this repo) 4.21B hybrid
dQwen3.5-9B-Base 8.95B hybrid
dQwen3-1.7B-Base 1.72B full attention (control)

Quickstart

Needs a CUDA GPU and transformers>=5.13 (tested with torch 2.7.1+cu128, flash-linear-attention 0.5.1).

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("UT-IFML/dQwen3.5-4B-Base", trust_remote_code=True, dtype=torch.bfloat16).cuda().eval()
print(model.generate("def fibonacci(n):", gen_length=512, stop_strings=["\ndef "]).text)

generate decodes the whole canvas at once, committing positions above a confidence threshold (tau=0.9); pass block_length=32 for left-to-right block decoding, or tau=None, steps_per_block=k for a fixed budget. The 50B-token checkpoint from the paper is revision="step25000-swa".

Citation

@article{xue2026dqwen,
  title   = {dQwen3.5: Hybrid-Attention Diffusion Language Models},
  author  = {Xue, Anton and Rout, Litu and Akella, Aditya and Klivans, Adam and Sanghavi, Sujay and Shakkottai, Sanjay},
  journal = {arXiv preprint arXiv:2609.20751},
  year    = {2026}
}

Configuration

Architecture
DiffuQwen35
Context length (tokens)
262,144
Layers
32
Hidden size
2,560
Feed-forward size
9,216
Attention heads
16
Key/value heads
4
Head dimension
256
Vocabulary size
248,320
Model type
diffuqwen35

Identity and Version

Repository
UT-IFML/dQwen3.5-4B-Base
Publisher
IFML
Task
Text generation
Modality
Text
Library
transformers
Parameters
4.2B parameters
Languages
Not stated by the source
Revision
58a4e7b4dedfa4f0309f7bb5c75383d494e84c3c
First published
2026-08-03
Last updated
2026-09-18

Files and Weights

13 files, 8.4 GB in total. The weights are 2 files totalling 8.4 GB in safetensors.

Weights2 files · 8.4 GB
Configuration5 files · 52.7 KB
Tokenizer2 files · 20.0 MB
Documentation1 file · 2.1 KB
Other2 files · 1.1 MB
Repository1 file · 1.6 KB
Every file
FileTypeSizeSHA-256
model-00001-of-00002.safetensorsWeights5.0 GB 07ac9350a3f6
model-00002-of-00002.safetensorsWeights3.4 GB 7105d3e569d4
argparse.jsonConfiguration866 B —
config.jsonConfiguration2.1 KB —
generation_config.jsonConfiguration179 B —
model.safetensors.index.jsonConfiguration35.6 KB —
modeling_dqwen3_5.pyConfiguration14.0 KB —
README.mdDocumentation2.1 KB —
chat_template.jinjaOther7.8 KB —
training_log.jsonlOther1.1 MB —
.gitattributesRepository1.6 KB —
tokenizer.jsonTokenizer20.0 MB 06b9509352d2
tokenizer_config.jsonTokenizer1.2 KB —

License and Download

License
apache-2.0
Access
Open weights, no gate
Download size
8.4 GB
Download from IFML

Released by IFML through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published8.4 GB
16-bit8.4 GB
8-bit4.2 GB
4-bit2.1 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About dQwen3.5-4B-Base

How much GPU memory does dQwen3.5-4B-Base need?

About 10.1 GB at 16-bit and 2.5 GB at 4-bit: the weights (4.2B parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run dQwen3.5-4B-Base on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use dQwen3.5-4B-Base commercially?

Yes. dQwen3.5-4B-Base is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

What is dQwen3.5-4B-Base's context length?

262,144 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Text generation

Huihui-NeoHorse-1-4B-abliterated

Huihui.ai

This is an uncensored version of TokenRhythm/NeoHorse-1-4B created with abliteration (see remove-refusals-with-transformers to know more about it). This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens. Layers 5-17 are being ablated (0-based indexing). The MTP and Visual components were extracted from the original Qwen/Qwen3.5-4B and can provide excellent support. If needed, you only need to copy the contents of MTP-Visual to overwrite the model directory. You can use this model in your applications by loading it with Hugging Face's transformers library: - Risk of Sensitive or Controversial Outputs: This model’s safety filtering…

Open weights apache-2.0 4.2B parameters 262,144 tokens transformers
View model

Model · Text generation

Qwen3-4B

Qwen

Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Built upon extensive training, Qwen3 delivers groundbreaking advancements in reasoning, instruction-following, agent capabilities, and multilingual support, with the following key features: - Uniquely support of seamless switching between thinking mode (for complex logical reasoning, math, and coding) and non-thinking mode (for efficient, general-purpose dialogue) within single model, ensuring optimal performance across various scenarios. - Significantly enhancement in its reasoning capabilities, surpassing previous QwQ (in thinking mode) and…

Open weights apache-2.0 4B parameters 40,960 tokens transformers
View model

Model · Text generation

Qwen3-4B-Instruct-2507

Qwen

We introduce the updated version of the Qwen3-4B non-thinking mode, named Qwen3-4B-Instruct-2507, featuring the following key enhancements: - Significant improvements in general capabilities, including instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage. - Substantial gains in long-tail knowledge coverage across multiple languages. - Markedly better alignment with user preferences in subjective and open-ended tasks, enabling more helpful responses and higher-quality text generation. - Enhanced capabilities in 256K long-context understanding. Qwen3-4B-Instruct-2507 has the following features: NOTE: This model supports only non-thinking…

Open weights apache-2.0 4B parameters 262,144 tokens transformers
View model

Model · Text generation

Qwen3-4B-Base

Qwen

Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Building upon extensive advancements in training data, model architecture, and optimization techniques, Qwen3 delivers the following key improvements over the previously released Qwen2.5: Qwen3-4B-Base has the following features: For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our blog, GitHub, and Documentation. The code of Qwen3 has been in the latest Hugging Face transformers and we advise you to use the latest version of transformers. With transformers<4.51.0, you will…

Open weights apache-2.0 4B parameters 32,768 tokens transformers
View model

Model · Text generation

qwen3-4b-base-dapo-v4

Reliquary

Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Building upon extensive advancements in training data, model architecture, and optimization techniques, Qwen3 delivers the following key improvements over the previously released Qwen2.5: Qwen3-4B-Base has the following features: For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our blog, GitHub, and Documentation. The code of Qwen3 has been in the latest Hugging Face transformers and we advise you to use the latest version of transformers. With transformers<4.51.0, you will…

Open weights apache-2.0 4B parameters 32,768 tokens transformers
View model

Model · Text generation

RL-fromscratch

Tangyunbo

We introduce the updated version of the Qwen3-4B non-thinking mode, named Qwen3-4B-Instruct-2507, featuring the following key enhancements: - Significant improvements in general capabilities, including instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage. - Substantial gains in long-tail knowledge coverage across multiple languages. - Markedly better alignment with user preferences in subjective and open-ended tasks, enabling more helpful responses and higher-quality text generation. - Enhanced capabilities in 256K long-context understanding. Qwen3-4B-Instruct-2507 has the following features: NOTE: This model supports only non-thinking…

Open weights apache-2.0 4B parameters 262,144 tokens transformers
View model