This is the model card of a transformers model that has been pushed on the Hub. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Use the code below to get started with the model. Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
Open-weight model · Image and text to text
Qwen3.5-2B-Base-heretic
by Roumiany aminroumiany/Qwen3.5-2B-Base-heretic
Qwen3.5-2B-Base-heretic is an open-weight model for image and text to text from Roumiany, released under Apache License 2.0. It has 2.2B parameters and a 262,144-token context. At 16-bit it needs about 5.3 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 56 downloads a month.
Over recent months, we have intensified our focus on developing foundation models that deliver exceptional utility and performance.
Parameters2.2B
Context262,144
Weights4.4 GB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads56
Runs On
What it takes to serve Qwen3.5-2B-Base-heretic (2.2B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 4.4 GB | 5.3 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 2.2 GB | 2.7 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 1.1 GB | 1.3 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 18, 2026.
Qwen3.5-2B-Base-heretic on every accelerator the SAVRN Index prices, at every precision
Model Card
By Roumiany, published under apache-2.0, revision 931aed126c4a.
Over recent months, we have intensified our focus on developing foundation models that deliver exceptional utility and performance. Qwen3.5 represents a significant leap forward, integrating breakthroughs in multimodal learning, architectural efficiency, reinforcement learning scale, and global accessibility to empower developers and enterprises with unprecedented capability and efficiency. For more details, please refer to our blog post Qwen3.5. If you find our work helpful, feel free to give us a cite.
Read Roumiany's full model card
This is a decensored version of Qwen/Qwen3.5-2B-Base, made using Heretic v1.4.0
[!TIP] This model is reproducible!
See the README in the
reproducedirectory for more information.
Abliteration parameters
| Parameter | Value |
|---|---|
| direction_index | 13.98 |
| attn.o_proj.max_weight | 1.49 |
| attn.o_proj.max_weight_position | 15.96 |
| attn.o_proj.min_weight | 0.03 |
| attn.o_proj.min_weight_distance | 13.69 |
| mlp.down_proj.max_weight | 1.10 |
| mlp.down_proj.max_weight_position | 15.49 |
| mlp.down_proj.min_weight | 0.80 |
| mlp.down_proj.min_weight_distance | 10.68 |
Performance
| Metric | This model | Original model (Qwen/Qwen3.5-2B-Base) |
|---|---|---|
| KL divergence | 0.0297 | 0 (by definition) |
| Refusals | 4/100 | 81/100 |
Qwen3.5-2B-Base
[!Note] This repository contains model weights and configuration files for the pre-trained only model in the Hugging Face Transformers format.
These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, etc.
The intended use cases are fine-tuning, in-context learning experiments, and other research or development purposes, not direct interaction. However, the control tokens, e.g.,
<|im_start|>and<|im_end|>were trained to allow efficient LoRA-style PEFT with the official chat template, mitigating the need to finetune embeddings, a significant optimization given Qwen3.5's larger vocabulary.
Over recent months, we have intensified our focus on developing foundation models that deliver exceptional utility and performance. Qwen3.5 represents a significant leap forward, integrating breakthroughs in multimodal learning, architectural efficiency, reinforcement learning scale, and global accessibility to empower developers and enterprises with unprecedented capability and efficiency.
Qwen3.5 Highlights
Qwen3.5 features the following enhancement:
-
Unified Vision-Language Foundation: Early fusion training on multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models across reasoning, coding, agents, and visual understanding benchmarks.
-
Efficient Hybrid Architecture: Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency and cost overhead.
-
Scalable RL Generalization: Reinforcement learning scaled across million-agent environments with progressively complex task distributions for robust real-world adaptability.
-
Global Linguistic Coverage: Expanded support to 201 languages and dialects, enabling inclusive, worldwide deployment with nuanced cultural and regional understanding.
-
Next-Generation Training Infrastructure: Near-100% multimodal training efficiency compared to text-only training and asynchronous RL frameworks supporting massive-scale agent scaffolds and environment orchestration.
For more details, please refer to our blog post Qwen3.5.
Model Overview
- Type: Causal Language Model with Vision Encoder
- Training Stage: Pre-training & Post-training
- Language Model
- Number of Parameters: 2B
- Hidden Dimension: 2048
- Token Embedding: 248320 (Padded)
- Number of Layers: 24
- Hidden Layout: 6 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))
- Gated DeltaNet:
- Number of Linear Attention Heads: 16 for V and 16 for QK
- Head Dimension: 128
- Gated Attention:
- Number of Attention Heads: 8 for Q and 2 for KV
- Head Dimension: 256
- Rotary Position Embedding Dimension: 64
- Feed Forward Network:
- Intermediate Dimension: 6144
- LM Output: 248320 (Tied to token embedding)
- MTP: trained with multi-steps
- Context Length: 262,144 natively and extensible up to 1,010,000 tokens.
Citation
If you find our work helpful, feel free to give us a cite.
@misc{qwen3.5,
title = {{Qwen3.5}: Towards Native Multimodal Agents},
author = {{Qwen Team}},
month = {February},
year = {2026},
url = {https://qwen.ai/blog?id=qwen3.5}
}
Configuration
- Architecture
- Qwen3_5ForConditionalGeneration
- Context length (tokens)
- 262,144
- Layers
- 24
- Hidden size
- 2,048
- Feed-forward size
- 6,144
- Attention heads
- 8
- Key/value heads
- 2
- Head dimension
- 256
- Vocabulary size
- 248,320
- Model type
- qwen3_5
Identity and Version
- Repository
- aminroumiany/Qwen3.5-2B-Base-heretic
- Publisher
- Roumiany
- Task
- Image and text to text
- Modality
- Image and text
- Library
- transformers
- Parameters
- 2.2B parameters
- Languages
- Not stated by the source
- Revision
- 931aed126c4af905fb8adde74ed403d5eb2dc8b6
- First published
- 2026-10-13
- Last updated
- 2026-10-16
Files and Weights
15 files, 4.4 GB in total. The weights are 1 file totalling 4.4 GB in safetensors.
Weights1 file · 4.4 GB
Configuration4 files · 13.1 KB
Tokenizer2 files · 20.0 MB
Documentation2 files · 8.8 KB
Other5 files · 956.8 KB
Repository1 file · 1.6 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model.safetensors | Weights | 4.4 GB | b1ff70cceab9 |
| config.json | Configuration | 2.7 KB | — |
| generation_config.json | Configuration | 116 B | — |
| processor_config.json | Configuration | 1.2 KB | — |
| reproduce/reproduce.json | Configuration | 9.0 KB | — |
| README.md | Documentation | 4.8 KB | — |
| reproduce/README.md | Documentation | 4.0 KB | — |
| chat_template.jinja | Other | 7.8 KB | — |
| reproduce/Qwen--Qwen3--5-2B-Base.jsonl | Other | 945.1 KB | — |
| reproduce/SHA256SUMS | Other | 84 B | — |
| reproduce/config.toml | Other | 1.9 KB | — |
| reproduce/requirements.txt | Other | 1.9 KB | — |
| .gitattributes | Repository | 1.6 KB | — |
| tokenizer.json | Tokenizer | 20.0 MB | a5cd9732badc |
| tokenizer_config.json | Tokenizer | 1.5 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 4.4 GB
Download from Roumiany
Released by Roumiany through its official repository on Hugging Face. Read the license.
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 4.4 GB |
| 16-bit | 4.4 GB |
| 8-bit | 2.2 GB |
| 4-bit | 1.1 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About Qwen3.5-2B-Base-heretic
How much GPU memory does Qwen3.5-2B-Base-heretic need?
About 5.3 GB at 16-bit and 1.3 GB at 4-bit: the weights (2.2B parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run Qwen3.5-2B-Base-heretic on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use Qwen3.5-2B-Base-heretic commercially?
Yes. Qwen3.5-2B-Base-heretic is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is Qwen3.5-2B-Base-heretic's context length?
262,144 tokens, from the maximum position embeddings in its published configuration.
Similar Models
The Label baseline of the DN-MOPD paper at Qwen3.5-2B continued to 160 updates (paper Table 5): multi-teacher on-policy distillation with label routing (each prompt is scored by the expert of its domain, every domain multiplier is 1). Released for comparison with DN-MOPD-Qwen3.5-2B; it is not the proposed method. Paper: Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation (arXiv:2609.35347, project page) · Code: github.com/LiXin97/DN-MOPD The full recipe, with the launch scripts for every row of the paper's tables, is in recipes/qwen3.5/ and docs/recipe.md. This model was trained and evaluated with the non-thinking chat format. Pass enablethinking=False to the…
A Qwen3.5-2B student trained with DN-MOPD (Domain-Normalized Multi-Teacher On-Policy Distillation) continued to 160 updates (paper Table 5). Three same-size RL experts (math, code, instruction following) teach one student on its own responses; each prompt is scored by the expert of its domain, and DN-MOPD rescales each domain's token-level feedback by its measured spread, wd = clip(σall / σd, 0.25, 4), so that no domain dominates the shared update. Paper: Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation (arXiv:2609.35347, project page) · Code: github.com/LiXin97/DN-MOPD The full recipe, with the launch scripts for every row of the paper's tables, is in…
This model was converted to MLX format from Qwen/Qwen3.5-2B using mlx-vlm version 0.3.12. Refer to the original model card for more details on the model.
Qwen3.5 with a two-layer MLP-Mixer decision head. Supports text, images and videos, and returns Choice, Noul and Score decisions. Multiple questions can share one state encoding with execution="sharedstate". Trained on millions of samples. Install Python 3.10+, PyTorch, Transformers 5.4.0, torchvision, Pillow and av. Flash Attention 2 is optional with a compatible CUDA build. The model contains its tokenizer, processor and custom inference code; a separate base-model download is unnecessary. Use attnimplementation="flashattention2" for Flash Attention, or execution="question" for independent questions. Video defaults to 16 frames. Image/video request examples and training instructions are…
We're excited to unveil Qwen2-VL, the latest iteration of our Qwen-VL model, representing nearly a year of innovation. SoTA understanding of images of various resolution & ratio: Qwen2-VL achieves state-of-the-art performance on visual understanding benchmarks, including MathVista, DocVQA, RealWorldQA, MTVQA, etc. Understanding videos of 20min+: Qwen2-VL can understand videos over 20 minutes for high-quality video-based question answering, dialog, content creation, etc. Agent that can operate your mobiles, robots, etc.: with the abilities of complex reasoning and decision making, Qwen2-VL can be integrated with devices like mobile phones, robots, etc., for automatic operation based on…