LT-OPD trains vision-language models with fewer visual tokens through on-policy self-distillation. This Qwen3.5-4B model combines CDPruner with an MLP that aggregates discarded tokens into their nearest retained tokens, preserving their original positions without adding tokens. Use Python 3.12 and the CUDA environment in the installation guide. Load the model with LT-OPD's visual-token compression runtime
Valen-4B is an open-weight model for image and text to text from Valen, released under Apache License 2.0. It has 4.5B parameters and a 262,144-token context. At 16-bit it needs about 10.9 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 204 downloads a month.
Valen-4B combines Qwen3.5-4B with a two-layer MLP-Mixer decision head. It predicts probability distributions for Choice, Noul (yes/no), and Score questions. A shared state can contain text, images, or video and serve multiple questions.
Parameters4.5B
Context262,144
Weights9.1 GB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads204
Runs On
What it takes to serve Valen-4B (4.5B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 9.1 GB | 10.9 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 4.5 GB | 5.4 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 2.3 GB | 2.7 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 18, 2026.
Valen-4B on every accelerator the SAVRN Index prices, at every precision
Model Card
By Valen, published under apache-2.0, revision 223660591947.
Valen-4B combines Qwen3.5-4B with a two-layer MLP-Mixer decision head. It predicts probability distributions for Choice, Noul (yes/no), and Score questions. A shared state can contain text, images, or video and serve multiple questions. Valen-4B 基于 Qwen3.5-4B 和两层 MLP-Mixer 决策头，支持图文、文本、视频，以及共享 state 的多题推理。 These weights include merged LoRA adapters. BF16 rounding can cause small probability differences from the original training checkpoints used for the reported benchmarks. The verified environment uses Python 3.10, PyTorch 2.6.0, Transformers 5.4.0, torchvision 0.21.0, Pillow, and av 16.1.0. Flash Attention 2 requires a compatible flash-attn installation. For a local copy, replace the…
Read Valen's full model card
Valen-4B combines Qwen3.5-4B with a two-layer MLP-Mixer decision head. It predicts probability distributions for Choice, Noul (yes/no), and Score questions. A shared state can contain text, images, or video and serve multiple questions.
Valen-4B 基于 Qwen3.5-4B 和两层 MLP-Mixer 决策头，支持图文、文本、视频，以及共享 state 的多题推理。
Training / 训练
- Two-stage SFT: decision-head warmup followed by joint LoRA, Mixer, and visual-merger fine-tuning. We train on millions of samples.
- Backbone: BF16. Mixer: FP32. Videos use 16 sampled frames.
Files / 权重格式
- Root directory: complete backbone with merged LoRA, trained visual merger, FP32 Mixer, tokenizer, processor, and standalone loading code.
These weights include merged LoRA adapters. BF16 rounding can cause small probability differences from the original training checkpoints used for the reported benchmarks.
Load merged weights / 加载完整合并权重
The verified environment uses Python 3.10, PyTorch 2.6.0, Transformers 5.4.0, torchvision 0.21.0, Pillow, and av 16.1.0. Flash Attention 2 requires a compatible flash-attn installation.
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained(
"Valen-Team/Valen-4B",
trust_remote_code=True,
dtype=torch.bfloat16,
attn_implementation="sdpa", # or "flash_attention_2"
).to("cuda").eval()
torch.set_float32_matmul_precision("highest")
torch.backends.cudnn.allow_tf32 = False
response = model.predict({
"state": "There is one cat in the room.",
"questions": {
"animal": {"type": "choice", "instructions": "Which animal is present?",
"criteria": {"cat": "A cat", "dog": "A dog"}},
"is_cat": {"type": "noul", "instructions": "There is a cat in the room."},
},
}, execution="shared_state")
print(response["answers"])
For a local copy, replace the repository ID with "./Valen-4B".
execution="question" scores questions separately. Images and video use the Valen messages, image_url, and video_url schema; relative media paths are resolved against media_root.
Configuration
- Architecture
- ValenQwenForDecisionMaking
- Context length (tokens)
- 262,144
- Layers
- 32
- Hidden size
- 2,560
- Feed-forward size
- 9,216
- Attention heads
- 16
- Key/value heads
- 4
- Head dimension
- 256
- Vocabulary size
- 248,320
- Model type
- valen_qwen
Identity and Version
- Repository
- Valen-Team/Valen-4B
- Publisher
- Valen
- Task
- Image and text to text
- Modality
- Image and text
- Library
- transformers
- Parameters
- 4.5B parameters
- Languages
- Not stated by the source
- Revision
- 22366059194719ad8c4270567569e1d559491fa3
- First published
- 2026-10-05
- Last updated
- 2026-10-13
Files and Weights
21 files, 9.1 GB in total. The weights are 3 files totalling 9.1 GB in safetensors.
Weights3 files · 9.1 GB
Configuration12 files · 113.9 KB
Tokenizer2 files · 20.0 MB
Documentation2 files · 13.9 KB
Other1 file · 7.8 KB
Repository1 file · 1.6 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model-00001-of-00003.safetensors | Weights | 4.0 GB | 21fef5c5b93c |
| model-00002-of-00003.safetensors | Weights | 4.0 GB | 9ec6792a776d |
| model-00003-of-00003.safetensors | Weights | 1.1 GB | c7b96a0d85eb |
| batching.py | Configuration | 3.0 KB | — |
| compiler.py | Configuration | 13.6 KB | — |
| config.json | Configuration | 3.5 KB | — |
| configuration_valen.py | Configuration | 737 B | — |
| data_types.py | Configuration | 258 B | — |
| heads.py | Configuration | 6.6 KB | — |
| model.safetensors.index.json | Configuration | 70.8 KB | — |
| modeling_valen.py | Configuration | 4.1 KB | — |
| processor_config.json | Configuration | 1.2 KB | — |
| responses.py | Configuration | 1.5 KB | — |
| runtime_model.py | Configuration | 5.3 KB | — |
| schema.py | Configuration | 3.3 KB | — |
| LICENSE | Documentation | 11.5 KB | — |
| README.md | Documentation | 2.4 KB | — |
| chat_template.jinja | Other | 7.8 KB | — |
| .gitattributes | Repository | 1.6 KB | — |
| tokenizer.json | Tokenizer | 20.0 MB | 87a7830d63fc |
| tokenizer_config.json | Tokenizer | 1.1 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 9.1 GB
Download from Valen
Released by Valen through its official repository on Hugging Face. Read the license.
Built From
- Derived from Qwen/Qwen3.5-4B
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 9.1 GB |
| 16-bit | 9.1 GB |
| 8-bit | 4.5 GB |
| 4-bit | 2.3 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About Valen-4B
How much GPU memory does Valen-4B need?
About 10.9 GB at 16-bit and 2.7 GB at 4-bit: the weights (4.5B parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run Valen-4B on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use Valen-4B commercially?
Yes. Valen-4B is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is Valen-4B's context length?
262,144 tokens, from the maximum position embeddings in its published configuration.
Similar Models
H2O-Lightning-4B is a 4B-parameter decision model from H2O.ai, built on Qwen/Qwen3.5-4B. It answers typed decision questions about a record (a document, a ticket, a policy, a conversation) and about images that come with the record (photos, screenshots, scanned documents, charts). It returns a probability for every option: - yes/no (noul): gives the probability that a statement is true; It runs on unmodified vLLM 0.30.0 with a small standard-library shim in front (h2olightningshim.py). Each decision is one forward pass and one output token, so the cost is input tokens only. Try it in your browser: live demo on Hugging Face Spaces (text and images; built by the Hugging Face team and running…
Vinci Piccolo is a small, open-weight chat model fine-tuned for character and honesty — the first model in the Vinci family from SimpleDirect. The character you'd want in an AI, open and small enough to run yourself. Try it: chat app — free · ollama run hf.co/simpledirect/Vinci-Piccolo-1.0-GGUF Weight-file size is not a runtime-memory requirement — model loading, KV cache, context length and batching all need memory beyond the weights. See Hardware requirements below for the figures we do give. Most fine-tuning optimizes for capability. Vinci Piccolo is fine-tuned for something else: a consistent character and an honest disposition. It is trained against a written, public Constitution that…
INT4 weight-only GPTQ quantization of Qwen/Qwen3.5-4B for memory-constrained single-GPU serving. Also the verifier for the (DFlash2 speculative decoding, τ 3.8 at the serving point / 4.1 greedy vs native MTP 2.9 / 3.0 on the same ladder). - W4A16 (INT4 weights, BF16 activations), group 128, act-order, compressed-tensors (pack-quantized) — vLLM Marlin kernels - Calibrated with llm-compressor dataset-native proportions from (~2.6M effective tokens). The exact rows are hash-pinned in — that pin is the citable calibration generation. (A -cascade suffix on local paths in the lab repo refers to the same generation; this repo carries only the current one and supersedes the initial pre-cascade…
This model is a fine-tuned version of Qwen/Qwen3.5-4B-Base on the Omni-Edu-70K dataset. The following hyperparameters were used during training: - learningrate: 5e-06 - trainbatchsize: 1 - evalbatchsize: 8 - distributedtype: multi-GPU - numdevices: 8 - gradientaccumulationsteps: 8 - totaltrainbatchsize: 64 - totalevalbatchsize: 64 - lrschedulertype: cosine - lrschedulerwarmupsteps: 0.1 - numepochs: 3.0 - Transformers 5.2.0 - Pytorch 2.10.0 - Datasets 4.0.0 - Tokenizers 0.22.2
The Label baseline of the DN-MOPD paper at Qwen3.5-4B continued to 160 updates (paper Table 5): multi-teacher on-policy distillation with label routing (each prompt is scored by the expert of its domain, every domain multiplier is 1). Released for comparison with DN-MOPD-Qwen3.5-4B; it is not the proposed method. Paper: Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation (arXiv:2609.35347, project page) · Code: github.com/LiXin97/DN-MOPD The full recipe, with the launch scripts for every row of the paper's tables, is in recipes/qwen3.5/ and docs/recipe.md. This model was trained and evaluated with the non-thinking chat format. Pass enablethinking=False to the…