Skip to page content
SAVRN
Search Plan a project

Open-weight model · Image and text to text

Qwen3.5-4B-RKLLM

by Hanzo Huang HanzoHuang/Qwen3.5-4B-RKLLM

Qwen3.5-4B-RKLLM is an open-weight model for image and text to text from Hanzo Huang, released under Apache License 2.0. Its published files total 17.8 GB. It draws 122 downloads a month.

Image and text on Rockchip RK3576 and RK3588, converted from Qwen/Qwen3.5-4B. Use the files for your board’s exact SoC. w4a16 uses 4-bit weights and 16-bit activations; w4a16g128 adds a weight group size of 128. w8a8 uses 8-bit weights and activations.

Parameters—
Context—
Weights1.3 GB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads122

Model Card

By Hanzo Huang, published under apache-2.0, revision fb8999e62f30.

Image and text on Rockchip RK3576 and RK3588, converted from Qwen/Qwen3.5-4B. Use the files for your board’s exact SoC. w4a16 uses 4-bit weights and 16-bit activations; w4a16g128 adds a weight group size of 128. w8a8 uses 8-bit weights and activations. File size below is download size, not total runtime RAM. For image input, also download the vision encoder for the same SoC. Keep the language model and vision encoder from this release together. ONNX files are export intermediates; they are not needed to run these RKNN/RKLLM models. With Docker installed on a matching Rockchip board, this example starts the RK3576 w4a16g128 model and its API on port 8001: The image includes its model files.…

Read Hanzo Huang's full model card

Image and text on Rockchip RK3576 and RK3588, converted from Qwen/Qwen3.5-4B.

  • Conversion toolkit: RKLLM Toolkit 1.3.0.
  • Inputs / output: text or image + text → text.
  • License: apache-2.0; the upstream model card describes its terms and intended use.

Choose your files

Use the files for your board’s exact SoC. w4a16 uses 4-bit weights and 16-bit activations; w4a16_g128 adds a weight group size of 128. w8a8 uses 8-bit weights and activations. File size below is download size, not total runtime RAM.

Board Quantization Size Language model
RK3576 w4a16_g128 3.71 GiB Qwen3.5-4B_RK3576_w4a16_g128.rkllm
RK3576 w8a8 5.18 GiB Qwen3.5-4B_RK3576_w8a8.rkllm
RK3588 w8a8 5.16 GiB Qwen3.5-4B_RK3588_w8a8.rkllm

For image input, also download the vision encoder for the same SoC. Keep the language model and vision encoder from this release together.

Board Size Vision encoder
RK3576 0.68 GiB Qwen3.5-4B_vision_RK3576.rknn
RK3588 0.66 GiB Qwen3.5-4B_vision_RK3588.rknn

ONNX files are export intermediates; they are not needed to run these RKNN/RKLLM models.

Run on your board

With Docker installed on a matching Rockchip board, this example starts the RK3576 w4a16_g128 model and its API on port 8001:

sudo docker run --rm -d --name rkllm \
  --privileged -v /dev:/dev -p 8001:8001 \
  ghcr.io/hanzo-huang/rkllm-docker/vlm/qwen3.5-4b:rk3576-w4a16-g128

The image includes its model files. See rkllm-docker for API requests, image inputs, other board variants, and configuration.

To download the files separately instead:

hf download HanzoHuang/Qwen3.5-4B-RKLLM \
  RK3576/Qwen3.5-4B_RK3576_w4a16_g128.rkllm \
  RK3576/Qwen3.5-4B_vision_RK3576.rknn \
  --local-dir ./models

These are native RKLLM/RKNN artifacts. Use a compatible RKLLM runtime and the model’s correct chat template when integrating them in another application.

Release notes

No per-variant board benchmark is published here. Quantized output may differ from the original model; check responses on your own workloads. Use the toolkit version listed above when checking runtime compatibility.

SHA-256 checksums 
aa4d34b42752a0e491ed891ce4b7f32a745631601a65b175f743f971bdb33482  RK3576/Qwen3.5-4B_RK3576_w4a16_g128.rkllm
736f4b1065b481bae9cbd86e6c9ed30222abba9c5a3cb2d8d68dd92037d82dfe  RK3576/Qwen3.5-4B_RK3576_w8a8.rkllm
715566bbee72b25d8c4912f6cf6256a8ac264a754511573a99a541487bbc06b4  RK3588/Qwen3.5-4B_RK3588_w8a8.rkllm
6692f52adaa7ee0ba892c7cbba01979750cb82cff8061a571f56e9d84c6d98a2  RK3576/Qwen3.5-4B_vision_RK3576.rknn
c286ef69266c11a2a2cedd881ad6a40415c17c8924962f07454359bb6d60e2ba  RK3588/Qwen3.5-4B_vision_RK3588.rknn

Credits

Original model: Qwen/Qwen3.5-4B. Conversion tools: Rockchip RKLLM.

Identity and Version

Repository
HanzoHuang/Qwen3.5-4B-RKLLM
Publisher
Hanzo Huang
Task
Image and text to text
Modality
Image and text
Library
rkllm
Parameters
Not stated by the source
Languages
Not stated by the source
Revision
fb8999e62f305298f0527a3369cb30849d7174f2
First published
2026-08-11
Last updated
2026-10-19

Files and Weights

9 files, 17.8 GB in total. The weights are 1 file totalling 1.3 GB in onnx.

Weights1 file · 1.3 GB
Configuration1 file · 2.8 KB
Documentation1 file · 3.6 KB
Other5 files · 16.5 GB
Repository1 file · 2.2 KB
Every file
FileTypeSizeSHA-256
Qwen3.5-4B_vision.onnxWeights1.3 GB e359c3579c5d
Qwen3.5-4B_data_quant.jsonConfiguration2.8 KB —
README.mdDocumentation3.6 KB —
RK3576/Qwen3.5-4B_RK3576_w4a16_g128.rkllmOther4.0 GB aa4d34b42752
RK3576/Qwen3.5-4B_RK3576_w8a8.rkllmOther5.6 GB 736f4b1065b4
RK3576/Qwen3.5-4B_vision_RK3576.rknnOther725.7 MB 6692f52adaa7
RK3588/Qwen3.5-4B_RK3588_w8a8.rkllmOther5.5 GB 715566bbee72
RK3588/Qwen3.5-4B_vision_RK3588.rknnOther704.6 MB c286ef69266c
.gitattributesRepository2.2 KB —

License and Download

License
apache-2.0
Access
Open weights, no gate
Download size
1.3 GB
Download from Hanzo Huang

Released by Hanzo Huang through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published1.3 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About Qwen3.5-4B-RKLLM

Can I use Qwen3.5-4B-RKLLM commercially?

Yes. Qwen3.5-4B-RKLLM is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

Similar Models

Non-uniform GGUF quantizations of a 512-expert MoE, produced with GSQ and RCO, with a vision projector for multimodal use. This repository provides GGUF quantizations of Qwen3.8-Flash-Next at four sizes, together with the model's vision projector (mmproj) for multimodal use. In contrast to uniform quantization, which applies a single quantization type to all weight tensors, each model here assigns a separate quantization type to every tensor. The assignment is obtained by a gradient-based search that allocates precision according to per-tensor sensitivity, subject to a total size budget. The resulting files are standard GGUF and run unmodified in llama.cpp, Ollama, and LM Studio. A…

Open weights apache-2.0 gguf
View model

Model · Image and text to text

Huihui-Qwen3.8-27B-abliterated-GGUF

Huihui.ai

This is an uncensored version of Qwen/Qwen3.8-27B created with abliteration (see remove-refusals-with-transformers to know more about it). This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens. The newly added Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki series come from ConwayResearch/Underdog-Saluki-27B-1.0 and ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF. Only layers 22 to 52 (0-based indexing) have been ablated, while the other layers remain unablated. It may come with a small disclaimer warning. Due to ConwayResearch/Underdog-Saluki-27B-1.0 not providing an imatrix file, we will attempt to perform ablation by mixing the GGUF…

Open weights apache-2.0 transformers
View model

Qwen3.8-27B uncensored by HauhauCS 0/465 Refusals. This is the Aggressive variant: direct answers, no refusal behavior, and minimal preamble on hard prompts. Every text GGUF preserves Qwen3.8's native NextN head, and this release adds HauhauCS FastMTP: a specific acceleration sidecar qualified across the complete quant lineup at maximum native context. Vision is included through the separate BF16 projector. No changes to datasets or intended capabilities. This release preserves Qwen3.8-27B's text, reasoning, agentic, image, and video capabilities while applying the HauhauCS Aggressive uncensoring profile. Pick Aggressive when you specifically want the model to get to the answer without…

Open weights apache-2.0
View model

Model · Image and text to text

Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF

Michał Piszczek

I built this quant because the ready-made FP4 file answered the wrong question. It was fast, but on my short WikiText-2 control it scored 6.4949 PPL. Plain Q40 scored 6.3798. The first higher-quality hybrid went too far the other way: good perplexity, 34.19 tok/s, and no comfortable room for 256K plus vision. This is the build that survived both gates. It is a 17.1 GB, 5.01 BPW mixed-precision GGUF of Qwen/Qwen3.8-27B. It keeps large, tolerant matrices in native NVFP4 and spends more bits on selected attention, Gated DeltaNet, and late FFN tensors. The trained MTP layer remains embedded in the same GGUF. This is not a fine-tune. I built the private calibration workload from 5,472 messages…

Open weights apache-2.0
View model

Model · Image and text to text

Qwen3.8-27B-GGUF

GGML Org

Run it in the Llama app: https://llama.app - https://huggingface.co/Qwen/Qwen3.8-27B - https://huggingface.co/incoai/Qwen3.8-27B-DFlash2 - add info

Open weights apache-2.0
View model

lm mmap --lazy-mode on -ngl 99 -c 32768 \ mmproj-Qwen3.8-Flash-Next-BF16.gguf --local-dir. llama-mtmd-cli -m IQ3S/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3S-00001-of-00002.gguf \ --mmproj mmproj-Qwen3.8-Flash-Next-BF16.gguf -lm mmap --lazy-mode on \ -md mtp-Qwen3.8-Flash-Next-Q80.gguf --draft-max 4 -lm mmap --lazy-mode on -ngl 99 -lm mmap --lazy-mode on keeps the n-gram table memory-mapped on disk: it is 28.8 GB and one row is read per token. Shard 1 wants to be resident (VRAM or RAM); shard 2 is fine on an SSD. The vision projector is a standard clip GGUF shared with the upstream tiers, 0.91 GB. Native context is 262,144 tokens and the KV cache grows with it, so start at -c 32768 and work…

Open weights apache-2.0 gguf
View model