中文 · Hugging Face · ModelScope
A quantized version of Qwen/Qwen3.5-4B for Jetson AGX Orin 64GB / CUDA SM87, deployed with Orinfer v0.2.0. It supports text, images and multiple images, streaming Chat Completions, native MTP, prefix caching, continuous batching and resumable JSONL offline inference. Translation requests use enable_thinking: false; thinking follows the original model's chat template.
Quantization and files
Two-stage residual E8P Q4 with coefficients [4, 1], signed-block128 rotation and beam-8 encoding over the original BF16 weights, paired with INT8 computation. Sensitive tensors retain the validated W8/FP16/BF16/FP32 mixed-precision policy.
Deployed read-only GPU data, including scales, derived constants and precision exceptions: 2,608,622,592 bytes, approximately 4.478 bits per recorded GPU weight parameter. Stored safetensors: 2.61 GB across 4 shards. Context capacity: 262,144 tokens; KV cache is INT8 and persistent GDN state is FP32. Quantization seed: 20261002.
The repository contains HF configuration/tokenizer assets, the native chat template, standard safetensors containers and cache/model.json. The descriptor fixes physical tensor layouts and the execution-package digest. The matching Rust .so, cubins and their source are distributed through GitHub Release.
Deploy
Download this model repository into ./model. Prebuilt deployment uses the aarch64 Rust CLI, NVIDIA CUDA Driver and the matching SM87 execution package.
Execution-package digest: 670ccbc06ee203a128969cd56da544b9b621009197e1a0f960511449ed8e5e79.
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/orinfer-v0.2.0-linux-aarch64.tar.xz
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/orinfer-v0.2.0-qwen3_5-int8-quality-sm87-670ccbc06ee2-execution.tar.xz
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
tar -xf orinfer-v0.2.0-linux-aarch64.tar.xz
./orinfer-v0.2.0-linux-aarch64/bin/orinfer package install ./orinfer-v0.2.0-qwen3_5-int8-quality-sm87-670ccbc06ee2-execution.tar.xz
./orinfer-v0.2.0-linux-aarch64/bin/orinfer validate-model ./model
./orinfer-v0.2.0-linux-aarch64/bin/orinfer serve ./model \
--model qwen3.5-4b --cuda-graph decode_only --mtp-drafts 3
The listener is [::]:8088. OpenAI Chat Completions uses /v1/chat/completions; set stream: true for SSE. Images use image_url, and thinking output uses reasoning_content. The CLI also provides Ollama-compatible APIs and orinfer batch for JSONL jobs. See the serving guide.
Measured performance
Jetson AGX Orin 64GB, 2026-10-21. Throughput is tok/s; startup is seconds.
| Prefill 512 |
Prefill 8K |
Single decode, MTP |
8-request decode |
Max requests / decode |
Serve → API (s) |
Serve → first output (s) |
| 3592.58 |
3416.98 |
90.71 |
360.49 |
128 / 1071.29 |
1.68 |
1.83 |
Single-request runs enable adaptive MTP up to 3 drafts, with 512/8192 input tokens and 257 fixed output tokens; the single-decode column uses 512-token input. Multi-request runs automatically switch the benchmark to MTP off, with 256 input and 65 output tokens per request. Decode timing starts after the first emitted step, and multi-request timing uses a first-output barrier. Each throughput value is a median of three warm runs after one warmup; fixed-length diagnostics ignore EOS. All use greedy, seed 20261002, decode_only, thinking off and prefix cache off. MTP speed depends on the task and acceptance rate.
Startup uses three fresh Rust serve processes and a 21-token translation request: process invocation to /health ready, then to the first nonempty SSE output. Existing file caches are preserved; SSE output can contain several tokens. The table reports medians, and file-cache effects plus individual large-model startup observations are in the performance reference.
Quality scope
Quantization-baseline diagnostics recorded on 2026-10-19: fixed-history comparison against original BF16, seed 20261002; mean target NLL delta in nats/token:
| Sample |
Positions |
Mean NLL delta |
| Text |
118 |
-0.01596 |
| Long input through 8K |
26 |
+0.16662 |
| Simple images |
17 |
-0.02180 |
These are small-sample diagnostics covering text, inputs through 8K and controlled simple images. Broader task benchmarks, complex image translation and full-context quality remain evaluation areas. Quality and execution-regression scopes are recorded separately in the performance reference.
License
Model weights retain Apache-2.0. Orinfer software and its separate execution package use LGPL-3.0-or-later; source and third-party notices ship with the release.
