中文 · Hugging Face · ModelScope
A quantized version of IndexTeam/Index-Translate-9B for Jetson AGX Orin 64GB / CUDA SM87, deployed with Orinfer v0.2.0. It supports text, images and multiple images, streaming Chat Completions, native MTP, prefix caching, continuous batching and resumable JSONL offline inference. Translation requests use enable_thinking: false; thinking follows the original model's chat template.
Quantization and files
Two-stage residual E8P Q4 with coefficients [4, 1], signed-block128 rotation and beam-8 encoding over the original BF16 weights, paired with INT8 computation. Sensitive tensors retain the validated W8/FP16/BF16/FP32 mixed-precision policy.
Deployed read-only GPU data, including scales, derived constants and precision exceptions: 5,473,766,976 bytes, approximately 4.536 bits per recorded GPU weight parameter. Stored safetensors: 5.47 GB across 6 shards. Context capacity: 262,144 tokens; KV cache is INT8 and persistent GDN state is FP32. Quantization seed: 20261002.
The repository contains HF configuration/tokenizer assets, the native chat template, standard safetensors containers and cache/model.json. The descriptor fixes physical tensor layouts and the execution-package digest. The matching Rust .so, cubins and their source are distributed through GitHub Release.
Deploy
Download this model repository into ./model. Prebuilt deployment uses the aarch64 Rust CLI, NVIDIA CUDA Driver and the matching SM87 execution package.
Execution-package digest: 821345ac103f4aa09ba98d2a09bd8d489d560f1a5ea09f73b396e1f102431efa.
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/orinfer-v0.2.0-linux-aarch64.tar.xz
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/orinfer-v0.2.0-qwen3_5-int8-quality-sm87-821345ac103f-execution.tar.xz
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
tar -xf orinfer-v0.2.0-linux-aarch64.tar.xz
./orinfer-v0.2.0-linux-aarch64/bin/orinfer package install ./orinfer-v0.2.0-qwen3_5-int8-quality-sm87-821345ac103f-execution.tar.xz
./orinfer-v0.2.0-linux-aarch64/bin/orinfer validate-model ./model
./orinfer-v0.2.0-linux-aarch64/bin/orinfer serve ./model \
--model index-translate-9b --cuda-graph decode_only --mtp-drafts 3
The listener is [::]:8088. OpenAI Chat Completions uses /v1/chat/completions; set stream: true for SSE. Images use image_url, and thinking output uses reasoning_content. The CLI also provides Ollama-compatible APIs and orinfer batch for JSONL jobs. See the serving guide.
Measured performance
Jetson AGX Orin 64GB, 2026-10-21. Throughput is tok/s; startup is seconds.
| Prefill 512 |
Prefill 8K |
Single decode, MTP |
8-request decode |
Max requests / decode |
Serve → API (s) |
Serve → first output (s) |
| 2213.52 |
2223.34 |
56.49 |
233.87 |
128 / 890.74 |
2.92 |
3.10 |
Single-request runs enable adaptive MTP up to 3 drafts, with 512/8192 input tokens and 257 fixed output tokens; the single-decode column uses 512-token input. Multi-request runs automatically switch the benchmark to MTP off, with 256 input and 65 output tokens per request. Decode timing starts after the first emitted step, and multi-request timing uses a first-output barrier. Each throughput value is a median of three warm runs after one warmup; fixed-length diagnostics ignore EOS. All use greedy, seed 20261002, decode_only, thinking off and prefix cache off. MTP speed depends on the task and acceptance rate.
Startup uses three fresh Rust serve processes and a 21-token translation request: process invocation to /health ready, then to the first nonempty SSE output. Existing file caches are preserved; SSE output can contain several tokens. The table reports medians, and file-cache effects plus individual large-model startup observations are in the performance reference.
Quality scope
Quantization-baseline diagnostics recorded on 2026-10-18: fixed-history comparison against original BF16, seed 20261002; mean target NLL delta in nats/token:
| Sample |
Positions |
Mean NLL delta |
| Text |
130 |
+0.02370 |
| Long input through 8K |
26 |
-0.07549 |
| Simple images |
17 |
+0.08327 |
The accepted optimized execution has a four-position, two-request comparison against the same quantized single-request reference: maximum absolute NLL delta 0.11037, mean delta 0.02494 and mean KL 0.00197; all selected tokens are in the reference top3. The numerical guard is 0.1, and the remaining batch/cancellation checks from that guarded run remain a separate acceptance item.
These are small-sample diagnostics covering text, inputs through 8K and controlled simple images. Broader task benchmarks, complex image translation and full-context quality remain evaluation areas. Quality and execution-regression scopes are recorded separately in the performance reference.
License
Model weights retain Apache-2.0. Orinfer software and its separate execution package use LGPL-3.0-or-later; source and third-party notices ship with the release.
