Qwen3.5-4B

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Qwen3.5-4B — LiteRT-LM

Qwen/Qwen3.5-4B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0). Same conversion rail as our Qwen3.5-0.8B — the GPU-delegable rank-≤4 gated-delta kernel — plus two 4B-specific fixes described in the conversion notes.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 4B model has 24 linear-attention + 8 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 8 attention layers keep KV (4096-token budget here). The upstream checkpoint is multimodal; this package is the text decoder only (vision tower and MTP heads dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical. 2026-10-09: metadata updated (chat template with a thinking switch, reasoning channel, supports_thinking); TFLite, tokenizer and executor metadata byte-identical.

The Mixed INT4 variant exists for RAM-constrained devices (requested in LiteRT-LM #1658 for 8 GB Android phones): 2.57 GB of weights instead of 4.10 GB, measured ~1.7 GB CPU peak on the 138-token probe. If you have the memory for int8, int8 answers harder prompts slightly better (measured below).

Correctness

All gates below were run on this exact file (or its float parent, same graph rail):

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac).
• iPhone 17 Pro (Metal GPU): the composite 8-question probe (all 8 questions in one 138-token prompt) answers 8/8, word-for-word identical to the HF fp32 reference run greedy with the same template.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.
• One Engine per conversation. On the Mixed INT4 file, a new Conversation opened on an Engine that already ran another conversation replied in 9 of 9 turns with nothing but the opening word its own system prompt asked for, against that word plus the answer in 9 of 9 on a fresh Engine (3 runs, litert-lm 0.18.0, Mac CPU). Create a new Engine for each conversation — the runtime reuses the prompt tokens a new conversation shares with the previous one and does not reset the recurrent state (my open report); a process that runs one conversation is unaffected.
• CPU digit test. "What is 17 times 23?" at the default rendering (thinking off), greedy, 3 fresh CPU processes per file: the int8 file answered 391 in 3 of 3, though one run repeated its check line after the answer up to the 400-token cap and another restated it three times; the Mixed INT4 file answered 391 in 3 of 3 with no repetition (litert-lm 0.18.0, Mac CPU). The check is here because the sibling Qwen3.5-2B int8 file looped on this question on CPU until its 2026-10-11 re-quantization (details on its card).

Mixed INT4 correctness (all gates re-run on Qwen3.5-4B_mixed_int4.litertlm):

• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac). Prompt-length robustness: 40/40 CPU, 20/20 GPU.
• iPhone 17 Pro: the composite 8-question probe answers 8/8 on both Metal GPU and CPU (the int8 file answers 6 of 8 on CPU on this probe).
• GSM8K, n=100 (0-shot CoT, greedy, 2048-token budget, identical harness for both files, measured on this exact export): int8 97/100, Mixed INT4 93/100. That 4-point gap is the real cost of int4 on this model — invisible to the sanity gates, visible on math word problems. We also measured blockwise-128 (90/100) and OCTAV variants (90–92/100); blockwise-32 min-max won and is what ships.

Usage

litert-lm run ./Qwen3.5-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-4B_int8.litertlm --backend gpu --cache no --prompt "..."

Prefill signatures: 1024, 256, 64, 16, 4, 1 (not the 0.8B's full 1–1024 ladder). Every exported signature is charged engine memory even if never called, and with this model's 248k vocabulary the full ladder's Metal programs push a 12 GB iPhone past its jetsam ceiling during engine creation. The reduced ladder keeps GPU peak at ~4.9 GB on iPhone 17 Pro with no correctness cost (the runtime just plans slightly coarser chunks).

Chat template note: both files carry the LiteRT-LM models/qwen3_5/chat_template.jinja at 87ab9334 with two edits: its format_content macro gains the tool_response branch of the current main version (it reads chat content as a string or as parts, text and tool_response), and enable_thinking defaults to false instead of true. With nothing set, the generation prompt opens each new assistant turn with an empty <think>\n\n</think> block (past assistant turns render without it), and a single-turn prompt (one user message, with or without a system prompt) renders byte for byte as before. enable_thinking=true pre-fills <think>\n; the reasoning is returned on the thought channel (<think>\n / \n</think>), and the text carries the answer only. Tested with litert-lm 0.18.0 on a Mac (CPU): with thinking on, both files return the reasoning on the channel and give 391 for 17 × 23. A two-turn conversation ("My name is Kenji." / "What is my name?") gets the name back with thinking unset and on; with thinking on, the int8 file's second reply greets the user by name ("Hello Kenji! It's great to meet you. …") instead of stating it. On 0.17.0 both files answer a plain prompt and a two-turn conversation. With thinking on, GPU and phones were not run; the updated mixed_int4 file answered a plain prompt on a Galaxy S26 (CPU, litert_lm_main built from the v0.18.0 tag); the card's GPU and iPhone results come from the previous files, whose TFLite sections and default prompt are byte-identical. The template renders tool definitions, but the files' llm_model_type is generic_model, so the runtime does not parse the model's tool calls: they come back as text, and supports_function_calling is false.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

On device (cold start, unplugged, thermal-nominal start, single runs, 138-token prompt, quality-gate harness):

(iPhone GPU: model load 66 s on first launch.)

Mixed INT4 (litert-lm benchmark 0.16.0, Apple M4 Max, same flags; iPhone rows are single cold runs of the 138-token quality probe, unplugged):

(iPhone GPU: model load 65 s on first launch. Pixel 8a: litert_lm_main v0.16.0, 260-token prompt / 204 decoded tokens, warm weight cache; peak is process VmHWM; engine init 30–50 s; TTFT scales with prompt length — 4.2 s on a 24-token prompt.)

Honest notes: - Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). We have not verified GPU execution on Qualcomm Adreno devices — if you are on a Snapdragon phone, use the CPU backend unless you have confirmed the GPU path on your own device. - GPU inference runs with fp32 activations (declared in the bundle; the fp16-activation formulation is unfinished for this family). That is where the GPU memory multiple over CPU comes from. - Pixel 8a (and 8 GB-class Android phones): the int8 file is not attempted, and not expected to fit. The 0.8B's fp32-expanded GPU programs already exceeded this phone's ~3.8 GB available memory at 963 MB on disk; the int8 file is 4.3× larger, and its 4.1 GB of int8 weights alone leave no CPU headroom on an 8 GB phone either. Treat this release as Apple-hardware-first; an Android row can follow on 12 GB+ devices. - The Mixed INT4 file runs on a Pixel 8a (8 GB) on CPU — measured, table above. Peak process memory 4.43 GB on a 7.75 GB-RAM phone, coherent long-form generation and correct QA output. Use the CPU backend on 8 GB phones: the GPU path expands to ~5.7 GB on Metal and Android GPU budgets are similar or tighter (the int8 2B already killed a Pixel 8a during GPU engine creation — do not try GPU on 8 GB devices). First run writes a 2.66 GB XNNPACK weight cache next to the model (one-time) — leave ~3 GB of storage free. - Mixed INT4 CPU prefill is slower than int8's (Mac: 100 vs 243 tok/s prefill, TTFT 2.6 vs 1.1 s at 256 tokens; iPhone CPU is the exception — 46.7 vs 24.0, faster). Decode is equal or faster everywhere. - On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT). - On quality: individual questions gate 8/8 on every backend. On the harder composite probe (8 questions in one prompt), GPU matches HF fp32 word-for-word; the CPU int8 path answers 6 of the 8 (every answer it gives is correct — it skips two). If you need maximum fidelity on complex prompts on CPU, ask for a float variant.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (shared with the 0.8B): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).
• Rank-4 head interleave (4B-specific): the 4B is the first Qwen3.5 with grouped value heads (32 value vs 16 key heads). Upstream expands q/k with repeat_interleave, whose rank-5 lowering the GPU delegate rejects outright — the 0.8B (ratio 1) never traces that branch. The patch re-expresses it as concat + rank-4 reshapes, bitwise-identical to the stock op.
• Reduced prefill ladder (4B-specific): see the signature note above.
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• Mixed INT4 recipe: post-hoc ai-edge-quantizer on the same float export — int4 BLOCKWISE-32 min-max on every FULLY_CONNECTED, overridden to int8 channelwise on the lm_head and the embedding lookup (one shared 248320×2560 vocab buffer, kept single). Block size and algorithm were chosen by measurement: b32/b128 × min-max/OCTAV all built from the same export and scored on GSM8K n=100 (93 / 90 / 92 / 90) — blockwise-32 min-max won. Convs and the delta rule stay float, prefer_activation_type fp32 as in the int8 file.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.