Qwen3.5-2B, auto-optimized by Fable
This is Qwen3.5-2B auto-optimized by Claude Fable for fast single-request text generation. Fable built and tuned the included qwen35_fast inference engine while keeping Qwen's original BF16 weights unchanged.
| Single H100, batch 1, 256 output tokens |
Decode tokens/s |
| Transformers eager |
50–51 |
| vLLM 0.29 |
429–437 |
| vLLM 0.29 + MTP |
527–915 |
| Fable-optimized engine |
582–848 |
Across 12 development workloads, the Fable engine delivered 14× the decode speed of Transformers eager and 1.02× the speed of vLLM with MTP (geometric means). On 12 held-out workloads, it reached 528–866 tokens/s and 1.01× vLLM with MTP.
Setup
Use Python 3.12 and an NVIDIA CUDA GPU. Download the model and install its dependencies:
pip install huggingface_hub
hf download islamassanov/Qwen3.5-2B-auto-optimized --local-dir qwen35-fable
pip install -r qwen35-fable/requirements.txt
Run the optimized engine:
import sys
from transformers import AutoTokenizer
path = "qwen35-fable"
sys.path.insert(0, path)
from qwen35_fast import Engine
tokenizer = AutoTokenizer.from_pretrained(path)
prompt = [{"role": "user", "content": "Explain how a CPU cache hierarchy works."}]
input_ids = tokenizer.apply_chat_template(
prompt, tokenize=True, add_generation_prompt=True, enable_thinking=False
)["input_ids"]
engine = Engine(path, spec_k=2, compile_blocks=True, fused_gdn=True)
output = engine.generate(
input_ids,
n_out=512,
eos_ids={tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|im_end|>")},
)
print(tokenizer.decode(output["tokens"], skip_special_tokens=True))
The original checkpoint also works with Transformers for Qwen's standard text and vision-language workflows; the speed figures above use qwen35_fast.
What Fable changed
- Captured the decode step in a CUDA graph to reduce launch overhead.
- Fused decode operations with
torch.compile and Triton Gated-DeltaNet kernels.
- Used the checkpoint's MTP head to propose two tokens per step, accepting each only when the target model makes the same greedy choice.
The optimized path runs batch-1, greedy, non-thinking text generation. The weights and tokenizer come unchanged from Qwen revision 15852e8c16360a2fea060d615a32b45270f8a8fc.
Benchmarks
Results are median decode tokens/s over five runs on one H100 SXM, with 256 generated tokens per prompt. Prompts cover prose, code, and structured output at 128, 512, 2,048, and 8,192 input tokens. Engines ran sequentially in the same session; decode timing starts after the first token.
| Development prompts |
Transformers eager |
vLLM + MTP |
Fable engine |
| Prose, 128–8,192 input tokens |
50–51 |
528–576 |
582–618 |
| Code, 128–8,192 input tokens |
50–51 |
637–736 |
661–738 |
| Structured, 128–8,192 input tokens |
50–51 |
824–915 |
807–848 |
Full per-prompt results, held-out measurements, and the IFEval regression are in RESULTS.md. The project repository contains the benchmark code and development history.
