motif-3-bh-galaxy

This package runs the Motif-3 LLM of Motif Technologies on one Tenstorrent Blackhole Galaxy (32 Blackhole chips).

• Model: a 314B-parameter MoE with 13.2B active parameters per token.
• 53 layers of Grouped Differential Latent Attention (GDLA).
• A 4-stream mHC residual.
• 384 routed experts and 1 shared expert, all PolyNorm MLPs, with top-8 routing.
• 1 MTP (multi-token-prediction) layer.
• Supports the following configurations:
• Concurrent users (batch): 1 to 32.
• Context: up to 32,768 tokens per request, with one KV pool of 262,144 tokens shared by all users.
• Two launches:
  • default: for all traffic.
  • mtp (opt-in): greedy requests decode up to 1.9× faster with the model's own MTP layer.
• An OpenAI-compatible API through vLLM: chat, completions, streaming, thinking on/off, tool calls.
• Built on tt-metal / ttnn as a plain vLLM model behind the vllm-tt-plugin.
• Mesh (4, 8): attention is 8-way tensor-parallel in 4 groups of 8 user lanes; the MoE is 32-way expert-parallel (12 experts per chip).
• Latent KV cache: 576 values per token and layer, in BFP8.
• Decode runs as captured Metal traces. The MTP launch captures two: the 32-lane trace and a 64-row verify trace.
• The sampler runs on the device and is exact (temperature, top-k, top-p, seeds).
• Two custom kernels through ttnn.generic_op: the exact Motif Sinkhorn of the mHC and an exact FP32 router.
• With the default launch, one user decodes at 10.3-10.8 tokens/s (92 ms per decode step). 32 users get 326 tokens/s together on 128 / 1,024 tokens.
• With the MTP launch, one user decodes at 20.7 tokens/s and 32 users get 458 tokens/s (greedy).

Weights: Motif-Technologies/Motif-3 · Port: changh95/tt-metal @ motif3-bh-galaxy · Plugin: changh95/vllm-tt-plugin @ motif3-bh-galaxy · Results: reports/

• This package runs on p150x32 (mesh BH-Galaxy, opened as (4, 8), fabric FABRIC_2D_TORUS_XY). It has two launches:
• default: chunked prefill, prefix caching, packed prefill and exact on-device sampling.
• mtp (opt-in): default plus MTP self-speculative decoding (K = 1) with the 64-row verify. Only greedy requests speculate.
• The package uses tt-model-manager 0.1.0 (manifest schema 5: a self-contained bundle with the ttnn, vLLM and vllm-tt-plugin wheels, all Python dependencies and the model code). No container image and no Docker are needed.

Quickstart

export TT_MODEL_MODELS_DIR=/big/disk/tt-model   # weights (≈630 GB) + venv + ≈344 GB device weight cache + kernel cache
tt-model pull  changh95/motif-3-bh-galaxy --with-weights
tt-model serve changh95/motif-3-bh-galaxy

• tt-model pull installs the bundle into its own Python 3.10 venv from the vendored wheels. It downloads the weights Motif-Technologies/Motif-3 @ 2ed2ed5c (≈630 GB BF16) into the bundle's HF cache. This repository does not contain the weights.
• If you already have the weights, skip the download: tt-model pull changh95/motif-3-bh-galaxy, then WEIGHTS_DIR=/path/to/Motif-3 tt-model serve changh95/motif-3-bh-galaxy. The server runs offline (HF_HUB_OFFLINE=1).
• The server uses port 20000. If that port is busy, the server uses the next free port. The model id is Motif-Technologies/Motif-3. The server is ready when it logs Application startup complete.
• First start: the server converts the BF16 weights and writes the device weight cache (TT_CACHE_PATH, default <install>/.tt_cache, ≈344 GB). It also compiles the kernels. This is an estimate of about 80 minutes:
• about 29 min to convert the 51 MoE layers (34 s per layer, measured);
• about 49 min for the kernel compiles (measured: 2,935 s to healthy with the weight cache in place and an empty kernel cache);
• more if the 630 GB must first be read from a slow disk.
• Later starts load both caches: 93 s from tt-model serve to healthy (measured through this bundle).
• Host requirements:
• A Blackhole Galaxy with its driver and firmware (measured on TT-KMD 2.10.0, firmware bundle 19.12.0.0) and 1G hugepages.
• An x86-64-v3 CPU and glibc ≥ 2.35.
• The SFPI 7.84.0 kernel compiler, at /opt/tenstorrent/sfpi or installed into the bundle's venv without sudo: bash "${TT_MODEL_MODELS_DIR:-$HOME/.cache/tt-model/models}/changh95/motif-3-bh-galaxy/sfpi_install.sh".
• tt-model-manager at or after c3cfaf8. Run tt-model with tt-smi off your PATH (see Limitations).

Run with tt-cli

tt serve in tt-cli 1.0.1 needs Docker, even for a self-contained bundle. So start the server with tt-model serve (above). tt-cli works for the other steps:

tt model info changh95/motif-3-bh-galaxy --hw p150x32     # the catalog entry; tt-cli does not detect a Galaxy, so pass --hw
tt-model serve changh95/motif-3-bh-galaxy &               # the server (no Docker)
curl -s localhost:20000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Motif-Technologies/Motif-3",
  "messages": [{"role": "user", "content": "Explain KV caching in two sentences."}],
  "max_tokens": 2048, "temperature": 1.0, "top_p": 0.95}'
kill -TERM %1                                             # stop: SIGTERM, then wait (the server closes the 32-chip mesh)

The server is vLLM's OpenAI-compatible server:

• POST /v1/chat/completions and POST /v1/completions, with "stream": true as in vLLM.
• Thinking is on by default. Turn it off for one request with "chat_template_kwargs": {"enable_thinking": false}.
• The motif reasoning parser puts the thinking in the reasoning field and the answer in content.
• Tool calls: pass tools; the motif tool parser (Hermes-style <tool_call> JSON, with repair of malformed arguments) returns tool_calls. Auto tool choice is on.
• Sampling: temperature, top_p, top_k and seed run on the device. Penalties, min_p, structured outputs and top-N logprobs use vLLM's exact host sampler (default launch only; see Limitations).
• GET /health, GET /v1/models and GET /metrics (vLLM's metrics, including the MTP acceptance counters) work as in vLLM.

How to change the configuration

• The launch is fixed when the server starts. To change it, stop the server and start it again with other values.
• tt-model serve passes every argument after the bundle id to vLLM, and runs the server in your environment. So you change the configuration with environment variables and vLLM flags; no file of the bundle changes.
• The configuration of a request (sampling, thinking, tools, max_tokens) can change for each request.

Where in the code (paths in metal/models/demos/motif3/):

To serve the MTP launch:

MOTIF3_SPEC_VERIFY=auto tt-model serve changh95/motif-3-bh-galaxy --speculative-config '{"method": "custom_class", "model": "vllm_tt_plugin.model_owned_drafter", "num_speculative_tokens": 1}'

To serve without prefix caching (for benchmarks that repeat identical prompts; the benchmark tables below were measured this way):

tt-model serve changh95/motif-3-bh-galaxy --no-enable-prefix-caching

To serve with per-row prefill (a request's tokens then do not depend on the other requests; see Limitations):

MOTIF3_PACKED_PREFILL=0 tt-model serve changh95/motif-3-bh-galaxy

Response

{"id": "chatcmpl-...", "object": "chat.completion", "model": "Motif-Technologies/Motif-3",
 "choices": [{"index": 0, "finish_reason": "stop",
              "message": {"role": "assistant", "reasoning": "<the thinking>", "content": "<the answer>",
                          "tool_calls": []}}],
 "usage": {"prompt_tokens": ..., "completion_tokens": ..., "total_tokens": ...}}

• reasoning holds the thinking and content holds the answer (the motif reasoning parser). With thinking off, reasoning is empty.
• The model stops on its end-of-turn token (EOS ids 0, 3 and 6 of generation_config.json).

Implementation implications

• Packed prefill. The short prompts of one prefill step run as one pass (up to 8,192 tokens), not one prompt after another. So a burst of 32 short prompts gets its first tokens after 2.4 s, not after 23 s. From the first MoE layer on, a row in a larger pass is computed slightly differently, so a request's tokens can depend on the other requests of its prefill step (see Limitations).
• Chunked prefill. A long prompt is prefilled in chunks of 8,064 tokens, with one decode step after each chunk. So a long prompt delays the running requests by one chunk, not by its whole prefill.
• Prefix caching. A prompt that starts with a computed prefix (a system prompt, an earlier turn) reuses its KV blocks, including blocks written during decode. To make this possible, decode writes each new KV row to all 4 lane groups (about 1.7 ms per step).
• On-device sampling. Each chip keeps its top-64 candidates. The device computes the exact softmax normalizer over the full vocabulary (220,160 tokens), the top-p prefix sums in FP32, and a seeded draw. If the device cannot certify a lane exact, the host samples that lane again (about 0.3 % of the steps at temperature 1.0 / top-p 0.95).
• MTP. The MTP layer drafts one token in the decode trace, and the next step verifies it.
• Below about 19 live users, the drafts use idle lanes of the 32-lane step.
• From 19 users on, every user drafts, and a 64-row step verifies 32 anchors and 32 drafts together. The 64-row step costs 1.12-1.18× a 32-lane step.
• The 64-row rows are bit-identical to the 32-lane rows, so greedy outputs are the same tokens as without MTP.
• Determinism. Prefill and decode are bitwise reproducible from run to run on the same fabric. The ttnn ring all-gather has a completion race on an even ring; the model routes the affected gathers through all_broadcast + concat (ring gather safe, about 0.3-0.5 ms per step).
• Traces. Decode runs as Metal trace replays captured at boot. Prefill runs eagerly in power-of-two buckets; the packed shapes are warmed at boot (+2.2 s).

Accuracy

tt-inference-server --workflow evals against the default launch (prefix caching on; lm-eval generation tasks; temperature 1.0, top-p 0.95, sampled on the device; outputs capped by the 32K context), measured 2026-10-04 on --limit-samples-mode ci-nightly subsets:

• One sampled run each on subsets, so the 95 % intervals are wide (Wilson: MATH-500 91.5-99.0, AIME 2024 70.2-98.8, GPQA Diamond 70.9-92.9).
• 2 of the 3 MATH-500 misses are grader format mismatches of correct answers (2516 vs 2516_8; 1±√19 written as two roots). The third MATH-500 miss and the AIME miss reached the output cap.
• All 155 requests completed: 0 failed, 0 preempted.

Benchmarks

• The tables are tt-inference-server run.py --workflow benchmarks (vLLM bench serve: random prompts of exactly ISL tokens, ignore_eos, temperature 0, streaming), measured 2026-10-04 on one Blackhole Galaxy, fabric FABRIC_2D_TORUS_XY.
• The server was tt-inference-server's dev spec: the same tt-metal commit, vllm-tt-plugin and serve flags as this bundle (the bundle adds the motif parsers).
• Prefix caching was off for every benchmark, because vllm bench repeats identical prompts and cache hits would flatter the time to first token. The rest is the shipped launch.
• t/s = output tokens per second of all users together; t/s/u = tokens per second per user (OSL / mean end-to-end latency); TTFT = mean time to first token; TPOT = mean time per output token after the first.
• [N users]: the KV pool seats fewer users at that length (tt-inference-server runs at most 262,144 // (ISL + OSL) users). A dash: not measured.
• The host had no other heavy workload during the benchmarks.

default launch, t/s (t/s/u)

default launch, TTFT ms / TPOT ms

mtp launch (greedy), t/s (t/s/u), × = t/s against default at the same point

mtp launch (greedy), TTFT ms / TPOT ms

• The MTP acceptance on these random-token prompts was 0.66-0.91. With real chat prompts, the MTP launch measured 1.88-1.92× at 1-16 users and 1.69× at 32 users (20.4 / 158.5 / 314.8 / 559.0 vs 10.6 / 84.1 / 167.0 / 330.7 tokens/s at 1 / 8 / 16 / 32 users, through vllm serve).
• This bundle itself, with the same vllm bench serve command (greedy 128 / 128): 10.25 t/s, TPOT 91.9 ms, TTFT 820 ms at 1 user, and 259.8 t/s, TPOT 94.8 ms, TTFT 3,724 ms at 32 users. tt-inference-server: 10.3 t/s, 92.1 ms, 783 ms and 259.4 t/s, 95.0 ms, 3,724 ms.
• reports/ has the tt-inference-server reports of these tables (JSON and markdown) and the project's result documents.

Serving features (through vllm serve of the same code, default launch unless noted)

tt-inference-server's prefix-cache benchmark (--prefix-cache, AIPerf) could not run: its AIPerf client pins transformers<5, and the Motif-3 tokenizer needs transformers 5. The prefix-caching row above is the project's own measurement.

Limitations

• MTP.
• Only greedy requests (temperature 0, no penalties) speculate. While a sampled or penalized request is running, no request speculates, so mixed traffic decodes at about the speed of default.
• With short outputs at 16-32 users, mtp is slower than default (0.84× at 16 users, 0.80× at 32 users on 128 / 128). The requests of a wave finish at different steps, so new requests arrive one at a time, and each one's prefill stalls every decode.
• At 8 and 16 users with long outputs, it drafts less often (7-15 % of the tokens came from steps without a draft, against 0.1-0.2 % at other loads), so the gain there is 1.32-1.45×, not about 1.8×.
• Long prompts at many users. Chunked prefill runs a decode step after each 8,064-token chunk. This caps the stall of the running requests (about 7 s instead of up to 47 s), but with 8K-16K prompts and 128-token outputs the total throughput is 6-14 % lower than the project's first build without chunking. A single 8K / 16K prompt's TTFT is 0.8 / 1.3 s longer.
• Packed prefill makes outputs depend on the other requests. A request's greedy tokens and seeded sampled tokens can depend on which requests share its prefill step. Every measured divergence was a near tie, and no judged answer changed. The same batch reproduces exactly. Start with MOTIF3_PACKED_PREFILL=0 if you need outputs that do not depend on other requests.
• Request options the server refuses (HTTP 400).
• Both launches: prompt_logprobs and prompt embeddings.
• mtp launch: logprobs, structured outputs (response_format JSON / JSON schema, and tool_choice "required" or a named function; "auto" works), bad_words, allowed_token_ids and min_tokens. vLLM itself refuses logit_bias, and min_p on sampled requests, on a speculative launch.
• Supported inputs. 32,768 tokens per request (the checkpoint supports 262,144) and one KV pool of 262,144 tokens (32 users × 8K or 8 users × 32K). Long reasoning traces at many users can be preempted and prefilled again.
• Whole Galaxy only. One engine uses all 32 chips; the mesh is fixed at (4, 8) (the device weight cache depends on the mesh shape).
• Disk. 630 GB of BF16 weights, ≈344 GB of device weight cache and the kernel cache. The server reads the BF16 weights only while a part of the cache is missing. Before it writes a part, it checks that 60 GB stay free.
• SFPI is not bundled. The kernels compile with the SFPI at <venv>/lib/python3.10/site-packages/ttnn/runtime/sfpi (where sfpi_install.sh puts it), else /opt/tenstorrent/sfpi. It must be version 7.84.0.
• tt-smi on PATH. tt-model-manager up to 7994e0a does not know the Galaxy board type tt-galaxy-bh and refuses this blackhole bundle (--force does not help). Run tt-model pull/serve with tt-smi off PATH, or shadow it: mkdir -p ~/.local/share/no-tt-smi && printf '#!/bin/sh\nexit 1\n' > ~/.local/share/no-tt-smi/tt-smi && chmod +x ~/.local/share/no-tt-smi/tt-smi, then PATH=~/.local/share/no-tt-smi:$PATH tt-model serve ….
• Working directory. Start tt-model serve from a directory without a models/ folder (not from inside a tt-metal checkout). python -m puts the working directory first on sys.path, so a local models/demos/motif3 would be served instead of the bundle's.
• tt-cli 1.0.1. tt serve needs Docker, and tt model list does not show this bundle (tt-cli reads the hardware only from container bundles); tt-model list shows it. tt model info works with --hw p150x32.
• Fabric. All numbers were measured with the FABRIC_2D_TORUS_XY fabric. On a Galaxy whose fabric comes up degraded (TORUS_Y), the model also runs; the project measured decode within about 1 % of TORUS_XY.
• No async scheduling. vLLM runs with --no-async-scheduling.
• Status: experimental (tt-inference-server EXPERIMENTAL).

TODO

• MTP drafting at 8-16 users
• With long outputs, 7-15 % of the tokens come from steps without a draft at 8 and 16 users. The earlier verify mode (packed) drafted on almost every step there (145.7 vs 122.8 t/s at 128 / 1,024 and 8 users).
• Plan: find which lanes the auto planner leaves without a draft below the crossover (c* = 19), and make it draft every idle lane as packed does.
• MTP with short outputs at many users
• Each new request's prefill stalls all decodes, and the MTP launch has more of them per second. Plan: pack the arriving prompts of several steps into one prefill pass, and measure the decode stall.
• Row-invariant MoE prefill
• A row in a packed pass is computed slightly differently from the same row alone, from the first MoE layer on. Fixed per-row program configs would make the outputs independent of the other requests again.
• Decode step
• The decode step is 1.0-2.7 ms longer than in the project's first build: the safe ring gathers (0.3-0.5 ms) and the sampler (1.36 ms). Plan: a fixed upstream ttnn all-gather would let the gathers use the native ring again.
• Longer context
• The checkpoint supports 262,144 tokens; this package serves 32,768. The prefill buckets and the KV pool size limit it.
• Prefix-cache benchmark
• tt-inference-server's AIPerf client needs transformers 5 for the Motif-3 tokenizer.

License

• Weights: Motif-Technologies/Motif-3, MIT License.
• This repository does not include the weights. tt-model downloads them into the bundle's HF cache.
• Port and server code (metal/): Apache-2.0 (tt-metal, ttnn, vllm-tt-plugin, vLLM).
• The motif tool and reasoning parsers in metal/models/demos/motif3/vllm_plugins/ are adapted from MotifTechnologies/vllm.
• The tt-inference-server patch in provenance/ is Apache-2.0.
• Built with Motif 3.

Provenance

• The next table shows the sources of the bundle.
• metal/models in this repository is byte-identical to the model code of the tt-metal commit below, and run.sh puts it on PYTHONPATH.