gpt-oss-20b-blackhole-fast

Derived from openai/gpt-oss-20b. Weights revision 6cee5e81. This represents the model implementation on Tenstorrent hardware. See the original model card for license, training, and evaluation details.

Runs on p150 (one Blackhole P150 chip), p150x2 (the two P150 ASICs of one P300C board) and p150x4 (the four P150 ASICs of two P300C boards, a QuietBox 2).

Same model, same serving stack and the same six profiles as tt-hous/gpt-oss-20b-p150, with faster decode: a lone request on P150x4, P150x2, P150 decodes through fused, DRAM-streaming decode ops (from tenstorrent/tt-metal#60161, adapted to this runtime and extended to one and two chips). On P150x4, 2 to 4 decoding requests share one pass of those ops, and 5 to 32 run the base package's batched decode with the routed experts streamed from the fused path's weight copies. Prefill takes the base package's code path with faster expert dispatch (same numerics). Prefix caching, tool calling, harmony reasoning and every other request run the base package's own code paths.

Status: Measured on this image: the device steps, serving tables, task scores and context-limit checks below come from one QuietBox 2 and the same harness as the served-package numbers they are compared with; the teacher-forced accuracy table comes from development builds of the same decode path.

Quickstart

uv tool install tenstorrent   # once: the Tenstorrent CLI, `tt`
tt model pull jashansinghTT/gpt-oss-20b-blackhole-fast
tt serve jashansinghTT/gpt-oss-20b-blackhole-fast -- --profile p150x4-throughput

The first start compiles kernels and converts the weights (several minutes); the server is ready when it logs Application startup complete. Then send requests with any OpenAI client; model is the weights id:

curl -s http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "openai/gpt-oss-20b",
  "messages": [{"role": "user", "content": "Summarize the attached report in five bullets."}],
  "max_tokens": 2048, "reasoning_effort": "medium", "stream": true}'

Which profile

Every profile is configured for the full 131,072-token context (checked on every profile: a 131,071-token prompt is admitted, a 131,073-token prompt gets HTTP 400), runs up to 32 concurrent requests and caches prefixes. The fused decode is used whenever exactly one request is decoding (on P150x4 also 2 to 4), on every profile of a fused mesh except p150-throughput, p150x2-throughput: p150-throughput keeps the base package's decode (its batched decode reads the base package's expert layout at full speed; one chip cannot hold both layouts), so a single user on that hardware gets the fused speed-up with p150. p150x2-throughput keeps the base package's decode (building the TP=2 fused path slows its batched decode by about 3 % at 8 users), so a single user on that hardware gets the fused speed-up with p150x2.

Serve profiles

One image serves every profile below; pick one with --profile.

KV cache: the pool of the 12 full-attention layers in 64-token blocks, as each profile's server allocated it at boot (vLLM's GPU KV cache size): it holds 2.03 (P150), 4.03 (P150x2), 8.06 (P150x4) full 131,072-token requests. The 12 sliding-window layers keep a short per-sequence ring on the device outside this pool. The base package allocates the same pool on this hardware: the fused decode did not reduce it.

The p150x2 profiles use devices 0 and 1, which must be the two ASICs of the same P300C board. The p150x4 profiles use devices 0 to 3, the four ASICs of two P300C boards connected as one ring.

Using it

The server exposes the OpenAI API at http://127.0.0.1:<port>/v1 (chat completions, completions and /v1/models). Pass "model": "openai/gpt-oss-20b". The reasoning comes back separately from the answer in message.reasoning; tool calling is enabled (--tool-call-parser openai). max_tokens covers reasoning and answer, with the base package's answer reserve (VLLM_HARMONY_FINAL_RESERVE_TOKENS).

Expected performance

Measured on a QuietBox 2 (two P300C boards, firmware 19.15.0.0, TT-KMD 2.10.0) with the base package's own latency ladder and bench client (random-token prompts of exactly ISL tokens to /v1/completions, streamed, output pinned to 128 tokens, greedy, Users concurrent requests, N = 4 per user, one warmup; medians; the final rows are the median of 3 passes, the first with the baseline's seeds). "Before" is tt-hous/gpt-oss-20b-p150 measured the same way on the same box; the device step is a traced batch-1 decode replay (mean over 256 tokens after warmup, context 128).

tt-lab is measured with its README protocol (device command time per token at positions 0-1; tt-lab streams BF16 attention and head weights, so it reads more bytes per token). PR #60161 runs on 4 chips only and streams BFP4 experts.

Changes since the previous release

This revision against the previous release of this package (tt-model/gpt-oss-20b-fast:e5e84232a967), measured with the same harness on the same box. Listed: device steps that moved by 2 % or more, and every ladder row whose decode (1 user), total output (2+ users) or TTFT moved by 5 % or more; every other row is within 5 % of the previous release.

P150x4 (p150x4-throughput)

P150x4 (p150x4)

P150x2 (p150x2-throughput) — base-package decode (see Which profile)

P150x2 (p150x2)

"Before" here is the served package's p150x2-throughput profile (the baseline has no p150x2 rows); at c=1 with prompts up to 8,192 tokens the two profiles schedule a request the same way.

P150 (p150-throughput) — base-package decode (see Which profile)

P150 (p150)

"Before" here is the served package's p150-throughput profile (the baseline has no p150 rows); at c=1 with prompts up to 8,192 tokens the two profiles schedule a request the same way.

Bytes and bandwidth

Weight bytes each decode token must read (from the weight formats and shapes, batch 1), the measured device step, and the bandwidth that implies per chip. Speed from fewer bytes is not the same as a more efficient engine: compare rows with the same formats (BFP8 experts) for engine efficiency.

Accuracy

Teacher-forced next-token agreement with the Hugging Face reference in fp32 on the CPU (4 prompts × 128 positions, the same harness for every variant): top-1 = the TT argmax equals the reference token, top-5 = it is in the reference top 5, PCC over the full vocabulary.

Task scores through the OpenAI chat API with lm-evaluation-harness (the base package's protocol: reasoning_effort medium, greedy, 4,096-token budget, seed 42, 32 concurrent requests unless noted):

Subset scores move by a few points between runs of the same model. The model's chat template puts the current date in every system prompt, so runs on different days send different prompts, and with 32 concurrent requests a request's greedy output also depends on how many requests decode with it (each batch size rounds differently). The P150 and P150x2 throughput profiles run the same model as the previous release (logits identical per batch size); their subset answers still differ from its runs, which were taken on another day. One subset prompt is 0.4 points of GSM8K and 1 point of IFEval.

Limitations

• The base package's limitations apply unchanged (hardware, KV capacity, output budget, host sampling for min_p / logit_bias / top_k above 32, no built-in tools).
• On P150 and P150x2 the fused decode serves one decoding request at a time; with 2 or more requests decoding, the base package's batched decode runs and per-user speed is the base package's. On P150x4 up to 4 decoding requests share the fused ops; 5 to 32 run the batched decode (with the routed experts streamed from the fused path's copies).
• Decode-only weight copies for the fused path take extra device DRAM (estimates from the weight formats: about 5.9 GB per chip on P150x4 for BFP8 experts, attention, router and LM head; about 1.3 GB on P150, where the routed experts stream from the base package's own tensors; on P150x2 the p150x2 profile's copies take about 11.9 GB per chip, the measured size of its decode-only weight cache). The KV pool is the base package's; it was not reduced (see Serve profiles).
• Requests that need the base package's batch-1 sampler (penalties, log-probs, top-k above 32, logit bias, grammars) run the base package's batch-1 decode instead of the fused one.
• On P150x2 only p150x2 uses the fused decode; p150x2-throughput keeps the base package's decode.
• p150x2 builds the TP=2 fused decode next to the base package's batched decode, which then runs slightly slower (8 users, ISL 128: 286 vs 296 tok/s total on p150x2-throughput). Use p150x2-throughput for concurrent traffic on that hardware.
• p150 keeps the expert layout the fused decode streams; the base package's batched decode (2-32 requests) reads that layout more slowly (32 users, ISL 128: 15.2 vs 15.9 tok/s/u on p150-throughput). Use p150-throughput for concurrent traffic on that hardware.

Credits

• Jerry Wang — the fused batch-1 decode for gpt-oss on Blackhole, tenstorrent/tt-metal#60161 (DRAM-streaming expert and linear kernels, fused all-reduce + residual + RMSNorm boundaries, LM head with on-device top-32 sampling). This package builds on it.
• Adam Housman — the tt-hous/gpt-oss-20b-p150 package this one extends: model code, vLLM integration, the six serve profiles, the benchmark and evaluation harness used for every number here.
• Matt Craighead — tenstorrent/tt-lab, whose gpt-oss runtime set the speed bar and showed what a DRAM-channel-local, few-sync decode can reach on this hardware.

Provenance

code/ in this repo is byte-identical to the model code inside the image.