Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150

Qwen3.6-35B-A3B — TT-native mixed BFP4/BFP8 (GPTQ experts), single P150, native 262K context

A hardware-specific, text-only checkpoint and custom runtime derived from Qwen/Qwen3.6-35B-A3B, revision 995ad96eacd98c81ed38be0c5b274b04031597b0, for one Tenstorrent Blackhole P150.

This is a community experimental release by Lottolabs, not an official Qwen or Tenstorrent release. It is not GGUF, AWQ, an AutoGPTQ/GPTQModel int4 checkpoint, bitsandbytes, NVFP4, or an ordinary Transformers checkpoint. transformers.from_pretrained() and hosted Inference Providers are not supported launch paths. The vision encoder of the upstream model is not included: the server accepts text only.

What is in this release (x6-fused-20261007)

• Checkpoint (checkpoint/, 21.7 GB, manifest sha256 439096088c8b…):
• routed experts (256 per layer × 40 layers) and the shared expert: BFP4, quantized with a per-expert GPTQ written for Tenstorrent block floating point (128 calibration sequences × 2,048 tokens of chat, code, instructions and prose, disjoint from the eval set);
• every dense projection (Gated DeltaNet in/out, attention q/k/v/o) and the LM head: BFP8, round-to-nearest;
• router, norms, conv, embeddings and MTP head norms: BF16 (lossless).
• The 40 text layers and the MTP head are included; the vision encoder is not. Routed experts ship in the grouped layout the runtime loads (expert_groups/, 16 experts per file).
• Runtime (runtime-image.tar.gz, 5.5 GB): two Docker images that share one base:
• single (lottolabs/qwen36-35b-tt-p150:single-x6, sha256:faa77aad3f71…): one sequence, MTP-1 speculative decoding for greedy and sampled requests (or --no-mtp), exact prefix caching, native 262,144-token context;
• batched (lottolabs/qwen36-35b-tt-p150:batched-x6, sha256:ce6eaecb81ae…): 2–16 concurrent sequences, non-speculative, chunked prefill, prefix caching.
• Both serve every stored BFP8 weight as BFP8 (no load-time re-quantization to BFP4) and use the Hugging Face Gated DeltaNet prefill rule (QWEN_GDN_DIAG_ALPHA=0 by default; the earlier runtime damped the chunked delta rule with 0.25).
• New in this release: fused decode kernels. Decode runs fewer, larger device programs (see Fused decode kernels): single mode 138.8 instead of 112.3 tok/s at a 128-token prompt; greedy outputs are bit-identical to x4. The checkpoint is unchanged from x3-gptq-20261004 and x4-sampling-20261005.
• Since x4: exact on-device sampling. Temperature, top-k, top-p, min-p and the presence, frequency and repetition penalties run on the P150 with vLLM's semantics, and sampled requests keep MTP-1 speculation through lossless speculative sampling (see Sampling).
• Launcher (launch.py): verified download, image load, checkpoint-bound caches and the exact serving command for both modes.

Precision as served

Quality

Teacher-forced on the P150 with the served weights, precision plan and BFP8 KV cache, against a CPU BF16 run of the original checkpoint (Hugging Face Transformers modules). Eval set: the 34-record subset of the 27B release (14 multi-turn chats, 12 Python files, 8 instructions) plus four 4,096-token passages of Newton's Opticks. Chat and instruction records are scored on the assistant tokens only.

This release vs CPU BF16:

Comparison (ΔPPL / top-1 / KL vs CPU BF16):

¹ Unpublished development build: round-to-nearest BFP4 experts, GDN in-projections in layers 3–39 and the LM head re-quantized to BFP4 at load, damped GDN prefill rule. ² Same runtime as this release (dense BFP8, HF GDN rule) with round-to-nearest BFP4 experts. ³ CPU BF16 model with exactly this release's quantized weight values substituted (BF16 activations); the gap between ³ and this release is runtime numerics (BFP8 KV cache, device kernels), not weights.

GSM8K (first 100 test questions, thinking disabled, greedy, 512 new tokens, same prompt token IDs): this release 87/100 in single mode and 87/100 in batched mode (16 slots); CPU BF16 original 90/100; earlier build 86/100. Single mode and BF16 agree on the extracted answer for 86/100 questions; 16 TT and 14 BF16 answers hit the 512-token limit.

These measurements compare the TT build with its BF16 source on fixed text; they are not a benchmark certification.

Speed

Single P150, greedy, 512 streamed output tokens, technical-essay prompt. Decode tok/s is the median over requests of client-observed tokens after the first. The probe first sends one warmup request (cold TTFT, full prefill), then repeats the same prompt; with prefix caching on, the repeats at 65K and 262K tokens resume from the prefix cache.

Single mode (MTP-1, default):

Decode is 7–9% slower than the earlier build because the dense projections and the LM head are now read as BFP8 instead of BFP4 (GPTQ experts are the same size as round-to-nearest ones). That is the price of the quality gain above.

The table was measured on the x3 runtime. This runtime (x6, fused decode kernels), same sampling bench (tools/probes/sample_bench.py, 512 tokens, decode tok/s p50): 138.8 / 137.5 / 117.8 / 58.8 tok/s at 128 / 2,048 / 32,768 / 261,632 prompt tokens (x4 runtime: 112.3 / 111.9 / 98.4 / 53.4). Prefill and TTFT are unchanged.

Batched mode (16 slots): aggregate output tok/s over all clients (includes prefill and queue drain) / median per-request decode tok/s:

Peak aggregate at 16 users and 128-token prompts: 731.3 tok/s (this runtime; x4 runtime in the same session 689.8). No request failed.

Fused decode kernels

This release changes only the runtime: decode runs fewer and larger device programs. Every fused program performs the same tile math in the same accumulation order as the programs it replaces, so greedy output is bit-identical to x4: model-level logits hashes of 96/96 single-mode decode steps equal the x4 runtime's, HTTP exactness passes 20/20 checks against the same non-speculative reference, and greedy texts at 128 / 2K / 32K / 262K prompts are identical. Quality (Quality) is therefore unchanged.

Single mode (one sequence, MTP-1):

• one-launch RMSNorm for every decode norm (width-sharded, 64 cores), replacing interleaved-to-sharded + norm + sharded-to-interleaved;
• bank-packed weights read on both NoCs for the LM head, the MTP draft head and the Gated DeltaNet input projection (each worker streams contiguous runs from one DRAM bank at a time);
• Gated DeltaNet core in one program: convolution front end, delta-rule recurrence and gated output norm for decode and the MTP verification pass (previously three programs per layer);
• greedy argmax on tiled logits without the vocabulary-wide untilize, and one program that commits the accepted MTP candidate of all 30 Gated DeltaNet layers (previously one per layer);
• MoE decode of one or two rows through device top-k routing and batched expert streams.

Device time per step at a 1,990-token context: decode 11.88 → 9.51 ms, MTP-1 cycle 15.12 → 12.16 ms. Over HTTP (greedy, 512 tokens, decode tok/s p50):

An independent re-run of the same image with the plain serving script measured 138.0 and 137.6 tok/s at 128 and 2,048 tokens (evidence/fusion/single/operator-rerun/).

Sampled presets on this runtime (MTP-1, decode tok/s p50, MTP acceptance in parentheses):

Batched mode (2–16 slots): the one-launch norm, the packed dual-NoC LM head and Gated DeltaNet input projection, the tiled greedy argmax (now up to 16 rows) and a single gate/up + down MoE expert program for 1–8 rows. The fused Gated DeltaNet core is single-sequence only: batched mode keeps each slot's state in DRAM and already runs its own batched front-end and recurrence programs. For a fixed batch (same prompts in the same slots, same step schedule) outputs are bit-identical to the x4 batched runtime: per-step logits hashes of 48 decode steps match at 1, 2, 4, 8 and 16 slots (evidence/fusion/tools/bench35b.py). Device time per greedy decode step, x4 → x6: 1 slot 12.13 → 10.85 ms, 4 slots 13.89 → 12.65 ms, 8 slots 15.49 → 14.05 ms, 16 slots 19.43 → 18.18 ms.

Aggregate output tok/s (greedy / thinking preset, 512 tokens, x4 runtime in the same session in parentheses):

GSM8K at 16 concurrent users, coding preset, seed 1: 87/100 (x4 runtime, same probe and seeds: 87/100). Seeded requests reproduced one at a time (passed).

Measurements, patches against the x4 runtime sources and the bench harnesses: evidence/fusion/.

Sampling

Both images sample on the device. The sampler reproduces vLLM's sampler (same order of operations and float32 arithmetic: repetition, frequency and presence penalties, temperature, min-p, top-k, top-p) on the full 248,320-token vocabulary with custom device kernels, draws with a counter-based Philox RNG, and reads back only token IDs. In single mode, sampled requests keep MTP-1 speculation: the draft token is sampled from the MTP head's distribution and accepted by speculative rejection sampling, which leaves the output distribution exactly that of non-speculative sampling.

Defaults. A request that sets no sampling fields gets the model's generation_config.json defaults: temperature 1.0, top_k 20, top_p 0.95 (vLLM fills unset fields from it; the server log says so at startup). Plain OpenAI-client requests are therefore sampled, not greedy, and run on the device path. Fields you set override these defaults one by one; an unset top_k stays 20. Send temperature: 0 for greedy decoding.

Presets (Qwen3.6 model card) and measured single-mode speed on the x4 runtime (MTP-1, 512 output tokens, decode tok/s p50 over requests, MTP acceptance in parentheses; tools/probes/sample_bench.py; this runtime's numbers are in Fused decode kernels):

¹ x3 runtime, thinking / coding / non-thinking: vLLM's host sampler, no speculation (MTP acceptance 0). Default-parameter requests ran at 29.6 tok/s there and 111.8 tok/s now (128-token prompt).

Sampled decode runs at 96–101% of greedy speed; the difference follows the MTP acceptance rate. With --no-mtp (non-speculative): 128 tokens 79.8 / 79.8 / 79.5, 2,048 tokens 78.7 / 79.0 / 79.0, 32,768 tokens 71.5 / 71.6 / 71.4, 261,632 tokens 43.0 / 43.1 / 43.1 tok/s (thinking / coding / non-thinking); plain greedy requests keep the previous greedy path there and decode at 57.6 tok/s (2,048 tokens).

Batched mode, sampled (thinking preset, 512 output tokens; aggregate output tok/s, x4 runtime vs the x3 runtime's host sampler; this runtime in Fused decode kernels):

Coding preset, 2,048-token prompts: 67 tok/s at 1 user, 275 tok/s at 16 users.

What runs on the device, and what falls back to the host sampler. Fallback is exact (vLLM's own sampler) but slower: about 31 tok/s in single mode, where the host path also gets no MTP speedup, and about 36 tok/s per sequence in batched mode.

• On the device: greedy (temperature < 1e-5), and random sampling with top_k from 1 to 64 at any temperature, together with any top_p, min_p, presence_penalty, frequency_penalty, repetition_penalty and seed; stop, max_tokens, n = 1 and both endpoints work as usual.
• On the host: logprobs or prompt_logprobs, logit_bias, allowed_token_ids, bad_words (chat endpoint), min_tokens until it is reached, structured outputs / guided decoding, and random sampling with top_k disabled (-1 or 0) or above 64.
• In batched mode, one ineligible request sends that decode step of the whole batch to the host sampler.
• Single mode with MTP-1 rejects min_p > 0, logit_bias and min_tokens > 1 with HTTP 400 ("The min_tokens, min_p, and logit_bias sampling parameters are not yet supported with speculative decoding."). This check is vLLM's own request validation for any speculative configuration; the previous runtime returns the same 400. Batched mode and single mode with --no-mtp accept these fields (min_p on the device, the other two on the host).

Seeds. A request with seed is reproducible: the same request to the same server configuration returns the same text (checked by repeating seeded requests in single and batched mode). Each draw is Philox4x32-10 keyed by the 64-bit seed and counted by absolute token position, so it does not depend on other requests or on timing; requests without a seed draw a random 64-bit seed. The same seed gives different (equally distributed) samples with and without MTP, because speculation consumes random numbers differently; in batched mode the logits, and therefore samples, can also change with the batch composition (see Limitations). The first token after prefill is drawn by vLLM's seeded host sampler.

Correctness evidence (evidence/sampling/):

• Device kernels against vLLM's sampler code run in float64 on the same logits, 7 parameter sets × 12 cases (penalties, min_p, top_k 1–64, negative presence, greedy with penalties): identical support, max absolute probability error 3.9e-08.
• 100,000 device draws per preset: chi-square p = 0.98, 0.82, 0.27, 0.98 (thinking, coding, nonthinking, minp_freq_rep); host replay of 2,000 device draws per preset: 0 mismatches.
• Speculative sampling on device, 100,000 verify cycles per preset: emitted-token chi-square p = 0.66, 0.51, 0.19; acceptance 0.6577 (theory 0.6581), 0.3877 (theory 0.3879), 0.8177 (theory 0.8159) (thinking, coding, nonthinking).
• End to end over HTTP (250 seeds × 2 prompts × 3 presets, 24 tokens; per-position two-sample chi-square on the first four tokens plus output length, 30 tests per comparison):
• single mode, MTP-1 speculative sampling vs non-speculative device sampling: minimum p = 0.071.
• single mode, device sampler vs vLLM's host sampler on the same server (requests with logprobs take the host path): minimum p = 0.032.
• single mode, device sampler vs the x3 runtime's host sampler: minimum p = 0.007.
• batched mode at 1 user, device sampler vs vLLM's host sampler (x3 runtime, same model): minimum p = 0.112.
• Batched and single mode do not sample the same distribution (minimum p < 0.0001 with the device sampler, 0.0002 with the x3 runtime's host sampler): their decode kernels produce slightly different logits, which shifts the odds between near-tied tokens (0.57 / 0.43 in single mode vs 0.38 / 0.58 in batched mode for the two leading second tokens of one prompt, coding preset, 250 and 250 samples). The device sampler matches the host sampler within each mode.
• Greedy exactness is unchanged: tools/probes/http_exact.py passes 20/20 in single mode with MTP-1 and 20/20 with --no-mtp, against the non-speculative reference of the x3 release.
• Parameter matrix (tools/probes/param_matrix.py, 37 combinations per server, run on single mode with MTP-1, --no-mtp and batched mode): every request returned the expected HTTP status; device or host routing matched the rules above (MTP acceptance in single mode, decode speed otherwise); logit_bias, allowed_token_ids and bad_words take effect on the host path; the previous runtime returns the same HTTP statuses (evidence/sampling/runs/pm2-x3/).

GSM8K with sampling (first 100 test questions, thinking disabled, 512 new tokens, per-question seeds; correct answers out of 100 for seeds 1 / 2 / 3):

Greedy GSM8K for comparison: 87/100 (single and batched), CPU BF16 90/100.

Long context and passkeys

Passkey probe (tools/probes/long_context35.py): a numbered-fact archive with three keys (beginning, middle, end), raw completion with thinking disabled, default single mode (MTP-1, BFP8 KV cache), cold prefill.

Teacher-forced caveat at 262K. HTTP retrieval succeeds at 262,016 tokens, but a teacher-forced check on the same prompt (evidence/passkey/tt-x3-bfp8kv.json) gives the correct bare answer CEDAR-7419-QUARTZ… a total log-probability of -27.2 on the TT build (earlier build -13.4; CPU BF16 -0.006). The largest loss is the first forced token CED (-10.9; then Z -5.6, OR -4.4), where the TT build prefers BEGIN (p = 0.96): it formats the answer with the key's label (BEGIN_PASSKEY: …) instead of the bare key; the labelled correct answer scores -14.2. Fidelity over the prompt itself is better than the earlier build's (top-1 agreement with BF16 on the first/middle/last 2,048 positions 0.670 / 0.718 / 0.783, earlier 0.468 / 0.696 / 0.756). Treat answers near the 262K limit as less certain than the HTTP passkey result suggests.

Position handling was checked in the runtime source: RoPE angles are computed in FP32 on the host from absolute positions (rope.py, same formula and precision as Transformers), the 262,144-row table covers every position, and prefill/decode use the same table.

Download and serve

Requirements:

• Linux with a supported Tenstorrent P150 driver (/dev/tenstorrent) and hugepage mounts at /dev/hugepages and /dev/hugepages-1G.
• Docker and Python 3.11 or newer (standard library only).
• Exclusive ownership of one P150: stop every other workload on the card first; the launcher never resets the device or stops containers.
• Host RAM: 64 GB or more recommended. Loading the model stages the converted weights through host memory (about 50 GB at peak).
• Disk space for:
• the 21.7 GB checkpoint and the 5.5 GB runtime archive (download);
• the loaded runtime images (two tags sharing one ~22 GB base);
• per mode, a writable converted-tensor cache (~21 GB) and JIT kernel cache (~5 GB), built on the first start.

curl --fail --location --output launch.py \
  https://huggingface.co/Lottolabs/Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150/resolve/main/launch.py

# Single mode (default): one sequence, MTP-1 speculative decoding, 262,144-token context, prefix caching on
python3 launch.py --cache-root /large-disk/qwen36-35b --device-ownership-confirmed

# Batched mode: 2-16 concurrent sequences (non-speculative, chunked prefill); rounded up to 2/4/8/16 slots
python3 launch.py --cache-root /large-disk/qwen36-35b --mode batched --max-num-seqs 16 --device-ownership-confirmed

# One sequence without speculation
python3 launch.py --cache-root /large-disk/qwen36-35b --no-mtp --device-ownership-confirmed

# Download and verify only (no Docker, no device) / print the Docker command
python3 launch.py --cache-root /large-disk/qwen36-35b --download-only
python3 launch.py --cache-root /large-disk/qwen36-35b --print-command

Other options: --max-model-len (128–262,144), --prefix-caching on|off, --port, --name, --revision, --tensor-cache, --kernel-cache.

The launcher is standard-library Python. Before it serves, it:

1. resolves the requested revision to an immutable commit;
2. downloads the release (anonymously, or with HF_TOKEN if set);
3. verifies every byte against runtime-release.json, including the 1,312 grouped expert files listed in the native manifest, and checks that the release pins exactly the image IDs and checkpoint manifest hash written into launch.py;
4. binds the tensor cache to this checkpoint and image (below);
5. loads the checksum-pinned runtime-image.tar.gz if needed and checks the Docker image ID of the selected mode.

The OpenAI-compatible API (model Qwen/Qwen3.6-35B-A3B, /v1/completions and /v1/chat/completions, Qwen3 reasoning and qwen3_coder tool-call parsers) binds to 127.0.0.1:8000. The first start of each mode converts weights into the tensor cache and compiles kernels; later starts reuse both. Keep the printed revision and pass it with --revision for exact replay.

Previous releases (x4, x3)

Both earlier releases use the same checkpoint and stay available at their tagged revisions:

• x4-sampling-20261005 (tag x4, revision d404b8452e3a428c9bc7f7885a8968d3336ebe68): runtime images lottolabs/qwen36-35b-tt-p150:single-x4 sha256:118e0d90fe6a… and lottolabs/qwen36-35b-tt-p150:batched-x4 sha256:a2254f2cae15…, exact on-device sampling without the fused kernels;
• x3-gptq-20261004 (tag x3, revision 84494bfa72cee6bdfe529efee43e054d2e668cf8): runtime images lottolabs/qwen36-35b-tt-p150:single-x3 sha256:1ac069f321f1… and lottolabs/qwen36-35b-tt-p150:batched-x3 sha256:3efee1291257…, vLLM's host sampler.

Each launch.py pins the image IDs of its own release and refuses another release's files, so fetch the launcher from that revision and pass the same revision:

curl --fail --location --output launch-x4.py \
  https://huggingface.co/Lottolabs/Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150/resolve/d404b8452e3a428c9bc7f7885a8968d3336ebe68/launch.py
python3 launch-x4.py --revision d404b8452e3a428c9bc7f7885a8968d3336ebe68 --cache-root /large-disk/qwen36-35b --device-ownership-confirmed

All releases can share one --cache-root: each revision downloads into its own snapshot directory, and the tensor and kernel caches are keyed by image ID.

Cache safety

The runtime's converted-tensor cache (TT_CACHE_PATH, including the MoE expert bank layout) is keyed by tensor name and shape only, not by content. A cache built from a different checkpoint of the same model, for example an earlier quantization, would be loaded silently and serve the wrong expert weights. launch.py therefore never shares a tensor cache across checkpoints or images:

• the default tensor cache is CACHE_ROOT/tensor-cache/<manifest-sha256[:16]>-<image-id[:12]>, i.e. tensor-cache/439096088c8bbceb-faa77aad3f71 for single mode and tensor-cache/439096088c8bbceb-ce6eaecb81ae for batched mode;
• every tensor cache records its checkpoint manifest hash and image ID in qwen36-cache-identity.json; the launcher refuses a cache with a different identity, and a non-empty directory without one (--tensor-cache pointing at an old cache fails instead of serving wrong weights);
• the JIT kernel cache holds compiled kernels only and is keyed by image ID.

If you serve this checkpoint with your own Docker command, give it an empty, dedicated TT_CACHE_PATH.

Verification

• Exactness. HTTP correctness probe (tools/probes/http_exact.py): six chat prompts × 320 greedy tokens, /v1/completions with the same token IDs, A/B/A isolation, greedy after sampled, cancellation mid-decode and mid-prefill (131,072-token prompt), image rejection. The non-speculative server (prefix caching off) recorded the reference (20/20 checks); single mode with MTP-1 reproduces it exactly with prefix caching on (20/20) and off (20/20).
• Prefix caching. Multi-turn probe at 2K/8K/32K/131K-token histories (turn 2 = turn 1 + reply + a new message): at 8K/32K/131K turn 2 resumes from the prefix cache (snapshots are taken at 2,048-token prefill-chunk boundaries, so the 2K history, shorter than one chunk, is prefilled again); 131K history: 129,024 of 131,322 turn-2 prompt tokens served from the cache, TTFT 82.4 s → 2.4 s. Every text equals the non-speculative, caching-off reference: yes (and yes with caching off).
• Checkpoint. 772/772 native tensors verified, 0 failures: the 200 GPTQ tensors decode bit-exactly (via ttnn.load_tensor) to the GPTQ values; every other tensor is byte-identical to the parent checkpoint, whose BFP tensors were re-derived on CPU from the BF16 original as the bit-exact round-to-nearest conversion; the loader's own read_manifest(verify_files=True) passes (checkpoint/provenance/gptq-checkpoint-verification.json, checkpoint/provenance/rtn-parent/).
• MTP-off fix (inherited from the earlier build): single-sequence serving with MTP off used to fail during warmup (TT_FATAL: Reads are not supported during trace capture); the single-device model now keeps the vocabulary table on device. Diff in evidence/mtp-off-fix/.
• Release package (x3). The previous package, with the same checkpoint files: a fresh download of revision 1d43a2e5 into an empty directory with the repository's own launch.py --download-only re-hashed 2034 files (27.2 GB, 0 disagreements with SHA256SUMS). With the local release tags removed, launch.py loaded runtime-image.tar.gz to the pinned image IDs and served from the download with fresh, checkpoint-bound caches: single mode passed 20/20 exactness checks against the reference above, decoded 112.3 tok/s at 128, 112.2 tok/s at 2,048, 98.5 tok/s at 32,768 prompt tokens, multi-turn texts equal to the reference (turn 2 served from the prefix cache at 8,192/32,768-token histories); batched mode (4 slots) served 24 requests at 1 and 4 users with 0 errors, peak aggregate 253 tok/s (evidence/public-download-verification.json, status: passed).
• Release package (x4). The previous package, with the same checkpoint files: a fresh, anonymous download of revision eab50dc0 (branch x4) into an empty directory with the repository's own launch.py --download-only (75 min) re-hashed 2034 files (27.2 GB, 0 disagreements with SHA256SUMS). With the local release tags removed, launch.py loaded runtime-image.tar.gz to the pinned image IDs and served from the download with fresh, checkpoint-bound caches. Single mode: 20/20 exactness checks against the x3 non-speculative reference; greedy decode 112.4 / 111.9 tok/s and thinking preset 112.3 / 108.9 tok/s (MTP acceptance 0.74 / 0.70) at 128 / 2,048 prompt tokens; repeated seeded requests returned identical texts on both endpoints and a different seed a different text. Batched mode (4 slots), thinking preset: 24 requests at 1 and 4 users, 0 errors, aggregate 251 tok/s at 4 users with 128-token prompts and 166 with 2,048; seeded requests reproduced one at a time (evidence/public-download-verification-x4.json, status: passed). Later commits change documentation and evidence only; launch.py and every file it downloads are identical.
• This package. A fresh, anonymous download of revision cefaf84c (branch x6) into an empty directory with the repository's own launch.py --download-only (80 min) re-hashed 2034 files (27.2 GB, 0 disagreements with SHA256SUMS). With the local release tags removed, launch.py loaded runtime-image.tar.gz to the pinned image IDs and served from the download with fresh, checkpoint-bound caches. Single mode: 20/20 exactness checks against the x3 non-speculative reference; greedy decode 138.6 / 137.5 tok/s and thinking preset 138.1 / 132.9 tok/s (MTP acceptance 0.74 / 0.70) at 128 / 2,048 prompt tokens; repeated seeded requests returned identical texts on both endpoints and a different seed a different text. Batched mode (4 slots), thinking preset: 24 requests at 1 and 4 users, 0 errors, aggregate 270 tok/s at 4 users with 128-token prompts and 176 with 2,048; seeded requests reproduced one at a time (evidence/public-download-verification-x6.json, status: passed). Later commits change documentation and evidence only; launch.py and every file it downloads are identical.

Limitations

• Text only. The vision encoder is not shipped; image and video inputs are rejected.
• Batched mode is not batch-invariant. With more than one active sequence, greedy outputs can differ from the same request served alone, because the batched kernels round differently with batch composition (measured: concurrent outputs differed from single-request outputs at 4 users, 8 users, 16 users). Single mode is deterministic and exact against the non-speculative reference (greedy); seeded sampled requests are reproducible in single mode. The same rounding differences make batched mode's sampling distribution measurably different from single mode's between near-tied tokens, even at one user (see Sampling).
• Teacher-forced 262K caveat. HTTP passkeys pass at 262,016 tokens, but the teacher-forced correct-answer log-probability there is low (see Long context); retrieval quality near the context limit is weaker than the pass suggests.
• Sampling fallbacks. logprobs, logit_bias, allowed_token_ids, bad_words, min_tokens, guided decoding and top_k disabled or above 64 use the slower host sampler (about 31 tok/s in single mode, no MTP speedup). Single mode with MTP-1 rejects min_p > 0, logit_bias and min_tokens > 1 (HTTP 400, vLLM validation); use --no-mtp or batched mode for them.
• One P150, owned exclusively. No multi-card support. Batched mode is non-speculative; MTP-1 is single-sequence only.
• Host RAM: 64 GB or more recommended; the model load peaks around 50 GB of host memory (container memory peak during the verification's first start: 11 GB, docker stats).
• Quality is measured, not certified: see the tables above. The remaining gap to BF16 is larger on code and long prose than on chat.
• Serving is verified with this repository's images only; the runtime sources are provenance and need the unpublished development parent images to rebuild.

Integrity and provenance

• runtime-release.json: the serving inventory, the runtime-archive checksum, the two pinned Docker image IDs and the checkpoint manifest hash.
• release-manifest.json and SHA256SUMS: the complete repository inventory and checksums.
• checkpoint/native_manifest.json: per-tensor format, precision and hashes, expert_groups (the shipped grouped expert files), ttquant (GPTQ method, calibration, per-layer GPTQ source hashes, parent manifest hash) and release_packaging (inherited from the parent checkpoint: the omitted monolithic expert files; its note refers to the parent's round-to-nearest verification).
• checkpoint/provenance/: the GPTQ tools (gptq-gptq35.py, gptq-build_calib.py, gptq-build_ckpt.py, gptq-verify_ckpt.py), calibration statistics (gptq-calib-stats.json), the checkpoint verification (gptq-checkpoint-verification.json), the original safetensors index, the BFP emulator and name mapping, and in rtn-parent/ the parent checkpoint's manifest as built, its CPU verification against the BF16 original and its first device smoke test.
• checkpoint/quantization-error.json and precision-summary.json.
• checkpoint/README.md is the upstream Qwen model card, kept unchanged (it is a hashed asset of the native manifest).
• runtime/: for each image, the Qwen-specific sources as extracted from the image (overlay/, with sources.json hashes), the build scripts of the image lineage (build/) and the third-party licenses. Serving uses the checksum-pinned archive.
• evidence/fusion/: the fused-kernel measurements (model-level logits hashes and device times, HTTP exactness and speed, batched-port exactness at 1–16 slots), the bench harnesses and chain scripts, and the source patches of this runtime against the x4 runtime sources (source/).
• evidence/sampling/: the sampling measurements (speed, GSM8K, distribution comparisons, parameter matrix, kernel tests), their chain scripts, and the source patches of the x4 runtime against the x3 runtime (source/).
• evidence/ and tools/: every measurement above, with the probes and scripts that produced it.

License and attribution

• Released under Apache-2.0 (LICENSE). The model is derived from Qwen/Qwen3.6-35B-A3B by the Qwen team (Alibaba Cloud), Apache-2.0; the upstream model card is kept as checkpoint/README.md.
• The runtime images contain tt-metal (Apache-2.0) and vLLM (Apache-2.0) with modifications; their license and notice files are in runtime/licenses/.
• GPTQ calibration text: UltraChat 200k, databricks-dolly-15k, alpaca-cleaned, Project Gutenberg and Python package sources; only the derived weights are distributed.