span-02

span-02 is a decision model: it answers typed questions about a state (JSON or text, plus up to 4 images) with a full probability distribution over the options, in the Decision Index /v1/systemone wire format. It is a fine-tune of Gemma 4 26B-A4B (mixture of experts, about 4B active parameters per token), quantized (NVFP4 experts, FP8 dense layers) and served by a custom Rust/CUDA engine for Blackwell GPUs (sm_120: RTX 5090, RTX PRO 6000) that is included in this repo (engine/).

Before answering it may think. If the top probability of the no-think answer is below 0.8, the model writes a greedy thought. Its length limit comes from a deterministic thought budget: a fixed cost model (think_cost.json) turns a 750 ms target into a per-question token cap that depends only on the request (prompt length, images, number of questions and of thinking questions), so the same request always gets the same thought. After a thought the answer is read out again, and the final distribution is the average of the two: 0.5 x the no-think distribution + 0.5 x the post-thought distribution, for every question type. A choice answer is the option with the highest final probability. Thoughts are decoded with speculative decoding (Gemma's public MTP draft). The verification is greedy, so the thought is token-identical to plain greedy decoding.

Outputs are typed: choice, noul (yes/no probability), score (ordinal levels) and number (a free-form number). The model never returns free text as an answer.

Quantization

This checkpoint is engine-native. It cannot be loaded by transformers or vLLM.

model.safetensors is 16.4 GB. GPU memory in use at serving time is about 26 GB.

Hardware

• An NVIDIA Blackwell GPU with compute capability 12.0 (sm_120a). Tested on an RTX PRO 6000 Blackwell (96 GB), where the latency below was measured, and on an RTX 5090 (32 GB).
• NVIDIA driver with CUDA 12.9 support (R575 or newer).
• To build: CUDA toolkit 12.8+ (nvcc) and Rust. No GPU is needed at build time.

Other GPU architectures are not supported, because the kernels use the sm_120a block-scaled FP4/FP8 tensor-core instructions.

Files

The MTP draft model is not in this repo. launch.sh downloads the public google/gemma-4-26B-A4B-it-assistant.

Quickstart

Docker:

hf download lihe07/span-02 --local-dir span-02 && cd span-02
docker build -t span-02 .
docker run --gpus all -p 8000:8000 -v "$PWD":/model:ro -e DLLM_MODEL_DIR=/model -v span02-data:/data span-02

Without -v "$PWD":/model ... DLLM_MODEL_DIR, the container downloads this repo itself. Pass -e HF_TOKEN for that while the repo is private.

Without Docker (CUDA 12.8+ toolkit and Rust installed):

hf download lihe07/span-02 --local-dir span-02 && cd span-02
(cd engine && NVCC=/usr/local/cuda/bin/nvcc cargo build --release -p dllm-server)
bash launch.sh          # serves on 0.0.0.0:8000; PORT / HOST / AR_API_KEY are optional

Startup takes about 10 s plus about 7 s of warmup. GET /health returns {"ok":true,...} when the server is ready.

API

POST /v1/systemone (POST /v1/decisions is an alias with the same body and response). The state field accepts any JSON value. images (optional) is a list of up to 4 images as data URIs or bare base64.

Question types: * choice: criteria maps each option key to its description (1 to 679 options). The answer is the argmax, its confidence = (p_max - 1/n) / (1 - 1/n), and the probabilities in criteria order. * noul: the probability that the statement is true. criteria is optional: {"true": ..., "false": ...}. * score: criteria is an ordered array of 2 to 10 level descriptions, from low to high. The answer is score = sum_i i x p_i, with the per-level probabilities, a legend and a confidence. * number (our extension of the format): a free-form number, no criteria. The model writes the number greedily after its (optional) thought, as it was trained (the plain decimal, e.g. 391 or 2.5, at most 32 tokens). The answer is {"type": "number", "number": 391.0, "text": "391", "confidence": 0.98}: number is the parsed value (null if the text is not a number), confidence = exp(mean log-probability of the generated answer tokens). The think gate is trained for option questions only, so number questions think only with "think": "on".

Request:

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "span-02",
  "state": {"customer": "My package arrived damaged and nobody answers my emails. I want a refund now.",
            "order": {"id": 123, "days_late": 4}},
  "questions": {
    "dept":   {"type": "choice", "instructions": "Which team should handle this?",
               "criteria": {"returns": "Exchanges, wrong or damaged items",
                            "shipping": "Delivery status, delays, lost packages",
                            "billing": "Charges, invoices, payment problems"}},
    "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
    "anger":  {"type": "score", "instructions": "How frustrated is the customer?",
               "criteria": ["Not frustrated: calm or positive", "Somewhat frustrated: mild annoyance",
                            "Very frustrated: angry, hostile or threatening to leave"]}}}'

Response:

{"model": "span-02",
 "answers": {
   "dept":   {"type": "choice", "choice": "returns", "confidence": 0.945,
              "probabilities": {"returns": 0.963, "shipping": 0.024, "billing": 0.013}},
   "refund": {"type": "noul", "noul": 0.99999},
   "anger":  {"type": "score", "score": 1.9999,
              "legend": {"0": "Not frustrated: calm or positive", "1": "Somewhat frustrated: mild annoyance",
                         "2": "Very frustrated: angry, hostile or threatening to leave"},
              "probabilities": {"0": 0.000008, "1": 0.00012, "2": 0.99987}, "confidence": 0.9998}},
 "usage": {"input_tokens": 412, "output_tokens": 3,
           "think": {"dept": false, "refund": false, "anger": false},
           "trace_tokens": {"dept": 0, "refund": 0, "anger": 0}, "latency_ms": 41.7}}

(The token counts and latency in this response are illustrative.)

Extensions to the Decision Index format (clients that follow the official schema can ignore them): * "think": "auto" | "on" | "off" in the request overrides the gate for that request. auto (the default) uses the tau 0.8 gate; on thinks for every question, off never thinks. * "budget_ms": <number> (0 to 600000) sets the wall-clock thinking budget of that request in milliseconds; 0 means no wall-clock budget (a thought ends at the end-of-thought token, EOS or the cap). Without it, a request with "think": "on" thinks to completion (up to the cap) and other requests use the server's --budget-ms. The deterministic --think-ms cap of the submission settings applies regardless; budget_ms only adds or removes a wall-clock deadline (the submission server has none), so requests without it stay deterministic. * "return_trace": true adds usage.traces = {question id: thought text} for the questions that thought. * With images, the response also has a top-level image_tokens field (the total image soft tokens). * usage.think and usage.trace_tokens report which questions thought and how long the thoughts were.

Errors are {"error": {"message": ...}}: 400 for invalid JSON or a context that is too long, 413 for a body over 64 MB, and 422 for schema errors, more than 4 images, an image that cannot be decoded, or too many options. The capacity errors carry the Decision Index capacity markers, so the DI client records unsupported.

Serving settings (Decision Index submission)

--max-len 16384 --budget-ms 0 --tau 0.8 --cap 1024 --think-ms 750 --think-cost think_cost.json
--image-soft-tokens 560 --serial --no-prefix-cache --spec-max-branches 16 --pages 12000 --kv-gb 24

• --budget-ms 0: there is no wall-clock limit; the clock is never read for thinking decisions.
• --think-ms 750 --think-cost think_cost.json: deterministic thought budget. Each request's per-question thought cap = floor((750 - fixed) / per_token(n)), where fixed is the modelled prompt, image and readout time of that request and per_token(n) the modelled time per thought token with n of its questions thinking together. It is a pure function of the request, so answers do not depend on the machine or its load. A thought ends at the end-of-thought token, at EOS, or at that cap (at most --cap 1024). A short single question gets about 330 thought tokens, a request with many thinking questions or a long prompt fewer.
• --tau 0.8: think when the no-think top probability is below 0.8. The final distribution is then 0.5 x no-think + 0.5 x post-thought (choice, score and noul alike).
• --spec-draft (passed by launch.sh): MTP speculative decoding of the thought with an adaptive draft depth and a pruned 16384-token draft vocabulary. It is lossless (token-identical to greedy decoding).
• --image-soft-tokens 560: the Gemma-4 image budget (about 1.3 MP; aspect ratio kept).
• --serial --no-prefix-cache --spec-max-branches 16: each answer is a deterministic function of its request. It does not depend on load or on earlier requests (for up to 16 questions of one request thinking at once). --serial runs one request at a time; concurrent clients queue.
• --pages 12000 --kv-gb 24: KV cache pools (full-attention layers 12000 x 16 = 192K tokens, 3.9 GB; sliding-window layers get the rest of --kv-gb 24, ~20 GB), sized so that every Decision Index request fits on its own under --serial (the largest needs about 3.7K full pages: many questions with long tails). The pool size does not change any answer. The server uses about 41 GB of GPU memory.

Latency

Measured with the submission settings on one RTX PRO 6000 Blackwell, one request at a time, as client wall time per request including HTTP (ms). The Decision Index rule requires the median, mean and p80 each to be at most 1000 ms.

The rule is met on every mix measured, with margin. Against a plain 256-token cap on the same machine and requests, accuracy is the same within noise (pooled paired difference -0.03 points, 95 % CI [-0.39, +0.34]) and the tail is much shorter (DI text requests over 1 s: 34 -> 4 of 400; slowest 2.8 s -> 1.07 s). Against a blend that puts the post-thought half on the post-thought top option only (one-hot), the averaged distributions have the same accuracy within noise (+0.13 points, 95 % CI [-0.06, +0.35], 3,707 choice questions, same thoughts) and are better calibrated (log-loss 0.409 -> 0.397, Brier score 0.200 -> 0.193; on the questions that thought 1.368 -> 1.290 and 0.734 -> 0.689). Speculative decoding runs at about 480 thought tokens/s at batch 1 on real thoughts on this GPU (2.1 ms per token).

Capacity limits

Requests over these limits are rejected with the DI capacity markers. Nothing is truncated. * Choice questions: 1 to 679 options. * Images: at most 4 per request (PNG, JPEG, GIF, BMP, TIFF or WebP). Each image uses at most 560 soft tokens. * Context: the prompt tokens + 562 per image + the reserved thought tokens + 3 must fit in 16384 tokens. The reservation is the deterministic cap of a lone thinking question of that size (about 330 tokens for a short prompt, about 145 at 16k prompt tokens). That allows up to about 16240 prompt tokens without images. * KV cache: a request whose prompt pages (shared prefix + every question's own tail) exceed the whole KV pool is rejected with 422 "maximum context length"; none of the DI 0.3 public requests does with the pools above. * Request body: at most 64 MB.

License

The weights are a derivative of Google's Gemma 4 (google/gemma-4-26B-A4B-it) and are distributed under the Gemma 4 license terms, which are Apache 2.0: ai.google.dev/gemma/docs/gemma_4_license. The serving engine in engine/ is Apache-2.0. It ports algorithms from vLLM, SGLang, FlashInfer and other projects, which are credited in NOTICE and in the source file headers.