Sieve-9B-Plus

Sieve-9B-Plus is a decision model from the Sieve family. You give it a piece of text or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question. It reads the state once, scores the options directly, and never generates text.

It is a LoRA adapter (rank 32) and a pointer head on Qwen/Qwen3.5-9B (revision c2022362), and it stores one calibration temperature per question category. The code is at github.com/sthanika-ai/Sieve.

How it works

• Backbone. Only the text part of Qwen3.5-9B is used; the model cannot produce text.
• Readout. A pointer head scores the end of each option against the decision point. A calibrated softmax gives the probabilities.
• Isolation. Each question runs as its own row from the state's cache, so questions never see each other.

Use

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0   # recommended: the fused kernel our runs used

from sieve import load_sieve, decide

m = load_sieve("sthanika-ai/Sieve-9B-Plus")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
       {"department": {"type": "choice", "instructions": "Which team handles this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

• Loading. Load it with sieve 0.2.0 or later; the adapter alone is not a text-generation model. The backbone is downloaded at its pinned revision on first use.
• GPU memory. About 20 GB in bf16. Longer states need more.
• Server. sieve-serve --model sthanika-ai/Sieve-9B-Plus --graphs.
• causal-conv1d. Without it, transformers falls back to a slower PyTorch convolution, whose probabilities can differ slightly. Install it to reproduce our numbers.
• Temperature per category. head.pt stores a global T = 1.109 and 23 temperatures (1.000 to 1.338), fitted by NLL on the calibration split (43,176 questions). A question's category is its type, its number of options (2, 3-4, 5-9, 10-25, 26+) and the kind of state (empty, text or JSON). A question takes the temperature of its most specific category in the table (choice:3-4:text, then choice:3-4, then choice), else the global T. Temperatures never change an answer, and temperature=1.0 gives raw probabilities. sieve versions before 0.2.0 ignore the table and use the global T.

Results

Decision Index 0.3

Sieve-9B-Plus scores 53.35 (raw index 64.39, breadth 51.95) on the public 0.3 suite of the Decision Index, on our own run with the official kit at 62d2f51 (0.3) and its scorer.

• Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported request and no error. The index averages 37 of the benchmarks.
• How the run was made. One complete run of the 0.3 suite on two H100 80GB GPUs, in four shards.
• Results. sthanika-ai/Sieve-9B-Plus-decision-index-results.
• Not on the leaderboard yet. The score is self-reported. On the 0.3 board the ranking uses the Full score, where this public suite counts 20% and the maintainers' private tests count 80%.

The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each benchmark's own metric and the skill is chance-corrected.

Calibration split

The calibration split chose the checkpoint and fitted the temperatures.

In a 2-fold check inside the calibration split (fit on one half, score the other), the per-category table and the single T scored the same: NLL 0.3079 vs 0.3078, ECE 0.0087 vs 0.0084.

Latency

• Per Decision Index request. In-process wall time, one request at a time, eager, over the 140,620 requests of the 0.3 run on H100 80GB GPUs, with two runners sharing each GPU: median 92 ms, mean 161 ms, 80th percentile 126 ms, 95th percentile 337 ms. This is not the maintainers' RTX PRO 6000 measurement.

Training

• Recipe.
• LoRA r=32, α=64, dropout 0.05 on the attention, MLP and linear-attention projections (12 modules), with the pointer head trained from scratch.
• AdamW, lr 5e-5 for the adapter and the head, OneCycle with 10% warm-up, 1 epoch, 32 records per step, bf16.
• States and questions up to 16,384 tokens each; longer records are dropped, never truncated (none were).
• 5,183 steps, 5.6 hours on two H100 80GB GPUs.
• Permuted choice options and "none of the above" augmentation, with a cross-entropy loss.
• Selection. The checkpoint with the highest accuracy on a fixed 2,000-question calibration sample, with NLL breaking ties. This rule was fixed before training, and it chose step 4,500. The benchmarks were never used to choose.
• Temperatures. Fitted after selection on the whole calibration split: the global T, then one T per category with at least 200 calibration questions.
• Data. 165,832 training records and 30,722 calibration records.
• Sources: public datasets and generated decision records.
• Not published. The training files are not part of this release; the SHA-256 of each file is in training_config.json.

Files

License

Apache-2.0, see LICENSE.