cx-decider-2b-v1

Verdict: effective, not noise. 0.9499 vs 0.6208 baseline on 4,328 held-out rows (+0.33, noise ≈ ±0.01). Confidence ≥ 0.9 band: 1.000 accuracy (n=1,213). Details in Part 6.

If you take one thing from this page: this model does not do customer service. It watches something else do customer service and vetoes bad steps. Everything below walks you through that idea, then the evidence, then the exact commands.

Part 1 — Start here: what is this and is it for you?

The 30-second version

Imagine a support agent (human or AI) working a ticket. The work is a loop: look up the caller, check their history, open a case, log the call. The dangerous mistake is doing steps out of order — logging a call when no case exists means the record vanishes.

This model sits beside that loop and answers four questions about the current situation:

1. next_tool — "which step comes next?" (look up / check history / open case / log call)
2. safe_to_log — "does a case exist, so logging is safe?" (yes / no)
3. needs_case — "does this request need a case at all?" (yes / no)
4. urgency— "how urgent is this?" (routine / normal / urgent / VIP — see the warning in Part 4)

Every answer comes with a confidence number — and that number has been measured to mean what it says: when this model says ≥0.9, it is right 100% of the time on the tested data (1,213 out of 1,213). That is the whole point. A normal AI says words; this one says "this answer, and here's how much to trust it."

Is this for you?

Yes, if: you run (or are building) an AI loop that calls tools for customers, and you want a cheap, fast checker on every step — or you just want a calibrated classifier for support-like decisions (intent, escalation, fraud flags; see the v2 set below).

No, if: you want a chatbot (this model cannot generate a single sentence — it only scores options you give it), you want urgency scoring (ours is broken, honestly disclosed below), or you need decisions about anything other than customer service (generic ability was not re-measured after tuning).

Important: what this is NOT (read before you misunderstand)

This is not a merge of two AI projects. Three separate things exist, and only one of them is in this download:

1. chendren/qwen2.5-3b-cx-lam — the actor. A Qwen2.5-3B model that writes out customer-service tool calls. Frozen; nothing in this release changed it.
2. This repo — the judge. A Qwen3.5-2B model that scores decisions. Its only link to the actor is that its training examples were derived from the actor's trajectory dataset. No actor weights, gradients, or outputs are in this checkpoint. Different model family, different size, different job — a merge is technically impossible, not just unattempted.
3. The pairing — a ~50-line script (decision_gate.py, in the training repo, described in Part 3) where the actor proposes and the judge disposes. The "system" exists only when you run that script. There is no combined checkpoint to download.

Part 2 — Try it in 5 minutes

You need Python 3.10+, ~8 GB disk (the base model downloads on first use), and the strands-decider package. No GPU required — it runs on Apple Silicon (MPS), CUDA, or CPU.

Step 1 — install:

pip install strands-decider

Step 2 — ask your first question. Save this as try.py:

from strands_decider.infer import load_engine

# device="mps" on a Mac, "cuda" on NVIDIA, "cpu" anywhere (slower)
eng = load_engine("chendren/cx-decider-2b-v1", device="mps")

# Describe the situation: what the customer wants + what has been done so far.
state = "Customer asks about renewal pricing. Completed so far: crm.screenpop."

r = eng.ask(state, {
    "next_tool": {
        "type": "choice",
        "instructions": "Which CRM tool should fire next in the service loop?",
        "criteria": {
            "crm.screenpop": "look up the contact and open their record",
            "crm.list_cases": "check open cases and recent call history",
            "crm.create_case": "open a new support case capturing the issue",
            "crm.log_call": "log the completed call disposition",
        },
    },
    "safe_to_log": {
        "type": "noul",
        "instructions": "A case already exists, so it is safe to log the call now.",
    },
})

print("next step: ", r.answers["next_tool"].choice,
      f"(confidence {r.answers['next_tool'].confidence:.3f})")
print("safe to log:", float(r.answers["safe_to_log"].noul))

Step 3 — run it:

python try.py

What you should see (numbers may differ in the third decimal across hardware):

next step:  crm.list_cases (confidence 0.997)
safe to log: 0.0

Reading the output. The model says: after looking up the contact, the next step is checking their history (list_cases), and it is 96.5% sure. Logging is not safe (0.0 = "no, no case exists yet") — because so far only a screen-pop happened. Both answers are correct: the recorded workflow these rows came from does screenpop → list_cases → create_case → log_call, and logging at this point would orphan the record.

Step 4 — break it on purpose. Change the state to:

state = "Customer asks about renewal pricing. Completed so far: crm.screenpop, crm.list_cases, crm.create_case."

Run again. Now next_tool should say crm.log_call at ~0.999 and safe_to_log should read ~1.0. You just walked the full loop: the judge follows the workflow state and flips its answers as the situation changes.

Step 5 — see the veto. Change the state back to just crm.screenpop done, but ask what happens if someone proposes logging anyway — that is Part 3.

The three question types, plainly

• choice = "pick one of these options." You name the options and describe each; the model returns the winner, every option's probability, and a confidence. No retraining to add options — they are read from your request.
• noul ("yes/no") = "is this statement true?" Returns a probability. 0.0 above means "false" — there is no separate confidence because the probability is the uncertainty.
• score = "rate this on my scale." You give ordered levels; it returns an expected value plus the distribution. (Ours has a documented flaw — Part 4.)

Part 3 — How the gate works (the actual use)

The model alone answers questions. The system is a loop — here it is, all of it:

loop until done:
    1. ACTOR proposes one tool call  ("I think we should log_call")
    2. JUDGE (this model) scores the state:
         next_tool   — what should actually fire?
         safe_to_log — may log_call fire yet?
    3. GATE decides:
         if tool is log_call and safe_to_log ≤ 0.5  →  run create_case instead  (veto)
         elif judge disagrees at confidence ≥ 0.9   →  run judge's tool         (override)
         else                                       →  run actor's tool         (trust)

Two design points that matter:

• The veto fires first, before the confidence comparison. That ordering is what makes "log before a case exists" impossible, even on states where the judge's own next_tool answer is wrong. Measured: on all three pre-case states, an early log_call was vetoed at safe_to_log ≈ 0.00. The one known next_tool error (Part 4, the s2 edge) is therefore unreachable in practice — the veto catches exactly that failure mode.
• Below 0.9 confidence the judge abstains and the actor's proposal stands. The measured bands say why: ≥0.9 is perfect (1.000, n=1,213), 0.5–0.9 is mediocre (0.729). Trusting the mid band would be trusting a coin with opinions. The 0.9 bar is the base paper's own "act without a check" band — adopted, not invented.

The reference implementation is decision_gate.py (~50 lines: gate_action(engine, state, lam_tool)), and test_full.py --suites G replays the whole matrix: 4 canonical steps pass correct proposals, 4 wrong proposals get corrected, VIP variants behave — 17/18 probes end-to-end (the 18th is the model's s2 miss, which the gate absorbs: the correct proposal still passes).

To serve it over HTTP for a multi-step loop: strands-decider serve chendren/cx-decider-2b-v1 --port 8000 (POST /v1/systemone). Asking both questions in one request costs ~0.24s total on M2 Max MPS — the second question reuses the first's work (shared-prefix cache), so it is nearly free.

Part 4 — What the training data was (and the two flaws it caused)

Three datasets, three roles. This section exists so you can judge the evidence rather than take our word for it.

Dataset A — inherited knowledge (not trained here). The starting checkpoint (StrandsAgents/strands-decider-2B-hobson-v19) trained on ~100k rows: 21 public classification sets plus 12,909 multi-step documents distilled from a frozen Qwen3.5-4B, plus generated questions and human-rated answer-adequacy rows. That is where generic decision ability comes from (JevBench 0.723) — and the frozen-KL term in our run (weight 0.3) exists solely to keep it while CX rows are added.

Dataset B — the CX loop rows (this release's training signal). cx_decisions_train.jsonl: 33,148 rows from 2,075 customer-service trajectories. How they were made: each recorded tool sequence (screenpop → list_cases → create_case → log_call) was sliced per step; the actual next tool became the label — no human wrote opinions, no LLM judged anything; the labels are the recorded behavior itself. Plus 8,300 "step-zero" rows (fresh request → first tool), added in phase 2 because testing caught a real bug: without them, every training state had ≥1 completed tool, so a fresh request was out-of-distribution and the model confidently overrode a correct screenpop at 0.9999. That failure, its diagnosis, and its fix are all in the test log.

Dataset C — the v2 applied rows (published, not yet trained into this checkpoint). chendren/cx-decisions-v2-data: 2,400 rows, 120 each across 20 applied decisions (intent, channel, escalation, churn, sentiment, fraud, PII, SLA, next-best-action, …). Four subagents invented realistic 2026 states and judged labels by inference — no scripts, no templates. This is the planned train set for v2 (4 loop + 20 applied = 24 decision types).

Flaw 1 (live with it): urgency. The urgency labels came from priority keywords in the source data — 99% defaulted to one class. The model learned the prior, not the rubric (acc 0.80, ECE 0.15). The gate never reads urgency. Do not threshold on it. V2's score spreads are even because of this lesson.

Flaw 2 (contained): the s2 edge. At screenpop → list_cases with renewal-worded requests, next_tool says log_call (~0.78) instead of create_case; billing/cancel wordings answer correctly. Below the 0.9 bar so never auto-executed, and the veto (Part 3) covers the failure — but anyone executing raw next_tool below 0.9 without the veto will log before a case exists. That is why the card says "use the gate, not the raw answer."

Held-out honesty. Test rows (4,328) come from 271 different trajectories with zero state-string overlap against train — checked by script, not assumed. The v19 baseline ran on the identical rows, so the +0.33 gap is apples-to-apples.

Part 5 — Training, exactly (for reproducers)

• Start: v19, continual. Kept its trained LoRA + head; set adapters + head trainable: 17,872,384 / 1,899,697,472 params (0.94%). Temperatures reset to 1.0, refit after.
• Run: two phases × 220 optimizer steps (440 total). Micro-batch 8 × grad-accum 2 (=16/step). Head LR 5e-4, LoRA LR 2.5e-5, AdamW β=(0.9, 0.95), cosine schedule, 3% warmup, grad-clip 1.0, head-only decay 0.01, max_length 1024, seed 0. Frozen-KL 0.3 every 4th step. Loss 0.598 → 0.159 (full trace: history.json).
• Machine: Apple M2 Max, MPS, torch 2.14.1, ~0.06 step/s (~2h/phase). Stock train() couldn't be used (hardcodes cuda-or-cpu; init_from wipes the head) — train_mac.py reuses the recipe's own collator, loss, KL reference, and optimizer instead.
• Calibration: refit on 800 held-out rows — global 0.5177; per-kind noul 0.25, choice 0.6246, score 0.9611. Fit-set ECE 0.0109 → 0.0013.
• Why continual, why Mac: the reused capability is the pointer head's generic option-binding (~100k rows + teacher distillation); CX rows just move one task from unseen to in-mix. From-scratch would re-learn genericity to reach the same point. The 0.94% footprint is why 440 steps sufficed.

Part 6 — Results and what they mean (read the table, then the story)

Full suite: test_results.json (32/33). Reproduce: python test_full.py --suites H,G,R,C,L.

Benchmarks in context (base results): the paper reports in-mix v14 at choice 0.960 / noul 0.961 / overall 0.888 vs unseen at ~0.725 / ~0.60 / ~0.65. Ours: 0.9991 / 0.9991 / 0.9499 — at or above in-mix, as expected for a now-in-mix task with a narrow 4-option rubric. Top band 1.000 vs their 0.952 — the routing convention transfers. Not transferred: v19's JevBench 0.723 (never re-run here — KL is intent, not evidence), urgency (prior artifact), long documents (family weak spot, hard tier 0.505, untested here).

The principle this demonstrates: a decision head binds to prompt-described options only within its training distribution. v19 on CX: 0.6208 (their unseen band) with a confident mis-route. Same architecture + CX rows: 0.9499 (their in-mix band). No prompt engineering substitutes for the corpus row — you move a task from the right column to the left one by training on it. Anyone adding a novel rubric to any decision model should expect right-column performance until they do.

Part 7 — When NOT to use it (do not skip)

1. Urgency scores — broken (Flaw 1). Nothing downstream should read them.
2. Non-CX decisions — generic retention unmeasured. v19's 0.723 is not this checkpoint's number until JevBench runs on it.
3. Raw next_tool below 0.9 without the veto — the s2 edge (Flaw 2) will eventually fire log_call early. Always gate.
4. Auto-executing at 0.5–0.9 — mid band is 0.729 here. Confirm or human-route.
5. Commercial use blind — source trajectories derive from bitext (research/non-commercial) via chendren/cx-lam-trajectories. Check before monetizing.
6. Long multi-step documents — untested, family weak spot.

Limits in brief: 2B class; one calibration set serves one distribution (these temperatures fit short CX states); single seed, 440 steps, no multi-seed noise estimate (reference noise ≈ ±3 JevBench tasks); public-data label noise inherited; no safety tuning beyond the base recipe.

Files in this repo

• head.safetensors + lora/ — the weights (use with Qwen/Qwen3.5-2B-Base, ~4.5 GB download on first use)
• strands_decider_config.json — architecture + fitted temperatures
• train_config.json / history.json / provenance.json — the exact run record (hyperparameters, loss trace, SHA-256 hashes)
• test_results.json — all 33 probes with details
• tokenizer.json, tokenizer_config.json, chat_template.jinja — serving files