clef-flash-jev-claude-distill

Clef-flash · Jev-style Claude distillation

A fine-tune of Cloudflare/clef-flash (Qwen3.5-9B text decoder + 122M joint-schema head). It is distilled from the 27B Cloudflare/clef teacher and Claude Opus 5.5 author labels on synthetic decision data, with a KL anchor to the original clef-flash so general knowledge is kept. On the same 32 public Decision Index benchmarks it scores above both its base model and TypeSafe Jev 1.13.

Release layout (same as Clef-flash, so the bundled loader works unchanged): merged bf16 backbone model-0000*-of-00004.safetensors + fine-tuned fp32 head joint_head.safetensors + configs, tokenizer and joint_schema_model.py. The un-merged PEFT adapter (r = 32, all decoder Linear) is in adapter/, with its head and training config in adapter_head/. Evaluation outputs are in evals/.

import importlib.util
spec = importlib.util.spec_from_file_location("jsm", "joint_schema_model.py")
jsm = importlib.util.module_from_spec(spec); spec.loader.exec_module(jsm)
model, processor = jsm.load_release_model(".", device="cuda")   # bf16 backbone; our evals cast the head with .float()
out = jsm.systemone(model, processor, {
    "state": "…",
    "questions": {"q1": {"type": "choice", "instructions": "…", "criteria": {"approve": "…", "block": "…"}}},
})

Results

Every number below uses the same code path for all models: backbone bf16, head fp32, identical rows.

Decision Index 0.3 public suite (local reproduction, 32 of 37 benchmarks, 123,049 rows)

Our reproduction matches the public board's Clef-flash scores on all 32 benchmarks within ±0.5. The five benchmarks we could not fetch (BPoMP, Humicroedit, cfcolor, HLE, HoVer) count as zero for every model, which is why the absolute index is below the board's 56.15; comparisons are like-for-like. Jev's numbers are the board's own run on the same rows.

Largest gains: RAGTruth 35.6 → 72.8, iSarcasmEval 19.7 → 50.5, CLINC150+OOS 67.0 → 95.2, When2Call +3.8, ANLI +3.3, GSM8K +3.3, BANKING77 +3.0. Knowledge benchmarks hold (MMLU-Pro +1.0, BBH +0.9, GPQA ±0). Known regressions: Home-appliance simulator −17.0 (88 cases), New Yorker caption matching −9.1, PhishNChips −7.8, Habermas −2.5, API-Bank −1.6.

typed-decisions (Jev official test set, 2,000 cases)

Same accuracy as Jev 1.13 on its own test set, with a far better calibrated distribution (KL 0.17 vs 1.44).

External benchmarks (English decision sets + multilingual; 1,000-sample subsets for the multilingual sets)

Mean over 32 sets: Clef-flash 86.44 → this model 86.47 (general ability held). Gains on decision-style sets (DecideBench few-shot 85.2 → 90.8, zero-shot 93.0 → 94.5, ANLI 57.2 → 62.8, KLUE-NLI +1.0, PAWS-X ko/zh +1.5); small losses on Global-MMLU ko/ja/zh (−1.7 to −1.9) and XNLI (en −1.9, zh −3.8). Clef-flash's English knowledge sets (MMLU, ARC, HellaSwag, WinoGrande, BANKING77, CLINC) look memorised (train ≈ test accuracy), so for those the goal is only to stay at the base level.

Training recipe

• Data: jaeyong2/jev-decision-claude-distill — 37,932 synthetic decision states / 150,827 typed questions (en/ko/ja/zh): 35 workflow domains, 12 format families, 6 general-ability families and 9 language-judgement families (sarcasm, stance, aspect sentiment, neutral-heavy NLI, clinical-trial NLI, out-of-scope intents, appliance commands, caption matching, evidence ranking).
• Targets: the 27B Clef distribution where it agrees with the author label; otherwise 0.7·onehot(label) + 0.3·p_27B; disputed items use Claude Opus 5.5 verbalised probabilities; unanswerable items keep the 27B distribution. Language-judgement items were label-balanced, re-judged by Opus where both models disagreed with the author, and mined for hard examples (details in the dataset card).
• Loss: soft cross-entropy + Brier to the target, plus 0.3 · KL(model ‖ clef-flash).
• LoRA r = 32 / α = 128 / dropout 0.05 on all text-decoder Linear layers, full fp32 head; 1 epoch (2–3 epochs lowered knowledge and tool scores in our ablation), 18 states per optimizer step, lr 1e-4 (head 5e-5), states truncated to 6,000 tokens, question-order shuffling and random question subsets. 4 × RTX 3090 (2 GPUs per replica, DDP), 16.7 h.

Licensing: Apache-2.0 (base and teacher models Apache-2.0; all data synthetic and commercially usable). No benchmark text was used for training.