jevify-qwen3.5-4b-base-readout-coh

Qwen3.5-4B-Base, readout fine-tuned (LoRA, + coherence)

A System One decision model: it does not write text. It reads a state, answers typed questions (choice, score, noul), and returns calibrated probability distributions your code can branch on. This repo is a rank-16 LoRA (21,233,664 parameters) on Qwen/Qwen3.5-4B-Base, merged into the weights at load.

How it was trained (readout fine-tuning). The model is trained on its own decision readout — the distribution over the allowed answers read at the answer position, one forward pass, no decoding — with the primitive's proper scoring rule, plus a coherence penalty (weight 1.0): every training question comes with automatically derived siblings (the options as yes/no questions, the negation, the threshold questions of a scale), and the de Finetti sure loss of the family's answers is penalised, so the model's answers to related questions stay mutually consistent. Options are shuffled per family. Training data: the train splits of the 16 non-held-out jev-bench sources (5,885 families, at most 400 records per source); lr 3e-05, 2 epochs, best epoch by validation loss (epoch 0), seed 0. A Tier 0 recipe (temperature per primitive, Noul bias, option-order permutations) was then fitted on validation splits. The six held-out sources (clinc150, arc_challenge, yelp5, measuring_hate_speech, fever_evidence, strategyqa_grounded) never appeared in training.

from jevify import load_jevified

model = load_jevified("Praveenrajus/jevify-qwen3.5-4b-base-readout-coh")
model.ask({"text": "The battery lasted two days on a single charge."},
          {"q": {"type": "noul", "instructions": "Is the review positive?"}})

jevify-serve --model Praveenrajus/jevify-qwen3.5-4b-base-readout-coh serves it as a drop-in for the TypeSafe SDK (TYPESAFE_BASE_URL=http://localhost:8000). The backbone is pulled from its own repo at load, pinned to commit 1001bb4d826a52d1f399e183466143f4da7b741b.

Results

Every number is on the jev-bench test splits (22,773 records) or the study's other test suites, scored the same way for every model; rows below this model are references from the same study.

Decisions and calibration

Coherence and invariance — sure loss: mean d² over 4,749 question families (0 = perfectly coherent); order flip: how often the top answer changes when options are shuffled; tag TVD: how much the distribution moves when option tags change from A–J to other identifiers.

Out of distribution — stated rules (LegalBench, rule given in the question), none-of-the-above when the gold option is removed, injected-instruction hijack rate, and three community Jev benchmarks.

Reproduction check. Loading this folder with load_jevified and re-scoring 72 jev-bench test records from six sources reproduced the training run's own test predictions: 0 changed choice answers, mean largest |Δp| 0.005, max 0.022 (the adapter is merged into bf16 weights at load).

results/ holds the raw files: test_metrics.json (per source), recipe.json, coherence.json, probes.json, tags.json, train.json (the training log) and summary.json (this model's row of the study table).

Limitations

• One training seed per repo branch; out-of-distribution numbers in particular vary between identical runs, so compare arms across seeds before drawing conclusions.
• The phishing benchmark's decision threshold shifts after fine-tuning (ranking, AUROC, is preserved); a one-number log-odds shift fitted on a handful of labelled emails repairs it.
• English only; the recipe was fitted on jev-bench validation splits and may need refitting on a very different domain.

Method, benchmark and findings: github.com/uspraveen/Jevify (docs/FINDINGS.md).