MATILDA JEV v1.5 by Maincode
All scores on this card are from the Decision Index 0.3 public suite (the public dataset of the current
leaderboard edition).
A BF16 JEV decision model with a 255-option readout. This is the
highest-scoring checkpoint in the completed MATILDA JEV experiments as of
9 October 2026: Decision Index 63.47 (0.3 public suite).
The model accepts a state and one or more decision questions. It supports
choice, noul (probability of true), and ordinal score. Inference is one
forward pass per question; the model has no text-generation head and does not
generate a chain of thought. The LoRA updates are already merged into the
weights. No adapter merge is required.
Loading
Use Python 3.12 or newer and install PyTorch for your accelerator. The validated
cluster environment uses PyTorch 2.14.0, Transformers 5.17.0, and an AMD Instinct
MI355X. Other dependencies are listed in requirements-runtime.txt.
import sys
from pathlib import Path
from huggingface_hub import snapshot_download
checkpoint = Path(snapshot_download('Maincode/matilda-jev-v1.5'))
sys.path.insert(0, str(checkpoint / 'runtime'))
from maincode_jev_serve.model import DecisionModel, answer
model = DecisionModel(checkpoint=checkpoint, device='cuda:0')
row = {
'state': 'The parcel arrived on schedule.',
'question': {
'type': 'choice',
'instructions': 'Classify delivery.',
'criteria': {'on_time': 'On time', 'late': 'Late'},
},
}
print(answer(row['question'], model.predict([row])[0]))
The loader uses the custom code bundled in this repository. AutoModel alone
loads the backbone; use DecisionModel to include the separate decision head.
The package retains the existing MATILDA configuration aliases and the underlying
Transformers implementation. Model weights, tokenizer vocabulary, input template,
answer codes, readout, and temperature match the evaluated checkpoint.
predict.py accepts JSON lines containing either state + question or
state + questions:
{"state":"Two plus two equals four.","questions":{"truth":{"type":"noul","instructions":"Is this statement true?"}}}
python /path/to/downloaded/model/predict.py --device cuda:0 < requests.jsonl
Evaluation
Decision Index 0.3 public suite: 140,178 of 140,178 scored requests answered across 37 benchmarks
(kit 62d2f51).
Saved predictions were independently rescored with identical results. The full run, including every
prediction, is in Maincode/matilda-jev-decision-index
under runs/matilda-jev-v1.5/. scores.json in this repository is the earlier 0.2.1 run
(62.76), kept for reference.
| Metric |
Jev |
MATILDA JEV v1 |
MATILDA JEV v1.5 |
| Decision Index (public) |
57.96 |
60.16 |
63.47 |
| Raw |
68.55 |
70.04 |
72.62 |
| Breadth |
57.27 |
59.21 |
62.45 |
| Knowledge & Reasoning |
53.87 |
46.84 |
49.83 |
| Language Understanding |
59.23 |
64.80 |
69.91 |
| Retrieval & Classification |
55.42 |
61.07 |
65.96 |
| Tools & Automation |
75.09 |
79.07 |
80.84 |
| Arts & Human Taste |
39.08 |
46.22 |
45.41 |
Jev is the public-suite part of its Decision Index 0.3 board entry (per-benchmark public results, as published in the
kit's tests/fixtures/board-0.3.json), not its Full score. MATILDA JEV v1 is
Maincode/matilda-jev-v1 (runs/matilda-jev-v1.3/ in the same dataset).
Area scores are chance-corrected skill multiplied by 100. These results are
diagnostic: earlier training in this model's lineage used benchmark-related
material, and this release was chosen after comparing experimental scores.
The numbers are not an independent held-out generalization estimate. The full
suite evaluated text decisions; vision inference was not validated for this
release.
