Inferno-Exp-1.0 is a 76.2M-parameter decoder-only base language model
trained from scratch by AlgoDriveAI. It is a math-first model: its training data centers on arithmetic
and math word problems, and it is geared toward the AxiomicLabs small-language-model benchmarks used in
SLM challenges, ArithMark 3.0 and ArithMark 2.0 (results below). Future versions will build on this math
foundation with improvements aimed at broader language understanding and logical reasoning.
|
|
| Parameters |
76,163,200 |
| Layers × width |
14 × 640 |
| Attention |
grouped-query, 8 query / 2 KV heads, head dim 80, full-head RoPE |
| Feed-forward |
plain SwiGLU, inner width 1536 |
| Context |
2048 tokens |
| Tokenizer |
Mistral-7B-v0.1 (padded to 32,064); <|endoftext|> separates documents |
| Embeddings |
input and output tied |
| Training |
AlgoDriveAI/SLM_v3, 2 epochs; AdamW, WSD schedule, label smoothing 0.05 |
| Checkpoint |
step 290,000 (vanilla_gqa_epochs_r1) |
This is a base model (no instruction tuning). The model code ships in the repo, so load it with
trust_remote_code=True.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("AlgoDriveAI/Inferno-Exp-1.0")
model = AutoModelForCausalLM.from_pretrained("AlgoDriveAI/Inferno-Exp-1.0", trust_remote_code=True)
ids = tok("Once upon a time, a little fox", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))
Evaluation
All scores are zero-shot on the checkpoint in this repo. The ArithMark benchmarks were scored with
AxiomicLabs' official scripts, unmodified; the language benchmarks with the
lm-evaluation-harness.
| Benchmark |
Metric |
Score |
Standard error |
Chance |
| ArithMark 3.0 (word problems) |
acc_norm |
54.20% |
±1.58 |
25% |
| ArithMark 2.0 (integer arithmetic) |
acc |
67.32% |
±0.94 |
25% |
| HellaSwag |
acc_norm |
27.46% |
±0.45 |
25% |
| ARC-Easy |
acc_norm |
40.24% |
±1.01 |
25% |
| ARC-Challenge |
acc_norm |
24.66% |
±1.26 |
25% |
| PIQA |
acc_norm |
56.58% |
±1.16 |
50% |
On the Open SLM Leaderboard's
Intelligence Index formula these scores give 14.16.
Inferno-Exp-1.0 is math-first, and the scores show it: it is strong on arithmetic but modest on general
commonsense and reading benchmarks, where its narrower training data gives it less to draw on. Upcoming
versions keep the math foundation and broaden the training mix to lift these language and reasoning scores.
ArithMark 3.0
Scored with bencharithmark-3.py on
ArithMark 3.0 (1,000 four-choice
continuation word problems, grades 1–5) at commit 6f6e59dd: float32, batch size 32,
max context 1,024, no BOS token, length-normalized log-likelihood as the primary metric.
| Metric |
Score |
| Length-normalized accuracy (acc_norm, primary) |
54.20% |
| Raw continuation accuracy |
54.20% |
| Standard error (acc_norm) |
±1.58 points |
| Random-choice baseline |
25.00% |
ArithMark 2.0
Scored with benchmark_arithmark-2.0.py on
ArithMark 2.0 (2,500 four-choice integer
arithmetic continuations such as (16 / 4) + 44 =) at commit 5bd986c7: the script's standard GPU
setting (bfloat16), batch size 16, max context 1,024, no BOS token, raw summed continuation
log-likelihood (the benchmark's only metric).
| Operators per expression |
Items |
Accuracy |
| 1 |
1,250 |
95.04% |
| 2 |
750 |
46.93% |
| 3 |
500 |
28.60% |
| Overall |
2,500 |
67.32% (1,683 / 2,500) |
Standard error ±0.94 points; random-choice baseline 25.00%.
Language understanding (lm-evaluation-harness)
Zero-shot, acc_norm, float32, batch size 16:
lm_eval --model hf \
--model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.0,trust_remote_code=True,dtype=float32" \
--tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 16
| Task |
Items |
acc_norm |
| HellaSwag |
10,042 |
27.46% ± 0.45 |
| ARC-Easy |
2,376 |
40.24% ± 1.01 |
| ARC-Challenge |
1,172 |
24.66% ± 1.26 |
| PIQA |
1,838 |
56.58% ± 1.16 |
Notes
- Generation re-reads the full context each step (no KV cache); fine at this size.
- The tokenizer does not add a BOS token; its
bos_token is set to <|endoftext|>, the document
boundary the model saw during training, for tools that prepend one.
