English | 中文 A 143M-parameter bidirectional Chinese↔English translation model, instruction-tuned on the Haidass1.5-143M base — the strongest zh⇄en translator at this scale among general chat-architecture models. Drafter-143M: a control model with identical configuration, data and training recipe, except that it starts from random initialization instead of the pretrained base — used to quantify the contribution of base-model pretraining. OPUS-MT models are single-directional — one independent 78M model per direction; "-" marks directions a model does not serve. The same models re-evaluated on FLORES+ devtest (released 2026; zero overlap with dev): devtest sentences do not overlap with dev.…
cagliostro-v3 is an open-weight model for text generation from Bench Labs, released under Apache License 2.0. It has 146M parameters and a 2,048-token context. At 16-bit it needs about 0.4 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index.
A 146M parameter decoder-only language model pretrained from scratch on 75B tokens of open web, synthetic textbook and mathematics data. It is the third model in the cagliostro line and the first to clear an Index of 26 on the Open SLM Leaderboard metric.
Parameters146M
Context2,048
Weights2.3 GB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads—
Runs On
What it takes to serve cagliostro-v3 (146M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 0.3 GB | 0.4 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 0.1 GB | 0.2 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 0.1 GB | 0.1 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 21, 2026.
cagliostro-v3 on every accelerator the SAVRN Index prices, at every precision
Model Card
By Bench Labs, published under apache-2.0, revision 870f5e95f092.
A 146M parameter decoder-only language model pretrained from scratch on 75B tokens of open web, synthetic textbook and mathematics data. It is the third model in the cagliostro line and the first to clear an Index of 26 on the Open SLM Leaderboard metric. Zero-shot, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script, on the exported float32 weights in this repository. The Index is the leaderboard's own formula, (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) and CombinedARC is the mean of ARC-Easy and ARC-Challenge. For context against other models at this scale, using the leaderboard's published…
Read Bench Labs's full model card
A 146M parameter decoder-only language model pretrained from scratch on 75B tokens of open web, synthetic textbook and mathematics data. It is the third model in the cagliostro line and the first to clear an Index of 26 on the Open SLM Leaderboard metric.
This is a mid-training checkpoint. The run is at 72.745B of 75B tokens and 80% of the way through its learning rate cooldown. Roughly 2.3B tokens remain. The numbers below are measured on the weights in this repository as they stand today, not on a finished model. They should improve slightly when the run completes, and this card will be updated when it does.
Results
Zero-shot, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script, on the exported float32 weights in this repository.
| Benchmark | Metric | Score |
|---|---|---|
| HellaSwag | acc_norm | 42.42 |
| ARC-Easy | acc_norm | 54.17 |
| ARC-Challenge | acc_norm | 27.99 |
| PIQA | acc_norm | 67.46 |
| ArithMark-3 | acc_norm | 43.80 |
| Open SLM Index | 26.27 |
The Index is the leaderboard's own formula, (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) and CombinedARC is the mean of ARC-Easy and ARC-Challenge.
For context against other models at this scale, using the leaderboard's published figures:
| Model | Params | Tokens | Index |
|---|---|---|---|
| SmolLM2-135M | 135M | 2T | 27.13 |
| cagliostro-v3 | 146M | 72.7B | 26.27 |
| SmolLM-135M | 135M | 600B | 25.74 |
| GPT-X2.5-135M | 135M | n/a | 25.17 |
| Haidass1.5-143M | 143M | n/a | 25.07 |
| BananaMind-2-Pro | 139M | 100B | 24.96 |
The flat stretch from 30B to 64B is not a stall. A warmup-stable-decay schedule holds the learning rate at its peak for the first 85% of the run, and constant peak learning rate depresses zero-shot multiple-choice accuracy even while validation loss keeps improving. The recovery from 22.0 to 26.3 is what the cooldown unlocks, not new knowledge appearing in the last 9B tokens.
The sharp drop in training loss at 63.75B is the data mixture changing, not the model improving. Validation loss on a fixed held-out set is the honest line, and it moves smoothly.
Model details
| Field | Value |
|---|---|
| Parameters | 146,352,000 |
| Non-embedding parameters | 85.7% |
| Layers | 30 |
| Hidden size | 640 |
| Intermediate size | 1,536 |
| Attention heads | 10 |
| Key/value heads | 5 |
| Attention | Grouped query attention with cross-head subspace attenuation |
| Activation | SwiGLU |
| Normalization | RMSNorm, eps 1e-6 |
| Positional encoding | RoPE, theta 100,000 |
| Context length | 2,048 |
| Vocabulary | 32,768 BPE |
| Embeddings | Tied input and output |
| Logit cap | 15.0 |
| Weights | float32 safetensors |
The architecture is defined in this repository. trust_remote_code=True is required because CagliostroForCausalLM is not part of transformers.
Training data
Two mixtures. The first covers the stable phase, the second takes over when the cooldown begins at 85% of the run.
| Source | Stable phase | Cooldown |
|---|---|---|
| FineWeb-Edu (deduplicated) | 43.7% | 37.0% |
| DCLM-Baseline | 28.3% | 5.0% |
| Cosmopedia v2 | 16.0% | 25.0% |
| FineMath 3+ | 5.0% | 15.0% |
| OpenMathInstruct-2 | 3.0% | 13.0% |
| InfiWebMath 3+ | 2.0% | 0.0% |
| SmolTalk | 2.0% | 5.0% |
Mathematics rises from 10% to 28% during the cooldown. No source exceeds 0.4 epochs across the full run, so nothing is repeated enough to memorize.
Training setup
| Field | Value |
|---|---|
| Optimizer | AdamW, weight decay 0.01 |
| Schedule | Warmup-stable-decay |
| Warmup | 2,000 steps |
| Stable phase | steps 2,000 to 648,498 at peak learning rate |
| Cooldown | 114,441 steps, cosine to zero |
| Tokens per step | 98,304 |
| Total steps | 762,939 |
| Precision | bfloat16 with float32 master weights |
| Hardware | one RTX 5090 |
| Throughput | 90,000 to 103,000 tokens per second |
| Wall clock | about 9 days |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "bench-labs/cagliostro-v3"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)
ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
This is a base model with no instruction tuning and no chat template. It completes text.
Reproducing the evaluation
pip install lm-eval
python -m lm_eval --model hf \
--model_args pretrained=bench-labs/cagliostro-v3,dtype=float32,trust_remote_code=True \
--tasks hellaswag,arc_easy,arc_challenge,piqa \
--num_fewshot 0 --batch_size 8 --device cuda:0
ArithMark-3 uses the script linked from the leaderboard, pointed at the same model id. Evaluate in float32. A bfloat16 round trip moves logits by about 1.3e-1 at this logit cap, which is enough to change borderline multiple-choice answers, while float32 agrees with the training weights to 3.2e-05.
Provenance
This repository holds the full training history. A checkpoint was pushed every 30 minutes from the first step, giving 363 commits between 11 and 20 September 2026. Any intermediate checkpoint can be retrieved by revision, and the benchmark trajectory in the chart above can be reproduced from them.
Limitations
English only. 2,048 token context. No instruction tuning, no safety tuning, no RLHF. At 146M parameters it confabulates freely and should not be relied on for factual questions. The mathematics ability measured by ArithMark is arithmetic and short symbolic work, not general mathematical reasoning.
License
Apache-2.0. The training data is drawn from FineWeb-Edu and FineMath (ODC-By), DCLM-Baseline and OpenMathInstruct-2 (CC-BY-4.0), Cosmopedia v2 and SmolTalk (Apache-2.0).
Configuration
- Architecture
- CagliostroForCausalLM
- Context length (tokens)
- 2,048
- Layers
- 30
- Hidden size
- 640
- Feed-forward size
- 1,536
- Attention heads
- 10
- Key/value heads
- 5
- Vocabulary size
- 32,768
- RoPE base
- 100000
- Model type
- cagliostro
Identity and Version
- Repository
- bench-labs/cagliostro-v3
- Publisher
- Bench Labs
- Task
- Text generation
- Modality
- Text
- Library
- transformers
- Parameters
- 146M parameters
- Languages
- en
- Revision
- 870f5e95f092dd13928cf39c29fe464be040261e
- First published
- 2026-09-11
- Last updated
- 2026-09-20
Files and Weights
12 files, 2.3 GB in total. The weights are 2 files totalling 2.3 GB in pt, safetensors.
Weights2 files · 2.3 GB
Configuration4 files · 12.3 KB
Tokenizer2 files · 2.3 MB
Documentation1 file · 6.5 KB
Other2 files · 201.6 KB
Repository1 file · 1.6 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| latest.pt | Weights | 1.8 GB | f78351f10c48 |
| model.safetensors | Weights | 585.4 MB | 11e31616df91 |
| config.json | Configuration | 743 B | — |
| configuration_cagliostro.py | Configuration | 1.4 KB | — |
| generation_config.json | Configuration | 174 B | — |
| modeling_cagliostro.py | Configuration | 10.0 KB | — |
| README.md | Documentation | 6.5 KB | — |
| index_curve.png | Other | 89.5 KB | — |
| loss_curve.png | Other | 112.1 KB | 8e3182e05ff3 |
| .gitattributes | Repository | 1.6 KB | — |
| tokenizer.json | Tokenizer | 2.3 MB | — |
| tokenizer_config.json | Tokenizer | 174 B | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 2.3 GB
Download from Bench Labs
Released by Bench Labs through its official repository on Hugging Face. Read the license.
Built From
- Trained on (disclosed) HuggingFaceTB/finemath
- Trained on (disclosed) HuggingFaceTB/smollm-corpus
- Trained on (disclosed) HuggingFaceTB/smoltalk
- Trained on (disclosed) mlfoundations/dclm-baseline-1.0
- Trained on (disclosed) nvidia/OpenMathInstruct-2
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 2.3 GB |
| 16-bit | 0.3 GB |
| 8-bit | 0.1 GB |
| 4-bit | 0.1 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About cagliostro-v3
How much GPU memory does cagliostro-v3 need?
About 0.4 GB at 16-bit and 0.1 GB at 4-bit: the weights (146M parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run cagliostro-v3 on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use cagliostro-v3 commercially?
Yes. cagliostro-v3 is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is cagliostro-v3's context length?
2,048 tokens, from the maximum position embeddings in its published configuration.
Similar Models
English | 中文 The instruction-tolerant sibling of DALabCommunity/Haidass-Translate-143M: same 143M zh⇄en translation training, plus 9.1% cleaned general-domain data (STEPFUN ShareGPT) mixed in. Translation scores are within 0.3 BLEU of the pure-translation version, and the model retains limited general instruction-following ability that the pure-translation version does not have. Drafter-143M: a control model with identical configuration, data and training recipe, except that it starts from random initialization instead of the pretrained base — used to quantify the contribution of base-model pretraining. OPUS-MT models are single-directional — one independent 78M model per direction; "-"…
Test the whole generation capabilities here: https://transformer.huggingface.co/doc/gpt2-large Pretrained model on English language using a causal language modeling (CLM) objective. It was introduced in and first released at this page. model. Content from this model card has been written by the Hugging Face team to complete the information they provided and give specific examples of bias. GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs…
SmolLM2 is a family of compact language models available in three size: 135M, 360M, and 1.7B parameters. They are capable of solving a wide range of tasks while being lightweight enough to run on-device. More details in our paper: https://arxiv.org/abs/2502.02737 SmolLM2 demonstrates significant advances over its predecessor SmolLM1, particularly in instruction following, knowledge, reasoning. The 135M model was trained on 2 trillion tokens using a diverse dataset combination: FineWeb-Edu, DCLM, The Stack, along with new filtered datasets we curated and will release soon. We developed the instruct version through supervised fine-tuning (SFT) using a combination of public datasets and our…
SmolLM2 is a family of compact language models available in three size: 135M, 360M, and 1.7B parameters. They are capable of solving a wide range of tasks while being lightweight enough to run on-device. More details in our paper https://arxiv.org/abs/2502.02737 SmolLM2 demonstrates significant advances over its predecessor SmolLM1, particularly in instruction following, knowledge, reasoning. The 135M model was trained on 2 trillion tokens using a diverse dataset combination: FineWeb-Edu, DCLM, The Stack, along with new filtered datasets we curated and will release soon. We developed the instruct version through supervised fine-tuning (SFT) using a combination of public datasets and our own…
Ru-Small-Instruct — экспериментальная компактная русскоязычная языковая модель класса SLM (Small Language Model) с объемом параметров ~0.2B (~165M). Разработана с упором на суверенность весов (Zero-Fingerprint): модель обучена с нуля без заимствования базовых чекпоинтов у сторонних корпоративных сетей (Llama 3 от Meta, Qwen от Alibaba, Mistral). Модель предназначена для исследований локального инференса, работы на маломощном оборудовании, CPU и мобильных чипах, где критичны нулевая задержка (Time-To-First-Token) и полная независимость весов. Для компактной модели в 165M параметров, обученной на одном домашнем GPU за 48 часов, способность держать роль, грамотно формулировать сложные термины…