ALEA R4 3B-A0.7B Base: pretraining checkpoints
Qwen3-MoE architecture; 3.32B parameters in total (3,315,557,632, tied input/output embeddings), 673M active per token.
These are intermediate base-model pretraining checkpoints, not instruction-tuned. Safety and behaviour status:
pending final checks. They are published for research: to follow what the model learns over training and to load
any checkpoint with stock transformers. They are not a release.
R4 is ALEA's next model after R3, trained only on
lawful data: U.S. legal, regulatory and financial public-record text from the KL3M Data Project, plus synthetic,
dictionary, knowledge and task data created by ALEA.
main holds only this card. Each checkpoint is its own branch; load one with revision=.
|
|
| Architecture |
stock Qwen3MoeForCausalLM (no trust_remote_code): 32 layers, d_model 1,280, 16 query / 8 KV heads (head_dim 128), QK-norm, RoPE (theta 1e6), 48 routed experts (FFN 512) with top-6 routing, tied input/output embeddings |
| Parameters |
3,315,557,632 total; ≈ 673M active per token |
| Context |
4,096 tokens |
| Tokenizer |
alea-v2b-sw32k: 32,768-token BPE with control tokens and a chat template; <\|start\|> is prepended automatically |
| Training |
OLMo-core on one NVIDIA GH200; 1,048,576 tokens per step; AdamW, LR 4e-4 constant after a 2,000-step warmup (WSD; decay comes later) |
| Weights |
bf16 safetensors (≈ 6.6 GB per branch) |
| License |
CC-BY-4.0 (WordNet-derived content: see WORDNET_LICENSE.txt) |
The tokenizer is the alea-v2b-sw32k source tokenizer with a BOS post-processor; token IDs are identical.
Load a checkpoint
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo, rev = "alea-institute/alea-r4-3B-A0.7B-Base-ckpts", "s4-step23000" # any branch below
tok = AutoTokenizer.from_pretrained(repo, revision=rev)
model = AutoModelForCausalLM.from_pretrained(repo, revision=rev, dtype=torch.bfloat16) # add device_map="auto" for GPU
inputs = tok("This Agreement shall be governed by and construed in accordance with", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tok.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
Tested with transformers 5.17. Any version with Qwen3-MoE support loads these weights (4.51 or later; before 4.56
pass torch_dtype= instead of dtype=). vLLM's Qwen3MoeForCausalLM also loads them.
Checkpoints
| branch |
step |
stage |
tokens seen |
held-out BPB |
knowledge probe all / legal |
ICL suite |
s1-step1000 |
1,000 |
S1 |
1.05B |
0.956 |
31.1 / 29.9 |
0.2 |
s1-step2000 |
2,000 |
S1 |
2.10B |
0.695 |
33.1 / 42.7 |
2.0 |
s1-step3000 |
3,000 |
S1 |
3.15B |
0.622 |
38.7 / 51.2 |
7.0 |
s1-step4000 |
4,000 |
S1 |
4.19B |
0.585 |
42.9 / 61.0 |
19.2 |
s2-step5000 |
5,000 |
S2 |
5.24B |
0.557 |
76.4 / 84.8 |
36.8 |
s2-step6000 |
6,000 |
S2 |
6.29B |
0.542 |
78.4 / 86.6 |
32.8 |
s2-step7000 |
7,000 |
S2 |
7.34B |
0.532 |
81.7 / 87.8 |
49.0 |
s2-step8000 |
8,000 |
S2 |
8.39B |
0.524 |
82.9 / 86.0 |
46.8 |
s2-step9000 |
9,000 |
S2 |
9.44B |
0.517 |
83.8 / 91.5 |
57.2 |
s2-step10000 |
10,000 |
S2 |
10.49B |
0.513 |
81.7 / 88.4 |
56.6 |
s2-step11000 |
11,000 |
S2 |
11.53B |
0.507 |
84.4 / 92.7 |
57.6 |
s2-step12000 |
12,000 |
S2 |
12.58B |
0.505 |
84.1 / 93.3 |
60.8 |
s2-step12400 |
12,400 |
S2 |
13.00B |
0.504 |
84.6 / 92.1 |
56.2 |
s3-step13000 |
13,000 |
S3 |
13.63B |
0.483 |
84.3 / 87.2 |
60.0 |
s3-step14000 |
14,000 |
S3 |
14.68B |
0.481 |
81.7 / 90.8 |
67.4 |
s3-step15000 |
15,000 |
S3 |
15.73B |
0.477 |
84.4 / 89.6 |
64.6 |
s3-step16000 |
16,000 |
S3 |
16.78B |
0.475 |
83.5 / 87.8 |
59.6 |
s3-step17000 |
17,000 |
S3 |
17.83B |
0.474 |
82.8 / 89.0 |
65.2 |
s3-step18000 |
18,000 |
S3 |
18.87B |
0.473 |
82.8 / 92.7 |
63.2 |
s3-step19000 |
19,000 |
S3 |
19.92B |
0.471 |
87.6 / 92.1 |
69.4 |
s3-step20000 |
20,000 |
S3 |
20.97B |
0.470 |
84.9 / 88.4 |
62.4 |
s3-step20800 |
20,800 |
S3 |
21.81B |
0.469 |
85.2 / 90.2 |
67.8 |
s4-step21000 |
21,000 |
S4 |
22.02B |
0.467 |
79.8 / 86.0 |
68.2 |
s4-step22000 |
22,000 |
S4 |
23.07B |
0.466 |
78.0 / 81.7 |
64.8 |
s4-step23000 |
23,000 |
S4 |
24.12B |
– |
– / – |
– |
Stages:
- S1: data segment 1 (steps 1-4,000)
- S2: data segment 1b (steps 4,001-12,400): a larger share of ALEA's own dictionary, knowledge and task data
- S3: data segment 2 (from step 12,401)
Metrics are ALEA's internal probes, run in bf16 on the training-framework checkpoint: held-out BPB is bits per
byte on a frozen held-out set (macro-average over 7 domains; lower is better); knowledge probe
is 4-way multiple-choice accuracy (%, PMI scoring, chance 25) over all items and the legal subset; ICL suite is
in-context task accuracy (%, 4-shot). They track progress within this run and are not comparable to public
benchmarks.
Conversion check
Every branch is pushed only after a parity check: the bf16 export, loaded in stock transformers with fp32 compute,
must match the training framework's own forward pass with the same bf16 weights on fixed prompts and a random-token
batch (max |Δlogit| ≤ 0.05, mean ≤ 0.003, identical greedy continuations). The largest max |Δlogit| so far is
3.3e-05. Each branch's parity.json has its numbers, including bf16-compute figures.
