alephllm-mini-beatrix-training

alephllm — Mini-Beatrix training runs

Live training ground for AlephLLM: signed-address (aleph) language models. Code, presets, trainer, and the full test array live in the source repo — github.com/AbstractEyes/alephllm (pip install git+https://github.com/AbstractEyes/alephllm, package geolip.alephllm). This repo holds what training produces, one prefix per craft:

<craft>/manifest.json                      what is trained, what is planned,
                                           tokens run, phase statuses, ckpt index
<craft>/checkpoints/step_XXXXXXXX.safetensors    bf16 weights
<craft>/checkpoints/fp8/step_XXXXXXXX.safetensors  fp8-e4m3 shipping variant
                                           (inference testing only — never train from these)
<craft>/resume/latest.pt                   full resume state: fp32 model, Muon+Adam
                                           optimizer states, data-stream position, RNG
<craft>/runs/                              TensorBoard event files (rendered in the
                                           Training metrics tab)
<craft>/arms/<stage>_stepN.safetensors     the stage arms (mini-beatrix-3 on): one
                                           detachable adapter per curriculum stage,
                                           shipped at every checkpoint
<craft>/reports/                           the boundary reads (held-out bpb, the toggle
                                           ledger, the chat frame, the probes, the rank
                                           profile) at every phase boundary
<craft>/code/                              the training code pinned to the commits that
                                           train the run (mini-beatrix-3 on)

Mission log

• mini-beatrix-3 — COMPLETE (2026-10-05; started 2026-09-20, relaunched 2026-09-27 on the engineered form and 2026-09-28 on amoe-lora 0.2.11). The v3 craft: 32 blocks, d 1024, 16 heads, context 4096, byte vocabulary with the trigram byte embedding, about 376M parameters. Every block carries a CausalSplatHUB (four address constellations, K 64 at D 128, the exact chunked scan at chunk 256) and an anchored bank (three dispatched experts at ff 1024, expert outputs zero-born, gates σ(−3)); the aleph head (256 × 256) is born by revival at step 0 (fitted to the first batch: 8.26 → 5.45 bpb at birth). Recipe: Muon on the transport matrices, pure Adam elsewhere, flat LR, bf16 autocast over fp32 masters, 262,144 tokens per step (micro-batch 2 × 16 chunks × two RTX 5090s, data-parallel), torch 2.8. The plan is 64.4B tokens / 245,670 steps: wikitext warmup 0.3B → fineweb-edu 20.9B → nine curriculum stages (s0–s8; 2.8 / 2.8 / 4.0 / 4.8 / 4.8 / 4.0 / 3.2 / 5.6 / 3.2B; the rule-chain stages draw on a minted 216-word lexicon, pair rate .5) → two anneal diets of 4.0B. The stage arms: from curriculum s1 on, each stage attaches a 13.7M-parameter relay adapter at every block (a projection to 128, a closed-form signed address over 16 slots × 16 atoms, an MLP 128-256-1024, a sigmoid gate born at −3, a zero-born head), trained beside the trunk under a quiet term — the KL from the arm-masked distribution to the armed one on held-out web text, at weight 2 — so every arm stays silent off its stage and detachable; masking is an exact bypass, and each arm ships as its own safetensors file at every checkpoint (arms/; see code/README.md for the format). Boundaries (held-out fineweb-edu bpb; the toggle ledger's bpb cost with each aleph mechanism switched off, bank / head / hub): warmup → fineweb at 1,145 steps: 2.3875; fineweb → s0 at 80,873 (21.2B): 0.9838, toggles +3.20 / +2.30 / +2.57; s0 → s1 at 91,555 (24.0B): 1.1178 (+.134 on the web holdout after 2.8B of stage text, the diet shift), toggles +3.02 / +2.06 / +3.13; s1 → s2 at 102,237 (26.8B): 1.1273, toggles +3.17 / +1.99 / +3.26; s2 → s3 at 117,496 (30.8B): 1.1000, toggles +2.89 / +2.14 / +3.68; s3 → s4 at 135,807 (35.6B): 1.1084, toggles +2.98 / +2.08 / +4.16; s4 → s5 at 154,118 (40.4B): 1.1273, toggles +3.08 / +1.96 / +4.42; s5 → s6 at 169,377 (44.4B): 1.1236, toggles +3.05 / +1.92 / +4.72; s6 → s7 at 181,585 (47.6B): 1.1498, toggles +3.26 / +1.87 / +5.10; s7 → s8 at 202,948 (53.2B): 1.0482, toggles +3.24 / +1.94 / +6.06; s8 → anneal at 215,156 (56.4B): 1.0780, toggles +3.34 / +1.96 / +6.42; anneal (no chat) → anneal (mix) at 230,415 (60.4B): 0.9951, toggles +3.40 / +1.94 / +5.86; the end at 245,674 (64.4B): 0.9861, toggles +6.13 / +1.90 / +6.41; zero loss spikes, the clip never reached, the only guard event a known false positive of the dispatch-entropy watch at the byte funnel in the first 3,000 steps. The stage arms read silent off-domain at every close so far (armed vs masked held-out bpb +0.0001 for the s1 arm at the s1 close; +0.0001 and −0.00001 for the s1 and s2 arms at the s2 close; +0.00001, +0.00003 and −0.00001 for the three arms at the s3 close, although one block's s2 gate had opened to about half by then: a gate moving without a measurable write). The engineering pause: a clean stop at 106,000 (2026-09-26 20:20 UTC) to install what the run needed to fit and move past three arms on 32 GB cards: the arms' intermediates recomputed in the backward (0.3 GB per arm instead of 3), the per-block adapter chain through torch.compile under inductor's precision-cast emulation (the plain compile produced a non-finite backward on the address stage; the emulated form passed a full gradient census at three and eight arms), and the quiet term sampled on 4 abstention chunks per step instead of 16 at the same per-step weight. Relaunched 2026-09-27 03:18 UTC from the 106,000 resume state: 6.8 s per step at two arms (11.35 before), both cards at 23 GB; the 16/16 and 16/4 forms are compared at every checkpoint (task curve, quiet term, the checkpoint toggles, the arm gates). Stage 3 opened at 117,496 (2026-09-28 01:00 UTC) with the third arm attached: three arms, both cards near 24 GB. At that boundary the compiled adapter chain silently fell back to eager (torch's recompile budget, 8 per function by default, exhausted by the boundary evaluation's variable-length inputs under a static compile), so steps 117,496 to 122,008 ran at 10.5 s per step; amoe-lora 0.2.11 raises the budget inside the library and routes only the training shapes through the compiled chain, and the run was relaunched from the 122,000 checkpoint on it at 2026-09-28 14:26 UTC (about 7.5 s per step at three arms; code/README.md has the details). Stage 4 opened at 135,807 (2026-09-29 19:31 UTC) with the fourth arm attached on the compiled chain: zero recompile warnings through the boundary evaluation, about 8.3 s per step at four arms, the cards at 24.3 / 23.5 GB. The arms detached: at the 148,000 checkpoint (2026-10-01 00:35 UTC) every stage arm was detached and the trunk continues the curriculum alone. An effect-test battery over 13 checkpoints (v3_preflight/effect_tests/) read each arm's write on its own stage rows within ±0.0003 bpb even with its gates forced open, while a fresh arm of the same shape trained on the frozen 91,555 trunk took 64% of that stage's gain in 400 steps: the co-trained trunk had absorbed each stage before its arm could, and the arms cost about half of every step. The s1-s4 arm files ship through 148,000 under arms/; from the relaunch on the mission file's stage-arm list is empty (the resume state's arm entries are ignored by design, the trunk weights and optimizer states continue exactly), about 4.2 s per step, the cards at 21.6 / 21.5 GB. Stage 5 opened at 154,118 (2026-10-01 07:40 UTC) on the trunk alone: the boundary evaluation took three minutes without arm gauges, the arithmetic probe rose from .20 to .47 over its own stage while the logic probe eased from .33 to .20, and the run continues at about 4.1 s per step. Stage 6 opened at 169,377 (2026-10-02 01:15 UTC); at that close the causal probe held at .53 while the arithmetic probe eased from .47 to .27. Stage 7 opened at 181,585 (2026-10-02 15:13 UTC); at that close the web holdout read 0.026 above the stage 5 close, the largest stage cost since stage 0 (mini-beatrix-2s also paid the most for this stage after stage 0), while the stage's own learning probe moved from .30 to .33 and the difference probe eased from .63 to .40. Stage 8 opened at 202,948 (2026-10-03 16:11 UTC); at that close the web holdout read 0.102 below the stage 6 close, repaying the previous stage's cost four times over (mini-beatrix-2s's mixed stage 7 repaid its stage 6 the same way), while the difference probe rose from .40 to .53 and the arithmetic probe from .30 to .40. The anneal (web text without chat rows) opened at 215,156 (2026-10-04 07:01 UTC); at the stage 8 close the web holdout read 0.030 above the stage 7 close, the stage's diet pulling the shared weights away from web text as the tryfail stage had, while the self-other probe rose from .33 to .43 and the logic probe eased from .30 to .17. The mixed anneal (chat rows back in) opened at 230,415 (2026-10-05 00:35 UTC); at the no-chat half's close the web holdout read 0.083 below the stage 8 close and under one bit per byte for the first time since the pretraining phase, while the arithmetic probe eased from .37 to .23 and the causal probe rose from .57 to .60. The run finished at 245,674 (2026-10-05 18:02 UTC); at the close the web holdout read 0.009 below the no-chat half's close and 0.002 above the pretraining phase's close, while the grounding probe eased from .47 to .37 and the self-other probe rose from .43 to .50; the bank's toggle at the close read nearly twice its level at every earlier boundary, while the per-checkpoint reads of the same toggle ran +3.4 to +4.1 through the mixed half, so that reading is to be repeated on the final weights before it is leaned on. Live: the chat Space serves the newest checkpoint (its stage arms end at 148,000). Shipped automodel: mini-beatrix-3 (the final weights as a loadable package, bf16 and fp32). Pre-mission screens under mini-beatrix-3/v3_preflight/ (the depth ladder, the fusion screen, the head screen, the anneal-rate screen, the guard certification, the coverage audit, the minted lexicon); the Colab form of the mission is mini-beatrix-3/beatrix_v3_colab.ipynb.

• mini-beatrix-2s-control — COMPLETE (2026-09-03). The softmax twin of mini-beatrix-2s: the identical preset with the hub list empty (16-head causal SDPA in every one of the 20 blocks; 257.4M vs the craft's 237.1M), the identical recipe and diet, 16.101B tokens / 61,422 steps, one craft per arm, under a pre-registered protocol whose headline is the endpoint validation. The endpoint (anneal_mix @61,422, same step, same eval): splat 1.1097 vs sdpa 2.8846 bpb, +1.775 sdpa deficit. The crossover holds at 11 of 11 same-gauge paired rows: the twin led the open (0.3B: sdpa 2.106 vs splat 2.363) and trailed from the fineweb boundary on (5.3B: splat 1.1172 vs sdpa 1.1763, +.059; s0 +.253; s1 +.137; s2 +.176; s3 +.300; s4 +.333; s5 +.926; s6 +1.926; s7 +1.932; anneal_nochat +1.812; anneal_mix +1.775). The s8 pair (1.0737 vs 3.0810) is struck: the two rows were measured on different gauges (the craft's on the pre-0.8.5 synthetic fallback, the control's on the fineweb-edu holdout). Every other row — every curriculum stage and both anneals — is on the one fineweb-edu holdout, so the control's slide is a single comparable series: 1.18 (fineweb) → 1.57 (s4) → 2.18 (s5) → 3.19 (s6) → 3.12 (s7) → 2.93 → 2.88. The control is a DESTABILIZED model at its endpoint, not a trailing one. Its pre-clip gradient norm (train/grad_norm, clip 1.0) first crossed the clip at step 17,600 — inside the fineweb phase, on the RTX PRO 6000, while it still led — then escalated stage by stage: medians 0.29 on fineweb → 6.5 (s0) → 2.4 → 12 → 13 → 24.5 → 126 (s5; nine steps above 1,000, max 11,877) → 471 (s6; max 41,070) → 162 → 145 → 167 → 215 (anneal_mix), clipped on 813 of 824 logged curriculum and anneal steps, while the craft's raw norm never reached the clip (0 of 824; max 0.62). The anneal is a DIET change (1B of prose, then 1B with the chat frame), not a learning-rate decay: train/lr_scale read 1.000 first to last on both crafts. Bank dependence at the end: control toggle_bank_off 6.11 vs the craft's 3.31; the control's born-null aleph head stayed buried at every row. Grade: an INSTANCE at one craft per arm; the protocol completed unaltered. The claim that SDPA is not an adequate representative of this model's attention requirement is AbstractPhil's (2026-09-02); its audit (2026-09-04) minted the specific form — under this recipe (Muon 2e-2 / Adam 3e-4 flat, clip 1.0, no QK-norm, no weight decay, no LR decay) the softmax twin is a destabilized control and the endpoint gap measures stability, not capacity — and holds the general form as a CANDIDATE behind five confounds: the recipe (the documented softmax logit-growth instability, fixed in the literature by a normalization or a decay), one seed with 20.3M extra parameters in the losing arm, the shape (mini-beatrix-1, thirteen softmax blocks and three hubs, trained clean under the identical configuration: post-warmup grad-norm max 0.88, no clipped step), the aleph-addressed expert banks present in both crafts (the control's dispatch entropy collapsed before its gradients did), and the learned position table on decay-free Muon. Provenance: session caps at 22,854 and 53,792 (reports/v2/session_cap_step*.json) — the first resume landed on a rare H100 (s0–s8 rows), a third session from the 56,000 checkpoint ran the two anneal rows on another card; the escalation began before any resume. Fifteen reports under mini-beatrix-2s-control/reports/v2/ pair with the craft's by stage and step; the paired chart is article_assets_ft2/01_control_crossover.png.

• mini-beatrix-2s — COMPLETE (2026-08-31). 16.101B tokens / 61,422 steps on one card: warmup + fineweb 5B + curriculum s0-s8 + two-phase anneal (distribution shift without the chat frame, then with it). Final val 1.1097 bpb (run best, set in the chat phase); zero loss spikes; 17 boundary/session reports under mini-beatrix-2s/reports/v2/. Shipped automodel: mini-beatrix-2s. Notable ops in the record: the first BOUNDARY-WRITE head revival (reports/v2/revival_step24860.json) and the archived pre-specials run0 (mini-beatrix-2s/archive/run0-nospecials/).

• mini-beatrix-1 — COMPLETE. The 3-hub hybrid rung; automodel.

The training code for mini-beatrix-3

mini-beatrix-3/code/ is a verbatim snapshot of the two libraries at the commits that train the run — geolip.alephllm 0.10.5 (alephllm @ e8b7440) and the arm library amoe 0.2.10 (amoe-lora @ f3db5d4) — plus the pod's mission file, the launch / relaunch / clean-stop scripts, the arm-shipping sidecar and the pinned package versions. Its README.md walks the knobs (abstain_chunks, adapter_recompute, adapter_compile, arm_spec_certified, stage_arms), the abstention term, and how to read an arm file. The trainer is resume-first: a fresh session pulls manifest.json + resume/latest.pt from this repo and continues.

The mission ladder

Each craft has a *-control twin (identical minus the aleph attention layers) — the running architecture control. For the 2s rung the twin is mini-beatrix-2s-control (the same preset with hub_layers: [], so 16-head causal SDPA in every block; 257.4M vs the craft's 237.1M), run to the same 16.101B-token endpoint under a pre-registered protocol — see the mission log; its paired boundary reports sit under mini-beatrix-2s-control/reports/v2/. Training is resume-first: sessions pull manifest.json + resume/latest.pt and continue where the last one stopped (the 1 and 2s rungs on Colab, the RTX 6000 Pro 96GB; the v3 rung on a two-card RTX 5090 pod, data-parallel).

The architecture in one paragraph (the mini-beatrix-1 era)

The 2s and v3 rungs differ: mini-beatrix-2s carries a CausalSplatHUB in every one of its 20 blocks and no softmax attention; its control twin carries 16-head SDPA in every block and no hubs (hub_layers: [] in its manifest); mini-beatrix-3 carries a four-constellation hub and an anchored bank in every one of its 32 blocks, a revival-born head, and the detachable stage arms. See the mission log, mini-beatrix-3/code/README.md, and the model card of mini-beatrix-2s.

Trigram byte embedding (dedicated pad row) → pre-norm stack of standard causal SDPA plus three CausalSplatHUB layers (causal linear attention through a 2K-half-axis signed address, exact chunked scan) → per-layer anchored FFN bank (always-on trunk + 3 dispatched experts, expert outputs zero-init so the dispatch is born contributing exactly zero, gates σ(−3), no balance machinery) → dual head whose aleph read enters at γ=0 and must earn its way in by gradient. Muon on transport weights + pure Adam elsewhere; flat LR; bf16 autocast over fp32 masters; fp8 is a shipping format, never a training format.

Reading the instruments

TensorBoard carries the full born-in gauge suite: per-layer hidden-state effective rank, consumed-address erank per hub layer, coefficient-of- variation load analysis per bank, sign census, gate/γ trajectories, anchor drift, denominator health, structural collapse flags (anchor merging, dispatch-entropy collapse, erank floor, loss spikes), canary recall (clean-protocol in-context binding probes), and the toggle ledger — bpb deltas with each aleph mechanism switched off, the causal record of what the addresses actually contribute. For mini-beatrix-3 the arms/abstention scalar is the quiet term (the live arms' KL per position on held-out text) and toggles/toggle_arms_off is the held-out cost of switching every arm off — the arms' off-domain footprint (2e-6 to 5e-6 bpb so far).

Related record: the attention-side measurement campaign lives at aleph-splat-0; the encoder-side anchored-bank record at alephlm-0.