beetle-phase-transition

Induction-head phase transition in BeetleLM

Per-checkpoint mechanistic metrics for Beetle language models, tracking how the induction circuit forms during training.

The files here have six different schemas, so they are exposed as separate configs. Loading the directory as a single table fails with a cast error — that is why the configs above exist, not a bug.

from datasets import load_dataset
traj = load_dataset("suchirsalhan/beetle-phase-transition", "trajectories")
abl  = load_dataset("suchirsalhan/beetle-phase-transition", "ablation")

Metrics

repeat_dependence is the column that matters for deciding whether a high-PS head is really an induction head: candidates score ~0.7, every other head ~0.0003. A head that attends "somewhere earlier" can beat chance on PS alone without needing the repeat.

Main findings

A phase transition. PS sits at the chance floor (0.028, which is exactly uniform causal attention at the mean query position) until ~step 8k, then rises sharply to 0.64–0.84 with 4–9 heads crossing threshold.

It is not about curriculum, or about language. Onset varies little across the five bilingual curricula, and monolingual models in Dutch, English, German, Italian and Chinese all show it at a comparable budget — so the transition is a property of optimisation and architecture, not of bilingual training.

The circuit is functional, not merely present. Ablating the top-3 induction heads takes associative recall from 0.052 to 0.001; ablating three entropy-matched control heads leaves it at 0.057.

A three-stage cascade. Previous-token heads form first (~step 1.5–2k), induction ~7.5k steps later, functional associative recall ~17k steps after that. The 2B models terminate inside stage one, which is why they show previous-token structure but never induction.

Caveats

• The 2B checkpoint ladder ends at steps 4096–7127, below the transition. Its flat PS is a truncation, not evidence of absence.
• Monolingual models use the fineweb3 corpus build; language and corpus are confounded in that comparison.
• Onsets one log-grid step apart (8192 vs 16384) are not resolvable at 20 checkpoints.
• No seeded runs exist at 24B or 2B; seeds are HumanScale-only, for B2/B3.
• Two scales only, so token budget and model scale are not separable.