beatrix-tokenizers

Beatrix tokenizers: surface arms for mini-beatrix-3

Beatrix (mini-beatrix-3, a 376M-parameter byte-level language model) reads raw UTF-8 bytes. A tokenizer-based model hands text around in its own spelling: a byte-level BPE such as Qwen3's stores every byte as a printable stand-in character, so a space becomes the two bytes of 'Ġ' and ' taco' becomes the one token 'Ġtaco'. Beatrix reads that spelling as a different text from the plain bytes, and the difference grows with depth.

A surface arm is a small detachable adapter (13.7M parameters, one module after each of her 32 blocks) trained so that she reads a spelling the way she reads the plain bytes: at each token's closing byte her state on the spelled surface is pulled toward her own state on the plain surface at the same token, while a quiet term keeps her reading of plain bytes, web text and the other arms' subjects unchanged. Her plain-byte states are the teacher; no external model's states are involved. The arm is tied to a spelling convention, not to a model: every tokenizer that uses GPT-2's byte map (Qwen 2.5 and 3 of every size, Llama 3, GPT-2 and kin) hands her the same bytes for the same text.

Layout

surface/<convention>/<run>/s10_qwen.safetensors   the arm (amoe anchor format; base_model_id alephllm/mini-beatrix-3@step245674)
surface/<convention>/<run>/stats.pt                the loss's standardization statistics (for the record)
surface/<convention>/<run>/results.json            every read of the run: the bars at every block, both caption draws, the curve
surface/<convention>/<run>/train.log               the run's log
surface/<convention>/<run>/closes/                 the arm at every close of the run
surface/<convention>/<run>/readout.md              the full readout (when present)

The byte-map convention is exact (the spelling read back through the map gives the text's bytes). The other three are lossy (case folding, re-spacing, normalization), so each token's closing byte on the plain surface comes from the tokenizer's offset mapping (the byte after the token's span in the text as written); the arm's registry row records the rule.

Runs named ..._shuf are the shuffled-pairing control (an arm trained against the wrong targets: a control that must fail), solo runs are trained on the bare trunk, untrained runs on a random-initialization copy of the trunk. The arms trained over the eight stage arms expect those arms mounted first; they live on the training repository (AbstractPhil/alephllm-mini-beatrix-training).

Use

from geolip.alephllm import load_trunk
from geolip.alephllm.arm_mount import mount_surface, masked
from geolip.alephllm.train.surface import read_spelled
from transformers import AutoTokenizer

model = load_trunk(245674, device="cuda")              # the trunk at its final step
prog = mount_surface(model, "qwen3")                   # the eight stage arms, then the surface arm, by the training route
tok = AutoTokenizer.from_pretrained(prog.surface["tokenizer"])
kw = reader_kwargs(prog.surface)                       # the arm's convention, its A-text rule and its site set
r = read_spelled(model, tok, ["a taco truck parked by the sea"], blocks=[12, 16, 20, 24], **kw)
r["states"][20]      # her states on the spelling at every token's closing byte (LayerNorm'd), one row per site
r["sites"]           # (text index, token position, token id) per row: the rows line up token by token with the tokenizer's ids
with masked(prog, [prog.surface["member"]]):           # the same text read as plain bytes, the arm off
    plain = read_spelled(model, tok, ["a taco truck parked by the sea"], blocks=[20], surface="A", **kw)

The registry names: qwen3, qwen3-B, qwen3-solo, qwen3-shuf, qwen3-untrained, qwen3-bytes, t5, t5-solo, t5-B, clip, clip-B, bert, bert-B (the family's rows but t5 are the solo runs: nothing to mount underneath) (arm_mount.SURFACE_ARMS; a name whose files are not on the repository yet fails at download). The untrained-copy control mounts on a random-initialization copy of the trunk (seed 0), never on the trunk.

Install: pip install "geolip-alephllm[train] @ git+https://github.com/AbstractEyes/alephllm@feat/qwen-surface-arm" (the branch that carries the surface module until its release) and pip install "amoe-lora @ git+https://github.com/AbstractEyes/amoe-lora".

Reads

Every run was read, before training and at every thousand steps, on held-out captions the arm never trained on (512 of each of two draws; the second draw repeats the first), one caption a row, at every token's closing byte, as the whitened Procrustes alignment between two readings (1 = the same geometry up to a rotation, 0 = unrelated):

• the gap: her reading of the spelling through the arm against her reading of the plain bytes with the arm off (the target), per block;
• silence: her reading of the plain bytes with the arm on against with it off, per block, beside the cost in bits per byte on web text and on the stage arms' subjects.

The readout files carry every number; the results files carry every block of every close.

Status

The bar for the gap was .85 at every served block from 12 on; the bar for silence .995 at every block beside +.012 bits per byte. The MSE form is the one carried forward.