solomonoff-paper

Self-Play Pretraining with Zero Data: Checkpoints

Model checkpoints for the paper Self-Play Pretraining with Zero Data.

Two randomly initialized transformers are trained in tandem: a generator proposes programs for a minimal universal Turing machine, and a learner is trained by next-token prediction on the executed byte sequences. No natural data is used at any point during training. These checkpoints are the learners from that process, released so that every result in the paper can be recomputed from the weights.

All models are byte-level (vocabulary 256) decoder-only Llama-style transformers with a 4096-token context.

Scaling ladder

The main self-play ladder, six model sizes. Learner weights are saved every 256 self-play rounds; sizes refer to non-embedding parameter counts.

Layout:

<size>/seed-<seed>/
    config.json               model architecture
    learner_<round>.pth       learner weights at a training round

Each learner_*.pth file is a dictionary with the training round (round) and the learner weights (learner_state_dict). Only ladder seeds that completed at least 8192 self-play rounds are included. Generator weights, optimizer states, and training code are not part of this release.

Baselines and ablations

Usage

import json, torch
from huggingface_hub import hf_hub_download

repo = "nourya-cohen/solomonoff-paper"
cfg = json.load(open(hf_hub_download(repo, "24M/seed-40354564/config.json")))
blob = torch.load(hf_hub_download(repo, "24M/seed-40354564/learner_8191.pth"),
                  map_location="cpu", weights_only=False)
state_dict = blob["learner_state_dict"]

The model class, evaluation code, and per-figure analysis scripts are in the companion repository: nourya-aliz/Solomonoff-Figures. Its scoring/ directory evaluates these checkpoints on the paper's byte-level benchmark suite end to end.

Citation

Citation entry to be added on publication.