compute-multipliers-checkpoints

Pretraining progress is mostly coming from data: checkpoints, data and code

The trained models, evaluation data and code behind Pretraining progress is mostly coming from data (Dwarkesh Patel and Jerry Han, 2026).

How much of the progress in pretraining from 2019 to 2025 has come from data versus model improvements? For each year we take a year-representative open model recipe (GPT-2 to OLMo-2) and a year-representative public data corpus (OpenWebText to Ultra FineWeb), and train combinations of them from scratch at compute budgets up to 1e19 FLOPs. We evaluate the models on OLMES, which aggregates 10 relatively easy benchmarks, mostly multiple-choice QA.

Update to the blog post. The blog counted OLMo-2's compute with the parameter count of its published model ladder, which also includes the output layer; every other recipe counts only the non-embedding parameters. Counted the same way for every recipe, the OLMo-2 runs used 13 to 65% less compute than their labels, so the model improvement is larger than the blog reported: 5.0x rather than 3.7x. The data improvement (12.0x) is unchanged, and data still accounts for most of the gain. Everything on this page uses the consistent count. The intervals here come from resampling the training seeds rather than the blog's parametric bootstrap, so they differ slightly from the blog's.

Results

• From 2019 to 2025, at 1e19 FLOPs, upgrading the data (OpenWebText to Ultra FineWeb) saves 12.0x [10.4, 13.7] compute, upgrading the model (GPT-2 to OLMo-2) 5.0x [4.1, 5.9], and both together 21.1x [16.4, 26.7]. Data improvements gave 2.4x [2.3, 2.5] more compute efficiency than model improvements.
• Year over year, the compute efficiency gain is 1.31x [1.27, 1.34] on the model side and 1.51x [1.48, 1.55] on the data side; measured jointly, 1.66x [1.59, 1.73].
• The gains from data and model improvements are mostly independent: on the grid of recipes × corpuses at 3.16e18 FLOPs, an additive model explains 89% of the variance in the OLMES score.

Brackets are 68% intervals over the 27 paired resamples of the three training seeds. python reproduce/findings.py recomputes every number above from this repository.

What is in this repository

Every run behind the results and figures: 267 runs, 259 with checkpoints.

Sets share runs (82 runs are in more than one). Eight learning-rate check runs (GPT-J, GPT NeoX, LLaMA and LLaMA 3 at 1e18) have no checkpoint because their weights were not kept; their records and scores are included.

checkpoints/<run_id>/model.safetensors   weights (float32)
checkpoints/<run_id>/run.json            the run's record: architecture, learning rate, tokens, loss curve
checkpoints.csv                          one row per run: set, recipe, corpus, budget, size, parameters, tokens,
                                         compute, seed, learning rate, held-out loss, OLMES and its ten task scores
data/curve_points.csv                    which runs make up each plotted point and grid cell
data/olmes_items.csv, data/olmes_item_outcomes.npz   per-item OLMES outcomes of every run
reproduce/                               findings.py, verify_scores.py, figures.py (CPU, seconds)
training/                                model and training code, recipes, corpus configs
load_example.py                          rebuilds a checkpoint

In checkpoints.csv, n_params is the non-embedding parameter count (attention and MLP weight matrices) and flops = 6 × n_params × tokens. n_params_planned is the count each run's token budget was set from; it equals n_params except for OLMo-2.

Reproducing the findings

pip install numpy matplotlib
python reproduce/findings.py        # every number in Results, checked against this page
python reproduce/verify_scores.py   # recomputes all 267 OLMES scores from the per-item outcomes
python reproduce/figures.py         # redraws the four figures

training/ retrains any run: training/README.md tokenizes a corpus from its pinned source (all but The Pile, whose document-recovery step is not included) and trains one configuration, and its demo rebuilds the exact specification of a checkpoint in this repository. Re-running the OLMES evaluation itself requires the public OLMES suite; the per-item outcomes let you check every score without it.

Methodology

• Compute budgets are 1e17, 3.16e17, 1e18, 3.16e18 and 1e19 FLOPs. Compute is C = 6ND, with N the non-embedding parameters counted the same way for every recipe and D the training tokens. Each run's tokens were set from its recipe's planned parameter count; OLMo-2's ladder also counts its output layer, so OLMo-2 runs sit 13 to 65% below their labelled budget, and the figures plot them where they are.
• At each budget we vary the number of parameters and keep the compute-optimal size, the one with the lowest held-out loss on the training corpus. Under a fixed recipe it barely changes with the corpus, so most corpuses reuse the size chosen on FineWeb Edu; GPT-2 on OpenWebText has its own sizes, and OLM CC 2022 uses the next larger size at 1e19.
• Every run shares the GPT-2 BPE tokenizer (50,257 vocab), a 2,048-token context and 262,144-token batches.
• Peak learning rate is swept at 5 anchor points (3 model sizes and 2 D/N ratios) and fit as lr(N, D/N) = lr₀ · (N/N₀)^a · ((D/N)/(D/N)₀)^b, with a common a and b and a recipe-specific lr₀; OLMo-2 starts from the rates prescribed by Ai2's model ladders. A second sweep around these rates adopted a better rate wherever three seeds confirmed it: GPT-J, GPT NeoX and LLaMA 3 at 1e18, GPT NeoX at 3.16e18, and LLaMA and OLMo-2 at 1e19. LLaMA at 1e18 kept its rate; two of its seeds there are re-runs at that rate (lrcontrol in their IDs).
• OLMES is scored on the full evaluation split of every task; a task's score is the better of its cloze and multiple-choice formulations, with MMLU averaged over its 57 subjects first.
• Every point uses three seeds (6198, 6199, 6200); repeated runs of a seed are averaged first. The grid uses one seed per combination except 15 cells.
• A compute multiplier is the ratio of the reference's compute to the compute at which the candidate's scaling curve first reaches the reference's score (interpolating linearly in log compute).

Caveats

• Scale: these are small models (up to 1e19 FLOPs). Scale-dependent gains, such as OLMo-2's layer and QK norms, and inference optimizations such as LLaMA 3's GQA do not show up as compute multipliers here. Tokenizer improvements are not studied.
• The recipes and corpuses are representative of each year, not necessarily the best.
• The multipliers are for OLMES; other benchmarks could give very different numbers.
• Seed noise (a median standard deviation of 0.3 to 0.4 OLMES points) is comparable to the gaps between neighbouring recipes, so single year-to-year steps are uncertain; the 2019 to 2025 comparisons above are well outside it. GPT NeoX scores below GPT-2 at 1e19, and The Pile below OpenWebText, both by extrapolated crossings. GPT-3 had training instabilities on The Pile.
• These are base models trained on web text with no safety tuning. Do not deploy them.

Notes

• Every run.json carries experiment_stage: "nonreportable_pilot" and reportable: false, and run IDs start with pilot-. These are the names of the launch path every training run went through, not a verdict on the run: every checkpoint here is a completed run.
• run.json fields: n_nonembed_block_matrix is n_params; n_compute, n_nonembed, n_source and n_compute_campaign are all n_params_planned; n_realized_trainable_excluding_wte counts every trainable parameter except the token embedding. flops_6nd_block_matrix is flops; flops_6nd is 6 × n_params_planned × tokens (the labelled budget); flops_exact adds the output layer and the attention-score matmuls. curve and extra_eval are (tokens, held-out loss) pairs on the training corpus; extra_eval_bins names the held-out split evaluated; wall_sec is the run's wall-clock time on one GPU.

Quickstart

import sys
from huggingface_hub import snapshot_download

rid = 'pilot-prod-w-v2025-olmo2-control-ultra-fineweb-en-300m-seed6198-700564379b14'  # OLMo-2 on Ultra FineWeb, 1e19
root = snapshot_download('j23h67/compute-multipliers-checkpoints',
                         allow_patterns=[f'checkpoints/{rid}/*', 'training/*', 'load_example.py'])
sys.path.insert(0, root)
from load_example import load
model, spec = load(f'{root}/checkpoints/{rid}')

Requires torch, safetensors and pyyaml. Models use the GPT-2 tokenizer (tiktoken.get_encoding('gpt2')) and a 2,048-token context.

Licence and citation

Weights, run records, data files and code: Apache-2.0. The corpuses were used under their upstream terms and no corpus text is included. The Pile and OLM CC 2022 have redistribution restrictions; check their terms before using the models trained on them beyond research.

@misc{patel_han_2026_pretraining_data,
  title  = {Pretraining progress is mostly coming from data},
  author = {Dwarkesh Patel and Jerry Han},
  year   = {2026},
  url    = {https://www.dwarkesh.com/p/pretraining-progress-is-mostly-data}
}