The trained models, evaluation data and code behind Pretraining progress is mostly coming from data (Dwarkesh Patel and Jerry Han, 2026). How much of the progress in pretraining from 2019 to 2025 has come from data versus model improvements? For each year we take a year-representative open model recipe (GPT-2 to OLMo-2) and a year-representative public data corpus (OpenWebText to Ultra FineWeb), and train combinations of them from scratch at compute budgets up to 1e19 FLOPs. We evaluate the models on OLMES, which aggregates 10 relatively easy benchmarks, mostly multiple-choice QA. Update to the blog post. The blog counted OLMo-2's compute with the parameter count of its published model…
Independent publisher
Jerry
j23h67
Models in Library1
Datasets in Library0
Models on Hugging Face2
Followers—
Compare the published evidence
This profile connects the 1 model and dataset record shown below to their upstream cards and sources. The library counts describe SAVRN's coverage, rather than this publisher's entire output.
Reported licenses: apache-2.0. Open a record to review its specific license and conditions.
Compare model specifications, source dates, and hardware estimates on the individual records. Publisher identity verification is separate from a model's quality, deployment readiness, or suitability for your work.