FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from FineWeb dataset. This is the 1.3 trillion version. To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We then used this classifier to retain only the most educational web pages. FineWeb-Edu outperforms FineWeb on popular benchmarks and shows the power of classifiers trained on synthetic data. The Dataset Curation section details the process for creating the dataset. You can find a deduplicated version of FineWeb-edu in SmolLM-Corpus. We find that the deduplication of this dataset doesn't have…
Publicly accessible
odc-by
n>1T
The FineWeb dataset consists of more than 18.5T tokens (originally 15T tokens) of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the datatrove library, our large scale data processing library. FineWeb was originally meant to be a fully open replication of RefinedWeb, with a release of the full dataset under the ODC-By 1.0 license. However, by carefully adding additional filtering steps, we managed to push the performance of FineWeb well above that of the original RefinedWeb, and models trained on our dataset also outperform models trained on other commonly used high quality web datasets (like C4…
Publicly accessible
odc-by
n>1T
Synthetic data generated by DataTrove: The finalized run produced 1,354,044,711 (≈1.35B) samples and generated 486,367,076,933 (≈486.4B) completion tokens. Final counts were computed from generated parquet outputs using examples/inference/countcompletiontokens.py and the runs in projects/datatrove/finephrasetokencounts//slurm/stats.json. Each sample includes standard fields such as: - text (source input text from FineWeb-Edu, not the generated output) - rolloutresults (list of generation result objects; one per rollout) - finishreason - text (generated transformed output; for single-rollout runs this is in rolloutresults[0].text) - usage - completiontokens - prompttokens…
Publicly accessible
odc-by
n>1M
FineWeb2 A sparkling update with 1000s of languages What is it? This is the second iteration of the popular FineWeb dataset, bringing high quality pretraining data to over 1000 languages. The FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments. In particular, on the set of 9 diverse languages we used to guide our processing decisions
Publicly accessible
odc-by
n>1T
