FineWeb2 A sparkling update with 1000s of languages What is it? This is the second iteration of the popular FineWeb dataset, bringing high quality pretraining data to over 1000 languages.
Rows17,264
Configurations12
Sizen>1T
Licenseodc-by
AccessPublicly accessible
Monthly Downloads95.9k
Dataset Card
FineWeb2 A sparkling update with 1000s of languages What is it? This is the second iteration of the popular FineWeb dataset, bringing high quality pretraining data to over 1000 languages. The FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments. In particular, on the set of 9 diverse languages we used to guide our processing decisions
Excerpt from the card by FineData, licensed odc-by.
Structure
aai_Latn 623 rows
| Split | Rows | Size |
|---|---|---|
| test | 8 | 28.3 KB |
| train | 615 | 2.9 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
aak_Latn 1,568 rows
| Split | Rows | Size |
|---|---|---|
| test | 8 | 34.7 KB |
| train | 1,560 | 4.7 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
aau_Latn 2,015 rows
| Split | Rows | Size |
|---|---|---|
| test | 23 | 35.5 KB |
| train | 1,992 | 2.9 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
aaz_Latn 2,584 rows
| Split | Rows | Size |
|---|---|---|
| test | 25 | 51.2 KB |
| train | 2,559 | 3.5 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
aba_Latn 199 rows
| Split | Rows | Size |
|---|---|---|
| test | 1 | 1.6 KB |
| train | 198 | 759.1 KB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
abk_Cyrl 2,696 rows
| Split | Rows | Size |
|---|---|---|
| test | 26 | 231.7 KB |
| train | 2,670 | 23.7 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
abq_Cyrl 980 rows
| Split | Rows | Size |
|---|---|---|
| test | 12 | 106.6 KB |
| train | 968 | 8.0 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
abs_Latn 213 rows
| Split | Rows | Size |
|---|---|---|
| test | 3 | 9.2 KB |
| train | 210 | 586.7 KB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
abt_Latn 4,131 rows
| Split | Rows | Size |
|---|---|---|
| test | 33 | 54.8 KB |
| train | 4,098 | 7.6 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
abx_Latn 918 rows
| Split | Rows | Size |
|---|---|---|
| test | 9 | 9.6 KB |
| train | 909 | 2.2 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
aby_Latn 1,085 rows
| Split | Rows | Size |
|---|---|---|
| test | 5 | 4.3 KB |
| train | 1,080 | 2.2 MB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
abz_Latn 252 rows
| Split | Rows | Size |
|---|---|---|
| test | 1 | 3.3 KB |
| train | 251 | 535.2 KB |
textstringidstringdumpstringurlstringdatestringfile_pathstringlanguagestringlanguage_scorefloat64language_scriptstringminhash_cluster_sizeint64top_langsstring
Details
- Repository
- HuggingFaceFW/fineweb-2
- Publisher
- FineData
- Task category
- Text generation
- Tags
- Not stated by the source
- Size category
- n>1T
- Languages
- aai, aak, aau, aaz, aba, abi, abk, abn
- Revision
- af9c13333eb981300149d5ca60a8e9d659b276b9
- Last updated
- 2025-10-27
License and Download
- License
- odc-by
- Access
- No access gate
Download from FineData
Released by FineData through its official repository on Hugging Face.