Training text, token files and checkpoints of small (124-million-parameter) Llama-architecture language models trained on FineWeb.
Rows1,234,391
Configurations1
Size107.8 GB
Licenseodc-by
AccessPublicly accessible
Monthly Downloads—
Dataset Card
By Liangzhi Li, published under odc-by, revision aedb027944dc.
FineWeb 124M pretraining material
Training text, token files and checkpoints of small (124-million-parameter) Llama-architecture language models trained on FineWeb.
| Folder | Contents |
|---|---|
corpora/fineweb/ |
Shards of FineWeb as released (fields text, id, url, dump, date, language_score), named by Common Crawl dump: dumps of 2019–2021 and of 2024–2025. |
corpora/cosmopedia/ |
Shards of Cosmopedia. |
tokens/gpt2/ |
fineweb_2019-2021.bin: the 2019–2021 FineWeb shards tokenized with the GPT-2 BPE; cosmopedia.bin: the Cosmopedia shards tokenized likewise; streams/: further token streams of FineWeb 2019–2021 text. Every file is a flat array of uint16 token ids; in fineweb_2019-2021.bin and cosmopedia.bin every document ends with token 50256. fineweb_2019-2021.meta.jsonl.gz gives, per document, its position, token count and FineWeb id, url and dump. |
tokens/gpt-neox/streams/ |
Token streams of FineWeb 2019–2021 text with the GPT-NeoX tokenizer (as used by Pythia), uint16. |
checkpoints/llama-124m/base/seed1–seed8/ |
Models trained from scratch on 1.5 billion tokens of FineWeb 2019–2021, one per seed (final/, and for seeds 1–4 also after about 495 and 1,005 million tokens). |
checkpoints/llama-124m/runs/run_NNN/ |
Further training runs of the same architecture (weights only). |
Checkpoints
Every checkpoint folder holds model.safetensors (float32) and config.json for transformers'
LlamaForCausalLM: 12 layers, hidden size 768, 12 attention heads, MLP size 2048, context 2048, vocabulary 50,304
(the GPT-2 vocabulary padded), RMSNorm, rotary embeddings (θ = 10,000), input and output embeddings tied.
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer, LlamaForCausalLM
sub = "checkpoints/llama-124m/base/seed1/final"
d = snapshot_download("Liangzhi-Li/fineweb-124m-pretraining", repo_type="dataset", allow_patterns=f"{sub}/*")
model = LlamaForCausalLM.from_pretrained(f"{d}/{sub}")
tok = AutoTokenizer.from_pretrained("gpt2")
Training of the base models: AdamW, learning rate 6e-4 with 300 warm-up steps and cosine decay to 6e-5, weight decay 0.1, gradient clipping 1.0, 524,288 tokens per step, 2,861 steps.
Licences
FineWeb: ODC-By 1.0, subject to the Common Crawl terms of use. Cosmopedia: Apache-2.0. Token files derived from them carry the same terms. Checkpoints: CC BY 4.0.
Structure
default 1,234,391 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,234,391 | 5.0 GB |
textstringidstringsrc_configstring
Details
- Repository
- Liangzhi-Li/fineweb-124m-pretraining
- Publisher
- Liangzhi Li
- Task category
- Not stated by the source
- Tags
- fineweb, pretraining, language-modeling
- Size category
- 1B<n<10B
- Languages
- en
- Revision
- aedb027944dca4b9a29a22fc089fe029d199fcb1
- Last updated
- 2026-10-16
Files
377 files, 107.8 GB in total.
Data215 files · 27.0 GB
Documentation1 file · 2.7 KB
Other160 files · 80.8 GB
Repository1 file · 2.5 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| checkpoints/llama-124m/base/seed1/final/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed1/tokens_0495M/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed1/tokens_1005M/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed2/final/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed2/tokens_0495M/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed2/tokens_1005M/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed3/final/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed3/tokens_0495M/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed3/tokens_1005M/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed4/final/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed4/tokens_0495M/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed4/tokens_1005M/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed5/final/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed6/final/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed7/final/config.json | Data | 550 B | — |
| checkpoints/llama-124m/base/seed8/final/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_001/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_002/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_003/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_004/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_005/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_006/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_007/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_008/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_009/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_010/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_011/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_012/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_013/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_014/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_015/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_016/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_017/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_018/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_019/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_020/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_021/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_022/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_023/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_024/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_025/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_026/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_027/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_028/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_029/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_030/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_031/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_032/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_033/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_034/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_035/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_036/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_037/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_038/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_039/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_040/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_041/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_042/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_043/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_044/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_045/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_046/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_047/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_048/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_049/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_050/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_051/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_052/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_053/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_054/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_055/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_056/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_057/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_058/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_059/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_060/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_061/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_062/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_063/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_064/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_065/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_066/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_067/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_068/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_069/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_070/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_071/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_072/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_073/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_074/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_075/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_076/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_077/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_078/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_079/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_080/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_081/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_082/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_083/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_084/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_085/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_086/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_087/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_088/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_089/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_090/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_091/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_092/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_093/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_094/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_095/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_096/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_097/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_098/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_099/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_100/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_101/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_102/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_103/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_104/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_105/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_106/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_107/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_108/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_109/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_110/config.json | Data | 550 B | — |
| checkpoints/llama-124m/runs/run_111/config.json | Data | 550 B | — |
| corpora/cosmopedia/train_0000.jsonl.gz | Data | 366.3 MB | b1d94e6f48db |
| corpora/cosmopedia/train_0001.jsonl.gz | Data | 366.2 MB | 991e037ca96e |
| corpora/cosmopedia/train_0002.jsonl.gz | Data | 366.2 MB | fef2675db737 |
| corpora/cosmopedia/train_0003.jsonl.gz | Data | 367.4 MB | 78e581d9d10f |
| corpora/cosmopedia/train_0004.jsonl.gz | Data | 367.4 MB | 3b2603250504 |
| corpora/cosmopedia/train_0005.jsonl.gz | Data | 367.5 MB | c728de1ebb5f |
| corpora/cosmopedia/train_0006.jsonl.gz | Data | 386.4 MB | d8d7ace81209 |
| corpora/cosmopedia/train_0007.jsonl.gz | Data | 386.5 MB | 561c65bda09f |
| corpora/cosmopedia/train_0008.jsonl.gz | Data | 386.6 MB | ea78b8a93cf2 |
| corpora/fineweb/CC-MAIN-2019-04_0000.jsonl.gz | Data | 170.2 MB | c7c8e7df448e |
| corpora/fineweb/CC-MAIN-2019-09_0000.jsonl.gz | Data | 169.9 MB | f5b8b188cf50 |
| corpora/fineweb/CC-MAIN-2019-13_0000.jsonl.gz | Data | 169.5 MB | 309954e371aa |
| corpora/fineweb/CC-MAIN-2019-18_0000.jsonl.gz | Data | 169.5 MB | 8197b66579d1 |
| corpora/fineweb/CC-MAIN-2019-22_0000.jsonl.gz | Data | 169.8 MB | 371e3e7c5d11 |
| corpora/fineweb/CC-MAIN-2019-26_0000.jsonl.gz | Data | 169.5 MB | d03438537f6f |
| corpora/fineweb/CC-MAIN-2019-30_0000.jsonl.gz | Data | 169.6 MB | 66fbf6555114 |
| corpora/fineweb/CC-MAIN-2019-35_0000.jsonl.gz | Data | 169.1 MB | 2531337719dc |
| corpora/fineweb/CC-MAIN-2019-39_0000.jsonl.gz | Data | 169.3 MB | 4e5fffcfde9d |
| corpora/fineweb/CC-MAIN-2019-43_0000.jsonl.gz | Data | 169.2 MB | 70c65ff9ef65 |
| corpora/fineweb/CC-MAIN-2019-47_0000.jsonl.gz | Data | 169.2 MB | 352160107a56 |
| corpora/fineweb/CC-MAIN-2019-51_0000.jsonl.gz | Data | 169.4 MB | cb3e42b105e9 |
| corpora/fineweb/CC-MAIN-2020-05_0000.jsonl.gz | Data | 169.0 MB | 79c3baf2f833 |
| corpora/fineweb/CC-MAIN-2020-10_0000.jsonl.gz | Data | 169.6 MB | 23bb57f9d680 |
| corpora/fineweb/CC-MAIN-2020-16_0000.jsonl.gz | Data | 168.6 MB | 610fc856d4f1 |
| corpora/fineweb/CC-MAIN-2020-24_0000.jsonl.gz | Data | 169.5 MB | cbbde151f991 |
| corpora/fineweb/CC-MAIN-2020-29_0000.jsonl.gz | Data | 168.7 MB | 2b114b5828d5 |
| corpora/fineweb/CC-MAIN-2020-34_0000.jsonl.gz | Data | 168.7 MB | 2628f9bdd7d4 |
| corpora/fineweb/CC-MAIN-2020-40_0000.jsonl.gz | Data | 169.1 MB | b73ff9227a10 |
| corpora/fineweb/CC-MAIN-2020-45_0000.jsonl.gz | Data | 169.0 MB | 6b8ecb3f181f |
| corpora/fineweb/CC-MAIN-2020-50_0000.jsonl.gz | Data | 169.0 MB | 9f2638f08808 |
| corpora/fineweb/CC-MAIN-2021-04_0000.jsonl.gz | Data | 168.8 MB | 55dfd0b840fa |
| corpora/fineweb/CC-MAIN-2021-10_0000.jsonl.gz | Data | 168.2 MB | 0fe403338284 |
| corpora/fineweb/CC-MAIN-2021-17_0000.jsonl.gz | Data | 167.9 MB | 3ea73b55cde7 |
| corpora/fineweb/CC-MAIN-2021-21_0000.jsonl.gz | Data | 167.8 MB | 3c79fe268a82 |
| corpora/fineweb/CC-MAIN-2021-25_0000.jsonl.gz | Data | 167.9 MB | 4ccd9dbc4b14 |
| corpora/fineweb/CC-MAIN-2021-31_0000.jsonl.gz | Data | 167.8 MB | ce86ba28d232 |
| corpora/fineweb/CC-MAIN-2021-39_0000.jsonl.gz | Data | 167.9 MB | 2cb6dc9e662a |
| corpora/fineweb/CC-MAIN-2021-43_0000.jsonl.gz | Data | 167.9 MB | 4b9f1c3dcd66 |
| corpora/fineweb/CC-MAIN-2021-49_0000.jsonl.gz | Data | 168.0 MB | c25888968b74 |
| corpora/fineweb/CC-MAIN-2024-10_0000.jsonl.gz | Data | 408.0 MB | 3b44b3515baf |
| corpora/fineweb/CC-MAIN-2024-10_0001.jsonl.gz | Data | 407.8 MB | 0b8db5aac8fb |
| corpora/fineweb/CC-MAIN-2024-10_0002.jsonl.gz | Data | 331.3 MB | 725556ea00a2 |
| corpora/fineweb/CC-MAIN-2024-18_0000.jsonl.gz | Data | 407.9 MB | dad9a67e4784 |
| corpora/fineweb/CC-MAIN-2024-18_0001.jsonl.gz | Data | 407.5 MB | 2eac59efd83f |
| corpora/fineweb/CC-MAIN-2024-18_0002.jsonl.gz | Data | 331.1 MB | 41845c867e8d |
| corpora/fineweb/CC-MAIN-2024-22_0000.jsonl.gz | Data | 408.6 MB | 608ae51a1095 |
| corpora/fineweb/CC-MAIN-2024-22_0001.jsonl.gz | Data | 408.1 MB | 8bdec89098af |
| corpora/fineweb/CC-MAIN-2024-22_0002.jsonl.gz | Data | 331.2 MB | 1f4089a2e593 |
| corpora/fineweb/CC-MAIN-2024-26_0000.jsonl.gz | Data | 407.9 MB | cfbf823eefa9 |
| corpora/fineweb/CC-MAIN-2024-26_0001.jsonl.gz | Data | 407.9 MB | b9c7c89584a0 |
| corpora/fineweb/CC-MAIN-2024-26_0002.jsonl.gz | Data | 331.6 MB | b9f78ca5cf02 |
| corpora/fineweb/CC-MAIN-2024-30_0000.jsonl.gz | Data | 408.2 MB | ebfbb4a72d65 |
| corpora/fineweb/CC-MAIN-2024-30_0001.jsonl.gz | Data | 407.3 MB | a2bf6dcad3f8 |
| corpora/fineweb/CC-MAIN-2024-30_0002.jsonl.gz | Data | 331.1 MB | 497db157083e |
| corpora/fineweb/CC-MAIN-2024-33_0000.jsonl.gz | Data | 408.4 MB | 77f5c185ae1e |
| corpora/fineweb/CC-MAIN-2024-33_0001.jsonl.gz | Data | 408.4 MB | 24a74062030f |
| corpora/fineweb/CC-MAIN-2024-33_0002.jsonl.gz | Data | 331.8 MB | b97cbfff6d69 |
| corpora/fineweb/CC-MAIN-2024-38_0000.jsonl.gz | Data | 407.4 MB | caa92102302b |
| corpora/fineweb/CC-MAIN-2024-38_0001.jsonl.gz | Data | 406.8 MB | 19bc5bfeabac |
| corpora/fineweb/CC-MAIN-2024-38_0002.jsonl.gz | Data | 330.6 MB | 489c84df69ba |
| corpora/fineweb/CC-MAIN-2024-42_0000.jsonl.gz | Data | 407.1 MB | 69da2d9d4579 |
| corpora/fineweb/CC-MAIN-2024-42_0001.jsonl.gz | Data | 406.9 MB | c10c404d3c81 |
| corpora/fineweb/CC-MAIN-2024-42_0002.jsonl.gz | Data | 330.7 MB | 622c04493750 |
| corpora/fineweb/CC-MAIN-2024-46_0000.jsonl.gz | Data | 406.3 MB | 57515d3f92a1 |
| corpora/fineweb/CC-MAIN-2024-46_0001.jsonl.gz | Data | 406.4 MB | b9a1a37080cd |
| corpora/fineweb/CC-MAIN-2024-46_0002.jsonl.gz | Data | 330.6 MB | 4178e4575238 |
| corpora/fineweb/CC-MAIN-2024-51_0000.jsonl.gz | Data | 408.3 MB | 180464101310 |
| corpora/fineweb/CC-MAIN-2024-51_0001.jsonl.gz | Data | 408.0 MB | e8cd25ac89e0 |
| corpora/fineweb/CC-MAIN-2024-51_0002.jsonl.gz | Data | 331.5 MB | 9b6ee2fb3174 |
| corpora/fineweb/CC-MAIN-2025-05_0000.jsonl.gz | Data | 406.0 MB | 098f24e01671 |
| corpora/fineweb/CC-MAIN-2025-05_0001.jsonl.gz | Data | 405.6 MB | 73cb90f2039a |
| corpora/fineweb/CC-MAIN-2025-05_0002.jsonl.gz | Data | 329.6 MB | c4c0423d2228 |
| corpora/fineweb/CC-MAIN-2025-08_0000.jsonl.gz | Data | 407.0 MB | e641e930e481 |
| corpora/fineweb/CC-MAIN-2025-08_0001.jsonl.gz | Data | 406.5 MB | 6b21a0c85653 |
| corpora/fineweb/CC-MAIN-2025-08_0002.jsonl.gz | Data | 330.1 MB | bf9666d15469 |
| corpora/fineweb/CC-MAIN-2025-13_0000.jsonl.gz | Data | 406.2 MB | af48f6ea333a |
| corpora/fineweb/CC-MAIN-2025-13_0001.jsonl.gz | Data | 405.8 MB | ec3efce87887 |
| corpora/fineweb/CC-MAIN-2025-13_0002.jsonl.gz | Data | 329.7 MB | ca9bb6ab2733 |
| corpora/fineweb/CC-MAIN-2025-18_0000.jsonl.gz | Data | 406.5 MB | 93d351e3f5f8 |
| corpora/fineweb/CC-MAIN-2025-18_0001.jsonl.gz | Data | 406.2 MB | 70daf490f1d2 |
| corpora/fineweb/CC-MAIN-2025-18_0002.jsonl.gz | Data | 330.1 MB | fbb703069632 |
| corpora/fineweb/CC-MAIN-2025-21_0000.jsonl.gz | Data | 406.2 MB | 2294dd366770 |
| corpora/fineweb/CC-MAIN-2025-21_0001.jsonl.gz | Data | 406.0 MB | b3469cd5c088 |
| corpora/fineweb/CC-MAIN-2025-21_0002.jsonl.gz | Data | 329.8 MB | e6c208655811 |
| corpora/fineweb/CC-MAIN-2025-26_0000.jsonl.gz | Data | 406.5 MB | a9c5891ca157 |
| corpora/fineweb/CC-MAIN-2025-26_0001.jsonl.gz | Data | 405.6 MB | 7fe3f0487704 |
| corpora/fineweb/CC-MAIN-2025-26_0002.jsonl.gz | Data | 329.7 MB | 06bb6c411c17 |
| tokens/gpt2/fineweb_2019-2021.meta.jsonl.gz | Data | 241.2 MB | d47923fed2fd |
| README.md | Documentation | 2.7 KB | — |
| checkpoints/llama-124m/base/seed1/final/model.safetensors | Other | 494.4 MB | 74ebdb53b846 |
| checkpoints/llama-124m/base/seed1/tokens_0495M/model.safetensors | Other | 494.4 MB | 177a1e270683 |
| checkpoints/llama-124m/base/seed1/tokens_1005M/model.safetensors | Other | 494.4 MB | 73f469b08cd5 |
| checkpoints/llama-124m/base/seed2/final/model.safetensors | Other | 494.4 MB | 33f081f3d7cf |
| checkpoints/llama-124m/base/seed2/tokens_0495M/model.safetensors | Other | 494.4 MB | fe88c1c1e80a |
| checkpoints/llama-124m/base/seed2/tokens_1005M/model.safetensors | Other | 494.4 MB | 38e48f239b05 |
| checkpoints/llama-124m/base/seed3/final/model.safetensors | Other | 494.4 MB | aca1df07cfc9 |
| checkpoints/llama-124m/base/seed3/tokens_0495M/model.safetensors | Other | 494.4 MB | eef5a342beaf |
| checkpoints/llama-124m/base/seed3/tokens_1005M/model.safetensors | Other | 494.4 MB | 3c16a5534d37 |
| checkpoints/llama-124m/base/seed4/final/model.safetensors | Other | 494.4 MB | acb533fcc4a7 |
| checkpoints/llama-124m/base/seed4/tokens_0495M/model.safetensors | Other | 494.4 MB | 6e4a757a9199 |
| checkpoints/llama-124m/base/seed4/tokens_1005M/model.safetensors | Other | 494.4 MB | a452976c837f |
| checkpoints/llama-124m/base/seed5/final/model.safetensors | Other | 494.4 MB | 9931b685693c |
| checkpoints/llama-124m/base/seed6/final/model.safetensors | Other | 494.4 MB | c43694436a86 |
| checkpoints/llama-124m/base/seed7/final/model.safetensors | Other | 494.4 MB | db9697ee3f09 |
| checkpoints/llama-124m/base/seed8/final/model.safetensors | Other | 494.4 MB | 44f6a686857e |
| checkpoints/llama-124m/runs/run_001/model.safetensors | Other | 494.4 MB | d9140ba31459 |
| checkpoints/llama-124m/runs/run_002/model.safetensors | Other | 494.4 MB | 1654f3b030d1 |
| checkpoints/llama-124m/runs/run_003/model.safetensors | Other | 494.4 MB | b3ec5fef701b |
| checkpoints/llama-124m/runs/run_004/model.safetensors | Other | 494.4 MB | 21cce53315da |
| checkpoints/llama-124m/runs/run_005/model.safetensors | Other | 494.4 MB | 1ff4dd0e84b8 |
| checkpoints/llama-124m/runs/run_006/model.safetensors | Other | 494.4 MB | 1a60629ee750 |
| checkpoints/llama-124m/runs/run_007/model.safetensors | Other | 494.4 MB | bf3be7c22347 |
| checkpoints/llama-124m/runs/run_008/model.safetensors | Other | 494.4 MB | afb9e36bd21a |
| checkpoints/llama-124m/runs/run_009/model.safetensors | Other | 494.4 MB | e33e0b4ba9b1 |
| checkpoints/llama-124m/runs/run_010/model.safetensors | Other | 494.4 MB | f3388b23fc85 |
| checkpoints/llama-124m/runs/run_011/model.safetensors | Other | 494.4 MB | daf85b2cc848 |
| checkpoints/llama-124m/runs/run_012/model.safetensors | Other | 494.4 MB | 5319e2b2aa28 |
| checkpoints/llama-124m/runs/run_013/model.safetensors | Other | 494.4 MB | 72f61f73e866 |
| checkpoints/llama-124m/runs/run_014/model.safetensors | Other | 494.4 MB | ae9d37eac56b |
| checkpoints/llama-124m/runs/run_015/model.safetensors | Other | 494.4 MB | f460750c89c9 |
| checkpoints/llama-124m/runs/run_016/model.safetensors | Other | 494.4 MB | e1071e60e35b |
| checkpoints/llama-124m/runs/run_017/model.safetensors | Other | 494.4 MB | c884d9a5abd5 |
| checkpoints/llama-124m/runs/run_018/model.safetensors | Other | 494.4 MB | bbd9ba11e934 |
| checkpoints/llama-124m/runs/run_019/model.safetensors | Other | 494.4 MB | 114aebb1e5f2 |
| checkpoints/llama-124m/runs/run_020/model.safetensors | Other | 494.4 MB | 5abc755b478c |
| checkpoints/llama-124m/runs/run_021/model.safetensors | Other | 494.4 MB | 41d421c7c5d4 |
| checkpoints/llama-124m/runs/run_022/model.safetensors | Other | 494.4 MB | 9478970bae34 |
| checkpoints/llama-124m/runs/run_023/model.safetensors | Other | 494.4 MB | f01322715248 |
| checkpoints/llama-124m/runs/run_024/model.safetensors | Other | 494.4 MB | d24f3c426359 |
| checkpoints/llama-124m/runs/run_025/model.safetensors | Other | 494.4 MB | 5f7f834b3209 |
| checkpoints/llama-124m/runs/run_026/model.safetensors | Other | 494.4 MB | aebe6734d593 |
| checkpoints/llama-124m/runs/run_027/model.safetensors | Other | 494.4 MB | 8a72d05c0c3d |
| checkpoints/llama-124m/runs/run_028/model.safetensors | Other | 494.4 MB | c04ae8097cfa |
| checkpoints/llama-124m/runs/run_029/model.safetensors | Other | 494.4 MB | 7f60be46bc7c |
| checkpoints/llama-124m/runs/run_030/model.safetensors | Other | 494.4 MB | 7b358ba370a0 |
| checkpoints/llama-124m/runs/run_031/model.safetensors | Other | 494.4 MB | 13c8e9a361c4 |
| checkpoints/llama-124m/runs/run_032/model.safetensors | Other | 494.4 MB | 3927dba1603a |
| checkpoints/llama-124m/runs/run_033/model.safetensors | Other | 494.4 MB | 97169adc52c7 |
| checkpoints/llama-124m/runs/run_034/model.safetensors | Other | 494.4 MB | 0c19a31ef1e6 |
| checkpoints/llama-124m/runs/run_035/model.safetensors | Other | 494.4 MB | feca8c88cdbc |
| checkpoints/llama-124m/runs/run_036/model.safetensors | Other | 494.4 MB | 9014312aacbe |
| checkpoints/llama-124m/runs/run_037/model.safetensors | Other | 494.4 MB | 868f4c81f806 |
| checkpoints/llama-124m/runs/run_038/model.safetensors | Other | 494.4 MB | 4e3dc7d63828 |
| checkpoints/llama-124m/runs/run_039/model.safetensors | Other | 494.4 MB | ff0045444eca |
| checkpoints/llama-124m/runs/run_040/model.safetensors | Other | 494.4 MB | 39f7710f8057 |
| checkpoints/llama-124m/runs/run_041/model.safetensors | Other | 494.4 MB | dec8a2a7cc02 |
| checkpoints/llama-124m/runs/run_042/model.safetensors | Other | 494.4 MB | 81e357f7cc9e |
| checkpoints/llama-124m/runs/run_043/model.safetensors | Other | 494.4 MB | b84e4ca34db4 |
| checkpoints/llama-124m/runs/run_044/model.safetensors | Other | 494.4 MB | bd2e4ea4ffa3 |
| checkpoints/llama-124m/runs/run_045/model.safetensors | Other | 494.4 MB | 4919b6a1bd45 |
| checkpoints/llama-124m/runs/run_046/model.safetensors | Other | 494.4 MB | b13269336ac7 |
| checkpoints/llama-124m/runs/run_047/model.safetensors | Other | 494.4 MB | f802abfba222 |
| checkpoints/llama-124m/runs/run_048/model.safetensors | Other | 494.4 MB | 6077bb464315 |
| checkpoints/llama-124m/runs/run_049/model.safetensors | Other | 494.4 MB | 3231af99e392 |
| checkpoints/llama-124m/runs/run_050/model.safetensors | Other | 494.4 MB | aaeb1981ccb8 |
| checkpoints/llama-124m/runs/run_051/model.safetensors | Other | 494.4 MB | 3f2a303d47d3 |
| checkpoints/llama-124m/runs/run_052/model.safetensors | Other | 494.4 MB | dae870c5e137 |
| checkpoints/llama-124m/runs/run_053/model.safetensors | Other | 494.4 MB | c4ea1f9965f3 |
| checkpoints/llama-124m/runs/run_054/model.safetensors | Other | 494.4 MB | 05597ed2a489 |
| checkpoints/llama-124m/runs/run_055/model.safetensors | Other | 494.4 MB | ac309616326a |
| checkpoints/llama-124m/runs/run_056/model.safetensors | Other | 494.4 MB | e5f167af1dcd |
| checkpoints/llama-124m/runs/run_057/model.safetensors | Other | 494.4 MB | cd6b9bb16ada |
| checkpoints/llama-124m/runs/run_058/model.safetensors | Other | 494.4 MB | 9afef3f1f7be |
| checkpoints/llama-124m/runs/run_059/model.safetensors | Other | 494.4 MB | 010aaf737cd3 |
| checkpoints/llama-124m/runs/run_060/model.safetensors | Other | 494.4 MB | dac65a6a6a6c |
| checkpoints/llama-124m/runs/run_061/model.safetensors | Other | 494.4 MB | fa10a3a77b87 |
| checkpoints/llama-124m/runs/run_062/model.safetensors | Other | 494.4 MB | 2a12013cd70e |
| checkpoints/llama-124m/runs/run_063/model.safetensors | Other | 494.4 MB | faede7ce5ddb |
| checkpoints/llama-124m/runs/run_064/model.safetensors | Other | 494.4 MB | 3161cd3b3a23 |
| checkpoints/llama-124m/runs/run_065/model.safetensors | Other | 494.4 MB | cb6104487c2c |
| checkpoints/llama-124m/runs/run_066/model.safetensors | Other | 494.4 MB | a39f16ca69f6 |
| checkpoints/llama-124m/runs/run_067/model.safetensors | Other | 494.4 MB | 3b16a8ac5c26 |
| checkpoints/llama-124m/runs/run_068/model.safetensors | Other | 494.4 MB | d2080096bed9 |
| checkpoints/llama-124m/runs/run_069/model.safetensors | Other | 494.4 MB | b68be3248cfa |
| checkpoints/llama-124m/runs/run_070/model.safetensors | Other | 494.4 MB | d9903330d304 |
| checkpoints/llama-124m/runs/run_071/model.safetensors | Other | 494.4 MB | c24d3ce02460 |
| checkpoints/llama-124m/runs/run_072/model.safetensors | Other | 494.4 MB | 707e15db525d |
| checkpoints/llama-124m/runs/run_073/model.safetensors | Other | 494.4 MB | f04b8f7fb569 |
| checkpoints/llama-124m/runs/run_074/model.safetensors | Other | 494.4 MB | f04b8f7fb569 |
| checkpoints/llama-124m/runs/run_075/model.safetensors | Other | 494.4 MB | f04b8f7fb569 |
| checkpoints/llama-124m/runs/run_076/model.safetensors | Other | 494.4 MB | f04b8f7fb569 |
| checkpoints/llama-124m/runs/run_077/model.safetensors | Other | 494.4 MB | f04b8f7fb569 |
| checkpoints/llama-124m/runs/run_078/model.safetensors | Other | 494.4 MB | f04b8f7fb569 |
| checkpoints/llama-124m/runs/run_079/model.safetensors | Other | 494.4 MB | f04ff511a02c |
| checkpoints/llama-124m/runs/run_080/model.safetensors | Other | 494.4 MB | f04ff511a02c |
| checkpoints/llama-124m/runs/run_081/model.safetensors | Other | 494.4 MB | db78a146fcbc |
| checkpoints/llama-124m/runs/run_082/model.safetensors | Other | 494.4 MB | def821ed9bcb |
| checkpoints/llama-124m/runs/run_083/model.safetensors | Other | 494.4 MB | db78a146fcbc |
| checkpoints/llama-124m/runs/run_084/model.safetensors | Other | 494.4 MB | db78a146fcbc |
| checkpoints/llama-124m/runs/run_085/model.safetensors | Other | 494.4 MB | def821ed9bcb |
| checkpoints/llama-124m/runs/run_086/model.safetensors | Other | 494.4 MB | db78a146fcbc |
| checkpoints/llama-124m/runs/run_087/model.safetensors | Other | 494.4 MB | 79ac25270195 |
| checkpoints/llama-124m/runs/run_088/model.safetensors | Other | 494.4 MB | 298a21256e2e |
| checkpoints/llama-124m/runs/run_089/model.safetensors | Other | 494.4 MB | 8f386ffe1d30 |
| checkpoints/llama-124m/runs/run_090/model.safetensors | Other | 494.4 MB | c7b6c55eeb2d |
| checkpoints/llama-124m/runs/run_091/model.safetensors | Other | 494.4 MB | 113836e8a6c8 |
| checkpoints/llama-124m/runs/run_092/model.safetensors | Other | 494.4 MB | ca53033626d6 |
| checkpoints/llama-124m/runs/run_093/model.safetensors | Other | 494.4 MB | f299359911fa |
| checkpoints/llama-124m/runs/run_094/model.safetensors | Other | 494.4 MB | 48e028063440 |
| checkpoints/llama-124m/runs/run_095/model.safetensors | Other | 494.4 MB | 8932e2f0d8d8 |
| checkpoints/llama-124m/runs/run_096/model.safetensors | Other | 494.4 MB | d9da96444b73 |
| checkpoints/llama-124m/runs/run_097/model.safetensors | Other | 494.4 MB | 65cacd8c5cd6 |
| checkpoints/llama-124m/runs/run_098/model.safetensors | Other | 494.4 MB | 9d202eeb9b81 |
| checkpoints/llama-124m/runs/run_099/model.safetensors | Other | 494.4 MB | 2643788f9d31 |
| checkpoints/llama-124m/runs/run_100/model.safetensors | Other | 494.4 MB | 67f9b1f1cb17 |
| checkpoints/llama-124m/runs/run_101/model.safetensors | Other | 494.4 MB | a3bb5ee3ab69 |
| checkpoints/llama-124m/runs/run_102/model.safetensors | Other | 494.4 MB | b0b67d52faeb |
| checkpoints/llama-124m/runs/run_103/model.safetensors | Other | 494.4 MB | c4dbc53072b0 |
| checkpoints/llama-124m/runs/run_104/model.safetensors | Other | 494.4 MB | b1ce384b11c1 |
| checkpoints/llama-124m/runs/run_105/model.safetensors | Other | 494.4 MB | 07c8e62ad6d6 |
| checkpoints/llama-124m/runs/run_106/model.safetensors | Other | 494.4 MB | 79a3004a6ce9 |
| checkpoints/llama-124m/runs/run_107/model.safetensors | Other | 494.4 MB | fa1fb09df762 |
| checkpoints/llama-124m/runs/run_108/model.safetensors | Other | 494.4 MB | 7e994fe3b6e8 |
| checkpoints/llama-124m/runs/run_109/model.safetensors | Other | 494.4 MB | 85d567bdda19 |
| checkpoints/llama-124m/runs/run_110/model.safetensors | Other | 494.4 MB | 8a16471eeab2 |
| checkpoints/llama-124m/runs/run_111/model.safetensors | Other | 494.4 MB | 371321212c52 |
| tokens/gpt-neox/streams/stream_001.bin | Other | 298.8 MB | d998427c0b87 |
| tokens/gpt-neox/streams/stream_002.bin | Other | 298.8 MB | 23418a8fb480 |
| tokens/gpt-neox/streams/stream_003.bin | Other | 298.8 MB | 36e25ec2fa7a |
| tokens/gpt-neox/streams/stream_004.bin | Other | 298.8 MB | fc0df7d50454 |
| tokens/gpt2/cosmopedia.bin | Other | 3.4 GB | 09e11ab3a492 |
| tokens/gpt2/fineweb_2019-2021.bin | Other | 5.3 GB | 8842f46ebdfe |
| tokens/gpt2/streams/stream_001.bin | Other | 349.6 MB | 86f0e8b597aa |
| tokens/gpt2/streams/stream_002.bin | Other | 349.6 MB | 350d8aec26ac |
| tokens/gpt2/streams/stream_003.bin | Other | 349.6 MB | e5c2ce1a2171 |
| tokens/gpt2/streams/stream_004.bin | Other | 349.6 MB | 3266b04a3cb7 |
| tokens/gpt2/streams/stream_005.bin | Other | 199.8 MB | af3a1ceb0d54 |
| tokens/gpt2/streams/stream_006.bin | Other | 199.8 MB | 56683de3d128 |
| tokens/gpt2/streams/stream_007.bin | Other | 299.6 MB | b8cc746708c4 |
| tokens/gpt2/streams/stream_008.bin | Other | 299.6 MB | 05fa9eb7e76e |
| tokens/gpt2/streams/stream_009.bin | Other | 299.6 MB | bfa5e11a2118 |
| tokens/gpt2/streams/stream_010.bin | Other | 299.6 MB | 223463f03843 |
| tokens/gpt2/streams/stream_011.bin | Other | 199.8 MB | 2ea0e9aa5796 |
| tokens/gpt2/streams/stream_012.bin | Other | 199.8 MB | a5cd0cb4109c |
| tokens/gpt2/streams/stream_013.bin | Other | 199.8 MB | 9197d3d99784 |
| tokens/gpt2/streams/stream_014.bin | Other | 199.8 MB | f7285e641c96 |
| tokens/gpt2/streams/stream_015.bin | Other | 299.6 MB | 219b905014b9 |
| tokens/gpt2/streams/stream_016.bin | Other | 299.6 MB | 29ccae3edd40 |
| tokens/gpt2/streams/stream_017.bin | Other | 299.6 MB | 52fd2b2f0fbb |
| tokens/gpt2/streams/stream_018.bin | Other | 299.6 MB | 1d76a1b0560b |
| tokens/gpt2/streams/stream_019.bin | Other | 299.6 MB | 219b905014b9 |
| tokens/gpt2/streams/stream_020.bin | Other | 359.6 MB | c5f373fad583 |
| tokens/gpt2/streams/stream_021.bin | Other | 359.6 MB | 43227383ac89 |
| tokens/gpt2/streams/stream_022.bin | Other | 359.6 MB | 22d78a25599a |
| tokens/gpt2/streams/stream_023.bin | Other | 359.6 MB | 1487822fa66d |
| tokens/gpt2/streams/stream_024.bin | Other | 339.6 MB | 99bed5b4aef8 |
| tokens/gpt2/streams/stream_025.bin | Other | 339.6 MB | 45887aafada5 |
| tokens/gpt2/streams/stream_026.bin | Other | 339.6 MB | f4daa474b1f6 |
| tokens/gpt2/streams/stream_027.bin | Other | 339.6 MB | 38caf2085378 |
| .gitattributes | Repository | 2.5 KB | — |
License and Download
- License
- odc-by
- Access
- No access gate
Download from Liangzhi Li
Released by Liangzhi Li through its official repository on Hugging Face.