An English-language pretraining corpus prepared for controlled experiments with approximately 125M-parameter GPT-2 models based on karpathy/nanoGPT. The name refers to the approximately 5B-token mixture selected before final BPE tokenization.
Dataset Card
An English-language pretraining corpus prepared for controlled experiments with approximately 125M-parameter GPT-2 models based on karpathy/nanoGPT. The name refers to the approximately 5B-token mixture selected before final BPE tokenization. With the included 12,288-token BPE tokenizer, the packaged nanoGPT training split contains 5,396,605,407 tokens. The dataset provides both reusable source text and ready-to-train nanoGPT binaries. - documentid: string - sourceid: string - text: largestring - 12,029 merges - 256 byte tokens - 3 reserved special tokens -: ID 12,285 -: ID 12,286 -: ID 12,287 - One separator appended after every packaged document The tokenizer was trained on a…
Excerpt from the card by Fabryka AI.
Details
- Repository
- SlayerLab/minimal-en-corpus-5b
- Publisher
- Fabryka AI
- Task category
- Not stated by the source
- Tags
- pretraining, nanogpt, gpt-2
- Size category
- 1M<n<10M
- Languages
- en
- Revision
- 8624adc77201c4ab022146800fed85128d93b5ed
- Last updated
- 2026-09-22
Files
34 files, 17.0 GB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| data/manifest.json | Data | 1.5 KB | — |
| manifests/final-splits.json | Data | 871 B | — |
| manifests/mixture.json | Data | 1.9 KB | — |
| manifests/parquet-export.json | Data | 2.5 KB | — |
| manifests/qa.json | Data | 2.8 KB | — |
| runmirror/board_6b.json | Data | 16.6 KB | — |
| runmirror/board_bpe16m_c.json | Data | 16.6 KB | — |
| tokenizer/nanogpt-12k.json | Data | 559.3 KB | — |
| train/shard-00000.parquet | Data | 760.6 MB | c520f922b7c1 |
| train/shard-00001.parquet | Data | 925.0 MB | 307b3116c943 |
| train/shard-00002.parquet | Data | 867.8 MB | 036866429cde |
| train/shard-00003.parquet | Data | 1.8 GB | bbeb75d3388a |
| train/shard-00004.parquet | Data | 383.3 MB | 601f12697ccd |
| train/shard-00005.parquet | Data | 523.7 MB | 76df0d3ccc6c |
| train/shard-00006.parquet | Data | 491.6 MB | 49f6128c1d8e |
| train/shard-00007.parquet | Data | 408.5 MB | 7ca2a9ca9a93 |
| validation/shard-00000.parquet | Data | 6.4 MB | 5abe9d603867 |
| LICENSE | Documentation | 728 B | — |
| README.md | Documentation | 7.6 KB | — |
| data/train.bin | Other | 10.8 GB | cb4c457d2639 |
| data/train.idx | Other | 31.7 MB | 3c8179598f8e |
| data/val.bin | Other | 10.5 MB | 8253a10857ca |
| data/val.idx | Other | 35.8 KB | — |
| runmirror/eval_6b.log | Other | 49.3 KB | — |
| runmirror/eval_c.log | Other | 68.4 KB | — |
| runmirror/train.log | Other | 303.9 KB | — |
| scripts/blimp_avg.py | Other | 624 B | — |
| scripts/byte_lm_eval.py | Other | 18.1 KB | — |
| scripts/byte_lm_eval_bpe.py | Other | 1.9 KB | — |
| scripts/read_eval.py | Other | 694 B | — |
| scripts/train_gpt_ref.py | Other | 17.2 KB | — |
| .gitattributes | Repository | 2.6 KB | — |
| manifests/tokenizer-training.json | Tokenizer | 1.1 KB | — |
| tokenizer/tokenizer.json | Tokenizer | 830.1 KB | — |
License and Download
- License
- Not stated by the source
- Access
- No access gate
Released by Fabryka AI through its official repository on Hugging Face.
Models Trained on This Dataset
- Trained on (disclosed)gollem-v5-ckpts