Dataset containing synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary. The models referred to in the paper were trained on TinyStories-train.txt (the file tinystories-valid.txt can be used for validation loss).
Dataset Card
By Ronen Eldan, published under cdla-sharing-1.0, revision f54c09fd2331.
Dataset containing synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary.
Described in the following paper: https://arxiv.org/abs/2305.07759.
The models referred to in the paper were trained on TinyStories-train.txt (the file tinystories-valid.txt can be used for validation loss). These models can be found on Huggingface, at roneneldan/TinyStories-1M/3M/8M/28M/33M/1Layer-21M.
Additional resources: tinystories_all_data.tar.gz - contains a superset of the stories together with metadata and the prompt that was used to create each story.
TinyStoriesV2-GPT4-train.txt - Is a new version of the dataset that is based on generations by GPT-4 only (the original dataset also has generations by GPT-3.5 which are of lesser quality). It contains all the examples in TinyStories.txt which were GPT-4 generated as a subset (but is significantly larger).
Evaluation_prompts.yaml: List of prompts used to evaluate our models (see paper)
Structure
default 2,141,709 rows
| Split | Rows | Size |
|---|---|---|
| train | 2,119,719 | 2.0 GB |
| validation | 21,990 | 17.5 MB |
Details
- Repository
- roneneldan/TinyStories
- Publisher
- Ronen Eldan
- Task category
- Text generation
- Tags
- Not stated by the source
- Size category
- Not stated by the source
- Languages
- en
- Revision
- f54c09fd23315a6f9c86f9dc80f725de7d8f9c64
- Last updated
- 2024-08-12
Files
14 files, 7.6 GB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| TinyStories-train.txt | Data | 1.9 GB | c5cf5e22ff13 |
| TinyStories-valid.txt | Data | 19.4 MB | 94e431816c4c |
| TinyStoriesV2-GPT4-train.txt | Data | 2.2 GB | 6418d412de72 |
| TinyStoriesV2-GPT4-valid.txt | Data | 22.5 MB | 6874bae9a4c1 |
| TinyStories_all_data.tar.gz | Data | 1.6 GB | 26cf7605aca1 |
| data/train-00000-of-00004-2d5a1467fff1081b.parquet | Data | 248.7 MB | 77cf780cebe5 |
| data/train-00001-of-00004-5852b56a2bd28fd9.parquet | Data | 248.2 MB | e2f173a6dcc2 |
| data/train-00002-of-00004-a26307300439e943.parquet | Data | 245.9 MB | cac73cd1a097 |
| data/train-00003-of-00004-d243063613e5a057.parquet | Data | 248.0 MB | 862514ae7572 |
| data/validation-00000-of-00001-869c898b519ad725.parquet | Data | 10.0 MB | 33406a620655 |
| README.md | Documentation | 1.1 KB | — |
| Evaluation prompts.yaml | Other | 11.8 KB | — |
| MLP_input_output.npy | Other | 819.2 MB | 047e494123cf |
| .gitattributes | Repository | 2.8 KB | — |
License and Download
- License
- cdla-sharing-1.0
- Access
- No access gate
Released by Ronen Eldan through its official repository on Hugging Face.
Models Trained on This Dataset
- Trained on (disclosed)SharperSwarm
- Trained on (disclosed)CasualSwarms