The TOFU dataset serves as a benchmark for evaluating unlearning performance of large language models on realistic tasks.
Rows6,120
Configurations12
Size6.3 MB
Licensemit
AccessPublicly accessible
Monthly Downloads95k
Dataset Card
By Locus Lab, published under mit, revision 324592d84ae4.
TOFU: Task of Fictitious Unlearning
The TOFU dataset serves as a benchmark for evaluating unlearning performance of large language models on realistic tasks. The dataset comprises question-answer pairs based on autobiographies of 200 different authors that do not exist and are completely fictitiously generated by the GPT-4 model. The goal of the task is to unlearn a fine-tuned model on various fractions of the forget set.
Quick Links
- Website: The landing page for TOFU
- arXiv Paper: Detailed information about the TOFU dataset and its significance in unlearning tasks.
- GitHub Repository: Access the source code, fine-tuning scripts, and additional resources for the TOFU dataset.
- Dataset on Hugging Face: Direct link to download the TOFU dataset.
- Leaderboard on Hugging Face Spaces: Current rankings and submissions for the TOFU dataset challenges.
- Summary on Twitter: A concise summary and key takeaways from the project.
Applicability
Structure
full 4,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 4,000 | 990.8 KB |
questionstringanswerstring
forget01 40 rows
| Split | Rows | Size |
|---|---|---|
| train | 40 | 10.8 KB |
questionstringanswerstring
forget01_perturbed 40 rows
| Split | Rows | Size |
|---|---|---|
| train | 40 | 67.6 KB |
questionstringanswerstringparaphrased_answerstringperturbed_answerSequenceparaphrased_questionstring
forget05 200 rows
| Split | Rows | Size |
|---|---|---|
| train | 200 | 52.3 KB |
questionstringanswerstring
forget05_perturbed 200 rows
| Split | Rows | Size |
|---|---|---|
| train | 200 | 319.0 KB |
questionstringanswerstringparaphrased_answerstringperturbed_answerSequenceparaphrased_questionstring
forget10 400 rows
| Split | Rows | Size |
|---|---|---|
| train | 400 | 106.5 KB |
questionstringanswerstring
forget10_perturbed 400 rows
| Split | Rows | Size |
|---|---|---|
| train | 400 | 644.4 KB |
questionstringanswerstringparaphrased_answerstringperturbed_answerSequenceparaphrased_questionstring
holdout01 40 rows
| Split | Rows | Size |
|---|---|---|
| train | 40 | 11.4 KB |
questionstringanswerstring
holdout05 200 rows
| Split | Rows | Size |
|---|---|---|
| train | 200 | 59.9 KB |
questionstringanswerstring
holdout10 400 rows
| Split | Rows | Size |
|---|---|---|
| train | 400 | 122.4 KB |
questionstringanswerstring
real_authors 100 rows
| Split | Rows | Size |
|---|---|---|
| train | 100 | 15.1 KB |
questionstringoption1stringoption2stringoption3stringoption4stringanswerstring
real_authors_perturbed 100 rows
| Split | Rows | Size |
|---|---|---|
| train | 100 | 13.7 KB |
questionstringanswerstringperturbed_answerSequence
Details
- Repository
- locuslab/TOFU
- Publisher
- Locus Lab
- Task category
- Question answering
- Tags
- unlearning, question answering, TOFU
- Size category
- 1K<n<10K
- Languages
- en
- Revision
- 324592d84ae4f482ac7249b9285c2ecdb53e3a68
- Last updated
- 2025-03-27
Files
20 files, 6.3 MB in total.
Data18 files · 6.3 MB
Documentation1 file · 4.1 KB
Repository1 file · 2.3 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| forget01.json | Data | 11.7 KB | — |
| forget01_perturbed.json | Data | 70.6 KB | — |
| forget05.json | Data | 56.4 KB | — |
| forget05_perturbed.json | Data | 334.2 KB | — |
| forget10.json | Data | 114.9 KB | — |
| forget10_perturbed.json | Data | 674.7 KB | — |
| full.json | Data | 1.1 MB | — |
| holdout01.json | Data | 12.3 KB | — |
| holdout05.json | Data | 64.3 KB | — |
| holdout10.json | Data | 131.0 KB | — |
| real_authors.json | Data | 21.9 KB | — |
| real_authors_perturbed.json | Data | 17.1 KB | — |
| retain90.json | Data | 963.0 KB | — |
| retain95.json | Data | 1.0 MB | — |
| retain99.json | Data | 1.1 MB | — |
| retain_perturbed.json | Data | 641.1 KB | — |
| world_facts.json | Data | 23.1 KB | — |
| world_facts_perturbed.json | Data | 18.1 KB | — |
| README.md | Documentation | 4.1 KB | — |
| .gitattributes | Repository | 2.3 KB | — |
License and Download
- License
- mit
- Access
- No access gate
Download from Locus Lab
Released by Locus Lab through its official repository on Hugging Face. Read the license.
Models Trained on This Dataset
- Trained on (disclosed)Llama-3.2-1B-Instruct-TOFU-forget10-NPO
- Trained on (disclosed)Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff-ILU
- Trained on (disclosed)Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff-SAM
- Trained on (disclosed)Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff