TruthfulQA is a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics.
Dataset Card
By TruthfulQA, published under apache-2.0, revision 741b8276f2d1.
Dataset Card for truthful_qa
Table of Contents
- Dataset Card for truthful_qa
- Table of Contents
- Dataset Description
- Dataset Summary
- Supported Tasks and Leaderboards
- Languages
- Dataset Structure
- Data Instances
- generation
- multiple_choice
- Data Fields
- generation
- multiple_choice
- Data Splits
- Dataset Creation
- Curation Rationale
- Source Data
- Initial Data Collection and Normalization
- Who are the source language producers?
- Annotations
- Annotation process
- Who are the annotators?
- Personal and Sensitive Information
- Considerations for Using the Data
- Social Impact of Dataset
- Discussion of Biases
- Other Known Limitations
- Additional Information
- Dataset Curators
- Licensing Information
- Citation Information
- Contributions
Dataset Description
- Homepage: [Needs More Information]
- Repository: https://github.com/sylinrl/TruthfulQA
- Paper: https://arxiv.org/abs/2109.07958
- Leaderboard: [Needs More Information]
- Point of Contact: [Needs More Information]
Dataset Summary
Structure
generation 817 rows
| Split | Rows | Size |
|---|---|---|
| validation | 817 | 473.4 KB |
multiple_choice 817 rows
| Split | Rows | Size |
|---|---|---|
| validation | 817 | 610.3 KB |
Details
- Repository
- truthfulqa/truthful_qa
- Publisher
- TruthfulQA
- Task category
- Multiple choice
- Tags
- Not stated by the source
- Size category
- n<1K
- Languages
- en
- Revision
- 741b8276f2d1982aa3d5b832d3ee81ed3b896490
- Last updated
- 2024-01-04
SAVRN's Notes on truthful_qa
TruthfulQA is an evaluation set, and the header numbers make the point: 1,634 rows across two configurations (generation and multiple_choice), 504.8 KB on disk, 4 files, yet 204.1k monthly downloads. The card describes 817 questions spanning 38 categories, including health, law, finance and politics. Nothing here is training volume; it exists to measure whether a model generates truthful answers under adversarial-style questioning.
Apache 2.0 permits commercial use, modification and redistribution, so the license is not the constraint. The constraint is benchmark hygiene: with only 817 questions, any model that has seen this set in pretraining or fine-tuning will post inflated truthfulness scores, and the set is small enough to audit by hand. Downloading it costs essentially nothing at half a megabyte; keeping it out of your training pipeline is the real work.
Files
4 files, 504.8 KB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| generation/validation-00000-of-00001.parquet | Data | 222.6 KB | dfb1004b8ab8 |
| multiple_choice/validation-00000-of-00001.parquet | Data | 271.0 KB | 23f08e230ca4 |
| README.md | Documentation | 9.6 KB | — |
| .gitattributes | Repository | 1.6 KB | — |
License and Download
- License
- apache-2.0
- Access
- No access gate
Released by TruthfulQA through its official repository on Hugging Face. Read the license.
Models Trained on This Dataset
- Trained on (disclosed)bobcat-flash-1.2