Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).
Rows217,110
Configurations12
Size270.0 MB
Licensemit
AccessPublicly accessible
Monthly Downloads779.7k
Dataset Card
By Center for AI Safety, published under mit, revision c30699e8356d.
Dataset Card for MMLU
Table of Contents
- Table of Contents
- Dataset Description
- Dataset Summary
- Supported Tasks and Leaderboards
- Languages
- Dataset Structure
- Data Instances
- Data Fields
- Data Splits
- Dataset Creation
- Curation Rationale
- Source Data
- Annotations
- Personal and Sensitive Information
- Considerations for Using the Data
- Social Impact of Dataset
- Discussion of Biases
- Other Known Limitations
- Additional Information
- Dataset Curators
- Licensing Information
- Citation Information
- Contributions
Dataset Description
- Repository: https://github.com/hendrycks/test
- Paper: https://arxiv.org/abs/2009.03300
Dataset Summary
Measuring Massive Multitask Language Understanding by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021).
Structure
abstract_algebra 116 rows
| Split | Rows | Size |
|---|---|---|
| test | 100 | 21.3 KB |
| validation | 11 | 2.2 KB |
| dev | 5 | 919 B |
questionstringsubjectstringchoicesSequenceanswerClassLabel
all 115,700 rows
| Split | Rows | Size |
|---|---|---|
| test | 14,042 | 4.1 MB |
| validation | 1,531 | 613.5 KB |
| dev | 285 | 125.4 KB |
| auxiliary_train | 99,842 | 123.4 MB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
anatomy 154 rows
| Split | Rows | Size |
|---|---|---|
| test | 135 | 34.6 KB |
| validation | 14 | 3.3 KB |
| dev | 5 | 1.0 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
astronomy 173 rows
| Split | Rows | Size |
|---|---|---|
| test | 152 | 48.8 KB |
| validation | 16 | 5.2 KB |
| dev | 5 | 2.1 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
auxiliary_train 99,842 rows
| Split | Rows | Size |
|---|---|---|
| train | 99,842 | 123.4 MB |
trainvalue
business_ethics 116 rows
| Split | Rows | Size |
|---|---|---|
| test | 100 | 35.2 KB |
| validation | 11 | 3.2 KB |
| dev | 5 | 2.3 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
clinical_knowledge 299 rows
| Split | Rows | Size |
|---|---|---|
| test | 265 | 68.6 KB |
| validation | 29 | 7.3 KB |
| dev | 5 | 1.3 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
college_biology 165 rows
| Split | Rows | Size |
|---|---|---|
| test | 144 | 51.5 KB |
| validation | 16 | 5.1 KB |
| dev | 5 | 1.6 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
college_chemistry 113 rows
| Split | Rows | Size |
|---|---|---|
| test | 100 | 26.8 KB |
| validation | 8 | 2.5 KB |
| dev | 5 | 1.4 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
college_computer_science 116 rows
| Split | Rows | Size |
|---|---|---|
| test | 100 | 45.4 KB |
| validation | 11 | 5.0 KB |
| dev | 5 | 2.9 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
college_mathematics 116 rows
| Split | Rows | Size |
|---|---|---|
| test | 100 | 27.0 KB |
| validation | 11 | 2.9 KB |
| dev | 5 | 1.6 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
college_medicine 200 rows
| Split | Rows | Size |
|---|---|---|
| test | 173 | 85.9 KB |
| validation | 22 | 8.3 KB |
| dev | 5 | 1.8 KB |
questionstringsubjectstringchoicesSequenceanswerClassLabel
Details
- Repository
- cais/mmlu
- Publisher
- Center for AI Safety
- Task category
- Question answering
- Tags
- Not stated by the source
- Size category
- 10K<n<100K
- Languages
- en
- Revision
- c30699e8356da336a370243923dbaf21066bb9fe
- Last updated
- 2024-03-08
Files
181 files, 270.0 MB in total.
Data178 files · 270.0 MB
Documentation1 file · 53.2 KB
Other1 file · 5.7 KB
Repository1 file · 2.3 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| abstract_algebra/dev-00000-of-00001.parquet | Data | 3.5 KB | e778f6090366 |
| abstract_algebra/test-00000-of-00001.parquet | Data | 10.0 KB | 92338cdcdb43 |
| abstract_algebra/validation-00000-of-00001.parquet | Data | 3.7 KB | b66df8764498 |
| all/auxiliary_train-00000-of-00001.parquet | Data | 47.5 MB | 5126bb35f06c |
| all/dev-00000-of-00001.parquet | Data | 76.5 KB | 2b19bde1ed8c |
| all/test-00000-of-00001.parquet | Data | 3.5 MB | 74a41822ce7d |
| all/validation-00000-of-00001.parquet | Data | 408.4 KB | 66cdf0b090cc |
| anatomy/dev-00000-of-00001.parquet | Data | 3.5 KB | 1774b4af192a |
| anatomy/test-00000-of-00001.parquet | Data | 20.1 KB | cb88f5920b67 |
| anatomy/validation-00000-of-00001.parquet | Data | 5.3 KB | f99c5b6b26a8 |
| astronomy/dev-00000-of-00001.parquet | Data | 4.9 KB | 73b897f92f1f |
| astronomy/test-00000-of-00001.parquet | Data | 28.3 KB | bcdfe1afcb94 |
| astronomy/validation-00000-of-00001.parquet | Data | 6.1 KB | 98b9cdb9327b |
| auxiliary_train/train-00000-of-00001.parquet | Data | 47.5 MB | ae6662576ef9 |
| business_ethics/dev-00000-of-00001.parquet | Data | 5.0 KB | 494ea1d948ca |
| business_ethics/test-00000-of-00001.parquet | Data | 21.6 KB | 7910a4b8236c |
| business_ethics/validation-00000-of-00001.parquet | Data | 5.1 KB | 3a899aef15e3 |
| clinical_knowledge/dev-00000-of-00001.parquet | Data | 3.7 KB | 105b492234b3 |
| clinical_knowledge/test-00000-of-00001.parquet | Data | 40.5 KB | 7bd3de302db3 |
| clinical_knowledge/validation-00000-of-00001.parquet | Data | 7.5 KB | 2337cba28303 |
| college_biology/dev-00000-of-00001.parquet | Data | 4.3 KB | 0cf3cbb4fde7 |
| college_biology/test-00000-of-00001.parquet | Data | 31.8 KB | 5b6920ddbbb4 |
| college_biology/validation-00000-of-00001.parquet | Data | 6.9 KB | f5010d257812 |
| college_chemistry/dev-00000-of-00001.parquet | Data | 4.0 KB | 4c689a238a27 |
| college_chemistry/test-00000-of-00001.parquet | Data | 17.9 KB | f012f90fc3ae |
| college_chemistry/validation-00000-of-00001.parquet | Data | 4.9 KB | e394d9086108 |
| college_computer_science/dev-00000-of-00001.parquet | Data | 6.8 KB | 6b35dffa1635 |
| college_computer_science/test-00000-of-00001.parquet | Data | 28.1 KB | 08b79916057e |
| college_computer_science/validation-00000-of-00001.parquet | Data | 6.3 KB | 967090233faa |
| college_mathematics/dev-00000-of-00001.parquet | Data | 5.2 KB | 46ee5c84b379 |
| college_mathematics/test-00000-of-00001.parquet | Data | 16.6 KB | f57b57e992d0 |
| college_mathematics/validation-00000-of-00001.parquet | Data | 5.0 KB | e668d0830970 |
| college_medicine/dev-00000-of-00001.parquet | Data | 4.8 KB | 85bf7f27b795 |
| college_medicine/test-00000-of-00001.parquet | Data | 42.5 KB | f21b3166975d |
| college_medicine/validation-00000-of-00001.parquet | Data | 9.0 KB | 4b5a104770d7 |
| college_physics/dev-00000-of-00001.parquet | Data | 4.5 KB | eec8b9851f44 |
| college_physics/test-00000-of-00001.parquet | Data | 18.6 KB | 81b8eafc776b |
| college_physics/validation-00000-of-00001.parquet | Data | 6.4 KB | 79f20c0d67a0 |
| computer_security/dev-00000-of-00001.parquet | Data | 4.3 KB | b491f6df6ed1 |
| computer_security/test-00000-of-00001.parquet | Data | 19.1 KB | a6ad5774acf5 |
| computer_security/validation-00000-of-00001.parquet | Data | 6.7 KB | 1eaed4c8c709 |
| conceptual_physics/dev-00000-of-00001.parquet | Data | 4.0 KB | 1503db271848 |
| conceptual_physics/test-00000-of-00001.parquet | Data | 25.0 KB | c66a861048a2 |
| conceptual_physics/validation-00000-of-00001.parquet | Data | 6.0 KB | c6df7c8fd416 |
| data.tar | Data | 166.2 MB | bec563ba4bac |
| dataset_infos.json | Data | 138.3 KB | — |
| econometrics/dev-00000-of-00001.parquet | Data | 4.5 KB | 5edd31ddb8ce |
| econometrics/test-00000-of-00001.parquet | Data | 24.5 KB | 63501e9de9f8 |
| econometrics/validation-00000-of-00001.parquet | Data | 7.0 KB | 52aceec5650c |
| electrical_engineering/dev-00000-of-00001.parquet | Data | 4.1 KB | 92416acbafa7 |
| electrical_engineering/test-00000-of-00001.parquet | Data | 17.6 KB | 3df176cabbd1 |
| electrical_engineering/validation-00000-of-00001.parquet | Data | 5.1 KB | f03b430fd953 |
| elementary_mathematics/dev-00000-of-00001.parquet | Data | 4.5 KB | 03b0ff28b651 |
| elementary_mathematics/test-00000-of-00001.parquet | Data | 41.1 KB | 9cb645300084 |
| elementary_mathematics/validation-00000-of-00001.parquet | Data | 9.4 KB | a33f741dae37 |
| formal_logic/dev-00000-of-00001.parquet | Data | 4.8 KB | 6928410ade5a |
| formal_logic/test-00000-of-00001.parquet | Data | 21.5 KB | a9f917e01461 |
| formal_logic/validation-00000-of-00001.parquet | Data | 6.6 KB | b5ae1527ce33 |
| global_facts/dev-00000-of-00001.parquet | Data | 3.6 KB | ee23f3bd9dcc |
| global_facts/test-00000-of-00001.parquet | Data | 11.5 KB | 9e4b5a22ad1c |
| global_facts/validation-00000-of-00001.parquet | Data | 4.2 KB | 48b84607d614 |
| high_school_biology/dev-00000-of-00001.parquet | Data | 4.9 KB | 4ae1c2d737b6 |
| high_school_biology/test-00000-of-00001.parquet | Data | 62.7 KB | 73600cc5c803 |
| high_school_biology/validation-00000-of-00001.parquet | Data | 10.6 KB | 5c7a4d5bbb0a |
| high_school_chemistry/dev-00000-of-00001.parquet | Data | 4.2 KB | 9ff0a3200962 |
| high_school_chemistry/test-00000-of-00001.parquet | Data | 33.3 KB | fde2345147be |
| high_school_chemistry/validation-00000-of-00001.parquet | Data | 8.3 KB | 0caed5473675 |
| high_school_computer_science/dev-00000-of-00001.parquet | Data | 6.5 KB | 431ad5ab2038 |
| high_school_computer_science/test-00000-of-00001.parquet | Data | 27.3 KB | 6e1b210896af |
| high_school_computer_science/validation-00000-of-00001.parquet | Data | 5.3 KB | 1e41b804c5d8 |
| high_school_european_history/dev-00000-of-00001.parquet | Data | 22.2 KB | 61cc946b0082 |
| high_school_european_history/test-00000-of-00001.parquet | Data | 142.5 KB | 2bcf46d76656 |
| high_school_european_history/validation-00000-of-00001.parquet | Data | 31.6 KB | 38bd62d9100b |
| high_school_geography/dev-00000-of-00001.parquet | Data | 3.9 KB | d3a732b15dc6 |
| high_school_geography/test-00000-of-00001.parquet | Data | 28.2 KB | d083f0a37e98 |
| high_school_geography/validation-00000-of-00001.parquet | Data | 6.2 KB | d1c34639bf3b |
| high_school_government_and_politics/dev-00000-of-00001.parquet | Data | 4.5 KB | 3877a26ca2e8 |
| high_school_government_and_politics/test-00000-of-00001.parquet | Data | 40.2 KB | 54bb3fcc3987 |
| high_school_government_and_politics/validation-00000-of-00001.parquet | Data | 8.3 KB | 28f197e5a0ab |
| high_school_macroeconomics/dev-00000-of-00001.parquet | Data | 4.0 KB | 6aa2bf96a9cf |
| high_school_macroeconomics/test-00000-of-00001.parquet | Data | 54.8 KB | a0f2d6a33352 |
| high_school_macroeconomics/validation-00000-of-00001.parquet | Data | 9.9 KB | 3219f345d00b |
| high_school_mathematics/dev-00000-of-00001.parquet | Data | 4.5 KB | 73034b3a7694 |
| high_school_mathematics/test-00000-of-00001.parquet | Data | 33.7 KB | 310628511fac |
| high_school_mathematics/validation-00000-of-00001.parquet | Data | 7.0 KB | 841cc40a133f |
| high_school_microeconomics/dev-00000-of-00001.parquet | Data | 3.8 KB | a8ae561bb6bf |
| high_school_microeconomics/test-00000-of-00001.parquet | Data | 38.8 KB | 8a55dc2fa10f |
| high_school_microeconomics/validation-00000-of-00001.parquet | Data | 7.2 KB | 5dce9d16182e |
| high_school_physics/dev-00000-of-00001.parquet | Data | 4.6 KB | 3b6fb5f8d1f1 |
| high_school_physics/test-00000-of-00001.parquet | Data | 33.0 KB | 48fc84060e4c |
| high_school_physics/validation-00000-of-00001.parquet | Data | 8.0 KB | 696ee856dae6 |
| high_school_psychology/dev-00000-of-00001.parquet | Data | 5.2 KB | ada3aebdb980 |
| high_school_psychology/test-00000-of-00001.parquet | Data | 92.8 KB | 9df4a744eb6f |
| high_school_psychology/validation-00000-of-00001.parquet | Data | 15.2 KB | 964f33208874 |
| high_school_statistics/dev-00000-of-00001.parquet | Data | 6.1 KB | 3b7c5d8b0c6c |
| high_school_statistics/test-00000-of-00001.parquet | Data | 58.0 KB | 4922828bda3a |
| high_school_statistics/validation-00000-of-00001.parquet | Data | 10.9 KB | cb74547d5d96 |
| high_school_us_history/dev-00000-of-00001.parquet | Data | 17.8 KB | dcc43231bf78 |
| high_school_us_history/test-00000-of-00001.parquet | Data | 154.9 KB | 7e3c242fa3b0 |
| high_school_us_history/validation-00000-of-00001.parquet | Data | 27.3 KB | 73b682d8cc00 |
| high_school_world_history/dev-00000-of-00001.parquet | Data | 10.2 KB | 5c37e72b49e3 |
| high_school_world_history/test-00000-of-00001.parquet | Data | 201.6 KB | abc9b0182e7c |
| high_school_world_history/validation-00000-of-00001.parquet | Data | 38.5 KB | e7647086c3e3 |
| human_aging/dev-00000-of-00001.parquet | Data | 3.7 KB | 7d5ef3e4af1f |
| human_aging/test-00000-of-00001.parquet | Data | 31.2 KB | 69b143d394fd |
| human_aging/validation-00000-of-00001.parquet | Data | 6.3 KB | bacec25cb2c8 |
| human_sexuality/dev-00000-of-00001.parquet | Data | 4.1 KB | e2f15b5006b8 |
| human_sexuality/test-00000-of-00001.parquet | Data | 23.2 KB | 49efe58e1311 |
| human_sexuality/validation-00000-of-00001.parquet | Data | 5.3 KB | 0fdf548c599d |
| international_law/dev-00000-of-00001.parquet | Data | 5.0 KB | a74c7a33d187 |
| international_law/test-00000-of-00001.parquet | Data | 29.5 KB | 05da704a004f |
| international_law/validation-00000-of-00001.parquet | Data | 7.1 KB | 32b9266ab1a3 |
| jurisprudence/dev-00000-of-00001.parquet | Data | 4.0 KB | 1291e1cbdaaa |
| jurisprudence/test-00000-of-00001.parquet | Data | 23.3 KB | 0f302db5178c |
| jurisprudence/validation-00000-of-00001.parquet | Data | 6.2 KB | 8ef2d3dcda61 |
| logical_fallacies/dev-00000-of-00001.parquet | Data | 4.1 KB | cba7736ad6b6 |
| logical_fallacies/test-00000-of-00001.parquet | Data | 23.0 KB | 87eefcf75ec8 |
| logical_fallacies/validation-00000-of-00001.parquet | Data | 6.5 KB | 042b2502c81a |
| machine_learning/dev-00000-of-00001.parquet | Data | 5.2 KB | b8882136a1a6 |
| machine_learning/test-00000-of-00001.parquet | Data | 19.7 KB | cd3d1ac01f11 |
| machine_learning/validation-00000-of-00001.parquet | Data | 6.2 KB | 090d14c981b1 |
| management/dev-00000-of-00001.parquet | Data | 3.6 KB | 772b281c0322 |
| management/test-00000-of-00001.parquet | Data | 14.7 KB | b5579485f3b8 |
| management/validation-00000-of-00001.parquet | Data | 4.5 KB | 3db139ecc101 |
| marketing/dev-00000-of-00001.parquet | Data | 4.3 KB | a00bdec21ca5 |
| marketing/test-00000-of-00001.parquet | Data | 37.3 KB | 3c975f1b0dc6 |
| marketing/validation-00000-of-00001.parquet | Data | 8.2 KB | 844b008134e3 |
| medical_genetics/dev-00000-of-00001.parquet | Data | 3.8 KB | a3e66d3e1619 |
| medical_genetics/test-00000-of-00001.parquet | Data | 16.4 KB | f5535475f3af |
| medical_genetics/validation-00000-of-00001.parquet | Data | 5.6 KB | b1e5b8381e2f |
| miscellaneous/dev-00000-of-00001.parquet | Data | 3.4 KB | b3a42ee21922 |
| miscellaneous/test-00000-of-00001.parquet | Data | 98.6 KB | 5665dde94d34 |
| miscellaneous/validation-00000-of-00001.parquet | Data | 13.2 KB | 00724923c0ea |
| moral_disputes/dev-00000-of-00001.parquet | Data | 4.4 KB | 7efc2d7512a7 |
| moral_disputes/test-00000-of-00001.parquet | Data | 60.9 KB | fc2a4270481c |
| moral_disputes/validation-00000-of-00001.parquet | Data | 10.7 KB | c88b8e7cd4a5 |
| moral_scenarios/dev-00000-of-00001.parquet | Data | 5.1 KB | c31c54bb7f8c |
| moral_scenarios/test-00000-of-00001.parquet | Data | 89.8 KB | e21cc9d8f5a1 |
| moral_scenarios/validation-00000-of-00001.parquet | Data | 14.9 KB | 8b222953f05e |
| nutrition/dev-00000-of-00001.parquet | Data | 5.0 KB | 4727ff1fb935 |
| nutrition/test-00000-of-00001.parquet | Data | 55.0 KB | 7ba291c21774 |
| nutrition/validation-00000-of-00001.parquet | Data | 9.0 KB | 044ca489a676 |
| philosophy/dev-00000-of-00001.parquet | Data | 4.2 KB | 40bd9dee29bb |
| philosophy/test-00000-of-00001.parquet | Data | 48.6 KB | 51a6715bca76 |
| philosophy/validation-00000-of-00001.parquet | Data | 9.2 KB | 6883bbc1930b |
| prehistory/dev-00000-of-00001.parquet | Data | 4.6 KB | 9f5ee161a94b |
| prehistory/test-00000-of-00001.parquet | Data | 54.3 KB | f4cf1e103e64 |
| prehistory/validation-00000-of-00001.parquet | Data | 9.9 KB | 7ef498da9ba7 |
| professional_accounting/dev-00000-of-00001.parquet | Data | 4.9 KB | 03ab25053042 |
| professional_accounting/test-00000-of-00001.parquet | Data | 69.5 KB | 1fe6da7a3a86 |
| professional_accounting/validation-00000-of-00001.parquet | Data | 12.9 KB | c2cf2ba25cde |
| professional_law/dev-00000-of-00001.parquet | Data | 15.1 KB | cb7bf53b1bcf |
| professional_law/test-00000-of-00001.parquet | Data | 1.0 MB | 81cfdb63639f |
| professional_law/validation-00000-of-00001.parquet | Data | 115.6 KB | d64b01ffb220 |
| professional_medicine/dev-00000-of-00001.parquet | Data | 8.4 KB | 899036c8f5d8 |
| professional_medicine/test-00000-of-00001.parquet | Data | 124.9 KB | 8c1c18b16c98 |
| professional_medicine/validation-00000-of-00001.parquet | Data | 19.9 KB | f218bc4485cb |
| professional_psychology/dev-00000-of-00001.parquet | Data | 4.7 KB | c61d4fb24acc |
| professional_psychology/test-00000-of-00001.parquet | Data | 132.6 KB | 91529edc6760 |
| professional_psychology/validation-00000-of-00001.parquet | Data | 22.1 KB | 85a3859b7186 |
| public_relations/dev-00000-of-00001.parquet | Data | 4.4 KB | 7dd78ceeccd2 |
| public_relations/test-00000-of-00001.parquet | Data | 20.6 KB | 9ce4ff437b2b |
| public_relations/validation-00000-of-00001.parquet | Data | 6.4 KB | 38c2f887dbd9 |
| security_studies/dev-00000-of-00001.parquet | Data | 7.5 KB | c7ab02234d03 |
| security_studies/test-00000-of-00001.parquet | Data | 114.1 KB | 38477f527d22 |
| security_studies/validation-00000-of-00001.parquet | Data | 18.7 KB | 65866e2efc98 |
| sociology/dev-00000-of-00001.parquet | Data | 4.2 KB | 4abb7ebab6f1 |
| sociology/test-00000-of-00001.parquet | Data | 43.9 KB | 0cdb1dab92e7 |
| sociology/validation-00000-of-00001.parquet | Data | 8.4 KB | aadeb1fdd88e |
| us_foreign_policy/dev-00000-of-00001.parquet | Data | 4.2 KB | da27347e8868 |
| us_foreign_policy/test-00000-of-00001.parquet | Data | 19.5 KB | 8051dfc5f49b |
| us_foreign_policy/validation-00000-of-00001.parquet | Data | 5.3 KB | e1149eaf277f |
| virology/dev-00000-of-00001.parquet | Data | 3.9 KB | 2790f42d02f0 |
| virology/test-00000-of-00001.parquet | Data | 27.3 KB | c59ea23f72b4 |
| virology/validation-00000-of-00001.parquet | Data | 7.0 KB | fcc8b6cd99a0 |
| world_religions/dev-00000-of-00001.parquet | Data | 3.3 KB | 5676e31ed09b |
| world_religions/test-00000-of-00001.parquet | Data | 18.9 KB | b59bdb0cb4b1 |
| world_religions/validation-00000-of-00001.parquet | Data | 4.9 KB | 769666ee1ad8 |
| README.md | Documentation | 53.2 KB | — |
| hendrycks_test.py | Other | 5.7 KB | — |
| .gitattributes | Repository | 2.3 KB | — |
License and Download
- License
- mit
- Access
- No access gate
Download from Center for AI Safety
Released by Center for AI Safety through its official repository on Hugging Face. Read the license.