BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics.
Rows12,000
Configurations12
Size4.0 MB
Licensecc-by-4.0
AccessPublicly accessible
Monthly Downloads153.3k
Dataset Card
By NYU Machine Learning for Language, published under cc-by-4.0, revision 877fba0801ff.
Dataset Card for "blimp"
Table of Contents
- Dataset Description
- Dataset Summary
- Supported Tasks and Leaderboards
- Languages
- Dataset Structure
- Data Instances
- Data Fields
- Data Splits
- Dataset Creation
- Curation Rationale
- Source Data
- Annotations
- Personal and Sensitive Information
- Considerations for Using the Data
- Social Impact of Dataset
- Discussion of Biases
- Other Known Limitations
- Additional Information
- Dataset Curators
- Licensing Information
- Citation Information
- Contributions
Dataset Description
- Homepage:
- Repository: https://github.com/alexwarstadt/blimp
- Paper: BLiMP: The Benchmark of Linguistic Minimal Pairs for English
- Paper: https://arxiv.org/abs/1912.00582
- Point of Contact: More Information Needed
- Size of downloaded dataset files: 29.58 MB
- Size of the generated dataset: 11.45 MB
- Total amount of disk used: 41.03 MB
Dataset Summary
BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars.
Supported Tasks and Leaderboards
Languages
Structure
adjunct_island 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 166.0 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
anaphor_gender_agreement 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 131.0 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
anaphor_number_agreement 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 140.0 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
animate_subject_passive 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 144.5 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
animate_subject_trans 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 127.9 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
causative 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 122.9 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
complex_NP_island 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 199.1 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
coordinate_structure_constraint_complex_left_branch 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 211.0 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
coordinate_structure_constraint_object_extraction 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 171.8 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
determiner_noun_agreement_1 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 156.2 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
determiner_noun_agreement_2 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 156.3 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
determiner_noun_agreement_irregular_1 1,000 rows
| Split | Rows | Size |
|---|---|---|
| train | 1,000 | 164.6 KB |
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32
Details
- Repository
- nyu-mll/blimp
- Publisher
- NYU Machine Learning for Language
- Task category
- Text classification
- Tags
- Not stated by the source
- Size category
- 10K<n<100K
- Languages
- en
- Revision
- 877fba0801ffb7cbd8c39c1ff314a46f053f6036
- Last updated
- 2024-01-23
Files
69 files, 4.0 MB in total.
Data67 files · 3.9 MB
Documentation1 file · 58.6 KB
Repository1 file · 1.2 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| adjunct_island/train-00000-of-00001.parquet | Data | 62.2 KB | 094b1a6c4d8e |
| anaphor_gender_agreement/train-00000-of-00001.parquet | Data | 39.2 KB | 3c3f1d8d9e6a |
| anaphor_number_agreement/train-00000-of-00001.parquet | Data | 41.5 KB | 5dc16d19b9b3 |
| animate_subject_passive/train-00000-of-00001.parquet | Data | 47.3 KB | d4c957edec77 |
| animate_subject_trans/train-00000-of-00001.parquet | Data | 49.7 KB | ef2f412c3f47 |
| causative/train-00000-of-00001.parquet | Data | 49.0 KB | cfec48a03e32 |
| complex_NP_island/train-00000-of-00001.parquet | Data | 78.2 KB | 4a01cfd34626 |
| coordinate_structure_constraint_complex_left_branch/train-00000-of-00001.parquet | Data | 67.9 KB | b776239fe670 |
| coordinate_structure_constraint_object_extraction/train-00000-of-00001.parquet | Data | 51.6 KB | 55d830a519e5 |
| determiner_noun_agreement_1/train-00000-of-00001.parquet | Data | 49.9 KB | eac75d95c9fa |
| determiner_noun_agreement_2/train-00000-of-00001.parquet | Data | 49.5 KB | 17520554fff0 |
| determiner_noun_agreement_irregular_1/train-00000-of-00001.parquet | Data | 47.3 KB | b73d7c301e81 |
| determiner_noun_agreement_irregular_2/train-00000-of-00001.parquet | Data | 47.4 KB | 73c404f947d0 |
| determiner_noun_agreement_with_adj_2/train-00000-of-00001.parquet | Data | 56.3 KB | 732f6dc785f2 |
| determiner_noun_agreement_with_adj_irregular_1/train-00000-of-00001.parquet | Data | 54.4 KB | 939567391e6e |
| determiner_noun_agreement_with_adj_irregular_2/train-00000-of-00001.parquet | Data | 54.1 KB | e71c6a9baf9c |
| determiner_noun_agreement_with_adjective_1/train-00000-of-00001.parquet | Data | 55.7 KB | dd61dd23aa32 |
| distractor_agreement_relational_noun/train-00000-of-00001.parquet | Data | 59.6 KB | 2e0cef46676c |
| distractor_agreement_relative_clause/train-00000-of-00001.parquet | Data | 77.9 KB | c23c03ee033d |
| drop_argument/train-00000-of-00001.parquet | Data | 40.0 KB | 4acf04febbe9 |
| ellipsis_n_bar_1/train-00000-of-00001.parquet | Data | 92.8 KB | 58edad00d53c |
| ellipsis_n_bar_2/train-00000-of-00001.parquet | Data | 98.9 KB | 7506bd80abb5 |
| existential_there_object_raising/train-00000-of-00001.parquet | Data | 76.6 KB | 1c63d5b77b5d |
| existential_there_quantifiers_1/train-00000-of-00001.parquet | Data | 51.6 KB | ac2e197d5530 |
| existential_there_quantifiers_2/train-00000-of-00001.parquet | Data | 52.1 KB | df307122fda8 |
| existential_there_subject_raising/train-00000-of-00001.parquet | Data | 59.5 KB | 1e281bf8a80a |
| expletive_it_object_raising/train-00000-of-00001.parquet | Data | 88.6 KB | 45a0a957726a |
| inchoative/train-00000-of-00001.parquet | Data | 39.8 KB | cb452cd6ee37 |
| intransitive/train-00000-of-00001.parquet | Data | 42.4 KB | f04f66e0a30a |
| irregular_past_participle_adjectives/train-00000-of-00001.parquet | Data | 36.7 KB | 77d65967bbf6 |
| irregular_past_participle_verbs/train-00000-of-00001.parquet | Data | 37.3 KB | f6acd480c04c |
| irregular_plural_subject_verb_agreement_1/train-00000-of-00001.parquet | Data | 50.7 KB | b4b07609019d |
| irregular_plural_subject_verb_agreement_2/train-00000-of-00001.parquet | Data | 42.7 KB | 66a1bf6bba6d |
| left_branch_island_echo_question/train-00000-of-00001.parquet | Data | 50.5 KB | 41a535903bc7 |
| left_branch_island_simple_question/train-00000-of-00001.parquet | Data | 50.3 KB | 8c718b6f37cb |
| matrix_question_npi_licensor_present/train-00000-of-00001.parquet | Data | 51.9 KB | a7ed0834104a |
| npi_present_1/train-00000-of-00001.parquet | Data | 52.0 KB | cca3e48e8a08 |
| npi_present_2/train-00000-of-00001.parquet | Data | 51.7 KB | 44cdf2e5c796 |
| only_npi_licensor_present/train-00000-of-00001.parquet | Data | 51.4 KB | 8e26a57bc8d8 |
| only_npi_scope/train-00000-of-00001.parquet | Data | 85.0 KB | c5c14f42c274 |
| passive_1/train-00000-of-00001.parquet | Data | 53.9 KB | 3bcb3d739721 |
| passive_2/train-00000-of-00001.parquet | Data | 40.5 KB | ee29e69ca55e |
| principle_A_c_command/train-00000-of-00001.parquet | Data | 67.9 KB | 6d18116ede2c |
| principle_A_case_1/train-00000-of-00001.parquet | Data | 61.1 KB | 6615f7f1572b |
| principle_A_case_2/train-00000-of-00001.parquet | Data | 56.4 KB | e3a13d2ab24f |
| principle_A_domain_1/train-00000-of-00001.parquet | Data | 59.1 KB | 5efcec2af824 |
| principle_A_domain_2/train-00000-of-00001.parquet | Data | 58.5 KB | 5de4982ab8ba |
| principle_A_domain_3/train-00000-of-00001.parquet | Data | 52.9 KB | 25227d39f616 |
| principle_A_reconstruction/train-00000-of-00001.parquet | Data | 44.5 KB | bd57fef2d8e9 |
| regular_plural_subject_verb_agreement_1/train-00000-of-00001.parquet | Data | 49.5 KB | 602d82e663e7 |
| regular_plural_subject_verb_agreement_2/train-00000-of-00001.parquet | Data | 43.4 KB | 47b11c839024 |
| sentential_negation_npi_licensor_present/train-00000-of-00001.parquet | Data | 54.8 KB | 2f2e30c84624 |
| sentential_negation_npi_scope/train-00000-of-00001.parquet | Data | 90.2 KB | 06afb479723e |
| sentential_subject_island/train-00000-of-00001.parquet | Data | 56.7 KB | a096e1ea8ff2 |
| superlative_quantifiers_1/train-00000-of-00001.parquet | Data | 48.5 KB | eba2e172058a |
| superlative_quantifiers_2/train-00000-of-00001.parquet | Data | 50.5 KB | 76eadb284f7c |
| tough_vs_raising_1/train-00000-of-00001.parquet | Data | 44.8 KB | defbb5c0cb9b |
| tough_vs_raising_2/train-00000-of-00001.parquet | Data | 61.5 KB | 01d9c6a6b0df |
| transitive/train-00000-of-00001.parquet | Data | 55.1 KB | 1c92b2316478 |
| wh_island/train-00000-of-00001.parquet | Data | 52.8 KB | 63e455edde12 |
| wh_questions_object_gap/train-00000-of-00001.parquet | Data | 70.0 KB | 5107d0b8a405 |
| wh_questions_subject_gap/train-00000-of-00001.parquet | Data | 71.6 KB | a58be1f30174 |
| wh_questions_subject_gap_long_distance/train-00000-of-00001.parquet | Data | 98.9 KB | ca30473b9569 |
| wh_vs_that_no_gap/train-00000-of-00001.parquet | Data | 71.7 KB | 92fc7e67b32b |
| wh_vs_that_no_gap_long_distance/train-00000-of-00001.parquet | Data | 95.5 KB | e1aa6fd1c740 |
| wh_vs_that_with_gap/train-00000-of-00001.parquet | Data | 60.3 KB | c905f68e4432 |
| wh_vs_that_with_gap_long_distance/train-00000-of-00001.parquet | Data | 84.1 KB | c2adf0cb1714 |
| README.md | Documentation | 58.6 KB | — |
| .gitattributes | Repository | 1.2 KB | — |
License and Download
- License
- cc-by-4.0
- Access
- No access gate
Download from NYU Machine Learning for Language
Released by NYU Machine Learning for Language through its official repository on Hugging Face. Read the license.