SAVRN
Search Contact SAVRN

Dataset · Text classification

blimp

by NYU Machine Learning for Language nyu-mll/blimp

BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics.

Rows12,000
Configurations12
Size4.0 MB
Licensecc-by-4.0
AccessPublicly accessible
Monthly Downloads153.3k

Dataset Card

By NYU Machine Learning for Language, published under cc-by-4.0, revision 877fba0801ff.

Dataset Card for "blimp"

Table of Contents

  • Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
  • Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
  • Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
  • Personal and Sensitive Information
  • Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
  • Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

Dataset Summary

BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars.

Supported Tasks and Leaderboards

More Information Needed

Languages

Read the full dataset card (1,083 words)

Structure

adjunct_island 1,000 rows

SplitRowsSize
train1,000166.0 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

anaphor_gender_agreement 1,000 rows

SplitRowsSize
train1,000131.0 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

anaphor_number_agreement 1,000 rows

SplitRowsSize
train1,000140.0 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

animate_subject_passive 1,000 rows

SplitRowsSize
train1,000144.5 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

animate_subject_trans 1,000 rows

SplitRowsSize
train1,000127.9 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

causative 1,000 rows

SplitRowsSize
train1,000122.9 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

complex_NP_island 1,000 rows

SplitRowsSize
train1,000199.1 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

coordinate_structure_constraint_complex_left_branch 1,000 rows

SplitRowsSize
train1,000211.0 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

coordinate_structure_constraint_object_extraction 1,000 rows

SplitRowsSize
train1,000171.8 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

determiner_noun_agreement_1 1,000 rows

SplitRowsSize
train1,000156.2 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

determiner_noun_agreement_2 1,000 rows

SplitRowsSize
train1,000156.3 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

determiner_noun_agreement_irregular_1 1,000 rows

SplitRowsSize
train1,000164.6 KB
sentence_goodstringsentence_badstringfieldstringlinguistics_termstringUIDstringsimple_LM_methodboolone_prefix_methodbooltwo_prefix_methodboollexically_identicalboolpair_idint32

Details

Repository
nyu-mll/blimp
Publisher
NYU Machine Learning for Language
Task category
Text classification
Tags
Not stated by the source
Size category
10K<n<100K
Languages
en
Revision
877fba0801ffb7cbd8c39c1ff314a46f053f6036
Last updated
2024-01-23

SAVRN's Notes on blimp

BLiMP is a challenge set for English grammar knowledge, built as minimal pairs isolating contrasts in syntax, morphology and semantics. The card describes 67 sub-datasets of 1,000 pairs each, while the page metadata shows 12,000 rows across 12 configurations and 4.0 MB, so verify which subset your loader is returning before quoting results. Downloads run 153.3k a month, largely as an eval fixture.

CC-BY-4.0 allows commercial use with attribution, and the entire set is 4.0 MB, so cost is irrelevant. The discipline that matters is eval hygiene: minimal-pair sets of this size memorize fast, and any fine-tuning on them voids your benchmark comparison. Keep it strictly in the evaluation slot, report the version hash in your results, and never let it leak into pretraining corpora.

SAVRN Research

Files

69 files, 4.0 MB in total.

Data67 files · 3.9 MB
Documentation1 file · 58.6 KB
Repository1 file · 1.2 KB
Every file
FileTypeSizeSHA-256
adjunct_island/train-00000-of-00001.parquetData62.2 KB094b1a6c4d8e
anaphor_gender_agreement/train-00000-of-00001.parquetData39.2 KB3c3f1d8d9e6a
anaphor_number_agreement/train-00000-of-00001.parquetData41.5 KB5dc16d19b9b3
animate_subject_passive/train-00000-of-00001.parquetData47.3 KBd4c957edec77
animate_subject_trans/train-00000-of-00001.parquetData49.7 KBef2f412c3f47
causative/train-00000-of-00001.parquetData49.0 KBcfec48a03e32
complex_NP_island/train-00000-of-00001.parquetData78.2 KB4a01cfd34626
coordinate_structure_constraint_complex_left_branch/train-00000-of-00001.parquetData67.9 KBb776239fe670
coordinate_structure_constraint_object_extraction/train-00000-of-00001.parquetData51.6 KB55d830a519e5
determiner_noun_agreement_1/train-00000-of-00001.parquetData49.9 KBeac75d95c9fa
determiner_noun_agreement_2/train-00000-of-00001.parquetData49.5 KB17520554fff0
determiner_noun_agreement_irregular_1/train-00000-of-00001.parquetData47.3 KBb73d7c301e81
determiner_noun_agreement_irregular_2/train-00000-of-00001.parquetData47.4 KB73c404f947d0
determiner_noun_agreement_with_adj_2/train-00000-of-00001.parquetData56.3 KB732f6dc785f2
determiner_noun_agreement_with_adj_irregular_1/train-00000-of-00001.parquetData54.4 KB939567391e6e
determiner_noun_agreement_with_adj_irregular_2/train-00000-of-00001.parquetData54.1 KBe71c6a9baf9c
determiner_noun_agreement_with_adjective_1/train-00000-of-00001.parquetData55.7 KBdd61dd23aa32
distractor_agreement_relational_noun/train-00000-of-00001.parquetData59.6 KB2e0cef46676c
distractor_agreement_relative_clause/train-00000-of-00001.parquetData77.9 KBc23c03ee033d
drop_argument/train-00000-of-00001.parquetData40.0 KB4acf04febbe9
ellipsis_n_bar_1/train-00000-of-00001.parquetData92.8 KB58edad00d53c
ellipsis_n_bar_2/train-00000-of-00001.parquetData98.9 KB7506bd80abb5
existential_there_object_raising/train-00000-of-00001.parquetData76.6 KB1c63d5b77b5d
existential_there_quantifiers_1/train-00000-of-00001.parquetData51.6 KBac2e197d5530
existential_there_quantifiers_2/train-00000-of-00001.parquetData52.1 KBdf307122fda8
existential_there_subject_raising/train-00000-of-00001.parquetData59.5 KB1e281bf8a80a
expletive_it_object_raising/train-00000-of-00001.parquetData88.6 KB45a0a957726a
inchoative/train-00000-of-00001.parquetData39.8 KBcb452cd6ee37
intransitive/train-00000-of-00001.parquetData42.4 KBf04f66e0a30a
irregular_past_participle_adjectives/train-00000-of-00001.parquetData36.7 KB77d65967bbf6
irregular_past_participle_verbs/train-00000-of-00001.parquetData37.3 KBf6acd480c04c
irregular_plural_subject_verb_agreement_1/train-00000-of-00001.parquetData50.7 KBb4b07609019d
irregular_plural_subject_verb_agreement_2/train-00000-of-00001.parquetData42.7 KB66a1bf6bba6d
left_branch_island_echo_question/train-00000-of-00001.parquetData50.5 KB41a535903bc7
left_branch_island_simple_question/train-00000-of-00001.parquetData50.3 KB8c718b6f37cb
matrix_question_npi_licensor_present/train-00000-of-00001.parquetData51.9 KBa7ed0834104a
npi_present_1/train-00000-of-00001.parquetData52.0 KBcca3e48e8a08
npi_present_2/train-00000-of-00001.parquetData51.7 KB44cdf2e5c796
only_npi_licensor_present/train-00000-of-00001.parquetData51.4 KB8e26a57bc8d8
only_npi_scope/train-00000-of-00001.parquetData85.0 KBc5c14f42c274
passive_1/train-00000-of-00001.parquetData53.9 KB3bcb3d739721
passive_2/train-00000-of-00001.parquetData40.5 KBee29e69ca55e
principle_A_c_command/train-00000-of-00001.parquetData67.9 KB6d18116ede2c
principle_A_case_1/train-00000-of-00001.parquetData61.1 KB6615f7f1572b
principle_A_case_2/train-00000-of-00001.parquetData56.4 KBe3a13d2ab24f
principle_A_domain_1/train-00000-of-00001.parquetData59.1 KB5efcec2af824
principle_A_domain_2/train-00000-of-00001.parquetData58.5 KB5de4982ab8ba
principle_A_domain_3/train-00000-of-00001.parquetData52.9 KB25227d39f616
principle_A_reconstruction/train-00000-of-00001.parquetData44.5 KBbd57fef2d8e9
regular_plural_subject_verb_agreement_1/train-00000-of-00001.parquetData49.5 KB602d82e663e7
regular_plural_subject_verb_agreement_2/train-00000-of-00001.parquetData43.4 KB47b11c839024
sentential_negation_npi_licensor_present/train-00000-of-00001.parquetData54.8 KB2f2e30c84624
sentential_negation_npi_scope/train-00000-of-00001.parquetData90.2 KB06afb479723e
sentential_subject_island/train-00000-of-00001.parquetData56.7 KBa096e1ea8ff2
superlative_quantifiers_1/train-00000-of-00001.parquetData48.5 KBeba2e172058a
superlative_quantifiers_2/train-00000-of-00001.parquetData50.5 KB76eadb284f7c
tough_vs_raising_1/train-00000-of-00001.parquetData44.8 KBdefbb5c0cb9b
tough_vs_raising_2/train-00000-of-00001.parquetData61.5 KB01d9c6a6b0df
transitive/train-00000-of-00001.parquetData55.1 KB1c92b2316478
wh_island/train-00000-of-00001.parquetData52.8 KB63e455edde12
wh_questions_object_gap/train-00000-of-00001.parquetData70.0 KB5107d0b8a405
wh_questions_subject_gap/train-00000-of-00001.parquetData71.6 KBa58be1f30174
wh_questions_subject_gap_long_distance/train-00000-of-00001.parquetData98.9 KBca30473b9569
wh_vs_that_no_gap/train-00000-of-00001.parquetData71.7 KB92fc7e67b32b
wh_vs_that_no_gap_long_distance/train-00000-of-00001.parquetData95.5 KBe1aa6fd1c740
wh_vs_that_with_gap/train-00000-of-00001.parquetData60.3 KBc905f68e4432
wh_vs_that_with_gap_long_distance/train-00000-of-00001.parquetData84.1 KBc2adf0cb1714
README.mdDocumentation58.6 KB—
.gitattributesRepository1.2 KB—

License and Download

License
cc-by-4.0
Access
No access gate
Download from NYU Machine Learning for Language

Released by NYU Machine Learning for Language through its official repository on Hugging Face. Read the license.