Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Abstract

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

Full paper on arXiv · Code

Details

arXiv identifier

2110.01518

Published

2021-10-04

Authors

Prajjwal Bhargava, Aleksandr Drozd, Anna Rogers

Models That Cite This Paper

• Described bybert-tiny
• Described bybert-small-pii-detection