UmBERTo is a Roberta-based Language Model trained on large Italian Corpora and uses two innovative approaches: SentencePiece and Whole Word Masking. Now available at github.com/huggingface/transformers Marco Lodola, Monument to Umberto Eco, Alessandria 2019 UmBERTo-Commoncrawl-Cased utilizes the Italian subcorpus of OSCAR as training set of the language model. We used deduplicated version of the Italian corpus that consists in 70 GB of plain text data, 210M sentences with 11B words where the sentences have been filtered and shuffled at line level in order to be used for NLP research. This model was trained with SentencePiece and Whole Word Masking. These results refers to…