Sicilian-Backt-ITA-to-SCN

Sicilian Backtranslations ITA-to-SCN

Published dictionaries have recorded the Sicilian language in a consistent orthography for several hundred years. Most notably: Pasqualino (1785), Traina (1868), Mortillaro (1876) and Nicotra (1883).

Nonetheless, the Sicilian language text that Facebook collected during the No Language Left Behind project (2022) exhibits extreme orthographic diversity which (when incorporated into a training dataset) causes a model to generate a patchwork of different Sicilian dialects with each phrase seemingly sourced from a different town or province.

And to make matters worse, very few of the Sicilian-English pairs that the NLLB project collected are translations, which makes the collection almost unusable as a source of parallel text.

To overcome those deficiencies and create a large dataset that one can use to train a Sicilian translation model, we treated the NLLB collection as two monolingual corpora and backtranslated (Sennrich et al. 2015) them.

Dataset Card -- backtranslations, intended use: ITA-to-SCN

Dataset Summary

This dataset contains 1,057,469 sequences from the Sicilian corpora backtranslated into Italian and scored on the task of Italian-to-Sicilian translation.

Like golf, a low score is a better score. The scores come from Sockeye's scorer, which estimates the negative log probability that the target subword sequence is a translation of the source subword sequence. So a score close to zero implies a probability close to one. Accordingly, a low score is a better score.

The dataset is a subset created from metadata for mined bitext released by Meta AI. The original contains bitext for 148 English-centric and 1465 non-English-centric language pairs using the stopes mining library and the LASER3 encoders (Heffernan et al, 2022). Subsequently, Allen AI prepared bilingual collections for Hugging Face and for OPUS. The dataset presented here contains 1,057,469 pairs from the OPUS collection.

Licensing Information

The dataset is released under the terms of ODC-BY. By using this, you are also bound to the respective Terms of Use and License of the original source.

Sources Cited

K. Hefferman et al (2022). "Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages."

V. Mortillaro (1876). Nuovo Dizionario Siciliano-Italiano.

V. Nicotra (1883). Dizionario Siciliano-Italiano.

NLLB Team et al (2022). "No Language Left Behind: Scaling Human-Centered Machine Translation."

M. Pasqualino (1785). Vocabolario siciliano etimologico, italiano e latino.

R. Sennrich, B. Haddow and A. Birch (2015). "Improving Neural Machine Translation Models with Monolingual Data."

A. Traina (1868). Nuovo vocabolario siciliano-italiano.