Sicilian-Backt-SCN-to-ITA

Sicilian Backtranslations SCN-to-ITA

Published dictionaries have recorded the Sicilian language in a consistent orthography for several hundred years. Most notably: Pasqualino (1785), Traina (1868), Mortillaro (1876) and Nicotra (1883).

Nonetheless, the Sicilian language text that Facebook collected during the No Language Left Behind project (2022) exhibits extreme orthographic diversity which (when incorporated into a training dataset) causes a model to generate a patchwork of different Sicilian dialects with each phrase seemingly sourced from a different town or province.

And to make matters worse, very few of the Sicilian-English pairs that the NLLB project collected are translations, which makes the collection almost unusable as a source of parallel text.

To overcome those deficiencies and create a large dataset that one can use to train a Sicilian translation model, we treated the NLLB collection as two monolingual corpora and backtranslated (Sennrich et al. 2015) them.

And for Sicilian-Italian data, we developed these backtranslations from WikiMatrix's Italian corpora.

Dataset Card -- backtranslations, intended use: SCN-to-ITA

Dataset Summary

This dataset contains 941,392 sequences from WikiMatrix's Italian corpora backtranslated into Sicilian and scored on the task of Sicilian-to-Italian translation.

Like golf, a low score is a better score. The scores come from Sockeye's scorer, which estimates the negative log probability that the target subword sequence is a translation of the source subword sequence. So a score close to zero implies a probability close to one. Accordingly, a low score is a better score.

Licensing Information

Facebook Research distributed the original mined data under the Creative Commons Attribution-ShareAlike license (CC BY-SA). By using this, you are also bound to the respective Terms of Use and License of the original source.

Sources Cited

V. Mortillaro (1876). Nuovo Dizionario Siciliano-Italiano.

V. Nicotra (1883). Dizionario Siciliano-Italiano.

NLLB Team et al (2022). "No Language Left Behind: Scaling Human-Centered Machine Translation."

M. Pasqualino (1785). Vocabolario siciliano etimologico, italiano e latino.

H. Schwenk et al (2021). "WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia."

R. Sennrich, B. Haddow and A. Birch (2015). "Improving Neural Machine Translation Models with Monolingual Data."

A. Traina (1868). Nuovo vocabolario siciliano-italiano.