unnamed-ar

Arabic subset of YODAS v3

The data/ar/ partition of espnet/yodas3, repackaged as parquet with the audio inline so it loads without a script:

from datasets import load_dataset
ds = load_dataset("oddadmix/unnamed-ar", split="train", streaming=True)

Every row is one long-form recording (median ~5 minutes) with its full transcript.

What was changed

• Audio container only. Upstream ships Opus in a WebM/Matroska container, which libsndfile cannot open. Here it is Opus in Ogg, remuxed with ffmpeg -c:a copy — the Opus packets are copied untouched, so the audio is bit-identical to upstream. 48 kHz.
• Metadata joined. Upstream keeps audio in data/ar/audio/*.tar and the 12 metadata columns in data/ar/metadata/*.parquet; this version carries both in one row.
• Dialect columns added (see below).
• Nothing was filtered, resampled, re-encoded, segmented, or otherwise altered.

Dialect labels

dialect is one of 13 Arabic dialects or msa, predicted from the transcript by Nawah-Dialect-BERT-6M (5.98M params, 0.9428 accuracy / 0.9025 macro-F1 on its own held-out test set):

bh Bahraini · dz Algerian · eg Egyptian · iq Iraqi · lb Lebanese · ly Libyan · ma Moroccan · msa Modern Standard Arabic · ps Palestinian · sa Saudi · sd Sudanese · sy Syrian · tn Tunisian · ye Yemeni

Because these are long-form recordings and the classifier's window is 192 tokens, the label is the average of the softmax over up to 8 windows spread across the whole transcript, rather than a judgement on the opening sentence. dialect_score is the winning class probability and dialect_probs carries all 14, so you can apply your own confidence threshold.

These labels are predictions, not ground truth. Three caveats worth knowing:

• They come from the text, so they reflect the language register of the transcript, not the speaker's accent. A dialect speaker reading prepared MSA copy is labelled msa.
• The transcripts are themselves machine-generated (YouTube ASR), so transcription error feeds into the label.
• Rows with no transcript are null, not guessed — roughly 7% of rows in the shard sampled.

transcript is a JSON string: a list of {start, duration, text, words: [{t, w}]} segments with millisecond timestamps, so the recordings can be cut into utterances without force-alignment. length is the recording duration in seconds. Note transcript_lang and locale_lang can disagree — the transcript language is the one to trust for Arabic.

Provenance and license

CC-BY-3.0, inherited from YODAS v3, which crawled audio originally released under that license. Cite the upstream authors:

YODAS v3 — https://huggingface.co/datasets/espnet/yodas3