unnamed-split-ar

Arabic YODAS v3, cut into utterance-length clips

The data/ar/ partition of espnet/yodas3, cut from long recordings (median ~5 minutes) into clips of 5-45 seconds, each with its own transcript. Ready to batch for ASR training without further segmentation.

from datasets import load_dataset
ds = load_dataset("oddadmix/unnamed-split-ar", split="train", streaming=True)

Audio is Opus 48 kbps mono at 16 kHz in Ogg.

Where the cuts come from

Upstream timestamps are YouTube caption cues, not utterance segments. Measured on a shard of data/ar:

• 71.8% of consecutive cues strictly overlap (median gap -2.36 s): start + duration is how long a cue stays on screen, not when its speech ends.
• Only 9.7% of cue boundaries have a real silence gap >= 0.3 s, so cutting on caption boundaries puts most cuts inside continuous speech.

Snapping caption boundaries to the quietest nearby window does not fix this: scored by an independent VAD, 54.8% of such cuts still landed in speech. So the cuts here are chosen the other way round -- Silero VAD finds the pauses, pauses at least 150 ms long are the only legal cut points, and the transcript is then assigned to whatever span results. Measured the same way: 0.0% of cut points land in speech.

Because text follows the cut, word timings do the assignment where the cue has them (68% of cues), and majority overlap otherwise. cues keeps the contributing cue fragments with times rebased to the clip and a partial flag, so word-level alignment stays derivable.

clean_cut is false when a stretch of speech longer than 45 s contained no pause and the cut had to be forced past the limit; those are dropped here. speech_ratio is the fraction of the clip the VAD calls speech.

start/end locate the clip in the source recording and source_id groups clips cut from the same one -- split by source_id, not by row, or the same speaker and topic land on both sides of your split.

Filtering applied

• Cues with no text, and consecutive duplicate cue text (some recordings repeat one channel-outro caption for minutes), are dropped.
• Non-speech markers ([موسيقى], [ضحك], ...) are stripped; clips left with no text go.
• Clips below 35% speech by VAD, and outside 1-30 characters per second, are dropped as silence or misalignment.

Dialect labels

dialect is one of 13 Arabic dialects or msa from Nawah-Dialect-BERT-6M, predicted on the whole source recording and copied to each of its clips -- a single clip carries too little text to classify reliably. dialect_probs has all 14 class probabilities.

These are predictions, not ground truth: they read the transcript's language register rather than the speaker's accent, and the transcripts are themselves machine-generated.

Provenance

CC-BY-3.0, inherited from YODAS v3. Audio was decoded from upstream Opus, resampled to 16 kHz mono and re-encoded, so it is a second lossy generation.