german-english-voice-acting-challenges

Five-language expansion and stable voice aliases

Already available: 38,427 English/German recordings with complete annotations and MOSS v2 targets. New expansion fully uploaded: 36,452 recordings in 161 archives. A further 177,808 recordings is the production target; completed counts are reported separately.

Five-language release, models and loading guide · Actual uploaded statistics · Stable 30-voice alias table · GENERAL/SCRIPT listening demo · Annotation pipeline

New completed shards in multilingual/data/{train,validation,test}/ contain the audio, all acoustic score/embedding sidecars, both MOSS captions, the default GENERAL/SCRIPT training prompt, complete voice-generation provenance and 12-codebook MOSS Audio Tokenizer v2 targets. The expansion adds French, Dutch and Spanish and extra performances in all five languages. Original data/ and tuning/ archives remain available. Canonical synthetic speaker aliases preserve earlier names and keep the exact provider voice. New annotations/media are CC BY 4.0; upstream model licenses remain their own.

38,427 synthetic acting recordings · 314.13 hours · 151 WebDataset TARs. English: 19,215; German: 19,212. Every sample includes complete acoustic predictions, whole-audio and interval-conditioned MOSS captions, ASR words, a verified timed procedural script, nine procedural wording variants, and six selectable MOSS CAPTION/TRANSCRIPT variants.

▶ Bilingual listening demo · Same-audio MOSS/procedural comparison · One complete annotation with audio and every score · Complete example JSON · Example audio · Exact prompt grammar

Generation and provenance

The recordings are synthetically generated voice acting from LAION's Voice-Acting-Pipeline, covering psychological, situational, physical and voice-style challenges. The delivery condition is Sparse vocal response: naturalistic emotion transitions, restrained imperfections, and occasional situation-appropriate vocal reactions. Audio remains byte-identical 128-kbit/s mono MP3. This release contains successful recordings; 67 planned challenge/language pairs are absent. The 30 generation voices were assigned to balance each subset/language; missing performances slightly change the successful distribution.

generation_metadata preserves the original provider/model identifiers, exact Gemini voice, language, scenario, script, script-writing prompts, speech-generation prompts and request configurations, usage records, source revision, and voice-assignment seed wherever present in the source. The readable title does not replace this provenance. Only credential fields and machine-local file addresses are removed. generation_metadata_original_sha256 identifies the untouched source metadata. Intended emotions and original scripts were withheld from annotation inference.

Annotation pipeline and model properties

1. Humaneness Ears Medium, pinned revision 818506970d93c809a3295a0a02c40dee8ff2bfc3, is a Whisper Small encoder-only multitask student; it does not perform ASR. It supplies 192 raw regression targets (40 emotions, 57 VoiceNet dimensions, genuineness/blend, EIV extras, AudioBox, DNSMOS and VoiceCLAP attributes), CPS, estimated burst count, 128D Orange timbre, 250D Orange identity, 20-ms burst probabilities, onset/duration proposals and event-local top-three labels from a 53-class head. Raw, training-standardized and mixed-reference-normalized outputs remain available.
2. MOSS Voice Captioner describes the complete audio. MOSS Vocal Burst Captioner receives that same complete audio plus both start and end seconds for each selected event. The repositories include the base weights, separate rank-32 LoRAs, inference/training code and recipes. These recordings were not their adapter-training corpus. Inference uses greedy decoding, threshold 0.6, minimum duration 0.08 s, suppression when overlap exceeds half the shorter interval, and no selected-event cap.
3. Parakeet v3 supplies German/English ASR and word endpoints. Procedural Voice Captions uses threshold 0.7, the 80,265-clip S10 + multilingual Emilia speech reference, two distinctive VoiceNet properties, up to one emotion with z ≥ +0.5, tempo and compact tags. Pleasantness comes from predicted AudioBox Content Enjoyment, not VoiceNet ESTH. Blend never declares burst absence. Speech-reference z-values used for wording do not overwrite canonical mixed-reference z-values.
4. All selected events retain their exact start/end seconds. MOSS and procedural locator thresholds can select different counts; training transcript variants use the same intervals as the corresponding MOSS event captions. Original acoustic proposals and earlier procedural surfaces are retained. Every new procedural script passes word-preservation and time-budget checks. An empty ASR result remains empty rather than being replaced with the intended generation script.

Teacher taxonomies and packages: EmoNet / Empathic Insight, VoiceNet taxonomy, VoiceNet predictors, genuineness, vocal-burst blend, Orange timbre/identity generation, AudioBox Aesthetics, DNSMOS. These are distilled student predictions, not a fresh execution of all teachers.

Choose your Humaneness Voice training surface

The six entries in humaneness_voice_training_formats.variants combine two caption choices and three transcript choices:

Each entry contains caption, transcript, and the serialized prompt: CAPTION: ... followed by TRANSCRIPT: "...". The plain transcript is verbatim ASR. Inline transcripts add round-bracket vocal events with measured durations and square-bracket pauses. Fully timed transcripts also add a duration on every spoken segment. The no-appended-burst choice retains the original whole-audio prose unchanged; that prose may itself mention vocal events. It does not mean the audio is burst-free.

Procedural captions separately use GENERAL: and SCRIPT:. The original Humaneness Voice guide uses square-bracket speech durations, round-bracket vocal-event durations, and square-bracket pauses. Timed overlapping bursts retain full durations inside an enclosing segment budget rather than advancing the clock twice. This is an explicitly documented extension of the historical linear parser; training consumers must use the supplied parser and absolute JSON event timestamps. See Humaneness Voice Small, the complete grammar, and code/prompt_format.py.

Nine procedural.procedural_caption_variants change only seeded general wording; they share the verified script and do not represent nine independently inferred annotations. Earlier procedural text is retained under previous_caption_surface, with previous sentence wording alongside updated wording.

WebDataset schema and reproduction

Step-by-step reproduction and training-format selection.

Each sample is <id>.mp3 plus <id>.json. Major JSON namespaces are audio, generation_metadata, performance, humaneness_ears_medium, moss, procedural, humaneness_voice_training_formats, and annotation_provenance. index.jsonl identifies each shard, language, exact voice, duration and event count. inventory.json records TAR sizes/counts/SHA-256; summary.json records full coverage. Both original inference settings and the new format lock are in provenance/. Reproduction modules and normalizers are in code/; model weights remain in the linked repositories.

import json, webdataset as wds
from huggingface_hub import snapshot_download
root = snapshot_download("laion/german-english-voice-acting-challenges", repo_type="dataset")
url = root + "/data/voice-acting-captioned-{00000..00150}.tar"
for sample in wds.WebDataset(url, shardshuffle=False):
    annotation = json.loads(sample["json"])
    prompt = annotation["humaneness_voice_training_formats"]["variants"]["with_burst_descriptions__timed"]["prompt"]
    voice = annotation["generation_metadata"]["voice"]
    audio_mp3_bytes = sample["mp3"]

Measured quality and limitations

On the same 50 acting audios, a blinded Gemini audio reviewer gave uncapped greedy MOSS 3.863/5 and compact procedural captions 2.704/5 under one shared rubric (global fidelity 30%, burst description 25%, timing 20%, event precision 15%, event recall 10%). A 25/25 split selected and confirmed the procedural configuration; its quality tied the previous prose overall. These are automated ratings, not human ground truth or precision/recall percentages. The complete report includes every audio, dimensions, configuration results, confirmation results and limitations. Historical twenty-audio tuning used a different rubric and is not directly comparable.

Format verification confirms numerical serialization and word preservation, not annotation accuracy. Predicted boundaries can include ordinary speech or miss quiet bursts; model prose can invent details. Raw regression scores and reference z-values are not probabilities. Conditional class softmax is not calibrated event confidence. Synthetic acting is not representative of all natural speech. No audio is manually adjudicated in this release. Audio and original generation metadata are retained additively; raw source archives are not rewritten.

License

Synthetic media and newly produced annotations: CC BY 4.0, LAION. Attribute this dataset, the original acting scenarios and linked pipeline/model authors. External implementations and base models retain their own licenses; the dataset license does not replace those licenses. LICENSE contains the complete CC BY 4.0 text.

Humaneness Voice tuning export (11 October 2026)

All 38,427 recordings (314.13 hours) additionally have the Humanness Voice Annotator GENERAL/SCRIPT prompt and MOSS Audio Tokenizer v2 targets for later Humaneness Voice Small tuning. This is an additive data-preparation release; no new TTS model training is reported.

Download token shards and annotations · Exact statistics · Training/loading guide · Explanatory page · Listen to the default format

The 151 additive WebDataset TARs live under tuning/data/. Each sample has a lightweight JSON training record, an int16 [frames,12] codes.npy, and a compressed canonical annotation retaining raw acoustic scores/embeddings, exact intervals, MOSS descriptions and local delivery cues. Original MP3s and all previous annotations remain under data/; match by ID and audio SHA-256. Generation metadata, including the precise synthetic voice, is retained.

The separately loaded MOSS Audio Tokenizer v2 encodes 48-kHz audio using its first 12 RVQ codebooks, one frame per 80 ms. Mono is duplicated to stereo, and the partial final frame is explicitly zero-padded. True length/padding and exact producer revisions are recorded. Tokens means the frame count, not the number of codebook indices. Decode and trim using original_samples_48khz.

Challenge-level splits keep English/German counterparts together; the 50 earlier listening scenarios are additionally reserved for test. These are future-TTS preparation splits, not speaker holdouts or independent component-model benchmark claims. Use the custom split-aware loader in the guide. The default Hugging Face Dataset Viewer continues to read the original audio archives; it does not expose the training splits.

New annotations and code are CC BY 4.0. Upstream model/code licenses remain their own; the neural codec is Apache 2.0. Model predictions remain estimates and should be reviewed before training.