audio-eval-suite

A small, standardized audio-robustness evaluation set packaged to drop into eval loops. It pairs fully synthetic, license-clean speech-like probe signals with 300 real-synthetic room impulse responses (RIRs) and a labeled degradation chain (reverberation + additive noise), so you can measure how a model's acoustic predictions hold up as rooms get more reverberant and noisier.

Think of it as a quick sanity/robustness check: one line to load, a concrete metric, and ground-truth room acoustics for every clip.

Dataset at a glance

Measured ground-truth ranges (from the underlying RIRs):

• T60: 0.286 – 8.395 s
• DRR: −16.73 – 22.77 dB
• C50: −5.00 – 27.97 dB
• reverb_bin distribution: mild 875, strong 600, clean 25

How it was built

Sources are synthetic speech-LIKE probe signals, not real speech. Each is a seeded excitation: harmonic (voiced) segments with a time-varying F0 (80–250 Hz), a 3–4 resonance formant filter (300–3500 Hz), and 3–7 Hz syllabic amplitude modulation, interleaved with fricative-like band-limited noise bursts (unvoiced) and short silences. This produces a speech-like ~4–8 Hz envelope while remaining fully reproducible from a seed and free of any speech-corpus licensing.

Degradation chain, all seeded and exactly labeled:

reverberant = fftconvolve(source, rir)        # trimmed to source length
noisy       = reverberant + white_noise(snr)  # snr in {10, 5, 0} dB

The RIRs are reused from RIR-Bench-Hard, a 2,600-RIR adversarial bank; this suite draws 300 of them stratified across the five room categories, preferring the harder DRR/T60 tails.

Loading

from datasets import load_dataset

ds = load_dataset("mandipgoswami/audio-eval-suite", split="test")
print(ds)
print(ds[0])            # one (id, condition) row with ground-truth acoustics

Audio is referenced by relative path in the audio / audio_path columns. To decode a clip:

import soundfile as sf
from huggingface_hub import snapshot_download
root = snapshot_download("mandipgoswami/audio-eval-suite", repo_type="dataset")
wav, fs = sf.read(f"{root}/{ds[0]['audio']}")   # fs == 16000

Data fields (test.parquet)

Reference tasks & metrics

Two concrete, scorable tasks ship with the set (full spec in tasks/README.md):

• Task A — Blind reverb-level classification. Predict reverb_bin (clean/mild/strong). Metric: top-1 accuracy (overall + per-condition).
• Task B — RT60 regression. Predict t60_s from a clip. Metric: MAE (and RMSE), scored on reverberant/noisy rows (clean excluded).

Trivial baselines: majority-bin for Task A, mean-t60_s for Task B.

Intended uses

• A fast robustness/sanity eval for dereverberation, robust-ASR front-ends, and room-acoustic estimators.
• A controlled probe for how performance degrades with reverberation and noise, with exact ground truth per clip.

Limitations & biases

• Sources are synthetic, not real speech. They reproduce speech-like spectro- temporal structure but contain no phonetic/linguistic content, so this set does not measure word-level ASR or speaker traits.
• Noise is stationary white at fixed SNRs; real-world noise is more varied.
• RIRs are synthetic (image-source / simulated), not measured in physical rooms.
• The suite is intentionally small — it is a sanity/robustness probe, not a training corpus.

Related

• RIR-Bench-Hard — the 2,600-RIR adversarial bank these RIRs come from.
• Part of the rirmega line of reproducible room-acoustics resources.

Citation

@misc{goswami2026audioevalsuite,
  title  = {Audio Eval Suite: A Reproducible Robustness Set for Reverberant,
            Noisy Audio},
  author = {Goswami, Mandip},
  year   = {2026},
  howpublished = {Hugging Face Datasets},
  note   = {\url{https://huggingface.co/datasets/mandipgoswami/audio-eval-suite}}
}

License

Released under CC-BY-4.0. Sources are synthetic and RIRs are synthetic, so the set carries no third-party speech-corpus restrictions.