GGUF quantizations of the CED family (Consistent Ensemble Distillation, Xiaomi) - SOTA-tier audio-tagging models that classify everyday sounds (baby cry, footsteps, glass breaking, alarms, dog bark,...) into the 527-class AudioSet ontology. These files run with ced.cpp, a standalone C++/ggml port (no Python, no PyTorch at inference), and with LocalAI via the ced backend. Converted from the mispeech/ced- checkpoints (Apache-2.0). CED is a plain AST/DeiT Vision Transformer over a log-mel spectrogram; the port is numerically equal to the PyTorch reference. One self-contained GGUF per size + quant (config, 527 labels, and the mel filterbank/window are all embedded). Pick by your accuracy/size…
Open-weight model · Audio classification
wav2vec2-base-superb-ks
by Superb superb/wav2vec2-base-superb-ks
This is a ported version of The base model is wav2vec2-base, which is pretrained on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.
Parameters—
Context—
Weights378.4 MB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads14.1k
Model Card
By Superb, published under apache-2.0, revision 372e0486cd83.
This is a ported version of The base model is wav2vec2-base, which is pretrained on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz. For more information refer to SUPERB: Speech processing Universal PERformance Benchmark Keyword Spotting (KS) detects preregistered keywords by classifying utterances into a predefined set of words. The task is usually performed on-device for the fast response time. Thus, accuracy, model size, and inference time are all crucial. SUPERB uses the widely used Speech Commands dataset v1.0 for the task. The dataset consists of ten classes of keywords, a class for silence, and an unknown class to include the…
Read Superb's full model card
Wav2Vec2-Base for Keyword Spotting
Model description
This is a ported version of S3PRL's Wav2Vec2 for the SUPERB Keyword Spotting task.
The base model is wav2vec2-base, which is pretrained on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.
For more information refer to SUPERB: Speech processing Universal PERformance Benchmark
Task and dataset description
Keyword Spotting (KS) detects preregistered keywords by classifying utterances into a predefined set of words. The task is usually performed on-device for the fast response time. Thus, accuracy, model size, and inference time are all crucial. SUPERB uses the widely used Speech Commands dataset v1.0 for the task. The dataset consists of ten classes of keywords, a class for silence, and an unknown class to include the false positive.
For the original model's training and evaluation instructions refer to the S3PRL downstream task README.
Usage examples
You can use the model via the Audio Classification pipeline:
from datasets import load_dataset
from transformers import pipeline
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
classifier = pipeline("audio-classification", model="superb/wav2vec2-base-superb-ks")
labels = classifier(dataset[0]["file"], top_k=5)
Or use the model directly:
import torch
from datasets import load_dataset
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor
from torchaudio.sox_effects import apply_effects_file
effects = [["channels", "1"], ["rate", "16000"], ["gain", "-3.0"]]
def map_to_array(example):
speech, _ = apply_effects_file(example["file"], effects)
example["speech"] = speech.squeeze(0).numpy()
return example
# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
dataset = dataset.map(map_to_array)
model = Wav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-base-superb-ks")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/wav2vec2-base-superb-ks")
# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")
logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
Eval results
The evaluation metric is accuracy.
| s3prl | transformers | |
|---|---|---|
| test | 0.9623 |
0.9643 |
BibTeX entry and citation info
@article{yang2021superb,
title={SUPERB: Speech processing Universal PERformance Benchmark},
author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
journal={arXiv preprint arXiv:2105.01051},
year={2021}
}
Configuration
- Architecture
- Wav2Vec2ForSequenceClassification
- Layers
- 12
- Hidden size
- 768
- Feed-forward size
- 3,072
- Attention heads
- 12
- Vocabulary size
- 32
- Stored precision
- float32
- Model type
- wav2vec2
Identity and Version
- Repository
- superb/wav2vec2-base-superb-ks
- Publisher
- Superb
- Task
- Audio classification
- Modality
- Audio
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- en
- Revision
- 372e0486cd83e6f0c05c20a27262e9ca09450d24
- First published
- 2022-03-02
- Last updated
- 2021-11-04
Files and Weights
5 files, 378.4 MB in total. The weights are 1 file totalling 378.4 MB in bin.
Weights1 file · 378.4 MB
Configuration2 files · 2.6 KB
Documentation1 file · 3.7 KB
Repository1 file · 1.2 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| pytorch_model.bin | Weights | 378.4 MB | 54b662ca0e56 |
| config.json | Configuration | 2.4 KB | — |
| preprocessor_config.json | Configuration | 215 B | — |
| README.md | Documentation | 3.7 KB | — |
| .gitattributes | Repository | 1.2 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 378.4 MB
Download from Superb
Released by Superb through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:2105.01051
- Trained on (disclosed) superb
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 378.4 MB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About wav2vec2-base-superb-ks
Can I use wav2vec2-base-superb-ks commercially?
Yes. wav2vec2-base-superb-ks is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
Similar Models
This is a spoken language recognition model trained on the VoxLingua107 dataset using SpeechBrain. The model uses the ECAPA-TDNN architecture that has previously been used for speaker recognition. However, it uses more fully connected hidden layers after the embedding layer, and cross-entropy loss was used for training. We observed that this improved the performance of extracted utterance embeddings for downstream tasks. The system is trained with recordings sampled at 16kHz (single channel). The code will automatically normalize your audio (i.e., resampling + mono channel selection) when calling classifyfile if needed. The model can classify a speech utterance according to the language…
Abstract: The recognition of accented speech still remains a dominant problem in Automatic Speech Recognition (ASR) systems. We approach the classification of accented English speech through the Emphasized Channel Attention, Propagation and Aggregation Time Delay Neural Network (ECAPA-TDNN) architecture which has been shown to perform well on a variety of speech tasks. Three models are proposed: one trained from scratch, another two models (one using data augmentation and a baseline model) fine-tuned from the checkpoints of speechbrain/spkrec-ecapa-voxceleb (VoxCeleb). Our results show that the model fine-tuned with data augmentation yield the best results. Most of the misclassifications…
The development log of our Music Audio Pre-training (m-a-p) model family: - 02/06/2023: arxiv pre-print and training codes released. - 17/03/2023: we release two advanced music understanding models, MERT-v1-95M and MERT-v1-330M, trained with new paradigm and dataset. They outperform the previous models and can better generalize to more tasks. - 14/03/2023: we retrained the MERT-v0 model with open-source-only music dataset MERT-v0-public - 29/12/2022: a music understanding model MERT-v0 trained with MLM paradigm, which performs better at downstream tasks. - 29/10/2022: a pre-trained MIR model music2vec trained with BYOL paradigm. Here is a table for quick model pick-up: The m-a-p models…
The development log of our Music Audio Pre-training (m-a-p) model family: - 02/06/2023: arxiv pre-print and training codes released. - 17/03/2023: we release two advanced music understanding models, MERT-v1-95M and MERT-v1-330M, trained with new paradigm and dataset. They outperform the previous models and can better generalize to more tasks. - 14/03/2023: we retrained the MERT-v0 model with open-source-only music dataset MERT-v0-public - 29/12/2022: a music understanding model MERT-v0 trained with MLM paradigm, which performs better at downstream tasks. - 29/10/2022: a pre-trained MIR model music2vec trained with BYOL paradigm. Here is a table for quick model pick-up: The m-a-p models…
This model combines the SpeechBrain ECAPA-TDNN speaker embedding model with an SVM classifier to predict speaker gender from audio input. The model was trained and evaluated on the VoxCeleb2, Mozilla Common Voice v10.0, and TIMIT datasets - Mozilla Common Voice v10.0 English validated test set: 92.3% accuracy The model was trained on VoxCeleb2 dataset: - Converted to WAV format, single channel, 16kHz sampling rate, 256 kp/s bitrate - Applied SileroVAD for voice activity detection, taking the first voiced segment You can install the package directly from GitHub: - Model was trained on celebrity voices from YouTube interviews - Performance may vary on different audio qualities or recording…