SAVRN
Search Contact SAVRN

Open-weight model · Audio classification

emotion-recognition-wav2vec2-IEMOCAP

by SpeechBrain speechbrain/emotion-recognition-wav2vec2-IEMOCAP

This repository provides all the necessary tools to perform emotion recognition with a fine-tuned wav2vec2 (base) model using SpeechBrain. It is trained on IEMOCAP training data. For a better experience, we encourage you to learn more about SpeechBrain.

Parameters—
Context—
Weights377.6 MB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads43.7k

Model Card

By SpeechBrain, published under apache-2.0, revision 117a9c3dff08.

This repository provides all the necessary tools to perform emotion recognition with a fine-tuned wav2vec2 (base) model using SpeechBrain. It is trained on IEMOCAP training data. For a better experience, we encourage you to learn more about SpeechBrain. The model performance on IEMOCAP test set is: This system is composed of an wav2vec2 model. It is a combination of convolutional and residual blocks. The embeddings are extracted using attentive statistical pooling. The system is trained with Additive Margin Softmax Loss. Speaker Verification is performed using cosine distance between speaker embeddings. The system is trained with recordings sampled at 16kHz (single channel). The code will…

Read SpeechBrain's full model card



Emotion Recognition with wav2vec2 base on IEMOCAP

This repository provides all the necessary tools to perform emotion recognition with a fine-tuned wav2vec2 (base) model using SpeechBrain. It is trained on IEMOCAP training data.

For a better experience, we encourage you to learn more about SpeechBrain. The model performance on IEMOCAP test set is:

Release Accuracy(%)
19-10-21 78.7 (Avg: 75.3)

Pipeline description

This system is composed of an wav2vec2 model. It is a combination of convolutional and residual blocks. The embeddings are extracted using attentive statistical pooling. The system is trained with Additive Margin Softmax Loss. Speaker Verification is performed using cosine distance between speaker embeddings.

The system is trained with recordings sampled at 16kHz (single channel). The code will automatically normalize your audio (i.e., resampling + mono channel selection) when calling classify_file if needed.

Install SpeechBrain

First of all, please install the development version of SpeechBrain with the following command:

pip install git+https://github.com/speechbrain/speechbrain.git@develop

Please notice that we encourage you to read our tutorials and learn more about SpeechBrain.

Perform Emotion recognition

An external py_module_file=custom.py is used as an external Predictor class into this HF repos. We use foreign_class function from speechbrain.pretrained.interfaces that allow you to load you custom model. 

from speechbrain.inference.interfaces import foreign_class
classifier = foreign_class(source="speechbrain/emotion-recognition-wav2vec2-IEMOCAP", pymodule_file="custom_interface.py", classname="CustomEncoderWav2vec2Classifier")
out_prob, score, index, text_lab = classifier.classify_file("speechbrain/emotion-recognition-wav2vec2-IEMOCAP/anger.wav")
print(text_lab)

The prediction tensor will contain a tuple of (embedding, id_class, label_name).

Inference on GPU

To perform inference on the GPU, add run_opts={"device":"cuda"} when calling the from_hparams method.

Training

The model was trained with SpeechBrain (aa018540). To train it from scratch follows these steps: 1. Clone SpeechBrain:

git clone https://github.com/speechbrain/speechbrain/
  1. Install it:
cd speechbrain
pip install -r requirements.txt
pip install -e .
  1. Run Training:
cd  recipes/IEMOCAP/emotion_recognition
python train_with_wav2vec2.py hparams/train_with_wav2vec2.yaml --data_folder=your_data_folder

You can find our training results (models, logs, etc) here.

Limitations

The SpeechBrain team does not provide any warranty on the performance achieved by this model when used on other datasets.

Citing SpeechBrain

Please, cite SpeechBrain if you use it for your research or business.

@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}

About SpeechBrain

  • Website: https://speechbrain.github.io/
  • Code: https://github.com/speechbrain/speechbrain/
  • HuggingFace: https://huggingface.co/speechbrain/

Identity and Version

Repository
speechbrain/emotion-recognition-wav2vec2-IEMOCAP
Publisher
SpeechBrain
Task
Audio classification
Modality
Audio
Library
speechbrain
Parameters
Not stated by the source
Languages
en
Revision
117a9c3dff08be81a3628eecf6a66b547ec1659b
First published
2022-03-02
Last updated
2024-07-23

Files and Weights

11 files, 378.2 MB in total. The weights are 2 files totalling 377.6 MB in ckpt.

Weights2 files · 377.6 MB
Configuration2 files · 7.7 KB
Documentation1 file · 4.2 KB
Other5 files · 559.1 KB
Repository1 file · 1.4 KB
Every file
FileTypeSizeSHA-256
model.ckptWeights13.2 KB e351bafc8c0a
wav2vec2.ckptWeights377.6 MB b6db170a487b
custom_interface.pyConfiguration6.0 KB —
hyperparams.yamlConfiguration1.6 KB —
README.mdDocumentation4.2 KB —
anger.wavOther201.1 KB —
hap.wavOther168.4 KB —
label_encoder.txtOther83 B —
neutral.wavOther60.5 KB —
sad.wavOther129.0 KB —
.gitattributesRepository1.4 KB —

License and Download

License
apache-2.0
Access
Open weights, no gate
Download size
377.6 MB
Download from SpeechBrain

Released by SpeechBrain through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published377.6 MB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About emotion-recognition-wav2vec2-IEMOCAP

Can I use emotion-recognition-wav2vec2-IEMOCAP commercially?

Yes. emotion-recognition-wav2vec2-IEMOCAP is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

Similar Models

Model · Audio classification

ced-gguf

Mudler

GGUF quantizations of the CED family (Consistent Ensemble Distillation, Xiaomi) - SOTA-tier audio-tagging models that classify everyday sounds (baby cry, footsteps, glass breaking, alarms, dog bark,...) into the 527-class AudioSet ontology. These files run with ced.cpp, a standalone C++/ggml port (no Python, no PyTorch at inference), and with LocalAI via the ced backend. Converted from the mispeech/ced- checkpoints (Apache-2.0). CED is a plain AST/DeiT Vision Transformer over a log-mel spectrogram; the port is numerically equal to the PyTorch reference. One self-contained GGUF per size + quant (config, 527 labels, and the mel filterbank/window are all embedded). Pick by your accuracy/size…

Open weights apache-2.0 ced.cpp
View model

Model · Audio classification

lang-id-voxlingua107-ecapa

SpeechBrain

This is a spoken language recognition model trained on the VoxLingua107 dataset using SpeechBrain. The model uses the ECAPA-TDNN architecture that has previously been used for speaker recognition. However, it uses more fully connected hidden layers after the embedding layer, and cross-entropy loss was used for training. We observed that this improved the performance of extracted utterance embeddings for downstream tasks. The system is trained with recordings sampled at 16kHz (single channel). The code will automatically normalize your audio (i.e., resampling + mono channel selection) when calling classifyfile if needed. The model can classify a speech utterance according to the language…

Open weights apache-2.0 speechbrain
View model

Model · Audio classification

accent-id-commonaccent_ecapa

Juan Pablo Zuluaga

Abstract: The recognition of accented speech still remains a dominant problem in Automatic Speech Recognition (ASR) systems. We approach the classification of accented English speech through the Emphasized Channel Attention, Propagation and Aggregation Time Delay Neural Network (ECAPA-TDNN) architecture which has been shown to perform well on a variety of speech tasks. Three models are proposed: one trained from scratch, another two models (one using data augmentation and a baseline model) fine-tuned from the checkpoints of speechbrain/spkrec-ecapa-voxceleb (VoxCeleb). Our results show that the model fine-tuned with data augmentation yield the best results. Most of the misclassifications…

Open weights mit speechbrain
View model

Model · Audio classification

MERT-v1-330M

Multimodal Art Projection

The development log of our Music Audio Pre-training (m-a-p) model family: - 02/06/2023: arxiv pre-print and training codes released. - 17/03/2023: we release two advanced music understanding models, MERT-v1-95M and MERT-v1-330M, trained with new paradigm and dataset. They outperform the previous models and can better generalize to more tasks. - 14/03/2023: we retrained the MERT-v0 model with open-source-only music dataset MERT-v0-public - 29/12/2022: a music understanding model MERT-v0 trained with MLM paradigm, which performs better at downstream tasks. - 29/10/2022: a pre-trained MIR model music2vec trained with BYOL paradigm. Here is a table for quick model pick-up: The m-a-p models…

Open weights cc-by-nc-4.0 transformers
View model

Model · Audio classification

MERT-v1-95M

Multimodal Art Projection

The development log of our Music Audio Pre-training (m-a-p) model family: - 02/06/2023: arxiv pre-print and training codes released. - 17/03/2023: we release two advanced music understanding models, MERT-v1-95M and MERT-v1-330M, trained with new paradigm and dataset. They outperform the previous models and can better generalize to more tasks. - 14/03/2023: we retrained the MERT-v0 model with open-source-only music dataset MERT-v0-public - 29/12/2022: a music understanding model MERT-v0 trained with MLM paradigm, which performs better at downstream tasks. - 29/10/2022: a pre-trained MIR model music2vec trained with BYOL paradigm. Here is a table for quick model pick-up: The m-a-p models…

Open weights cc-by-nc-4.0 transformers
View model

Model · Audio classification

gender_cls_svm_ecapa_voxceleb

Gregory Koushnir

This model combines the SpeechBrain ECAPA-TDNN speaker embedding model with an SVM classifier to predict speaker gender from audio input. The model was trained and evaluated on the VoxCeleb2, Mozilla Common Voice v10.0, and TIMIT datasets - Mozilla Common Voice v10.0 English validated test set: 92.3% accuracy The model was trained on VoxCeleb2 dataset: - Converted to WAV format, single channel, 16kHz sampling rate, 256 kp/s bitrate - Applied SileroVAD for voice activity detection, taking the first voiced segment You can install the package directly from GitHub: - Model was trained on celebrity voices from YouTube interviews - Performance may vary on different audio qualities or recording…

Open weights apache-2.0
View model