wav2vec2-emotion-recognition

wav2vec2-emotion-recognition (Robust Fine-Tuning)

This model is a fine-tuned version of facebook/wav2vec2-base-960h for Speech Emotion Recognition (SER).

It has been trained using a Frozen Feature Extractor strategy to preserve the model's acoustic understanding while adapting to emotion detection. This approach ensures stable performance and prevents "Catastrophic Forgetting," achieving nearly 80% accuracy on the validation set.

Update: The "Calm" and "Neutral" classes have been merged to improve classification consistency, resulting in 7 distinct emotion classes.

Model Description

• Model Architecture: Wav2Vec2 with a frozen CNN feature extractor and a trainable sequence classification head.
• Language: English
• Task: Speech Emotion Recognition (SER)
• Fine-tuned from: facebook/wav2vec2-base-960h
• Class Merging: The "Calm" emotion has been merged into "Neutral" to reduce label ambiguity.

Datasets

The model was trained on a combined dataset of ~12,000 audio files from: - TESS - CREMA-D - SAVEE - RAVDESS

Performance Metrics

Note: Achieved stable convergence using a Cosine Learning Rate Scheduler and Frozen Feature Extractor.

Supported Emotions (7 Classes)

The model classifies audio into one of the following emotions:

1. Angry
2. Disgust
3. Fear
4. Happy
5. Neutral (includes Calm)
6. Sad
7. Surprise

Training Configuration

The model was fine-tuned using the following "Robust" configuration to ensure stability on Google Colab T4 GPUs:

• Feature Extractor: FROZEN (Locked weights to preserve pre-trained acoustic features)
• Epochs: 10
• Learning Rate: 3e-5
• Scheduler: Cosine Decay (Smooth landing)
• Batch Size: 4 (Physical) / 16 (Effective via Gradient Accumulation)
• Optimizer: AdamW
• Precision: fp32 (Standard precision to prevent gradient underflow)

Limitations

Audio Requirements:

• Sampling Rate: 16kHz (Model will hallucinate if input is not 16kHz. Resampling is required.)
• Duration: < 10 seconds recommended (Longer files should be chunked).
• Environment: Best results with clear speech; background noise may reduce confidence.

Demo

Try the model in your browser: Hugging Face Space: Audio Emotion Recognition

FINETUNING NOTEBOOOKS

Notebook

Contact

• GitHub: DGautam11
• LinkedIn: Deepan Gautam
• Hugging Face: @Dpngtm

For issues and questions, feel free to: 1. Open an issue on the Model Repository 2. Comment on the Demo Space

Usage

```python from transformers import AutoModelForAudioClassification, Wav2Vec2Processor import torch import torchaudio import numpy as np

1. Load Model and Processor

model_id = "Dpngtm/wav2vec2-emotion-recognition" model = AutoModelForAudioClassification.from_pretrained(model_id) processor = Wav2Vec2Processor.from_pretrained(model_id)

2. Load Audio

Replace with your file path

audio_path = "path_to_audio.wav" speech_array, sampling_rate = torchaudio.load(audio_path)

3. Resample to 16kHz (CRITICAL STEP)

if sampling_rate != 16000: resampler = torchaudio.transforms.Resample(sampling_rate, 16000) speech_array = resampler(speech_array) sampling_rate = 16000

4. Handle Stereo (Convert to Mono)

if speech_array.shape[0] > 1: speech_array = torch.mean(speech_array, dim=0, keepdim=True)

5. Process & Predict

inputs = processor(speech_array.squeeze(), sampling_rate=16000, return_tensors="pt", padding=True)

with torch.no_grad(): logits = model(**inputs).logits

6. Decode Result

predicted_id = torch.argmax(logits, dim=-1).item() id2label = model.config.id2label predicted_label = id2label[predicted_id]

print(f"Predicted Emotion: {predicted_label}")