Speech Emotion Recognition

A speech emotion classifier with InstanceNorm2d speaker normalization, Context CNN dialog modeling, and Bi-LSTM + Attention layers.

Project claim

Detect human emotional states from spoken audio in real-time.

Role

Lead Architect (Core Branches)

Focus signals

+15.8% accuracy gainRTF 0.00008 (Real-time)Focal Loss tuning

Proof signal 1

+15.8% accuracy over baseline

Proof signal 2

RTF = 0.00008 (fully real-time)

Proof signal 3

Focal Loss minority F1 improvement

Challenge

Audio features suffer from static representation gaps, class imbalances, speaker tone variability, and cross-corpus noise.

Solution

Designed a dual-branch network separating Log-Mel spatial spectrogram features and MFCC temporal coefficients, applying speaker normalization and multi-utterance dialogue context.

Build notes

Extracted 64 Mel-bands for spatial features and 20 coefficients with delta/delta-delta for temporal paths.

Used focal loss and label smoothing to overcome severe class imbalances in minority emotional states.

Performed data augmentation using time masking, frequency masking, and Gaussian noise to build robustness.

Tools + stack

PyTorch, Librosa DSP, 2D CNN, Bi-LSTM, and Attention.

PyTorchLibrosaPythonDockerGit

Accuracy gain

+15.8% vs MLP

Real-time factor

0.00008 (RTF)

Dataset scale

11,076 samples

Architecture flow

Step 1

audio source

Step 2

log-mel cnn

Step 3

mfcc dense

Step 4

context cnn

Step 5

bi-lstm attention

Step 6

focal classification

Dual-Branch Architecture

Branch 1 runs a 2D CNN over 2D Log-Mel spectrograms to capture spectral texture. Branch 2 runs Dense layers over 1D MFCC coefficients to capture temporal details. The outputs are fused with Context vectors.

Experimental Generalization

Evaluated on IEMOCAP (clean) and MELD (noisy Friends show dialogues). The v3 model maintained 54.5% accuracy under heavy background noise, proving robust real-world generalization.

Decision signals

Filter out speaker tone variance to capture true emotional shifts

Model multi-utterance conversational context to track trajectories

Ensure low inference footprint for mobile and edge deployments

Outcomes

Trained a Dual-Branch CNN model achieving 55.9% accuracy on IEMOCAP and 54.5% on IEMOCAP+MELD noisy data.

Integrated InstanceNorm2d to remove speaker-specific tone biases and focus training on emotional variance.

Built a dialog-level Context CNN encoding previous 3 utterances to model conversational trajectory changes.