EMOTION RECOGNITION BY IMAGE AND HUMAN SPEECH PROCESSING

Elizaveta Plyshevskaya1, Elena Kozlova1

1 Faculty of Radiophysics and Computer Technologies, Belarusian State University, Minsk, Belarus

Emotion recognition problem is hard to solve and formulate. People express emotions in many ways: by face mimics, gestures, speech intonations and content. Also physical state of a person is different when experiencing different emotions. So it is important to analyze different components when recognizing emotions.

The idea of the present study is to improve an emotion recognition algorithm using Intel RealSense technology [1] that recognizes emotion by analyzing image of the person's face by adding speech emotion recognition module to the system. Analyzing both speech and face components allows to get a more accurate result, especially in such cases, as, for example, when a person is lying or has a face injury.

Emotion recognition algorithm using Intel RealSense consists of six steps and uses the emotion classification described in Paul Ekman[2] studies and the mimic codes of emotional states. The picture is grabbed from the camera, than a face and facial landmark points are detected. As it is done the program counts the key values such as mouth and eyes being open in percents. Then the classification starts and the result is presented.

Emotion recognition by speech is performed by a convolutional neural network for small-footprint keyword spotting, because it is comparatively simple, fast and quick to train. At first spoken words are defined by grouping the audio samples into segments and calculating the strength of frequencies across a set of bands. Then the model gets a spectrogram for an audio sample. Spectrogram can be treated as a single-channel image, as it appears to be a two-dimensional array, created by vectors that are actually one-dimensional arrays of frequency strengths for each audio segment of an audio sample.

Figure 1
Fig. 1. Emotion recognition system with two modules.

Average accuracy for emotion recognition by image is about 80% when the system recognizes seven emotions with video frame rate about 30 fps, when average accuracy for emotion recognition by speech is about 82% for 10000 epochs of learning using dataset with 21000 audio samples.

Combining and analyzing the recognition results of the two modules can help to increase the average accuracy and get more reliable results about the emotional state of a person.


[1] E. Tarakanova, E. Kozlova, Emotion recognition algorithm based on Intel RealSense technology (Trans. Open Readings Programme and Abstracts, 2017).

[2] P. Ekman, W. Friesen, Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues (Trans. Piter Publishing house, Russia, 2010).