Most of the known computer systems which determine the emotional state of a person are based on obtaining data from a facial image in a video sequence. A more promising direction is based on the analysis of multiple streams: facial images, gestures, postures, speech and biometrical data. The proposed system allows obtaining information by analyzing facial expressions, speech and biometrical data of the object in real time.
The system (figure 1) consists of four main modules: data acquisition module, data processing module, classification module, and a data output module. Let us consider in more detail each component.

Data acquisition module From the analyzed object the data is fed to the input of video, audio, and biometric channels. Each channel carries out a set of unique operations on the input data. The video channel performs the selection and capture of the face in the video stream, the segmentation of key areas that demonstrate muscle movement. The audio channel carries out the primary selection of the useful signal (human speech) in the audio stream. The biometric channel supplies a stream of data on skin temperature, blood pressure, and heart rate.
As a rule, the analyzed object locates in a noisy environment, therefore there is a non-zero probability of obtaining poor-quality data for each channel, up to the complete impossibility of using them. To take this feature into account, each channel must calculate its own coefficient of content relevance $I_{\text{ch}}$ and transfer it to the processing module.
Data processing module Such a module receives at its input previously prepared information for each of the channels and outputs normalized vectors $X_{\text{set}}$ of data attributes at the output. Such vectors resulting from digital signal processing are unique to each non-verbal component. For example, for the mimic component, the vector contains information about the change in the location of the supporting segments on the face, which characterize the dynamics of contraction of the facial muscles. For speech - the volume, pitch and frequency response of speech.
Additional $Y_{\text{set}}$ features to $X_{\text{set}}$ feature vectors take into account the age, educational characteristics of a person, his gender, nationality, etc. since recent studies [1] cast doubt on the thesis of the invariance of the expression of emotions by different people.
Classification Module The main output parameter, the $P_{\text{res}}$ vector, reflects the probability that the non-verbal component corresponds to a certain type of emotion from a given set of considered emotions $E_{\text{set}}$. Functional dependencies $F_{\text{ch}}: X_{\text{set}} \rightarrow P_{\text{ch}}$, where ch is the corresponding non-verbal channel, the values of the feature vectors are mapped onto probability vectors $P_{\text{ch}}$. For such a functional display, classifiers [2], unique for each channel, which are based on a training data set, are used. Further, the output of the classifiers, $P_{\text{ch}}$ vectors, must be normalized and summed to obtain the integral vector $P_{\text{res}}$.
Data output module As a result, we can determine the parameter $E_{\text{res}} \in E_{\text{Set}}$ corresponding to the dominant type of emotion highlighted in the general set of possible emotional states
The output parameterization used also allows obtaining secondary output data - valency and tonality.