Facial Emotion Recognition Using Context Based Multimodal Approach

 Abstract —Emotions play a crucial role in person to person interaction. In recent years, there has been a growing interest in improving all aspects of interaction between humans and computers. The ability to understand human emotions is desirable for the computer in several applications especially by observing facial expressions. This paper explores a ways of human-computer interaction that enable the computer to be more aware of the user’s emotional expressions we present a approach for the emotion recognition from a facial expression, hand and body posture. Our model uses multimodal emotion recognition system in which we use two different models for facial expression recognition and for hand and body posture recognition and then combining the result of both classifiers using a third classifier which give the resulting emotion . Multimodal system gives more accurate result than a signal or bimodal system


I. INTRODUCTION
Different people express their feelings in a different way under different circumstances (different context).The human sciences contain a bank of literature on emotion which is large, but fragmented [1][6] [7] [8].The main sources which are relevant to our approach are in psychology and linguistics, with some input from biology.Emotions play an important role in human-to-human communication and interaction, allowing people to express themselves beyond the verbal domain.Some study in perceiving facial emotions has fascinated the human computer interaction environments.In recent years, there has been a growing interest in improving all aspects of interaction between humans and computers especially in the area of human emotion recognition by observing facial, voice, and physiological signals, where the different modalities are treated independently.Here we present a multimodal approach in which we use two different models one for recognizing the emotion using facial expression and second for hand and body posture as context.The design of above approach is shown in Figure 1.

II. METHODOLOGY
Recently, researchers have also turned to emotional body language, i.e. the expression of emotions through human body poses and/or body motion.An implicit assumption common to the work on emotional body language is that body language is only a different means of expressing the same set of basic emotions as facial expressions.Using a set of emotional body language stimuli, which was originally prepared for neuro scientific studies, we show that human observers, as expected, perform very well on this task, and construct a model of the underlying processing stream.The model is then tested on the same stimulus set.The data we use for our work is should based on the database which was originally created FABO [9] bimodal database consisting of combined face and body expressions recorded simultaneously.Which is as shown in Figure 2.Here segmentation process is applied based on a background subtraction method on image in order to obtain the silhouette of the upper body.We then apply thresholding, noise cleaning, morphological filtering and connected component labeling.We extract the face and the hands automatically from image, by exploiting skin color information.The hand position consists of the position of the centroid and in-plane rotation.We employ the Camshift algorithm [11] for tracking the hands and predicting their locations in image.Orientation feature helps to discriminate between different poses of the hand together with the edge density information.These body features we give to the classifier as input to get the emotion.

Facial Emotion Recognition Using Context
Based Multimodal Approach As you can check in Figure 1, our approach have two different models 1. Facial expression recognition system (FERS).
III. RELATED WORK

Facial expression recognition system (FERS).
The leading study of Ekman and Friesen formed the basis of visual facial expression recognition.Their studies suggested that anger, disgust, fear, happiness, sadness and surprise are the six basic prototypical facial expressions recognized universally [2].Here we consider eight emotional states: Anger, Despair, Interest, Pleasure, Sadness, Irritation, Joy and Pride.We choose this set of features in order to obtain emotions.Block diagram of the process to find the features from face is as shown in Figure 3. Initially a face detection algorithm is applied to find out the face from given image.Face detection is to identify all image regions which contain a face regardless of its threedimensional position, orientation, and lighting conditions.Such a problem is challenging because faces are no rigid and have a high degree of variability in size, shape, color, and texture [4].Figure 4 shows the Face features extraction system.Ekman and Friesen [5] have produced a system for describing -all visually distinguishable facial movements,‖ called the Facial Action Coding System or FACS.
It is based on the enumeration of all -action units‖ (AUs) of a face that cause facial movements [10].There are 46 AUs in FACS that account for changes in facial expression .The combination of these action units result in a large set of possible facial expressions.Table I shows Some AU and their associated facial change obtained from Ekman's study [12].
Table I Some AU and their associated facial change obtained from Ekman's study [12].
Recognition of facial expressions can be achieved by categorizing a set of such predetermined features as in FACS.Here we take a input face which is an outcome of face detection algorithm .We extract the facial action units from face using FACS .These feature points then given to the classifier which also takes input from body posture recognition system to find out emotion.As we extract Facial Action Units from face the same way we extract the body posture and hand postures as Body Action Units (BAU) .We use Clamshift Algorithm [11] to extract BAU's from image as shown in Table II.The block diagram of body posture recognition system as shown in the Figure 5.We then classified the data from expressive face and body into labeled emotion categories using Bayesian classifier.Emotion modulates almost all modes of human communication -facial expression, gestures, posture, tone of voice, choosing of words, respiration, skin temperature and clamminess, etc. Emotions can significantly change the message: sometimes it is not what was said that is the most important, but how it was said.Faces tend to be the most visible form of emotion communication, but they are also most easily controlled in response to different social situations when compared to the voice and other ways of expression.As noted by Picard2 affect recognition is most likely to be accurate when it combines multiple modalities, information about the user's context, situation, goal, and preferences.A combination of low-level features, high-level reasoning, and natural language processing is likely to provide the best emotion inference The reason as to why the system trained with body gesture features proved to be most successful may reside in the fact that, in the corpus of acted emotional expressions, each is represented by a specific type of gesture: participants were provided with specific instructions in order to perform different gestures for each emotion.While this choice was made in order to build a system capable of recognizing different types of body gestures based on movement expressivity, it may have made the discrimination of emotions from body gesture easier than using facial and speech features [3].

V. CONCLUSION
The addition of body gesture information to facial expression for emotion recognition is novel.Consideration of multiple modalities is helpful when some modality feature values are missing or unreliable.By taking all of these aspects into account, we hope to be able to develop into the near future multimodal context-sensitive systems that are smart, perceptually aware, recognize the context in which they act, can adapt to their users, and can understand how they feel, and respond appropriately

Figure 1 .
Figure 1.Block diagram for multimodal approach system.

Figure 2 .
Figure 2.Examples of affective body gestures (from the FABO database).

Figure 5
Figure 5 Bayesian classifier for emotion

Table II .
Emotion and the respective Facial expression and Body posture