Real Time Facial Expression Recognition Using Webcam and SDK Affectiva
DOI:
https://doi.org/10.9781/ijimai.2017.11.002Keywords:
Classification, Emotion recognition, Real Time Recognition, Face Detection, Facial ExpressionAbstract
Facial expression is an essential part of communication. For this reason, the issue of human emotions evaluation using a computer is a very interesting topic, which has gained more and more attention in recent years. It is mainly related to the possibility of applying facial expression recognition in many fields such as HCI, video games, virtual reality, and analysing customer satisfaction etc. Emotions determination (recognition process) is often performed in 3 basic phases: face detection, facial features extraction, and last stage - expression classification. Most often you can meet the so-called Ekman’s classification of 6 emotional expressions (or 7 - neutral expression) as well as other types of classification - the Russell circular model, which contains up to 24 or the Plutchik’s Wheel of Emotions. The methods used in the three phases of the recognition process have not only improved over the last 60 years, but new methods and algorithms have also emerged that can determine the ViolaJones detector with greater accuracy and lower computational demands. Therefore, there are currently various solutions in the form of the Software Development Kit (SDK). In this publication, we point to the proposition and creation of our system for real-time emotion classification. Our intention was to create a system that would use all three phases of the recognition process, work fast and stable in real time. That’s why we’ve decided to take advantage of existing Affectiva SDKs. By using the classic webcamera we can detect facial landmarks on the image automatically using the Software Development Kit (SDK) from Affectiva. Geometric feature based approach is used for feature extraction. The distance between landmarks is used as a feature, and for selecting an optimal set of features, the brute force method is used. The proposed system uses neural network algorithm for classification. The proposed system recognizes 6 (respectively 7) facial expressions, namely anger, disgust, fear, happiness, sadness, surprise and neutral. We do not want to point only to the percentage of success of our solution. We want to point out the way we have determined this measurements and the results we have achieved and how these results have significantly influenced our future research direction.Downloads
References
P. Augustyniak et al., “Seamless tracing of human behavior using complementary wearable and house-embedded sensors,” Sensors, vol. 14, no. 5, pp. 7831–7856, 2014.
K. Bahreini, R. Nadolski, and W. Westera, “Towards multimodal emotion recognition in e-learning environments,” Interactive Learning Environments, 2014, pp. 1–16.
V. Bettadapura, “Face Expression Recognition and Analysis: The State of the Art,” arXiv, Tech. Rep., 2012, pp. 1–27.
Y. Chen, R. Hwang, and C. Wang, “Development and evaluation of a web 2.0 annotation system as a learning tool in an e-learning environment,” Computers & Education, vol. 58, no. 4, pp. 1094–1105, 2011.
C. Darwin, The Expression of the Emotions in Man and Animals. Oxford University Press, 1998.
P. Ekman, “All emotions are basic,” in The Nature of Emotion, P. Ekman and R. J. Davidson, Eds. Oxford University Press, 1992, pp. 15–19.
P. Ekman, “Strong evidence for universals in facial expressions: A reply to Russell’s mistaken critique,” Psychological Bulletin, vol. 115, no. 2, pp. 268–287, 1994.
P. Ekman and W. Friesen, Facial Action Coding System: Investigator’s Guide. Consulting Psychologists Press, 1978.
B. Fasel and J. Luettin, “Automatic facial expression analysis: A survey,” Pattern Recognition, vol. 36, no. 1, pp. 259–275, 2003.
M. Feidakis, T. Daradoumis, and S. Caballé, “Emotion measurement in intelligent tutoring systems: What, when and how to measure,” in Proceedings of the Third International Conference on Intelligent Networking and Collaborative Systems, 2011, pp. 807–812.
G. Giannakakis, M. Pediaditis, D. Manousos, E. Kazantzaki, F. Chiarugi, P. G. Simos, and M. Tsiknakis, “Stress and anxiety detection using facial cues from videos,” Biomedical Signal Processing and Control, vol. 31, pp. 89–101, 2017, doi:10.1016/j.bspc.2016.06.020.
M. X. Huang, J. Li, G. Ngai, and H. V. Leong, “StressClick: Sensing stress from gaze-click patterns,” in Proceedings of the 2016 ACM Multimedia Conference (MM ’16), 2016, pp. 1395–1404, doi:10.1145/2964284.2964318.
Y. Juan, S. Dobson, and S. McKeever, “Situation identification techniques in pervasive computing: A review,” Pervasive and Mobile Computing, vol. 8, no. 1, pp. 36–66, 2012.
D. Keltner, Born to Be Good: The Science of a Meaningful Life. W. W. Norton & Company, 2009.
A. Kumar, A. Kumar, S. K. Singh, and R. Kala, “Human activity recognition in real-time environments using skeleton joints,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 3, no. 7, pp. 61–69, 2016.
J. Li, G. Ngai, and V. Hong, “Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics,” Applied Computing Review, vol. 16, no. 3, pp. 37–49, 2016.
Y. F. Li, J. Zhang, and W. Wang, Active Sensor Planning for Multiview Vision Tasks. Springer, 2008.
A. A. Liu et al., “Coupled hidden conditional random fields for RGB-D human action recognition,” Signal Processing, pp. 74–82, 2015.
M. Magdin, M. Turcani, and L. Hudec, “Evaluating the emotional state of a user using a webcam,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 4, no. 1 (Special Issue), pp. 61–68, 2016.
D. McDuff, E. R. Kaliouby, T. Senechal, M. Amr, J. F. Cohn, and R. Picard, “Affectiva-MIT facial expression dataset (AM-FED): Naturalistic and spontaneous facial expressions collected ‘in-the-wild’,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2013, pp. 881–888, doi:10.1109/CVPRW.2013.130.
D. McDuff, A. Mahmoud, M. Mavadati, M. Amr, J. Turcot, and E. R. Kaliouby, “AFFDEX SDK: A cross-platform real-time multi-face expression recognition toolkit,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), 2016, pp. 3723–3726, doi:10.1145/2851581.2890247.
M. Pantic and L. J. M. Rothkrantz, “Automatic analysis of facial expressions: The state of the art,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1424–1445, 2000.
F. Pasquale, G. Percannella, and M. Vento, “Graph matching and learning in pattern recognition in the last 10 years,” International Journal of Pattern Recognition and Artificial Intelligence, vol. 28, p. 1450001, 2014.
J. A. Russell, “Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies,” Psychological Bulletin, vol. 115, no. 1, pp. 102–141, 1994.
T. Senechal, D. McDuff, and E. R. Kaliouby, “Facial action unit detection using active learning and an efficient non-linear kernel approximation,” in Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), 2016, pp. 10–18, doi:10.1109/ICCVW.2015.11.
V. B. Semwal et al., “An optimized feature selection technique based on incremental feature analysis for bio-metric gait data classification,” Multimedia Tools and Applications, vol. 76, no. 22, pp. 24457–24475, 2016.
N. Sharma and T. Gedeon, “Modeling observer stress for typical real environments,” Expert Systems with Applications, vol. 41, no. 5, pp. 2231–2238, 2014, doi:10.1016/j.eswa.2013.09.021.
N. Sharma and T. Gedeon, “Objective measures, sensors and computational techniques for stress recognition and classification: A survey,” Computer Methods and Programs in Biomedicine, vol. 108, no. 3, pp. 1287–1301, 2012, doi:10.1016/j.cmpb.2012.07.003.
M. Suk and B. Prabhakaran, “Real-time facial expression recognition on smartphones,” in Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2015, pp. 1054–1059.
M. Trojahn, F. Arndt, M. Weinmann, and F. Ortmeier, “Emotion recognition through keystroke dynamics on touchscreen keyboards,” in Proceedings of the 15th International Conference on Enterprise Information Systems (ICEIS), vol. 3, 2013, pp. 31–37.
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2001, pp. 511–518.
L. Xia, C.-C. Chen, and J. K. Aggarwal, “View invariant human action recognition using histograms of 3D joints,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2012.
Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, “A survey of affect recognition methods: Audio, visual, and spontaneous expressions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39–58, 2009, doi:10.1109/TPAMI.2008.52.
Z. Zhang, “Feature-based facial expression recognition: Sensitivity analysis and experiment with a multi-layer perceptron,” International Journal of Pattern Recognition and Artificial Intelligence, vol. 13, no. 6, pp. 893–911, 1999.
Nordic APIs, “20 emotion recognition APIs that will leave you impressed—and concerned,” Nov. 3, 2017. [Online]. Available: http://nordicapis.com/20-emotion-recognition-apis-that-will-leave-youimpressed-and-concerned/
Face Detection and Recognition, “Face Detection Home Page,” Nov. 3, 2017. [Online]. Available: http://facedetection.com
Affectiva, “Developer Portal,” Nov. 3, 2017. [Online]. Available: http://developer.affectiva.com
Downloads
Published
-
Abstract165
-
PDF46
Submit Article
Information
