A Convolutional Neural Network for Compound Micro-Expression Recognition

Yue Zhao, Jiancheng Xu, Yue Zhao, Jiancheng Xu

Abstract

Human beings are particularly inclined to express real emotions through micro-expressions with subtle amplitude and short duration. Though people regularly recognize many distinct emotions, for the most part, research studies have been limited to six basic categories: happiness, surprise, sadness, anger, fear, and disgust. Like normal expressions (i.e., macro-expressions), most current research into micro-expression recognition focuses on these six basic emotions. This paper describes an important group of micro-expressions, which we call compound emotion categories. Compound micro-expressions are constructed by combining two basic micro-expressions but reflect more complex mental states and more abundant human facial emotions. In this study, we firstly synthesized a Compound Micro-expression Database (CMED) based on existing spontaneous micro-expression datasets. These subtle feature of micro-expression makes it difficult to observe its motion track and characteristics. Consequently, there are many challenges and limitations to synthetic compound micro-expression images. The proposed method firstly implemented Eulerian Video Magnification (EVM) method to enhance facial motion features of basic micro-expressions for generating compound images. The consistent and differential facial muscle articulations (typically referred to as action units) associated with each emotion category have been labeled to become the foundation of generating compound micro-expression. Secondly, we extracted the apex frames of CMED by 3D Fast Fourier Transform (3D-FFT). Moreover, the proposed method calculated the optical flow information between the onset frame and apex frame to produce an optical flow feature map. Finally, we designed a shallow network to extract high-level features of these optical flow maps. In this study, we synthesized four existing databases of spontaneous micro-expressions (CASME I, CASME II, CAS(ME)2, SAMM) to generate the CMED and test the validity of our network. Therefore, the deep network framework designed in this study can well recognize the emotional information of basic micro-expressions and compound micro-expressions.

Keywords: 3D-FFT; CNN; EVM; FACS; TV-L1 optical flow; compound micro-expressions.

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
This Compound Facial Expressions of Emotion (CFEE).
Figure 2
Figure 2
The framework of the proposed method.
Figure 3
Figure 3
Compound facial expressions in real environments (left: “disgustedly surprised”, right: “fearfully surprised”).
Figure 4
Figure 4
The generation process of CMED: (a) Description of Positively Surprised; (b) Description of Positively Negative; (c) Description of Negatively Surprised; (d) Description of Negatively Negative.
Figure 4
Figure 4
The generation process of CMED: (a) Description of Positively Surprised; (b) Description of Positively Negative; (c) Description of Negatively Surprised; (d) Description of Negatively Negative.
Figure 5
Figure 5
The compound micro-expression database.
Figure 6
Figure 6
Comparison of ME sequences at different magnification factors.
Figure 7
Figure 7
Optical flow maps of six MEs in CASME Ⅱ database.
Figure 8
Figure 8
Overall framework of proposed network.
Figure 9
Figure 9
Recognition performance using different magnification factor.
Figure 10
Figure 10
Comparison of different magnification method.
Figure 11
Figure 11
Optical flow feature maps with different λ and Nscales.
Figure 12
Figure 12
Recognition performance using different input graph on CMED. Magnified not magnificated.
Figure 13
Figure 13
The measurement of confusion matrix: (a) the basic ME database; (b) the CMED.

References

    1. Martin C.W., Ekman P. The Philosophy of Deception. 3rd ed. Oxford University Press; Oxford, UK: 2009. pp. 118–136.
    1. Ekman P., Wallace V.F. Constants across cultures in the face and emotion. J. Personal. Soc. Psychol. 1971;17:124–129. doi: 10.1037/h0030377.
    1. Zhao G., Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 2007;29:915–928.
    1. Sze T.L., Kok S.W. Micro-expression recognition using apex frame with phase information; Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference; Kuala Lumpur, Malaysia. 12–15 December 2017.
    1. Yong J.L., Jin K.Z., Wen J.Y. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition. IEEE Trans. Affect. Comput. 2016;7:299–310.
    1. Sze T.L., Raphael W.P., John S. Optical strain based recognition of subtle emotions; Proceedings of the 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS); Kuching, Malaysia. 1–4 December 2014.
    1. Bruce V., Young A.W. Messages from Facial Movements. 2nd ed. Psychology Press; London, UK: 2012. pp. 153–208.
    1. Ekman P., Friesen W.V. Pictures of Facial Affect. Consulting Psychologists Press; Saint Paul, MN, USA: 1976. pp. 124–152.
    1. Hjortsjo C.H. Man’s Face and Mimic Language, Lund. Studentlitterature; Stockholm, Sweden: 1970. pp. 78–91.
    1. Du S., Tao Y., Martinez A.M. Compound facial expressions of emotion. Proc. Natl. Acad. Sci. USA. 2014;111:1454–1462.
    1. Du S., Martinez A.M. Compound facial expressions of emotion: From basic research to clinical applications. Dialogues Clin. Neurosci. 2015;17:443–455.
    1. Yan W.J., Wu Q., Liu Y.J., Wang S.J., Fu X. CASME Database: A Dataset of Spontaneous Micro-Expressions Collected from Neutralized Faces; Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG); Shanghai, China. 22–26 April 2013.
    1. Yan W.J., Li X., Wang S.J., Zhao G. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE. 2014;9:102–110. doi: 10.1371/journal.pone.0086041.
    1. Li X., Pfister T., Huang X., Zhao G., Pietikäinen M. A Spontaneous Micro-expression Database: Inducement, collection and baseline; Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG); Shanghai, China. 22–26 April 2013.
    1. Qu F., Wang S.J., Yan W.J., Fu X. CAS(ME)2: A Database of Spontaneous Macro-expressions and Micro-expressions; Proceedings of the 2016 International Conference on Human-Computer Interaction; Toronto, ON, Canada. 17–22 July 2016.
    1. Davison A.K., Lansley C., Costen N., Tan K., Yap M.H. SAMM: A Spontaneous Micro-Facial Movement Dataset. IEEE Trans. Affect. Comput. 2018;9:116–129. doi: 10.1109/TAFFC.2016.2573832.
    1. Wang Y., See J., Phan R.C.-W., Oh Y. LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition; Proceedings of the 12th Asian Conference on Computer Vision; Singapore. 1–5 November 2014; Cham, Switzerland: Springer; 2015.
    1. Sze T.L., John S., Kok S.W. Automatic Micro-expression Recognition from Long Video Using a Single Spotted Apex; Proceedings of the 2016 Asian Conference on Computer Vision International Workshops; Taipei, Taiwan. 20–24 November 2016; Cham, Switzerland: Springer; 2017.
    1. Li Y., Huang X., Zhao G. Can Micro-Expression be Recognized Based on Single Apex Frame; Proceedings of the 2018 International Conference on Image Processing; Athens, Greece. 7–10 October 2018.
    1. Sze T.L., Gan Y.S., Wei C.Y. OFF-ApexNet on Micro-expression Recognition System. arXiv. 20181805.08699
    1. Sze T.L., Gan Y.D., John S. Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition; Proceedings of the 14th IEEE International Conference on Automatic Face and Gesture Recognition; Lille, France. 14–18 May 2019.
    1. Huang X., Wang S., Zhao G., Piteikainen M. Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection; Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop (ICCVW); Santiago, Chile. 7–13 December 2015.
    1. Huang X., Zhao G., Hong X., Zheng W., Pietikäinen M. Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing. 2016;175:564–578. doi: 10.1016/j.neucom.2015.10.096.
    1. Matthew S., Sridhar G., Vasant M., Dmitry G. Towards macro- and micro-expression spotting in video using strain patterns; Proceedings of the 2009 Conference: Applications of Computer Vision (WACV); Snowbird, UT, USA. 7–8 December 2009.
    1. Sze T.L., John S., Raphael C.W. Subtle Expression Recognition Using Optical Strain Weighted Features; Proceedings of the Asian Conference on Computer Vision 2014 Workshops; Singapore. 1–2 November 2014.
    1. Mei W., Weihong D. Deep Visual Domain Adaptation: A Survey. Neurocomputing. 2018;312:135–153.
    1. Samira E.K., Xavier B., Pascal L., Caglar G., Vincent M. EmoNets: Multimodal deep learning approaches for emotion recognition in video. J. Multimodal User Interfaces. 2016;10:99–111.
    1. Liu A., Yang Y., Sun Q., Xu Q. A Deep Fully Convolution Neural Network for Semantic Segmentation Based on Adaptive Feature Fusion; Proceedings of the 2018 5th International Conference on Information Science and Control Engineering (ICISCE); Zhengzhou, China. 20–22 July 2018.
    1. Liu P., Han S., Meng Z., Tong Y. Facial Expression Recognition via a Boosted Deep Belief Network; Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition; Columbus, OH, USA. 23–28 June 2014.
    1. Kim Y., Lee H., Provost E.M. Deep learning for robust feature generation in audiovisual emotion recognition; Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing; Vancouver, BC, Canada. 26–31 May 2013.
    1. Patel D., Hong X., Zhao G. Selective deep features for micro-expression recognition; Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR); Cancun, Mexico. 4–8 December 2016.
    1. Dae H.K., Wissam J.B., Jinhyeok J., Yong M.R. Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition. IEEE Trans. Affect. Comput. 2019;10:223–236.
    1. Ekman P., Wallace V.F. Facial Action Coding System: A Technique for the Measurement of Facial Movement. 3rd ed. Consulting Psychologists Press; Saint Paul, MN, USA: 1978. pp. 123–148.
    1. Ekman P., Rosenberg E. What the Face Reveals. 2nd ed. Oxford University Press; New York, NY, USA: 2005. pp. 78–106.
    1. Ekman P., Oster H. Facial expressions of emotion. J. Nonverbal Behav. 1979;17:236–248. doi: 10.1146/annurev.ps.30.020179.002523.
    1. Liu C., Torralba A., Freeman W.T., Durand F. Motion magnification. ACM Trans. Graph. 2005;34:519–526. doi: 10.1145/1073204.1073223.
    1. Wu H.Y., Rubinstein M., Shih E., Guttag J., Durand F., Freeman W. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 2012;31:65–83. doi: 10.1145/2185520.2185561.
    1. Wadhwa N., Rubinstein M. Phase-Based Video Motion Processing. ACM Trans. Graph. 2013;32:145–160. doi: 10.1145/2461912.2461966.
    1. Sze T.L., See J. Less is More: Micro-expression Recognition from Video using Apex Frame. Signal Process. Image Commun. 2018;62:82–92.
    1. Christopher Z., Thomas P., Horst B. A Duality Based Approach for Realtime TV-L1 Optical Flow. Pattern Recognit. 2007;9:214–223.
    1. Javier S.P., Enric M.L., Gabriele F. TV-L1 Optical Flow Estimation. Image Process. Line. 2013;3:137–150.
    1. Jia X., Gengming Z. Joint Face Detection and Facial Expression Recognition with MTCNN; Proceedings of the 2017 4th International Conference on Information Science and Control Engineering (ICISCE); Changsha, China. 21–23 July 2017.
    1. Horn B.K., Schunck B.G. Determining optical flow. Artif. Intell. 1981;17:185–203. doi: 10.1016/0004-3702(81)90024-2.
    1. Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 2012;25:1097–1105. doi: 10.1145/3065386.

Source: PubMed

스폰서 및 공동 작업자

건강 상태

약물 개입

3
구독하다