Incremental Learning Based Classification for Facial Expression Recognition

Abstract

The present study focuses on the facial expression recognition. Communication is fundamental to humans. Many scientific research studies have shown that most part of the human communication is nonverbal (55% to 93%). The next generation computing; such as, pervasive computing, and ambient intelligence, needs to develop human- centered systems that readily react to multimodal human communication occurring naturally. This bulk of information through nonverbal communication is ignored in traditional human-computer-interaction (HCI) and sufficed on user's intentional input only. A system is needed, which has the ability to identify and realize the intentions and emotions as expressed by social and affective indicators. The research on facial expression recognition (FER) has been under focus in computer vision field for a couple of decades; however, there are many questions that need to be answered. This thesis addressed a few of them. Facial expressions are of two types; spontaneous and posed. The present study showed that these two types of expressions are different in many aspects. The factors such as lighting, pose, head movement, cultural variations etc. make spontaneous expressions more difficult and challenging to recognize. The objective of the study is to develop a system that is robust enough to such variations. A major deficiency in FER area is the unavailability of a database that can be a representative of all such variations. Researchers believe that this goal is far away to be achieved. So, in the absence of such database, we proposed incremental learning as a good alternate solution. With the incremental learning capability, the proposed systems have ability to adjust themselves in any environment and culture. Furthermore, we started to develop a facial expression database for various cultures. We proposed three FER systems based on incremental learning and conducted a vast range of experimentation and comparisons. A multinomial classifier is proposed and developed to optimize the nearest neighbor classifier based on template matching. Various similarity measures are studied and compared. A dynamically weighted majority voting (DWMV) mechanism is proposed to create better generalization in ensemble systems that is necessary for real world scenarios. Diversity is probably the most desired property of ensemble based systems. We proposed and developed a diversity boosting based algorithm to construct ensemble classifier for high performance. Detailed performance comparisons on widely adopted facial expression databases along with spontaneous vs posed expression comparisons are performed. Most studies in this area used same databases for training and testing, and showed good results with no cross dataset evaluations. We conducted a vast range of experiments on six benchmark databases (MUG, MMI, CK, CK+, FEEDTUM, JAFFE) plus our own multi-cultural database. Cross database experiments performed and showed soundness of our proposed systems. We compared the results of our proposed systems with latest and previously proposed FER techniques. The results showed the soundness of our proposed methods. All these investigations and contributions provide useful insight into enhancing the robustness and efficiency of FER systems, and making them to perform better in real world applications.