Emotion Recognition of Facial Expressions and Hand Gestures

Method

Participants     

The five participants were chosen from friends of the interviewer. All five were female, aged between 28 and 53, with a mean age of 41 years. Two participants identified as being Maori, two as being NZ European and one identified as being European (English). Four out of five of the participants had children. Three participants had achieved academically to an undergraduate level, one participant had a 7^th form level of secondary education and one participant held no formal academic qualifications. All five participants were in employment with their occupations being; machine operator, early childhood educator, general manager, healthcare worker and a naval sailor. Participation was voluntary with no incentives offered.

Materials

The experiment was constructed to measure the participants recognition of facial expressions with and without hand gestures. There were eight photos presented (Appendix A), the first four photos presented an actors expressions without hand gestures, the final four photos presented the actors expressions accompanied by hand gestures. The participants were to choose one expression they felt was being expressed in each photo from; anger, fear, sadness, disgust, happiness and surprise. The interview was conducted via an online quiz format using google forms (Appendix B). Each section was displayed individually and required information to be input by the participants before they could proceed to the following section.

Procedure

All five participants accessed the quiz/experiment individually via an online link. The first section introduced the interviewee, what the quiz/experiment involved and why it was being conducted. The second section provided the instructions for the participants, what was required of them and gave them the opportunity to ask any questions before proceeding. The next section was the informed consent form which the participants were requested to print out, sign and return to the interviewee before they were permitted to proceed.

The eight black and white photos, showed an actor expressing a range of different emotions; anger, fear, sadness, happiness and surprise. The first four photos presented to the participants showed a range of facial expressions that had no hands showing and the final four photos showed a range of facial expressions that were accompanied with hand gestures. This was done to determine if the use of body language/hand gestures, alongside facial expressions made it easier to recognise the emotion someone is expressing compared to recognition of emotions with no hand gestures. Each photo presented was accompanied by the question 'What emotion do you think the person is feeling?' and gave five options for the participants to choose one from; anger, fear, sadness, disgust, happiness and surprise.

At the end of the quiz, all the participants were thanked for their participation and given the correct emotions for each photo that the actor was portraying. A brief statement was made to explain that there were no right or wrong answers due to the purpose of the experiment.

Results

The results shown in Table 1, Table 2 and Figure 1 were collected via the results of the online quiz/experiment, with the data collected being calculated automatically by the online quiz/experiment program. The results of which were available only to the interviewee in both an individual view (Appendix C) or an overall summarised view of all participants results (Appendix D). Analysis of the data found that there was no overall increase in correct judgements of emotions of the photographs displaying hand gestures, compared to the correct judgements of emotions of photos with no hand gestures.

As shown in Table 1, when the four photographs showing facial expressions with no hand gestures were presented to the participants, photo 1; sadness, had one incorrect recognition of fear and photo 3; surprise, had two incorrect recognitions of happiness. Photos 2; anger and photo 4; happiness all had the correct emotion recognised.

In Table 2, when the four photographs showing facial expressions accompanied by hand gestures were presented to the participants, photo 5; fear, had one incorrect recognition of surprise, photo 7; anger, had one incorrect recognition of disgust and photo 8; surprise, had one incorrect recognition of fear. Photo 6; sadness, was the only photograph that all five participants recognised the correct emotion being expressed.

Overall the number of correct emotion recognitions between the photographs showing expressions with no hand gestures and the photographs showing expressions with hand gestures presented no difference between the two different sets of photographs (Figure 1). Out of a possible twenty correct answers, both sets had seventeen overall correct answers.

Discussion

It was hypothesised that hand gestures may help the in the recognition of emotion expression and that the participants of the study would show an increase of correct recognitions of the emotion expressed from the photographs showing the facial expressions with hand gestures compared to the photographs showing just the facial expressions. The hypothesis was formed from theories of well-known studies by Ekman and Friesen (Constants across cultures in the face and emotion, 1971) in which they studied the universality of facial expressions, Wallbott's (Bodily expression of emotion, 1998) study of emotion expression in posture, movement and gestures, and the studies of Van den Stock, Righart and de Gelder (Body expressions influence recognition of emotions in the face and voice, 2007), which looked at not just the facial expressions but the whole body expression.

The tests conducted in this study revealed that there was no increase in emotion recognition regardless of whether the hand gestures were present or not. The overall results as presented in Figure 1 show that there was no increase in the amount of correct emotion recognition when using hand gestures alongside facial expression, compared to the amount of correct emotion recognitions of the photographs showing just facial expressions without hand gestures, with both sets of photographs having seventeen out of twenty correct recognitions of emotion.

The study by Ekman and Friesman (1971), produced what is considered to be the most commonly accepted facial expressions; happiness, surprise, sadness, fear, disgust and anger. Their study showed that individuals shown photos of these expressions, generally made the same correct recognition of the emotion being portrayed. This is supported by the findings of this study as shown in Table 1; where there where only three incorrect recognitions of the emotions expressed, and Table 2; where again only three incorrect recognitions were made.

Further research by Van der Stock, Righart and de Gelder (2007) extended this research by presenting phtotos to participants of facial expressions with accompanying body language. Their findings showed that "recognition of facial expressions is influenced by the accompanying body language" (p. 491). These findings were also supported by the previous work of Wallbott (1998) who ascertained a correlation between some emotions and patterns of movement and certain postures. It is these findings that are in contrast to the results of this study. As shown in Figure 1; where both sets of photographs had seventeen out of twenty correct recognitions of the emotion being expressed, this showed no increase in the recognition of the emotions being expressed between the photos of just facial expressions and those that displayed hand gestures alongside the facial expressions.

It is felt that the amount of participants in the experiment may not have been large enough to draw a strong conclusion from the limited amount of information that was being sought. To address this, it may be advantageous to increase the number of participants to achieve a true reflection of the information. Another possibility is the use of static images of emotion, with recent research by Keltner and Codaro (Understanding Multimodal Emotional Expressions: Recent Advances in Basic Emotion Theory, 2015) on emotion recognition revealing a trend towards a more multimodal view; that emotion is not static and that fluidity of movement needs to be viewed to better recognise emotion. With this in mind, the possibility of displaying video clips of expressions may provide a clearer outcome to recognising the subtleties of emotion and hand gestures.

Research by Russell (Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies., 1994) has suggested that rather than narrowing the participants options to match a single emotion to the portrayed expression, that allowing the participant to explain in their own words what they think the emotion being expressed may provide a clearer insight. For example photo 6 (Appendix A), which is portraying sadness; the participant may explain this as the person seeming fearful by the way they have closed in on themselves with their hands close to their face and they are sad because they are scared. This possible explanation allows the interviewer to note exactly what the participant is viewing in regards to the portrayed emotion.

The overall findings from this study have shown no increase of participants recognising emotion when presented with images showing facial expressions with accompanying hand gestures compared to photos with only facial expressions. The results revealed a high number of correct recognitions across all of the photographs presented by the participants with thirty-four out of the forty photographs being correctly identified, which supports previous research pertaining to the recognition of facial expressions. But as there was no increase between the two sets of photographs, with both achieving seventeen out of twenty correct recognitions of emotion, this current study cannot show that emotion recognition is increased with the use of accompanying hand gestures.

While the hypothesis has not been supported in this study, it has raised some interesting questions and possibilities for future research. Can emotions and hand gestures be accurately recognised from static images? Is a one choice per static image, presenting a true representation of participants observations, or is it a case of just picking a random choice from the options given? It is felt that by expanding the method of portraying the expressions, hand gestures and emotions and the method in which participants are able to respond to what is being portrayed to them, may provide a clearer insight into the recognition of emotions and whether or not accompanying hand gestures increase these recognitions.

References 

Ekman, P. (1999). Facial Expressions. In T. Dalgleish, & M. J. Power (Eds.), Handbook of Cognition and Emotion (pp. 301-320). Chichester, UK. doi: 10.1002/0470013494.ch16, UK: John Wiley & Sons, Ltd.

Ekman, P., & Friesmen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124-129.

Keltner, D., & Cordaro, D. T. (2015). Understanding Multimodal Emotional Expressions: Recent Advances in Basic Emotion Theory. Emotion Researcher. Retrieved March 10, 2017, from http://emotionresearcher.com/understanding-multimodal-emotional-expressions-recent-advances-in-basic-emotion-theory/

Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115(1), 102-141.

Van den Stock, J., Righart , R., & de Gelder, B. (2007). Body expressions influence recognition of emotions in the face and voice. Emotion, 7(3), 487-494.

Wallbott, H. G. (1998). Bodily expression of emotion. European Journal Of Social Psychology, 28(6), 879-896.

Appendix

Appendix A

Photographs

Appendix B

Static copy of online quiz

Appendix C

Examples of individual results view

Appendix D

Example of collective/summarised results view