Stress recognition identifying relevant facial action units through explainable artificial intelligence and machine learning

Abstract

Background and objective: Facial cues and expressions constitute a component of bodily responses that provide useful information about one's stress levels. According to the Facial Action Coding System, they can be modelled consistently in terms of fundamental facial muscle movements, called facial Action Units (AUs). This article investigates automatic acute stress recognition based on AUs using conventional Machine and Deep Learning techniques.

Methods: We created a new experimental dataset containing 58 participants performing 4 experimental phases and 11 stress and non-stress tasks in which the proposed system performs automatic facial AUs recognition. A computational feature selection method was employed to select a robust relevant AU combinations subset, which integrated with conventional Machine Learning and Deep Learning methods using the Layer-Wise Relevance Propagation algorithm to assess and model the implication of AUs under acute stress conditions. Ordinal modelling was used following the pairwise transformation to establish a common reference based on the personalized values of each participant.

Results: The results indicate that, under acute stress conditions, participants' faces presented significantly more AUs and with greater intensity compared to neutral conditions. The most relevant combination of AUs to each stress type was computationally identified, ranked and selected. The mean yielded classification accuracy of stress condition versus neutral achieved across all experimental tasks was greater than 93%.

Conclusions: There are specific combinations of AUs that are relevant to the stress conditions of each experimental phase leading in each case to better neutral and stress separability.

Keywords: Deep learning; Explainable artificial intelligence; Facial action units (AU); Machine learning; Stress.