Emery Pierson

Baptiste Chopin

Kévin Feghoul

Estephe Arnaud

Santana Deise

Thomas Maia

Besnier

Keywords: physiological signals, facial landmarks, emotion recognition, facial expression recognition, stress/pain detection, self-supervised learning, metric learning, manifold

Emotion recognition is a subfield of affective computing and a critical research direction for the development of human-centered artificial intelligence, which targets the identification and interpretation of human emotions through machines in an objective and automatic manner. Emotions can be expressed and conveyed through a variety of channels which can be broadly categorized into externally observable behavioural modalities such as facial expressions, body postures, speech and intrinsic physiological modalities such as brain activities, skin conductance, heart rate, etc. Therefore, emotion recognition methods have been developed accordingly based on these modalities. Recent advances in sensor technology and human-computer interaction systems offer the possibility of migrating the deployment of emotion recognition systems from the laboratory to the real world. However, most of the existing research is still directed towards the former. In this context, we are interested in designing emotion recognition algorithms for real-life scenarios from physiological, behavioural and multimodal perspectives and exploring the implications and impact of their interventions on practical applications.

In this thesis, from the physiological perspective, we propose a self-supervised multimodal representation learning method for wearable emotion recognition based on peripheral physiological signals to cope with the overfitting problem posed by limited labelled data and the bias of inaccurate real-world labelling. From the behavioural perspective, we suggest a non-Euclidean metric learning method for 2D facial landmark-based expression recognition to tackle problems such as pose variation and intra-class variation. From a multimodal perspective, we present a deep geometric framework based on a symmetric positive definite matrix representation for multimodal stress and pain detection to address the fusion issue of physiological and behavioural modalities. Additionally, we also realize the deployment of emotion recognition in a real-life scenario, i.e. medical simulation training, where the previously proposed algorithm is integrated into a graphical user interface to test its feasibility and validity on the collected multimodal data, while exploring its pedagogical effects.

Résumé

La reconnaissance des émotions est un sous-domaine de l'informatique affective et une direction de recherche critique pour le développement de l'intelligence artificielle centrée sur l'homme, qui vise l'identification et l'interprétation des émotions humaines par des machines de manière objective et automatique. Les émotions peuvent être exprimées et transmises par divers canaux qui peuvent être classés en deux grandes catégories : les modalités comportementales observables de l'extérieur, telles que les expressions faciales, les postures corporelles et la parole, et les modalités physiologiques intrinsèques, telles que les activités cérébrales, la conductivité de la peau, le rythme cardiaque, etc. Des méthodes de reconnaissance des émotions ont donc été développées sur la base de ces modalités. Les récentes avancées en matière de technologie des capteurs et de systèmes d'interaction homme-machine offrent la possibilité de faire passer le déploiement des systèmes de reconnaissance des émotions du laboratoire au monde réel. Cependant, la plupart des recherches existantes sont encore orientées vers le premier. Dans ce contexte, nous nous intéressons à la conception d'algorithmes de reconnaissance des émotions pour des scénarios de la vie réelle d'un point de vue physiologique, comportemental et multimodal et à l'exploration des implications et de l'impact de leurs interventions sur des applications pratiques. intégré dans une interface utilisateur graphique pour tester sa faisabilité et sa validité sur les données multimodales collectées, tout en explorant ses effets pédagogiques.

Mots clés: signaux physiologiques, repères faciaux, reconnaissance des émotions, reconnaissance des expressions faciales, détection du stress et de la douleur, apprentissage auto-supervisé, apprentissage métrique, manifold.

iii First and foremost, I would like to thank my supervisor, Mohamed Daoudi, for his invaluable guidance, encouragement, and unwavering support throughout my research. He has demonstrated an exceptional ability to connect with people on a personal level, and his kindness and understanding have created a supportive and positive research environment. Moreover, I have benefited greatly from his approach to balancing life and research, which will be an asset for the rest of my life.

Secondly, I would like to express my sincere gratitude to my co-supervisor, Ali Amad, for his patient listening and unstintingly positive feedback during every meeting. His experience and expertise in the field of cognitive science has been instrumental in enabling me to continue my exploration and research in cross-disciplinary areas.

I would also like to thank Prof. Juan-Carlos Álvarez Paiva from the University of Lille for his invaluable contributions to my research on facial expression recognition. His solid knowledge and expertise in the field of mathematics have been essential in advancing my understanding of complex mathematical concepts and their applications for learning tasks. I would like to express my special thanks to the members of the PhD committee, Prof. Gilles Lebuffe, Prof. Irene Cheng, Prof. Denis Hamad, Assistant Prof. Shaun Canavan, Associate Prof. Stefano Berretti, and Associate Prof. Deise Santana Maia for their time, insights, and precious feedback of my research. The horizontal axis of each subplot is the number of randomly selected samples from each class, varying from 1 to 1000, while the vertical axis is the corresponding average accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Different architectures used in the ablation studies of fusion strategies. (GAP:

1D global average pooling applied before classification.) . . . . . . . . . . . . 3.9 Evaluation results of the robustness of the SSL methods in the presence of missing modalities. The horizontal axis of each subplot represents the name of the missing modality, while the vertical axis represents the drops in model performance compared to the case of complete modalities, where the metrics of the vertical axes in the first and second rows are accuracy and F1-score, respectively. (ns: no significant difference; * : p < 0.05, the more asterisks, the more significant the difference.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Overview of the proposed framework. First, the SPD matrix sequences that incorporate the correlation information between multimodal data (i.e. physiological and behavioural signals) can be extracted from the segmented data records.

Subsequently, the tangent space mapping projects the SPD matrix sequences to the vector sequences in the tangent space. Finally, these vectors can be used as input to the LSTM-based classification network for stress/pain recognition. . . ⋆ : optimal matrix.) 6.2 Confusion matrix using two optimal metrics, i.e., Fubini-Study metric and Euclidean metric obtained after metric learning for the CK+ dataset. (Angry (An); Contempt (Co); Disgust (Di); Fear (Fe); Happy (Ha); Sad (Sa); Surprise (Su).

Underlining indicates superior performance of the proposed metric.) . . . . . . 6.3 Confusion matrix using two optimal metrics, i.e., Fubini-Study metric and Euclidean metric obtained after metric learning for the Oulu-CASIA dataset. (Angry (An); Disgust (Di); Fear (Fe); Happy (Ha); Sad (Sa); Surprise (Su). Underlining indicates superior performance of the proposed metric.) . . . . . . . . . 6.4 Comparison with state-of-the-art geometric methods on the CK+ dataset. . . . .

Motivation

Emotions are sets of complex physiological, cognitive and behavioral responses that are triggered by internal or external stimuli. They perform a central role in human daily life and significantly influence mechanisms such as attention, decision-making, learning, memorisation and perception. Therefore, the identification of emotional states is crucial for a thorough understanding of human intelligence, behaviour and cognition [START_REF] Bota | A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals[END_REF][START_REF] Schmidt | Wearable affect and stress recognition: A review[END_REF]. In 1995, Picard's pioneering work [START_REF] Picard | Affective Computing[END_REF] in affective computing laid the foundation for enabling machines to intelligently recognise, interpret, simulate, and respond to human emotions. It is an interdisciplinary field that integrates computer science, psychology, social and cognitive sciences. One of the main topics that emerged from it is emotion recognition [START_REF] Poria | A review of affective computing: From unimodal analysis to multimodal fusion[END_REF] which attempts to empower computers with the ability to automatically infer human emotions and has become increasingly essential for the advancement of human-centred artificial intelligence.

In recent years, rapid advances in sensors, algorithms, and computing resources are driving the deployment of emotion recognition systems from the laboratory to the real world. The corresponding application scenarios include:

• Healthcare: Emotion recognition provides diagnosis, prediction, and proactive intervention for psychological disorders such as depression [START_REF] Ayata | Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems[END_REF][START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF];

• Automotive: Emotion recognition can assist in driving by identifying whether drivers are fatigued or aggressive in order to ensure their safety [START_REF] Silva | In-vehicle driver recognition based on hand ecg signals[END_REF][START_REF] Healey | Detecting stress during real-world driving tasks using physiological sensors[END_REF];

• Human-computer interaction: Emotion recognition can interpret emotional cues to enhance the user experience of various interactive devices such as robots, games, smart speakers, etc [START_REF] Liu | A facial expression emotion recognition based human-robot interaction system[END_REF][START_REF] Granato | Feature extraction and selection for real-time emotion recognition in video games players[END_REF];

• Marketing and advertising: Emotion recognition enables precise marketing or the design of specialized advertising by interpreting the emotional responses of the target customers [START_REF] Liu | Eeg-based emotion-adaptive advertising[END_REF];

In addition to the above examples, another compelling application is Emotion-aware education [START_REF] Li | Research on leamer's emotion recognition for intelligent education system[END_REF][START_REF] Hammoumi | Emotion recognition in e-learning systems[END_REF], which aims to incorporate emotion recognition into education programs by monitoring the students' emotional state to provide better quality tutoring. Previous studies [START_REF] Bower | Mood and memory[END_REF][START_REF] Harley | An integrated emotion-aware framework for intelligent tutoring systems[END_REF][START_REF] Feidakis | Endowing e-learning systems with emotion awareness[END_REF] have suggested a strong association between emotions and learning procedures. In [START_REF] Immordino-Yang | We feel, therefore we learn: The relevance of affective and social neuroscience to education[END_REF], emotions are considered to be a major impact factor in learning. In addition, the impact of different types of emotions on the learning process varies, with positive emotions influencing aspects such as motivation, concentration and self-regulation of learning, while negative emotions are detrimental to students' performance and achievement [START_REF] Hammoumi | Emotion recognition in e-learning systems[END_REF]. One of the most representative negative emotions is stress, defined in Hans Selye's pioneering work as "the non-specific response of the body to any demand for change" [START_REF] Selye | The stress of life[END_REF], and commonly referred to as a state of high arousal. According to the Yerkes-Dodson law [START_REF] Yerkes | The relation of strength of stimulus to rapidity of habit-formation[END_REF], a person's arousal level tends to lead to a decrease in performance after a certain threshold has been exceeded (i.e. when in a state of stress). An illustration is shown in Fig. 1.1. The evidence reviewed here indicates the demand to introduce automatic recognition of students' emotions, especially their stress states, into the educational sessions, which is indeed the intention of this thesis.

Low High Arousal

Weak

Strong

Performance

Fatigue, sleepiness

Optimal arousal

Strong stress, anxiety, impaired performance Figure 1.1: The correlation between arousal levels and performance described in Yerkes-Dodson law [START_REF] Yerkes | The relation of strength of stimulus to rapidity of habit-formation[END_REF]. Excerpt from [START_REF] Saeed | Deep physiological arousal detection in a driving simulator using wearable sensors[END_REF].

More specifically, this thesis concerns medical simulation training for health professionals at the PRESGAE Health Simulation Centre1 . During the simulation training, learners may experience stressful states due to factors such as "high fidelity" medical scenarios, complex medical tasks, etc., which can result in decreased performance and learning efficiency. Therefore, the main objective of this thesis is to develop a new tool for identifying the emotions, especially stress, felt by learners during simulations, thereby offering them targeted debriefings to improve their learning. In the end, two sub-goals are concluded from the main objective:

• Technical objective: emotions are multi-faceted and involve changes in behavior, physiology, and psychology. Thus, novel automatic emotion recognition algorithms suitable for real-life settings can be designed and evaluated from behavioural, physiological, and multimodal perspectives to achieve a more thorough exploration.

• Practical objective: this sub-goal can be further divided into three steps: firstly, emotion data needs to be collected under real-life conditions, secondly, an emotion recognition tool (i.e. a graphical user interface) that integrates the designed intelligent algorithms needs to be developed and tested under real-life conditions, and finally, the pedagogical impact of this tool needs to be analyzed and evaluated.

Challenges

A wide range of human emotional information can be captured, broadly classified into two categories: behavioural modality-based (e.g., facial expressions, body gestures, audio, etc) and physiological modality-based (e.g., electroencephalography (EEG) signals and a series of peripheral physiological signals), representing controllable external manifestations and uncontrollable internal variations under certain emotions, respectively. Considering the necessity of switching from in-lab settings to real-world applications, we focus on peripheral physiological signals and facial landmarks for unimodal emotion recognition, while the multimodal approach is based on the fusion of physiological modalities (i.e., peripheral physiological signals) and two behavioral modalities (i.e., facial expressions and body movements represented by facial landmarks and accelerations, respectively). The reason for selecting these modalities is further explained in Section 2.2. In terms of emotion modality, automatic emotion recognition methods are accordingly differentiated into behaviour-based approach, physiology-based approach and multimodal (behavioural+physiological) approach. In the following, we list the challenges encountered by the different types of approaches for deploying accurate and robust emotion recognition systems in a practical setting.

Physiological chanllenges

-Feature engineering of multivariate signal Physiology-based emotion recognition is a complex time-series analysis involving multichannel heterogeneous signals, which typically require domain knowledge in neuropsychology [START_REF] Zhang | Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review[END_REF]. To achieve favorable recognition performance, critical features need to be extracted and selected from each variable, which turns out to be computationally expensive and time-consuming, especially for traditional machine learningbased algorithms. Furthermore, the combinations of multimodal features are generally based on early or late fusion which cannot simultaneously capture intra-and inter-modal correlations.

-Inaccurate annotation and noisy data Existing peripheral physiological datasets were mainly collected in a laboratory environment, where the experimental protocols were carefully designed to stimulate the expected emotional responses and obtain accurate labeling (the bound-ary between emotional and non-emotional data is quite clear based on the protocol). When shifting the focus to real-life scenarios, the continuous nature of emotions makes it challenging to determine the beginning and end of emotional events. In addition, subjective errors introduced by self-evaluation also contribute to the uncertainty of the labels. Besides, unrestricted body movements can lead to signal artefacts which significantly degrade data quality [START_REF] Bota | A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals[END_REF].

-Limited labeled data and low subject diversity Labeled data volume and subject diversity are critical to obtaining models with generalization capabilities, particularly for supervised deep learning models, whose automatic representation learning can effectively address the plague of inefficient feature engineering. To date, the majority of emotion-related physiological datasets are based on EEG signals, which are not applicable in real-life scenarios. In addition, current emotion recognition studies typically use physiological data covering no more than 50 subjects [START_REF] Bota | A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals[END_REF].

Behaviroual chanllenges

-Pose variations In an unconstrained environment, the head pose may not be frontal to the camera, and this variation in appearance results in facial displacement, scale changes, in-plane and out-of-plane rotations, preventing us from directly using the detected facial feature points for facial expression analysis [START_REF] Ding | Pose-invariant face recognition with homography-based normalization[END_REF]. From the perspective of landmark configurations, these changes can be summarised as rigid transformations of the face shape in the 3D case and more complex projection transformatios in the 2D case. In addition, pose changes are often accompanied by other factors such as self-occlusion, texture distortion, etc.

-Intra-class variation inter-subject differences due to gender, age, culture, etc. often hinder the analysis of facial points, resulting in large intra-class variations. This is manifested as different individuals expressing the same facial expression in different ways or, in a more extreme case, the same person presenting the same expression in different ways at different moments.

Multimodal chanllenges

Since the manifestation and evolution of emotions involves signals from multiple domains, recent research has turned to behaviroual-physiological fusion with the expectation of leveraging complementary properties between modalities to achieve more robust predictions [START_REF] Baltrušaitis | Multimodal machine learning: A survey and taxonomy[END_REF]. In addition, multimodal-based approaches can deal with the problem of missing modalities. For example, when dealing with complex emotions such as stress, the intensity of facial expressions is commonly weak, whereas physiological information can assist in identifying emotions.

-Lack of multimodal data Currently, the emotion recognition community struggles with a lack of public available datasets which contain both behavioural and physiological data. Therefore, existing efforts tend to concentrate on a single perspective (i.e., either behavioural or physiological).

-Varying data structure Multimodal data also suffers from diversity of data structures. For example, the facial expression videos captured by the camera are a 2D matrix sequences with a high-spatial, low temporal resolution, while the peripheral physiological signals captured by the biosensors are a 1D time series with a lower-spatial, higher-temporal resolution.

-Multimodal fusion A major issue is how to efficiently fuse multimodal features to improve model performance. Simply concatenating feature vectors to generate multimodal representations is the most frequently used technique for integrating behavioral and physiological data.

However, the resulting vectors are typically high-dimensional, associated with redundancy, and fail to effectively capture intra-modal correlations. An alternative way in existing research is decision-level fusion, where features from different modalities are extracted independently to provide the corresponding decisions. However, the interactions between the modalities are not well established during this process.

Thesis contributions

In order to achieve the main objective of this thesis, i.e. to develop novel emotion recognition algorithms and to facilitate their deployment in the real world, the challenges mentioned in the previous section should be addressed. To this end, the contributions achieved in this work can be grouped accordingly into technical and practical levels. Secondly, unlike most supervised learning-based approaches, a self-supervised learning (SSL) scheme was adopted to (3) address the problem of de-generalisation due to the limited amount of labeled data and the small number of subjects, (4) get rid of the interference caused by inaccurate labeling and various noises in real-life scenarios, and

(5) obtain an effective classification model that can be applied to a range of emotionally relevant downstream tasks.

• From the behavioural viewpoint, we propose a non-Euclidean metric learning method that can be applied to facial expression recognition. Different from traditional landmarkbased approaches which construct the facial shape representation in Euclidean space, we considered the equivalence classes of 2D landmark configurations, which can be considered as points in the complex projective space to (1) obtain a more robust shape representation that is invariant under the class of affine transformations which contains rotation, scaling, translation, and (2) better measure the similarity between different facial shapes by using non-Euclidean metrics. Moreover, a metric learning algorithm was applied to

(3) reduce the intra-class variations while increasing the inter-class distances to achieve a more discriminative feature space.

• From the multimodal viewpoint, we presented a novel LSTM-based geometric framework for multimodal stress and pain detection tasks. Unlike commonly used early fusion or late fusion strategies, we computed the symmetric positive definite (SPD) matrix of multimodal data as input to the learning model to (1) cope with structural differences between visual and physiological data and obtain compact, high-order multimodal representations, and (2) simultaneously capture intra-and inter-modal correlations at different instants. To the best of our knowledge, this is the first use of the geometry of SPD matrices to merge physiological and behavioural signals.

Practical level Emotion recognition in unconstrained conditions, i.e. during medical simulation training, was performed and the corresponding contribution was also three-fold:

• From the experimental viewpoint, a large-scale multimodal dataset containing motion signals and peripheral physiological signals was collected via a wearable device for emotion analysis, where a series of psychological questionnaires were assigned to subjects to generate ground truth for partial data.

• From the implementation viewpoint, an emotion analysis tool, i.e., a graphical user interface was developed to (1) load, synchronize and display video and multimodal signal recordings, (2) execute the proposed emotion recognition algorithms and (3) test its feasibility in a real-life setting.

• From the pedagogical viewpoint, the impact of the intervention of the emotion recognition system on pedagogy was analyzed and evaluated.

Thesis outline

This thesis is organised as follows: In Chapter 2, we conducted a systematic literature review of emotion representations, emotion-related modalities, and emotion recognition approaches based on physiological, behavioural and multimodal modalities.

Emotion models

For the purpose of interpreting and quantifying human emotional states, two types of generic emotion models have been established by psychologists: the categorical model and the dimensional model, in which emotions are either denoted by discrete categories or mapped into a multidimensional space, respectively [START_REF] Wang | A systematic review on affective computing: emotion models, databases, and recent advances[END_REF]. The emotion recognition tasks involved in this thesis and previous related studies are all based on these two emotion models.

Categorical models

The categorical models define emotions as discrete classes. This type of approach assumes that there exists a set of basic emotions that are innate to humans, whose expression and recognition is generally undifferentiated for individuals from different races or cultures [START_REF] Colombetti | From affect programs to dynamical discrete emotions[END_REF]. A series of studies were carried out to build the collection of basic emotions, the most prominent of which are Ekman's basic emotion model [START_REF] Ekman | An argument for basic emotions[END_REF] and Plutchik's 'wheel of emotions' [START_REF] Plutchik | A psychoevolutionary theory of emotions[END_REF]. Ekman proposed six basic emotions: joy, sadness, anger, fear, disgust, and surprise, based on the following attributes: 1) People are born with emotions rather than acquired; 2) People show the same emotions when they are in the same situations; 3) People exhibit analogous physiological or behaviroual patterns when expressing the same emotions. Plutchik's emotional wheel model [START_REF] Plutchik | A psychoevolutionary theory of emotions[END_REF] contains 8 primary bipolar emotions: joy vs sadness, anger vs fear, trust vs disgust, surprise vs anticipation. Fig. 2.1(a) shows an illustration of the 'wheel of emotion'. The intensity of the emotion ranges from weak to strong, as it shifts from the periphery to the centre of the wheel. In addition, more complex emotions can be also generated by combining primary emotions, e.g., the combination of joy and trust is love.

Dimensional models

For categorical models, complex emotions are more difficult to process since they cannot cover all emotion. Moreover, the continuity of changes in emotional states might not be well captured by discrete models. To address the above issues, researchers have turned to the dimensional models which describe emotions as points in a space of different dimensions. Russell [START_REF] Russell | Affective space is bipolar[END_REF] proposed a two-dimensional model, also known as the circumplex model, which is the most frequently adopted dimensional model for identifying emotions [START_REF] Schmidt | Wearable affect and stress recognition: A review[END_REF]. In this model, each emotional state is mapped as a discrete point in a two-dimensional space, with horizontal and vertical coordinates represented by valence and arousal, respectively. The valence axis represents the emotions ranging from unpleasant (negative) to pleasant (positive), and the arousal axis ranges from passive (low) to active (high), indicating its intensity level [START_REF] Bota | A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals[END_REF]. For example, stress can be defined as a specific emotional state of negative valence and active arousal. A graphical representation of the circumplex model is shown in Fig. 2

Emotion related modalities

Automatic emotion recognition can be realized on the basis of behavioural or physiological indicators of the human body.

Behaviroual modalities

Behavioural responses to emotional states are actions that can be voluntarily controlled or changed and externally observed [START_REF] Alberdi | Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review[END_REF]. Facial expressions [START_REF] Jain | Extended deep neural network for facial emotion recognition[END_REF], body gestures [START_REF] Piana | Real-time automatic emotion recognition from body gestures[END_REF], speech [START_REF] Huang | Speech emotion recognition using deep neural network considering verbal and nonverbal speech sounds[END_REF],

etc, are representative modalities of conveying one's emotional state, with the former two appearing to provide richer relevant information. A body language-based emotion recognition study indicates that non-verbal signals such as facial expressions, body posture, gestures, etc, have a more significant impact on the delivery of information than verbal signals, accounting for 65% of the total.

The facial expression captured by the camera is a typical non-verbal indicator of emotional states projected from human physical appearance. It is generally the result of an action or combination of actions formed by facial muscles and organs. Previous research [START_REF] Mehrabian | Communication without words[END_REF] indicated that facial expressions are the most dominant form in which humans convey their emotions during social communication. According to the pioneering work of Ekman and Friesen [START_REF] Ekman | Constants across cultures in the face and emotion[END_REF],

there is a high degree of consistency in facial expressions across human groups of different ethnic and cultural backgrounds. In addition, they adopted the facial action coding system (FACS) that systematically defines facial expression classification criteria by describing the facial muscle movements and concluded six basic emotions: joy, sadness, anger, fear, disgust, and surprise. The strong connections between emotions and facial expressions established by neuroscience and psychology research have inspired researchers in the fields of computer vision and machine learning to explore automatic facial expression recognition algorithms. This also explains the reason for selecting facial expressions as the input modality for behaviour-based and multimodal emotion recognition in this thesis.

Another compelling visual modality that delivers human emotional information is body posture, which represents the movement of body parts and joints, It can normally be captured by cameras, Kinect sensors, or motion capture systems. Though most emotion analysis studies consider facial expressions as the primary source, there is still a body of literature dedicated to uncovering the link between body gestures and emotions. For example, a person's inner emotion changes can be expressed through the placement of hands or legs and the way they perform actions such as walking, sitting or standing [START_REF] Noroozi | Survey on emotional body gesture recognition[END_REF]. The position of the head is also one of the emotion-related cues, a raised chin, for example, may indicate an arrogant attitude and a sense of superiority [START_REF] Balaban (ghitan | The Definitive Book of Body Language by Allen & Barbara Pease[END_REF]. However, different from facial expressions, body posture seems to be greatly influenced by cultural or gender differences. An alternative way of obtaining information on human posture and activity for emotional analysis is to record 3-axis accelerometer (ACC) data from a smartphone or smartwatch. The ACC was initially used in wearable activity recognition systems, recent research in emotion recognition has applied it to the acquisition of contextual information on user actions [START_REF] Schmidt | Wearable affect and stress recognition: A review[END_REF]. In addition, some research [START_REF] Rojas | Stress recognition: A step outside the lab[END_REF] has demonstra0ted the feasibility of detecting stress states by estimating the intensity level of activities. In our work, it is applied to the multimodal emotion recognition task.

Physiological modalities

Physiological reactions triggered by emotions form a natural part of the human body functioning which are involuntary and involve internal manifestations that are imperceptible to the naked eye, such as changes in heart rate, respiratory rate, etc. Therefore, the physiological measures can provide more objective decisions for emotion recognition systems [START_REF] Wang | A systematic review on affective computing: emotion models, databases, and recent advances[END_REF]. The physiological modalities primarily consist of electroencephalography (EEG) signal captured from Central Nervous System (CNS) and a series of signals measured from the Peripheral Nervous System (PNS) such as electrocardiogram (ECG), respiration (RESP), blood volume pressure (BVP), electrodermal activity (EDA), skin temperature (TEMP), etc.

The central nervous system as indicated by the yellow area in Fig. 2.2, consisting of the brain and spinal cord, is the main part of the human nervous system, where the electrical activity of brain neurons can be accessed through EEG signals, providing information highly relevant to emotions [START_REF] Bota | A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals[END_REF]. However, the acquisition of EEG data requires the support of bulky and invasive head-mounted sensors, which typically consist of a large number of electrodes placed in different areas of the scalp, making their implementation in real-life scenarios quite challenging [START_REF] Schmidt | Wearable affect and stress recognition: A review[END_REF].

In addition, they are highly sensitive to physiological artefacts (e.g. blinking) and electrostatic artefacts introduced by the use of electrodes [START_REF] Jerritta | Physiological signals based human emotion Recognition: a review[END_REF]. The peripheral nervous system refers to all nerves apart from the brain and spinal cord (as shown by the blue area in Fig. 2.2). In response to external or internal emotional stimuli, the autonomous nervous system (ANS), as one of the primary components of the PNS, is activated, thereby triggering a series of changes in peripheral physiological signals. For instance, elevated respiratory rate and increased electrodermal activity appear when subjects are in a high arousal state (e.g., anger, stress, fear) [START_REF] Schmidt | Wearable affect and stress recognition: A review[END_REF]. The peripheral physiological signals that are frequently used for emotion recognition include:

• Electrocardiography (ECG): is a recording of the electrical activity generated during the contraction or relaxation of the cardiac muscles. It is a crucial indicator in emotion recognition research which is commonly applied to estimate parameters related to the cardiac cycle such as the heart rate (HR), heart rate variability (HRV), etc.

• Blood Volume Pulse (BVP): photoplethysmography detects changes in blood volume by the degree to which the artery absorbs the light it emits, thus inferring cardiac cycle information.

• Respiration (RESP): a measurement of exhaled or inhaled air volume and respiratory rate by contraction or relaxation of a chest strap fixed near the chest cavity.

• Electrodermal Activity (EDA): this signal reflects sweat secretion by measuring skin conductance. It can be broken into two basic components: tonic and phasic, providing slowly habituating measures of arousal and moment-by-moment measures of arousal which reflect stimulus-specific responses [START_REF] Alexander | Separating individual skin conductance responses in a short interstimulus-interval paradigm[END_REF], respectively.

• Skin Temperature (TEMP): is an indicator of the "fight or flight" response produced by exposure to a stressful situation. A drop in temperature of the extremities can be observed when under this response [START_REF] Schmidt | Wearable affect and stress recognition: A review[END_REF].

In contrast to EEG signal, the measurement of peripheral physiological signals can be accomplished with convenient and less invasive wearable sensors such as wristbands, patches, smart clothes, etc. Thus, their applicability in real life is greatly encouraged. Overall, we chose peripheral signals for physiological-based emotion recognition in this thesis.

Behavioural signal vs Physiological signal

The application of behavioural or physiological modalities in emotion recognition systems has its own benefits and drawbacks. Behavioural signals have received considerable attention due to their non-invasive nature in data acquisition. However, their reliability cannot be verified, as people can autonomously control these physical manifestations to hide their true emotions (i.e., social masking) [START_REF] Shu | A Review of Emotion Recognition Using Physiological Signals[END_REF]. Peripheral physiological measures are still slightly more intrusive than non-contact behavioural modalities, however, they allow for a long-term objective assessment of the subject's true internal emotional state. Moreover, considering practical context, they may be more suitable than visual data such as facial expressions, whose performance may be significantly influenced by the location of the camera and the computational cost of image or video analysis is more expensive than time series. Naturally, there is a third option for emotion recognition systems, i.e., the integration of behavioural and physiological modalities, which generally results in more reliable decisions and can cope with missing modalities [START_REF] Poria | A review of affective computing: From unimodal analysis to multimodal fusion[END_REF].

Emotion

Physiological data preprocessing

The primary part of pre-processing is signal filtering, with the purpose of eliminating the impact of noise due to electromagnetic interference, motion artefacts, etc [START_REF] Shu | A Review of Emotion Recognition Using Physiological Signals[END_REF]. Since the data acquisition setting in the literature is diverse, resulting in inconsistent frequencies of the obtained data, hence there is not yet a unified standard for filter selection. For example, low-pass filters with cutoff frequencies of 1 Hz and 2.5 Hz were applied for RESP and EDA signals in [START_REF] Valenza | The role of nonlinear dynamics in affective valence and arousal recognition[END_REF].

Schmidt et al. [START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF] employed a 0.1 Hz to 0.35 Hz bandpass filter for RESP signal while a 5 Hz low-pass filter for the raw EDA data. In [START_REF] Yan | Emotion classification with multichannel physiological signals using hybrid feature and adaptive decision fusion[END_REF], Butterworth bandpass filters with different orders and cutoff frequencies were utilized for ECG, EMG and BVP signals. After the filtering operation, the clean signal record is generally split into small data segments via a sliding window, from which meaningful patterns are subsequently extracted for identification. The time interval between an emotional stimulus and the physiological response it triggers tends to vary with different factors such as individual, signal modality, etc [START_REF] Schmidt | Wearable affect and stress recognition: A review[END_REF]. Therefore, it is challenging to define an appropriate window size for emotion recognition systems. In the work of Kreibig [START_REF] Kreibig | Autonomic nervous system activity in emotion: A review[END_REF], they observed that the most common average duration of physiological responses was 60 s or 30 s in a survey of 134 publications. Other frequently occurring average intervals are based on 0.5 s, 10 s, 120 s, 180 s, or 300 s. Once the data is segmented, manual or automated feature engineering is performed to extract discriminative representations for emotion classification. For subject-independent recognition algorithms, the data are further normalized to remove interindividual differences in physiological variables. The common normalization techniques are min-max normalization or z-score normalization [START_REF] Bota | A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals[END_REF]. The former scales the original data to the [0, 1] interval through the maximum and minimum values in the subject's data, while the latter converts the mean and standard deviation of the samples to 0 and 1, respectively.

Supervised machine learning-based methods

Traditional machine learning approaches typically require extracting and selecting the most relevant features from pre-processed physiological signals and feeding them into a series of classification algorithms for emotion recognition. This process can be further divided into four stages: feature extraction, feature selection, feature fusion and classification, which will be presented accordingly in the subsequent sections.

1) Feature Extraction

A majority of early research in wearable emotion recognition was based on machine learning, aiming to predict emotional states using crucial features manually extracted from preprocessed peripheral physiological signals in the time-domain, frequency-domain, time-frequency domain and non-linear domain as input to classification algorithms.

-Time domain From the temporal perspective, Hernandez et al. [START_REF] Hernandez | Call center stress recognition with person-specific models[END_REF] employed a series of statistical featues from EDA time series such as duration, max, min, mean, standard deviation, signal slope between the first and last points to detect stress in the workplace with a wearable wrist sensor. For ECG and BVP signals reflecting cardiac cycle activity, three important parameters: heart rate (HR), inter beat interval (IBI) and Heart Rate Variability (HRV) can be derived depending on the position of the signal peaks, from which rich emotionally relevant information can be computed. Bong et al. [START_REF] Bong | Analysis of electrocardiogram (ecg) signals for human emotional stress classification[END_REF] suggested three time-domain features: heart rate (HR), mean R peak amplitude (MRAmp), and mean R-R intervals (MRRI) extracted from the ECG signal for a two-stage emotional stress detection. Kim et al. [START_REF] Kim | Emotion recognition based on physiological changes in music listening[END_REF] calculated a series of statistical ECG features: mean, the standard deviation of all normal-to-normal(NN) intervals (SDNN), the standard deviation of the first difference of the HRV, the number of pairs of successive NN intervals differing by more than 50 ms (NN50), etc., for arousal-based emotion recognition. Zhang et al. [START_REF] Zhang | Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing[END_REF] utilized 7 statistical features: min, max, mean, median, standard deviation, minRatio and maxRatio from EMG and EDA signals.

-Frequency domain An alternative method of extracting effective features is to convert the time domain signal to the frequency domain. Fast Fourier Transform (FFT) [START_REF] Kim | Emotion recognition based on physiological changes in music listening[END_REF] was applied on HRV decuced from ECG signal to generate spectral features such as dominant frequency, power of very low frequency band (VLF: 0.003-0.04Hz), low frequency band (LF: 0.04-0.15Hz) and high frequency band (HF: 0.15-0.4Hz), and ratio of power LF/HF. Jerritta et al. [START_REF] Jerritta | Emotion recognition from electrocardiogram signals using hilbert huang transform[END_REF] performed Hilbert Huang Transform (HHT) on normalized QRS derivative ECG data for calcultaing low and high frequency features.

-Time-Frequency domain Some studies [START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF][START_REF] Guendil | Emotion recognition from physiological signals using fusion of wavelet based features[END_REF] focused on the time-frequency analysis using Wavelet Transform (WT). Guendil et al. [START_REF] Guendil | Emotion recognition from physiological signals using fusion of wavelet based features[END_REF] employed WT to decompose each signal into six levels and the resulting WT coefficients were treated as emotion-related components in the six frequency bands. Xie et al. [START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF] calculated WT related features: min, max, mean, and standard deviation in wavelet coefficients after decomposition for ECG, EMG and EDA signals.

-Non-linear domain Apart from the traditional temporal/spectral features, non-linear models have also been suggested for emotion recognition. Rubin et al. [START_REF] Rubin | Time, frequency complexity analysis for recognizing panic states from physiologic time-series[END_REF] derived 19 non-linear ECG features based on complex analysis such as sample entropy, maximal lyapunov exponent, correlation dimension for classifying states of panic and pre-panic. Valenza et al. [START_REF] Valenza | The role of nonlinear dynamics in affective valence and arousal recognition[END_REF] extracted various non-linear features based on Recurrence Plot (RP), Deterministic Chaos (DC) and Detrended Fluctuation Analysis (DFA) on ECG, EDA, and RESP signals for recognizing multilevel aoursal/valence states. Their experimental results indicated that the non-linear features enhanced the performance in comparison with the common time-frequency domain features.

2) Feature Selection and Reduction

Due to the complexity and individual variability of emotions, recognition methods often incorporate multiple physiological modalities to provide better performance [START_REF] Wang | A systematic review on affective computing: emotion models, databases, and recent advances[END_REF], resulting in high-dimensional vectors that cause the curse of dimensionality. Furthermore, some redundant features are not beneficial for identification which may lead to overfitting problems and cause weak generalization of the learning model [START_REF] Shu | A Review of Emotion Recognition Using Physiological Signals[END_REF]. To solve these problems, feature selection algorithms are commonly applied to select the most relevent features for emotion recognition tasks. Some filter methods were employed, which are independent of the learning model and filter features by statistical metrics. For example, Yan et al. [START_REF] Yan | Emotion classification with multichannel physiological signals using hybrid feature and adaptive decision fusion[END_REF] used Mutual Information (MI) to reduce the feature dimension. Ayata et al. [START_REF]Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems[END_REF] suggested the use of mRMR (minimum redundancy maximum relevance) algorithm to select the most appropriate feature subset for emotion classification. However, the drawback of the filter method is that it completely ignores the impact of the selected features on classification algorithm performance [START_REF] Bota | A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals[END_REF]. To address this issue, wrapper methods were proposed which asses the quality of selected features by introducing a classifier. In the work of [START_REF] Kim | Emotion recognition based on physiological changes in music listening[END_REF], Sequential backward selection (SBS) was applied, which is a top-down method that iteratively removes a feature from the entire feature set whose elimination improves the classification performance until the number of features is satisfied. Xie et al. [START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF] selected a similar approach to SBS, Sequential forward selection (SFS), which starts with an empty set. Their experimental results showed that by applying SFS, the performance of the classifier is significantly improved by about 30%. However, the high computational cost associated with exhaustive search on features make wrapper methods impractical for real-life applications [START_REF] Bota | A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals[END_REF]. In contrast to these previous time-consuming methods, some approaches adopted dimensionality reduction techniques, such as Principal Component Analysis (PCA) [START_REF] Valenza | The role of nonlinear dynamics in affective valence and arousal recognition[END_REF] which maps data to a lower dimensional space while retaining the components that contribute most to the variance. To avoid the information loss arising from mapping, a Kernel Principal Component Analysis (KPCA) method was applied in [START_REF] Zhang | Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing[END_REF], which first projects features into a high-dimensional space using Radial Basis Function (RBF) kernel for non-linear information extraction, and then reduces the dimensions similarly to PCA.

3) Feature Fusion

Features from different physiological modalities are generally fused in an early fashion or a late fashion to yield superior performance over unimodal recognition methods. Early fusion [START_REF] Valenza | The role of nonlinear dynamics in affective valence and arousal recognition[END_REF][START_REF] Kim | Emotion recognition based on physiological changes in music listening[END_REF][START_REF] Zhang | Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing[END_REF][START_REF] Guendil | Emotion recognition from physiological signals using fusion of wavelet based features[END_REF][START_REF] Lan Chen | Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers[END_REF] is a simple and intuitive strategy, where unimodal features are concatenated to form a single high-dimensional vector as input to the classifier. However, this approach is struggling with missing data and multimodal asynchrony issue [START_REF] Shu | A Review of Emotion Recognition Using Physiological Signals[END_REF]. Decision fusion can effectively tackle these problems, where classifiers learn independently on unimodal data and the final decision is a combination of predictions from all classifiers. Majority voting [START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF][START_REF]Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems[END_REF][START_REF] Anusha | Physiological signal based work stress detection using unobtrusive sensors[END_REF] is widely deployed for emotion recognition, where the most frequent prediction is served as the final decision. However, this approach ignores the relationships between different modalities, since each classifier is trained independently. To enhance the robustness of the decision, an adaptive decision fusion strategy [START_REF] Yan | Emotion classification with multichannel physiological signals using hybrid feature and adaptive decision fusion[END_REF] was proposed which first constructed sub-classifier weights by computing accuracy matrix, correlation coefficient, instability coefficient and linearly merged the weighted classification results from each classifier. Though this method allow us to capture the cross-modal interaction between different predictions, however, the connections among multimodal features are ignored.

4) Classification

Common machine learning based classifiers in the literature are Support Vector Machine (SVM) [START_REF] Yan | Emotion classification with multichannel physiological signals using hybrid feature and adaptive decision fusion[END_REF][START_REF] Hernandez | Call center stress recognition with person-specific models[END_REF][START_REF] Bong | Analysis of electrocardiogram (ecg) signals for human emotional stress classification[END_REF][START_REF] Zhang | Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing[END_REF][START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF][START_REF] Guendil | Emotion recognition from physiological signals using fusion of wavelet based features[END_REF][START_REF] Rubin | Time, frequency complexity analysis for recognizing panic states from physiologic time-series[END_REF][START_REF] Lan Chen | Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers[END_REF][START_REF] Anusha | Physiological signal based work stress detection using unobtrusive sensors[END_REF][START_REF] Cheng | A novel ecg-based real-time detection method of negative emotions in wearable applications[END_REF], K-Nearest Neighbour (K-NN) [START_REF] Jerritta | Physiological signals based human emotion Recognition: a review[END_REF][START_REF] Bong | Analysis of electrocardiogram (ecg) signals for human emotional stress classification[END_REF][START_REF] Zhang | Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing[END_REF][START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF][START_REF] Rubin | Time, frequency complexity analysis for recognizing panic states from physiologic time-series[END_REF][START_REF] Anusha | Physiological signal based work stress detection using unobtrusive sensors[END_REF][START_REF] Cheng | A novel ecg-based real-time detection method of negative emotions in wearable applications[END_REF], Random Forest (RF) [START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF][START_REF] Rubin | Time, frequency complexity analysis for recognizing panic states from physiologic time-series[END_REF][START_REF]Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems[END_REF][START_REF] Cheng | A novel ecg-based real-time detection method of negative emotions in wearable applications[END_REF], Gradient Boosting Decision Tree (GBDT) [START_REF] Zhang | Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing[END_REF][START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF][START_REF] Cheng | A novel ecg-based real-time detection method of negative emotions in wearable applications[END_REF], Linear Discriminant Analysis (LDA) [START_REF] Kim | Emotion recognition based on physiological changes in music listening[END_REF][START_REF] Anusha | Physiological signal based work stress detection using unobtrusive sensors[END_REF], etc. Among these learning models, the support vector machine (SVM) is the most frequently applied in the field of physiological emotion sensing [START_REF] Schmidt | Wearable affect and stress recognition: A review[END_REF]. Some studies [START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF][START_REF] Lan Chen | Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers[END_REF][START_REF] Cheng | A novel ecg-based real-time detection method of negative emotions in wearable applications[END_REF], have found that the SVM can achieve better performance in emotion recognition than other classifiers. For instance, Cheng et al. [START_REF] Cheng | A novel ecg-based real-time detection method of negative emotions in wearable applications[END_REF] evaluated various classifiers using ECG and HRV features from linear, non-linear, time and frequency domains, mong which SVM achieved the highest accuracy of 79.51% on the task of detecting positive and negative emotions. In [START_REF] Xie | Wt feature based emotion recognition from multi-channel physiological signals with decision fusion[END_REF], wavelet transform features of different modalities were extracted for the binary emotion classification, and ultimately the best accuracy of 94.81% was achieved by fusing the SVM classifiers from ECG and EMG signals. Nevertheless, some studies [START_REF] Zhang | Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing[END_REF][START_REF] Rubin | Time, frequency complexity analysis for recognizing panic states from physiologic time-series[END_REF] have revealed that the ensemble methods may yield superior performance. In [START_REF] Rubin | Time, frequency complexity analysis for recognizing panic states from physiologic time-series[END_REF], the fusion of time-domain, frequency-domain and non-linear ECG features was fed into a series of classifiers for evaluation, where the RF demonstrated the best performance, achieving an accuracy of 97.2% and 90.7% for panic and pre-panic detection, respectively. Zhang et al. [START_REF] Zhang | Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing[END_REF] implemented four classifiers: GBDT, SVM, KNN and Gaussian Naive Bayes (GaussianNB) for performance evaluation. With the property of auto-interaction over multiple feature sets,

The GBDT outperformed other classifiers in all experimetal settings, achieving an accuracy of 93.42% in 4-class emotion recognition task.

5) Discussion of supervised machine learning-based methods

Table 2.1 briefly summarizes the supervised machine learning-based approaches. While these approaches have demonstrated impressive results, they present certain limitations in the feature engineering process. First, emotion-related features are required to be extracted for each modality. This relies heavily on domain knowledge which results in difficulties of transferring the model to other tasks, especially in the multimodal case. Second, the most reliable features should be selected to cope with the curse of dimensionality introduced by multimodal data while obtaining better recognition performance. However, this process is quite time-consuming and may lead to loss of information. In addition, the above two steps limit the flexibility of fusing multimodal data at different levels since one has to always consider the modalities involved as well as the features extracted and selected. -Fully connected The 1D concatenated multimodal vector can be passed directly through fully connected deep networks to learn complex transformations. For example, Saeed et al. [START_REF] Saeed | Personalized driver stress detection with multitask neural networks using physiological signals[END_REF] proposed a Multi-Task fully connected Neural Network (MT-NN) for personalized driver stress -Recurrent Recurrent neural networks (RNNs) and their gated variants such as the long short-term memory (LSTM) are traditional deep neural networks for processing time-series data, whose decisions at current time steps are influenced by past time steps, thus capturing the temporal dependencies of elements in large scale sequences [START_REF] Bradbury | Quasi-recurrent neural networks[END_REF]. Several efforts have been conducted using recurrent models to fully explore the temporal relationships in physiological sequences. For instance, Awais et al. [START_REF] Awais | Lstm-based emotion detection using physiological signals: Iot framework for healthcare and distance learning in covid-19[END_REF] -Hybrid In order to take advantage of both the parallel computation of CNNs and the longterm dependency modelling of RNNs, some researches have adopted hybrid models on emotion detection tasks. For example, Keren et al. [START_REF] Keren | End-to-end learning for dimensional emotion recognition from physiological signals[END_REF] proposed a CNN-LSTM model where convolution and pooling operations first compressed the input sequence to obtain a compact high-level representation, which is subsequently fed into the LSTM to further capture underlying temporal correlations in sequential data. In experiments of identifying arousal and valence, the hybrid model consistently produced better results, compared to methods based on hand-crafted features.

In general, early fusion proved to be the preferred option for most emotion recognition methods based on peripheral physiological signals due to its simplicity of installation, where the joint representation can be extracted directly from the combined multimodal input, emphasizing the importance of inter-modal interactions. However, it neglects the necessity of designing special sub-models to further capture intra-modal properties.

2) Intermediate Fusion

Early fusion treats multimodal data as a whole, where marginal representations are not learned explicitly and hence cannot differentiate which modality the learned features are derived from. normalization to efficiently learn discriminative temporal features, and eventually the marginal representations of the two branches are concatenated and sent to a fully-connected layer with softmax function for predicting emotional arousal and valence. Zhu et al. [START_REF] Zhu | Emotion recognition from physiological signals using multi-hypergraph neural networks[END_REF] designed a emotion recognition framework based on multi-hypergraph neural networks (MHGNN). Three hypergraphs were first built from three physiological modalities (EEG, EOG and EMG), where the vertices and edges of the graph represent the subjects described by the stimuli and the latent correlations between subjects, respectively. Then, two-layer hypergraph neural networks acting on each modality extracted representations encoding both intra-modal and cross-subject correlations. Finally, a dense layer accepted all marginal representations and classified them into different emotion categories. In [START_REF] Liao | Multimodal physiological signal emotion recognition based on convolutional recurrent neural network[END_REF], a hybrid model was proposed for emotion recognition in arousal-valence space, where 2D-CNN was applied to extract spatial features from multichannel EEG signals and temporal features of multiple peripheral signals were captured via LSTM, and ultimately multimodal features were concatenated for fully connected-based classification.

Intermediate fusion can

The marginal fusion-based approach described above fully leveraged intra-modal correlations, however, cross-modal correlations may not be effectively modelled. Therefore, there exists the potential to further enhance recognition performance by exploring inter-modal cooperation.

-Joint Marginal representations from different branches can also be concatenated and passed through a shared encoder to obtain a joint representation for classification, which exploits the complementary and collaborative nature of the modalities. Zhang et al. [START_REF] Zhang | Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine[END_REF] proposed an emotion recognition framework based on regularized deep fusion of kernel machine. Low-level unimodal representation obtained from ensemble deep kernel machine optimization (eDKMO) and bimodal joint representations aggregated by a fully connected layer formed the final representation set, which are fed into the global fusion layer with regularization to generate the final fused representation for classification. This type of method captures cross-modal relationships by including additional integration layers to further enhance the recognition performance.

However, this increases the complexity of the model which may result in overfitting, especially on limited physiological data.

Overall, intermediate fusion extracts marginal and joint representations in a hierarchical

fashion to simultaneously capture intra-and inter-modal relationships. However, few researches have explored the performance of emotion recognition purely based on peripheral physiological signals.

3) Late Fusion

Considering that different modalities can provide distinct information for emotion recognition, separate sub-models can be trained on unimodal data with their resulting decisions being combined to produce the final decision. According to the fusion techniques of sub-decisions, we divide the existing emotion recognition methods into two categories, namely majority votingbased methods and weighted voting-based methods, both of which will be discussed separately in the following.

-Majority voting The majority voting is considered to be the most intuitive late fusion solution, with the most voted category as the classification result. Dar et al. [START_REF] Dar | Cnn and lstm-based emotion charting using physiological signals[END_REF] proposed a hybird framework for multimodal emotion recognition. A 2DCNN-based model and a 1DCNN-LSTM-based model were applied to EEG signals and peripheral physiological signals (i.e., ECG, EDA) respectively to generate modality-specific decisions. The final predictions regarding the quadratic classification of arousal-valence were determined based on the majority voting strategy. There are certain limitations to this approach, e.g. a subset of the classifiers (which are quantitatively dominant) may occasionally give the incorrect classification at the same time [START_REF] Shahzad | Comparative analysis of voting schemes for ensemblebased malware detection[END_REF].

-Weighted voting An alternative approach for decision fusion is to average the outputs of all sub-models, implying that all branches are allocated equal weights. Hssayeni et al. [START_REF] Hssayeni | Multi-modal physiological data fusion for affect estimation using deep learning[END_REF] explored the effectiveness of this fusion technique, in which a 1DCNN-based architecture was trained independently on different peripheral physiological signals. The class probabilities generated by the CNNs were averaged as the final output of the model. In experiments for esti-mating positive and negative emotions, this late fusion approach exhibited better performance against early fusion. Considering that the contribution of different modalities to emotion recognition may be unequally distributed, different weights should be assigned to individual subdecisions. For example, Li et al. [START_REF] Li | Exploring temporal representations by leveraging attention-based bidirectional lstm-rnns for multi-modal emotion recognition[END_REF] proposed a variable weight fusion strategy for multimodal emotion recognition. EEG, ECG and EDA signals were first converted into spectrograms, an Attention-based Bi-LSTM model was then applied to extract the most relevant time-frequency domain features from each signal. Finally, the predictions from all sub-models are fed into the fully connected layer to automatically learn the corresponding weights for each modality. In the classification experiments of arousal and valence states, the proposed fusion strategy significantly improved the performance of the equal weight scheme.

On the whole, late fusion focuses on marginal representations which can encode intramodal correlations and this is particularly effective for data with certain modalities missing or extremely heterogeneous data. However, its major drawback is its inability to capture the interaction between different modalities, especially for highly relevant physiological signals in particular emotional states.

4) Discussion of supervised deep learning-based methods

In the previous sections we reviewed multimodal physiological emotion recognition approaches based on the three fusion strategies, a brief summary of which is presented in Table 2.2. Among the three fusions, early fusion and late fusion can only capture inter-and intra-modal correlations, respectively, while intermediate fusion is encouraging for efficient encoding of both intra-modal and inter-modal associations at different levels. However, it can be clearly observed from Table 2.2 that there are no existing intermediate fusion methods that specifically target low-frequency peripheral physiological signals. From the perspective of real-life applications, EEG signals are inappropriate in this context due to their strong invasive nature. Hence, there is a need to develop such fusion methods that rely solely on peripheral physiological signals. Moreover, for supervised deep learning methods, training a sufficiently accurate and generalizable model commonly depends on a large amount of labeled data, which is challenging for physiological data, as the annotation is time-consuming, expensive, and requires the intervention of domain experts. In addition to the quantitative limitation, the quality of the collected labels can also be a problem. This is particularly true in real-life situations where the ground truth of emotions is based on self-report, which can introduce subjective errors.

Other learning paradigm based methods

In the previous section, we have demonstrated that deep neural network-based approaches are effective in feature extraction and fusion, thus achieving superior performance over traditional machine learning-based approaches. However, developing fully-supervised deep fusion models for emotion recognition in a mobile environment using only wearable peripheral physiological signals still suffers from the following problems: 1) How to train a generalised model using limited labelled data? and 2) How to deal with the biases introduced by inaccurate and uncertain ground truth collected in real-life settings? Since we always have access to large amounts of unlabeled data, recent researches have shifted towards other learning schemes: unsupervised learning, or semi-supervised learning with the intention of leveraging unlabelled data to discover the true distribution of samples. In general, unsupervised learning uses entirely unlabelled data, while semi-supervised learning uses a mixture of unlabelled and labelled data. In the following, we will present physiological emotion recognition methods based on these two learning strategies.

1) Unsupervised

Unsupervised learning algorithms expect to automatically seek hidden structures in data without relying on supervision from labels [START_REF] Dike | Unsupervised learning based on artificial neural network: A review[END_REF]. A common technique technique applied in unsuper-vised learning-based emotion recognition methods is data clustering, which aims to allocate data to different clusters by maximizing intra-cluster similarity while minimizing inter-cluster similarity [START_REF] Grira | Unsupervised and semi-supervised clustering: a brief survey[END_REF]. For instance, Birjandtalab et al. [START_REF] Birjandtalab | A non-eeg biosignals dataset for assessment and visualization of neurological status[END_REF] explored the performance of unsupervised emotion recognition algorithms using multimodal wearable physiological signals consisting of EDA, HR and arterial oxygen level (SPO2). A Gaussian mixture (GMM) model was applied to the clustering of multimodal features, ultimately achieving an accuracy of over 84% in the task of differentiating between four emotional states. Fiorini et al. [START_REF] Fiorini | Unsupervised emotional state classification through physiological parameters for social robotics applications[END_REF] examined the validity of unsupervised learning models with ECG, HR and brain activity signals. Multimodal features extracted from the time and frequency domains were fed into three unsupervised algorithms:

K-Means, K-medoids and Self-organizing maps. Although the above methods can solve the problems associated with data labelling, however they are all based on hand-crafted features.

For the automatic learning of more complex non-linear representations, there is an increasing trend towards the use of deep neural networks for unsupervised learning. A typical model is an autoencoder, which extracts meaningful representations through the compression and reconstruction of the unlabeled data. Several studies have explored the feasibility of this technique for emotion recognition. In [START_REF] Ross | Unsupervised multi-modal representation learning for affective computing with multi-corpus wearable data[END_REF], stacked convolutional autoencoders (SAE) were applied independently on unlabelled ECG and EDA data to obtain generalized latent representations for arousal classification, achieving better performance than the fully supervised approaches. Though this method effectively modeled the heterogeneity of multimodal signals, i.e. using different models to extract valid unimodal features, however, it neglected the collaborative and complementary nature of multimodality. Different from the previous approach, Zhang et al. [START_REF] Zhang | Corrnet: Fine-grained emotion recognition for video watching using wearable physiological sensors[END_REF] presented a correlation-based emotion recognition algorithm (CorrNet), where intra-modal features are first obtained with separate convolutional autoencoders, followed by covariance and cross-covariance computation between each pair of modalities to obtain inter-modal features.

However, these unsupervised learning methods based on autoencoders did not introduce supervised signals in pre-training. As a result, the learned representations may contain patterns that are irrelevant to emotions, resulting in unsatisfactory performance.

2) Semi-supervised

Semi-supervised learning typically uses both a large amount of unlabelled data and a small amount of labelled data, assuming that the presence of unlabelled data contributes to a better learning process [START_REF] Van Engelen | A survey on semi-supervised learning[END_REF]. Depending on the training strategy, semi-supervised learning methods can be classified into one-stage based or multi-stage based [START_REF] Schmarje | A survey on semi-, self-and unsupervised learning for image classification[END_REF]. Most of the semi-supervised emotion recognition methods belong to the latter category, which use unlabelled data for pretraining to provide a good initialisation of the model, followed by fine-tuning on labelled data. For example, Luo et al. [START_REF] Luo | Semi-supervised cross-subject emotion recognition based on stacked denoising autoencoder architecture using a fusion of multi-modal physiological signals[END_REF] presented a semi-supervised learning algorithm for Valence-Arousal-Dominance emotion recognition based on EEG signals and peripheral physiological signals. Stacked denoising autoencoders (SDAE) was first pre-trained and then fine-tuned in a supervised manner to generate robust representations that were ultimately served as input to KNN and SVM for classification. Finally, the proposed method showed slight improvements over other supervised and semi-supervised methods in the performance comparisons. However, as with [START_REF] Ross | Unsupervised multi-modal representation learning for affective computing with multi-corpus wearable data[END_REF], it adopted independent autoencoders for reconstruction of different modalities and thus failed to effectively capture the cross-modal correlation of multiple data. Yu et al. [START_REF] Yu | Semi-supervised learning and data augmentation in wearablebased momentary stress detection in the wild[END_REF] proposed a semi-supervised multimodal stress detection framework. A LSTM autoencoder-based (LSTM-AE) model was first trained on labelled samples and the resulting latent representations were clustered with a GMM model, which was subsequently applied to select unlabelled samples with a similar distribution to the labelled data. Later, the selected unlabelled data was applied to pre-train the LSTM-AE model providing the initial parameters of the encoder in the supervised architecture. In the end, the proposed method enhanced the classification performance by 7.7% to 13.8%, compared to the supervised approaches. Though the methods described above have yielded promising results, allowing us to obtain robust and generalized emotion recognition models with limited labeled data. However, their approach regarded the multimodal data as a whole. The autoencoder-based model was designed to compress and reconstruct the concatenated hand-crafted feature sequences, thus ignoring the correlation information within the modalities.

3) Discussion of non-supervised learning based methods

In the above sections, we explore non-supervised learning approaches for emotion recognition, some of which have shown competitive or even better performance compared to fully supervised algorithms, offering a potential to eliminate overfitting effects associated with limited labeled data. Table 2.3 summarizes the corresponding methods demonstrated in the previous sections.

As can be observed from the table, first, the number of non-supervised studies, whether based on traditional machine learning algorithms or deep neural networks, remains significantly lower than that of fully supervised ones. Second, for deep unsupervised/semi-supervised learning methods, the autoencoder appears to be a popular model option for learning generalized representations. However, these methods cannot simultaneously model the heterogeneity and collaboration of multimodal physiological signals when pre-trained with unlabeled data; in other words, they typically employ a single model or independent models for representation learning, thus ignoring intra-and inter-modal correlations, respectively. In summary, how to learn meaningful representations from large amounts of unlabeled data while effectively integrating multimodal physiological signals are crucial factors for accurate and robust emotion recognition in real-life settings.

Emotion recognition based on facial expressions

The process of recognizing human emotions based on facial cues is referred to as facial expression recognition (FER) and its corresponding methods can be broadly categorized into two The acquired emotion-related facial images are subjected to a pre-processing consisting of face detection and alignment to remove irrelevant variations of facial expression from the data so as to normalize the visual semantic information [START_REF] Li | Deep facial expression recognition: A survey[END_REF]. A brief introduction to pre-processing is provided in Section 2.

Visual data preprocessing

In this section, we briefly introduce the two basic steps before performing facial expression recognition. Given an image containing the subject, the first step of data preprocessing is to determine the location of the region of interest (ROI), i.e., the face. One of the most widely employed face detection algorithms is the Viola and Jones (V&J) object detection framework, which can achieve accurate results on data collected in a controlled environment [START_REF] Martinez | Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition[END_REF]. When migrating to real-world applications, this task tends to be challenging, as the human head posture may be non-frontal and obscured by hand motions or glasses, etc. Some robust face detection algorithms [START_REF] Yan | Face detection by structural models[END_REF]105] based on the Deformable Parts Model framework [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF] were developed to cope with this situation. Once face detection is accomplished, facial landmark detection algorithms such as the Active Appearance Model (AAM) and the Supervised Descent Method (SDM) are then employed to locate points of critical facial areas. The obtained facial points are further applied to compute transformations (e.g., Procrustes transformation) with predefined facial shapes to align face images, thus eliminating variations including translations, uniform scaling, and rotations [START_REF] Li | Deep facial expression recognition: A survey[END_REF]. At the end of these two operations, the normalized shapes can now be exploited to extract facial representations for recognizing emotions.

Point-based feature support

Approaches using point-based representations generally select specific facial landmarks from local regions (i.e., primary components of the face) which are further exploited to derive effective features. Depending on whether dependencies between facial landmarks were considered, such methods can be further classified into independent feature-based and non-independent feature-based. For example, Asmara et al. [START_REF] Asmara | Study of drmf and asm facial landmark point for micro expression recognition using klt tracking point feature[END_REF] tracked 20 facial points located in the eyebrow, eye and mouth regions in the facial expression video, and calculated the shifted Euclidean distance of the coordinate points between consecutive frames. In this case, the motion features of each facial landmark were encoded independently for facial expression classification.

Considering the relationship between facial points when expressing emotions, Munasinghe [START_REF] Munasinghe | Facial expression recognition using facial landmarks and random forest classifier[END_REF] calculated the distances between the points within the eyebrow and mouth regions as crucial information for identifying emotions. Taking into account the variation in facial sizes, the obtained distance vectors were subsequently normalized by a specific distance in each region.

In the end, a random forest (RF) classifier achieved an average accuracy of 90% in predicting four emotions (i.e., anger, happiness, sad, and surprise). Huang et al. [START_REF] Huang | Hybnet: A hybrid network structure for pain intensity estimation[END_REF] proposed a 1D convolution-based neural network to explore the geometric information in facial landmarks.

Vectors of normalized distances between facial points and a reference point (i.e., the point on the tip of the nose) were first computed and subsequently fed into 1D convolutional layers and pooling layers to extract discriminative features for pain intensity estimation. Ryumina and Karpov [START_REF] Ryumina | Facial expression recognition using distance importance scores between facial landmarks[END_REF] proposed a deep geometric facial expression recognition framework based on Long Short-Term Memory (LSTM). They first selected random samples of facial expressions and calculated the Euclidean distances between facial landmarks. Subsequently, three emsemble classifiers: Random Forest Classifier (RF), Extra Trees Classifier (ET) and AdaBoost Classifier (AB) were adopted to calculate the average importance score of the features for filtering the most relevant distances. An LSTM-based deep neural network was applied to capture the temporal relationships in the selected facial distance sequences and achieved superior expression recognition performance than the appearance-based approaches.

In contrast to the previous studies based on facial landmark pairs, Ghimire et al. [START_REF] Ghimire | Recognition of facial expressions based on tracking and selection of discriminative geometric features[END_REF] pro-posed a triangle-based representation for a better description of the relationship between points.

Three landmarks were randomly chosen as the triangle's vertices, and the derived representation contained four components, i.e., two side lengths and two angle differences between the current frame and the first frame. Ultimately, the most discriminative triangle features were selected by a multi-class AdaBoost with Extreme Learning Machine (ELM) classifier and applied to the SVM-based classification. Palestra et al. [START_REF] Palestra | Improved performance in facial expression recognition using 32 geometric features[END_REF] employed a hybrid representation containing 32 geometric features for facial expression recognition. The distance between two points, the slope of the line connecting the two points, the area of the irregular polygon, and the ratio of the major and minor axes of the defined ellipse were extracted from the regions of eyebrows, eyes, cheeks, nose, and mouth, respectively, as inputs to a series of classifiers. As a result, the best performance of up to 95.46% was obtained by the RF classifier. Though the above local representation-based approaches exhibited promising performance in facial expression recognition, the geometric features extracted from selected facial landmarks cannot fully describe the complex associations between facial components.

Local region-based feature support

This type of approach typically divides facial landmarks into several groups based on the main components of the face, from which corresponding local features are extracted and combined to form the final representation for the learning task. Liu et al. [START_REF] Liu | Facial expression recognition using hybrid features of pixel and geometry[END_REF] applied separate LSTM models to process landmarks detected in facial sub-regions such as eyes, eyebrows, and mouth, where landmark coordinate series were fed into the LSTM cells to explore their relative position dependencies. In the end, the geometric features extracted from seven local regions were concatenated and weighted by a convolution-based attention model for expression recognition. Apart from the above concatenation-based methods for integrating local features, some methods employ graph models to capture the relationship between facial local areas. Wang et al. [START_REF] Wang | Capturing complex spatio-temporal relations among facial muscles for facial expression recognition[END_REF] proposed a facial expression recognition framework based on the Interval Temporal Bayesian Network (ITBN), where they defined a facial expression as a set of primitive events that occur continuously or in parallel, with primitive events being the movements of facial points to approximate local muscle activities. The ITBN model established the spatiotemporal dependencies between local facial components by constructing a directed graph with facial primitive events as nodes, resulting in superior recognition performance in comparison with classical appearance-and geometry-based approaches. Although the approaches described above take into account the connections between facial regions, however, the selection of these specific regions is subjective and diverse. There may be an optimal division of regions that may require the intervention of expert knowledge. Furthermore, the division strategy may also vary with specific tasks. Moreover, these approaches may lack robustness to variations in the head pose as it does not regard the context of the whole face.

Full face-based feature support

Unlike the previous approaches based on features extracted from special points or facial local regions, this type of approach treats the entire facial area as a whole, where the shape representations are extracted from the coordinates of all facial landmarks.

1) Shape representation in Euclidean space

Some approaches tend to construct the global shape representation in Euclidean space, which is typically represented by a multidimensional vector. For instance, Jain et al. [START_REF] Jain | Facial expression recognition with temporal modeling of shapes[END_REF] treated all facial landmarks as a whole, i.e., a high-dimensional vector composed of 2D coordinates.

Latent Dynamic Conditional Random Fields (LDCRFs), an undirected graph model was then selected as a classifier, permitting the parallel encoding of motion patterns within a single facial expression and between different expressions. Experimental results showed that LDCRFs obtained more robust performance on landmark-based shape features than on appearance features. Lorincz et al. [START_REF] Lorincz | Emotional expression classification using time-series kernels[END_REF] presented a kernel-based facial expression recognition method. The 3D landmarks captured from facial expression videos were regarded as multidimensional time series, and two kernels: the Dynamic Time Warping (DTW) kernel and the Global Alignment (GA) kernel were applied to capture the similarity information between the sequences. Additionally, they also adopted the alternating projection method on the former to ensure the positive semi-definite nature of the kernel. Ultimately, Gram matrices induced by these two kernels were applied for SVM-based emotion classification. In general, the above approaches encoded the facial shape as a Euclidean vector consisting of 2D or 3D coordinates of the landmarks for ex-pression recognition. However, this commonly results in a high-dimensional representation. In the work of [START_REF] Lorincz | Emotional expression classification using time-series kernels[END_REF], they employed the PCA technique for the compression of the shape descriptor, which in turn led to information loss. To solve this problem, some approaches adopted deep neural networks for effective representation learning from coordinate vectors. Qiu et Wan [START_REF] Qiu | Facial expression recognition based on landmarks[END_REF] first selected specific reference points to normalize the 68 detected facial points, and then the coordinates of the 2D landmark configurations were fed into a fully connected neural network in the form of 1D vectors of size 136 to classify facial expressions. Jung et al. [START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF]

2) Shape representation on manifold

Different from previous methods that use flattened coordinate vectors in Euclidean space as the face descriptor, this class of methods builds compact, discriminative facial shape representations on non-linear manifolds that are invariant to affine transformations [START_REF] Cherian | Positive Definite Matrices:Symmetric positive definite (SPD) matrices Data RepresentationData representationand Applications to Computer Vision[END_REF]. One successful case is the covariance descriptor [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF][START_REF]Pedestrian detection via classification on riemannian manifolds[END_REF][START_REF] Cavazza | Kernelized covariance for action recognition[END_REF] in the field of computer vision, which efficiently fuses high-dimensional features by computing the corresponding covariance matrix, and this is what we refer to as compactness. The resulting covariance representations are the symmetric positive definite (SPD) matrices that will induce the manifold structure by defining the appropriate metric [START_REF] Zhao | Landmark manifold: Revisiting the riemannian manifold approach for facial emotion recognition[END_REF]. More specifically, the set of SPD matrices forms a cone-shaped Riemannian manifold, on which each covariance-based representation can be considered as a point [3]. Fig. 2.9 shows an illustration of the corresponding Riemannian manifold. Consequently, one can obtain more precise similarity measures between Riemannian representations rather than Euclidean ones, which is related to better discriminative power. In addition, representations based on Riemannian geometry are invariant under affine deformation, thus offering a possibility for robust facial shape analysis. Hence, some studies have turned their attention to constructing shape representations in nonlinear spaces based on the foregoing merits. These methods typically embed the motion of the facial landmarks in the time-parameterized curves according to the Riemannian geometry of manifold data. For example, Taheri et al. [START_REF] Taheri | Towards view-invariant expression analysis using analytic shape manifolds[END_REF] proposed a robust facial expression recognition approach based on shape manifold analysis to eliminate the disturbance due to viewpoint variations. The global facial deformation evoked by the emotional expression can be registered by a sequence of facial landmarks, which were further treated as a curve on the Grassmann manifold. The 2D landmark configurations in successive frames were considered as adjacent points on the curve, the connections between which were characterized by the velocity vectors in the tangent space. Consequently, the shape representation derived on the manifold can better discriminate between different facial expressions than that in Euclidean space. Tanfous et al. [START_REF] Tanfous | Sparse coding of shape trajectories for facial expression and action recognition[END_REF] conducted a view-invariant facial expression analysis using geometry of non-linear manifolds. Continuous 2D landmark configurations detected in facial expression videos were first modeled as trajectories in Kendall's shape space, which were subsequently encoded as sparse time series in Euclidean vector space by Riemannian sparse coding and dictionary learning (SCDL) to suit traditional linear classification algorithms. In the end, the proposed method achieved competitive results in macro-and micro-facial expression recognition tasks. Kacem et al. [START_REF] Kacem | A Novel Geometric Framework on Gram Matrix Trajectories for Human Behavior Understanding[END_REF] presented a novel facial expression recognition framework based on the geometry of the space of Gram matrices. Gram matrices computed from a sequence of facial landmarks were considered as points on the Riemannian manifold of positive semidefinite matrices of fixed-rank, where the matrices of adjacent frames were joined by pseudo-geodesics to model the facial movements. The Dynamic Time Warping (DTW) technique was adopted to measure the similarity between trajectories, which was finally fed into a pairwise proximity function SVM (ppfSVM)-based classification. Szczapa et al. [START_REF] Szczapa | Automatic estimation of self-reported pain by interpretable representations of motion dynamics[END_REF] proposed a Riemannian framework for pain intensity estimation using facial landmarks. The Gram matrices were first computed from the combination of the face point coordinates and the velocities between consecutive frames. The matrix sequences were then be regarded as curves on the Riemannian manifold, where a curve-fitting algorithm was applied to cope with missing or incorrect data. The input data for Support Vector Regression (SVR)-based pain intensity prediction was finally provided by Global Alignment Kernel (GAK) through the calculation of the similarity scores between trajectories. Overall, the methods discussed above incorporate the intrinsic geometry of the shape space, therefore yielding facial representations that are robust to affine transformations. They generally employed a fixed metric to measure the similarity between shape representations. However, the selected metric may not always be optimal for a specific task or dataset. In such cases, the underlying relationships between facial shapes may not be captured effectively. [START_REF] Taheri | Towards view-invariant expression analysis using analytic shape manifolds[END_REF] Non-linear Manifold Hand-crafted [START_REF] Tanfous | Sparse coding of shape trajectories for facial expression and action recognition[END_REF] Non-linear Manifold Hand-crafted [START_REF] Kacem | A novel geometric framework on gram matrix trajectories for human behavior understanding[END_REF] Non-linear Manifold Hand-crafted [START_REF] Szczapa | Automatic estimation of self-reported pain by interpretable representations of motion dynamics[END_REF]

Discussion of landmark-based FER methods

We classified facial landmark-based emotion recognition methods into three types based on the range of facial feature support and introduced them accordingly in the previous sections. Table 2.4 provides a brief summary of these methods. First, we notice that most of the methods still construct their facial feature representations in Euclidean space, which are commonly disturbed by affine transformations such as translation, scaling, and rotation. Some full face-based methods solved this issue by defining the facial shapes on non-linear manifolds, thus achieving more robust representations to face deformations. However, these methods use predefined metrics to describe the similarity relationships between different facial shapes. Thus they are not flexible enough to adapt to the variable data structures in different tasks or datasets. In addition, all existing methods do not explicitly address the significant intra-class variability presented by facial expression images in the wild. Hence, there exists the possibility of further enhancing the discriminative power of feature space to obtain better recognition performance. The principles of the first three fusions are consistent with those mentioned in the previous section 2.3.3, while the last one combines the early fusion and late fusion. The details of these multimodal approaches will be discussed below.

Emotion recognition based on multimodal signals

Early Fusion

Similar to the early fusion mentioned in Section 2.3.3, data from the behavioral and physiological domains are concatenated and treated as a single input for emotion recognition. Li et al. [START_REF] Li | Facial expression recognition based on electroencephalogram and facial landmark localization[END_REF] proposed a multimodal fusion framework for emotion recognition based on EEG signals and facial expressions. Energies of different channels are extracted by DWT as EEG features, while a series of geometric features such as distances between facial landmarks, angles of the corner of the lip and slopes of the eyebrows are selected as facial features. Ultimately, the SVM classifier with fused feature vectors as input obtained an improved performance over that using solely EEG data. Werner et al. [START_REF] Werner | Automatic pain recognition from video and biomedical signals[END_REF] collected facial distances and gradient-based features from video frames which were combined with the statistical features calculated from biological signals. The resulting multimodal vectors were employed to train a random forest model for pain assessment. Experimental results indicated that the combination of facial and physiological data improved unimodal performance in most cases, especially when predicting high pain intensity. Gil-Martin et al. [START_REF] Gil-Martin | Human stress detection with wearable sensors using convolutional neural networks[END_REF] introduced cube root (CR) and constant Q transform (CQT) in the computation of the spectrum for wearable physiological and motion signals and fed the corresponding spectrum variants as single-channel images into a 2D convolutional and fully connected based neural network for multimodal emotion recognition. Experimental results showed that the proposed model achieved significant performance improvements on multimodal data compared to the unimodal settings. However, early fusion approaches sometimes fail to deliver performance gains. Schmidt et al. [START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF] collected physiological and motion data using a wrist device and a chest device and extracted various time-and frequency-domain multimodal features for training a range of common machine learning classifiers. However, the results of experiments conducted on both devices demonstrated that the simple concatenation of features from different modalities did not lead to improved performance.

Intermediate Fusion

Marginal features computed from physiological and behavioral modalities are either applied directly to prediction or used to jointly learn advanced joint representations. For instance, Wu et al. [START_REF] Wu | Multimodal fused emotion recognition about expressioneeg interaction and collaboration using deep learning[END_REF] The obtained optimal architecture performed better on multimodal data compared to the setup using only physiological data. Lai et al. [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF] proposed a residual temporal convolution-based deep neural network (Res-TCN) to capture the effective features of multimodal data. Two fusion strategies were implemented, the first of which fed sensor data from all modalities as a whole into the Res-TCN model for predicting emotions, while the second combines the marginal representations derived from the independent convolutional models for the final classification. In comparison experiments, the latter exhibited superior performance.

Late Fusion

This type of approach generally integrates decisions derived from physiological and behavioral models. Saffaryazdi et al. [START_REF] Saffaryazdi | Using facial micro-expressions in combination with eeg and physiological signals for emotion recognition[END_REF]

Hybrid Fusion

Hybrid fusion can be regarded as a mixture of early fusion and late fusion. Two hybrid fusion strategies have been found in the literature for multimodal emotion recognition, and their corresponding illustrations are shown in Fig. 2.10. Regarding the first strategy, Zhong et al. [4] proposed a multimodal emotion recognition framework based on hybrid fusion of facial expressions and peripheral physiological signals. The facial features extracted from different views were first fused with a series of physiological signal features within each data segment. Next, separate weak learners were trained on the fused features and assigned a decision for each segment. The final decision was a combination of all the weak learners' decisions. The proposed fusion strategy was compared with the common early and late fusion strategies and obtained better performance in predicting emotional valence and arousal. Concerning the second strategy, Gjoreski et al. [5] developed a multimodal stress detection method suitable for the real-life setting. First, a stress detector was trained on biological data collected from the laboratory environment. Meanwhile, an action recognizer learned the subject's behavioral information from the ACC (3-axis acceleration) signal. Then, physiological features, the output of the stress detector and action recognizer were fused and fed into a series of machine learning algorithms for continuous pressure prediction. Finally, the experimental results indicated that the incorporation of behavioral information could enhance recognition performance in unconstrained scenes.

Discussion of multimodal emotion recognition methods

We introduced in the previous sections the emotion recognition approaches integrating physiological data and behavioral data (i.e., facial expressions and body motion from ACC signal), which are briefly listed in Table 2 This chapter shows technical contributions on physiological aspects, where a transformerbased self-supervised learning framework for wearable emotion recognition is proposed, which can solve the overfitting problem due to limited supervised data while performing effective multimodal fusion. In the following, first, we introduce wearable emotion recognition and its challenges in Section 3.1. Next, Section 3.2 shows self-supervised learning-based approaches in the field of emotion recognition and related research on multimodal signal fusion. In Section 3.3 we describe the proposed approach. Sections 3.4 and 3.5 present the datasets used and the corresponding experimental results on emotion recognition, respectively. Finally, Section 3.6 is a summary of our work.

Introduction

Emotion recognition is an emerging field of research aimed at developing algorithms and technologies to identify human emotions from various data sources. Most current methods of emotion recognition rely on physical or physiological indicators of the human body. In contrast to physical signals such as facial expressions [START_REF] Jain | Extended deep neural network for facial emotion recognition[END_REF] and speech [START_REF] Huang | Speech emotion recognition using deep neural network considering verbal and nonverbal speech sounds[END_REF], physiological responses under certain emotional states are involuntary and therefore provide more objective decisions for identification systems [START_REF] Wang | A systematic review on affective computing: emotion models, databases, and recent advances[END_REF]. The physiological modalities primarily consist of electroencephalogra- However, given that most of them are trained in a supervised manner, it is challenging to obtain generalizable models using limited labeled data, especially in daily life settings, where standard protocols for obtaining accurate emotion labels are not yet well defined. Besides, each specific supervised task requires training the deep model from scratch and its knowledge transfer ability on other tasks is not satisfactory [START_REF] Sarkar | Self-supervised ecg representation learning for emotion recognition[END_REF]. Self-supervised learning (SSL), as an emerging learning paradigm, eliminates the need for extensive manual labeling and has demonstrated comparable or even superior performance to supervised learning methods in areas of computer vision (CV), natural language processing (NLP). Several SSL-based efforts [START_REF] Banville | Uncovering the structure of clinical EEG signals with self-supervised learning[END_REF][START_REF] Jiang | Self-supervised contrastive learning for eeg-based sleep staging[END_REF][START_REF] Cheng | Subject-aware contrastive learning for biosignals[END_REF] have been done for emotion recognition using EEG signal, but they are not suitable for practical scenes. Only one work [START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF] as a feature extractor to generate more generalized multimodal representations for emotion classification tasks. The overview of the proposed approach is illustrated in Fig. 3.1. To validate the effectiveness of our method and the knowledge transferability across different datasets, we pre-trained the proposed model on a large-scale unsupervised emotion dataset PRESAGE collected in unrestricted real-life scenarios and evaluated its performance on three public emotion recognition datasets. Overall, our contributions can be summarized as follows:

• We proposed a novel self-supervised learning (SSL) framework to learn generalized representations from a large number of unlabeled samples to cope with the overfitting problem on small-scale physiological data.

• We adopt an intermediate fusion strategy based on temporal convolution and transformer, capable of modeling both the heterogeneity and cross-modal correlation of physiological signals to effectively fuse multimodal data.

• We outperformed state-of-the-art supervised or self-supervised learning-based approaches in various emotion-related classification tasks involving mental stress, affective states, arousal, and valence. Moreover, our model was proven to be more accurate and stable on limited labeled data than fully-supervised models. In addition, multiple ablation studies have been performed to investigate the effectiveness of our method. The proposed SSL model is first pre-trained with signal transform recognition as the pretext task to learn generalized multimodal representation. The encoder part of the resulting pretrained model is then served as a feature extractor for downstream tasks which is frozen or fine-tuned on the labeled samples to predict emotion classes.

Related Work

More recently, researchers have shifted their gaze from the invasive EEG signal to peripheral physiological signals which can be captured by lightweight and more invisible devices.

Among numerous peripheral signals, electrocardiogram (ECG), blood volume pressure (BVP), electrodermal activity (EDA), etc, captured from the cardiac system and electrical skin activity are crucial emotion-related modalities and have proven their validity in several studies [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF][START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF][START_REF] Huynh | StressNAS: Affect State and Stress Detection Using Neural Architecture Search[END_REF][START_REF] Zhang | Corrnet: Fine-grained emotion recognition for video watching using wearable physiological sensors[END_REF][START_REF] Jerritta | Emotion recognition from electrocardiogram signals using hilbert huang transform[END_REF]. However, exploration of the wearable emotion recognition is not yet well established. In Section 2.3, we reviewed methods based on fully-supervised, unsupervised and semi-supervised learning. Among them, fully supervised methods based on deep neural networks struggle with the risk of overfitting and inaccurate emotion labeling, while unsupervised learning methods have unsatisfactory performance due to their weak supervision and there is a lack of exploration for semi-supervised learning algorithms. In addition, the fusion strategies of most existing approaches are ineffective in capturing both intra-and inter-modal correlations of multimodality. In the following sections we will first detail a novel solution to the problem of overfitting in the low-data regime, namely self-supervised learning, followed by multimodal data fusion methods.

Self-supervised learning (SSL) for limited labelled data

To solve the overfitting problem introduced by the limited available data, one common solution is data augmentation, i.e., applying different transformations on the original samples to obtain more abundant data. However, performing data augmentation on a dataset consisting of a limited number of subjects could not introduce inter-subject variability during training and ultimately yield a generalized model [START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF]. Recently, a compelling branch in the field of unsupervised representation learning is self-supervised learning (SSL), which can effectively address the de-generalization issue posed by insufficient labeled data. Unlike unsupervised learning which does not involve any labelled data, SSL is designed with a series of pretext tasks that introduce self-supervision to unlabelled data, enabling more effective representation learning for downstream tasks. Each unsupervised sample is automatically labeled through inherent dependencies and associations between the data without human intervention [START_REF] Liu | Self-supervised learning: Generative or contrastive[END_REF]. The SSL model pre-trained on pseudo-labeled data is considered as powerful feature extractor for a variety of downstream tasks. In the domains of computer vision and natural language processing, SSLbased work such as SimCLR [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], Word2Vec [START_REF] Schneider | wav2vec: Unsupervised pretraining for speech recognition[END_REF], and BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] have exhibited competitive and even superior performance on a range of tasks. However, few studies have investigated the performance of SSL models on peripheral physiological signal data. Sarkar et Etemad [START_REF] Sarkar | Self-supervised ecg representation learning for emotion recognition[END_REF] introduced a self-supervised representation learning framework for ECG-based emotion recognition, where the 1DCNN-based multi-task deep neural network is pre-trained with the objective of identifying the signal transformation types applied to unlabeled data. Their study indicated that the pretext task based on transformation recognition can enable the model to better cope with potential variation factors in the data. However, not all time steps of a signal sequence are associated with the downstream target event (i.e., a specific emotion). Thus, how to filter out irrelevant information during SSL for downstream tasks is an unsolved problem. Exploiting the synchrony of multimodal emotional responses is a potential solution. More specifically, multimodal physiological signals exhibit correlated or consistent temporal changes when emotions are elicited. In this way, modeling the correlation of multimodal signals in SSL can facilitate the capture of emotion-related components in unlabeled data. Regarding multimodal emotion recognition, Dissanayake et al. [START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF] proposed a self-supervised contrastive learning approach, which aims to approximate the positive pairs while pushing the negative pairs away from each other. However, their SSL model is obtained by pretraining each modality independently, and thus again ignores the cross-modal correlations. Therefore, more effective multimodal fusion strategies need to be developed for SSL-based wearable emotion recognition.

Multimodal data fusion for emotion recognition

Multimodal data fusion strategies can be generally categorized into: early fusion, intermediate fusion, and late fusion. Most existing approaches for multimodal emotion recognition are based on early fusion, where multimodal data are combined as a whole before performing a learning task. Joint representations can be extracted directly from concatenated vectors with deep models such as 1DCNN [START_REF] Santamaria-Granados | Using deep convolutional neural network for emotion detection on a physiological signals dataset (amigos)[END_REF] and Bi-LSTM [START_REF] Zitouni | Arousalvalence classification from peripheral physiological signals using long short-term memory networks[END_REF], which allow for encoding inter-modal correlations. However, since unimodal features are not learned explicitly (i.e., the heterogeneity of the multimodal signal is ignored), this fusion strategy is not effective in capturing intra-modal correlations. Late fusion-based approaches [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF][START_REF] Ayata | Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems[END_REF] integrate the decisions of multiple independent learning models to predict emotion categories. Thus, in contrast to early fusion, this fusion approach ignores the connections and interactions between modalities.

Different from the previous fusion approaches, intermediate fusion enables both intra-and inter-modal correlation, where independent feature extractors are first applied to different modalities and the obtained unimodal features are then aggregated in an additional fusion module to further learn the joint representation. A variety of options exist for this fusion module. For example, Shu and Wang [START_REF] Shu | Emotion recognition through integrating eeg and peripheral signals[END_REF] adopt ed the restricted Boltzmann machine (RBM) model to learn the joint probability distribution of multimodal low-level features to encode cross-modal information exchanges. Zhang et al. [START_REF] Zhang | Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine[END_REF] modeled the associations between multimodal features by introducing a regularization term to the objective function. More recently, the transformers have also gained popularity in intermediate fusion-based approaches [START_REF] Wu | Leveraging multi-modal interactions among the intermediate representations of deep transformers for emotion recognition[END_REF][START_REF] Tsai | Multimodal Transformer for Unaligned Multimodal Language Sequences[END_REF][START_REF] Ding | Sparse fusion for multimodal transformers[END_REF] for video, audio and text. Regarding studies on emotion recognition, Wu et al. [START_REF] Wu | Leveraging multi-modal interactions among the intermediate representations of deep transformers for emotion recognition[END_REF] proposed a multimodal Recursive Intermediate Layer Aggregation (RILA) model, which was applied between layers of unimodal deep transformers to capture interactions across modalities through the integration of multimodal intermediate representations. In this work, the transformers were employed to provide valid intermediate features. At the same time, they have also proved to be effective in merging multimodal data [START_REF] Tsai | Multimodal Transformer for Unaligned Multimodal Language Sequences[END_REF][START_REF] Ding | Sparse fusion for multimodal transformers[END_REF]. The attention mechanism can capture advanced patterns shared across modalities, thus exhibiting advantages over naive fusion strategies such as concatenation. In terms of practicality, multimodal emotion recognition based on the video, audio and text may not be well suited to real-life scenarios, as it requires considerable com-putational resources for long-term video stream analysis. In contrast, wearable physiological signals can consistently predict emotions in a low-cost and objective way. However, the validity of transformer-based models has not been well established for wearable emotion recognition.

Meanwhile, video, audio and text-based approaches cannot be directly migrated to physiological data due to differences in data structures. In addition, they are susceptible to overfitting problems as they generally have a relatively deep architecture and follow a fully-supervised setup.

Proposed Method

Overview

Our goal is to employ unlabeled data for capturing generic representations of multimodal physiological signals in order to address the de-generalization problem introduced by a limited number of labeled samples. Hence, we propose a self-supervised learning (SSL) scheme using signal transformation recognition as a pretext objective. An illustration of the proposed approach is shown in Fig.

3.3.2.A Pretext Task: signal transformation recognition

Signal transformation recognition was adopted as the pretext task in SSL, which proved to be effective in learning generalized representations for downstream tasks such as action recognition [START_REF] Saeed | Multi-task self-supervised learning for human activity detection[END_REF] and emotion recognition [START_REF] Sarkar | Self-supervised ecg representation learning for emotion recognition[END_REF]. The random transformations used in the previous SSL methods are one of the common data augmentation techniques for time series, which can generally be classified into two categories: magnitude domain transformations and time domain transformations. The former interferes with the signal values while preserving the time step order, whereas the latter mainly affects the time scale. Previous evaluations of SSL models based on individual transformation recognition [START_REF] Sarkar | Self-supervised ecg representation learning for emotion recognition[END_REF][START_REF] Saeed | Multi-task self-supervised learning for human activity detection[END_REF] have indicated that Noise addition and Scaling ranked highly for magnitude domain transformations, while Permutation and Time-warping performed outstandingly well among time domain transformations. Meanwhile, according to the review of time series augmentation strategies [START_REF] Um | Data augmentation of wearable sensor data for parkinson's disease monitoring using convolutional neural networks[END_REF][START_REF] Iwana | An empirical survey of data augmentation for time series classification with neural networks[END_REF], though most of the suggested transformations have been adopted in previous SSL-based work, two transformations have not been thoroughly evaluated: Magnitude-warping and Cropping. Ultimately, we selected the five transformations: Permutation, Time-warping, Noise addition, Magnitude-warping and Cropping for the pretext task. The reason why Scaling was omitted is that Magnitude-warping can be seen as a special variant of Scaling1 .

The above signal transformations are performed on all three modalities and the resulting transformed signal data is fed into the proposed SSL model as input along with the original multimodal signal data. Fig. 3.2 shows the effect of these deformations on a sample of the EDA signal. Details of each transformation are described in subsequent paragraphs. Here, for simplicity, we write the above-mentioned 1D signal x m uniformly as x(t), where t represents the time step. Magnitude domain transformations:

• Gaussian noise addition: The original signal x(t) is disturbed by white Gaussian noise z(t), which can be extracted from a zero-mean normal distribution N (0, σ 2 ). By assigning a preferred signal-to-noise ratio (SNR), the variance σ 2 (i.e., the average power of the noise) of the distribution N can be derived from the following formula 10 (P sig -SN R)/10 , where P sig is the average power of the signal. In the end, the noised signal is calculated as x(t) + z(t).

• Magnitude-warping: The magnitudes of the original signal are altered by a random smooth curve formed by cubic spline interpolation function ϕ(•). In the end, the transformed signal can be calculated as x(t) • ϕ(x(t)).

Time domain transformations:

• Permutation: The original signal is split into n non-overlapping segments x(t) = {x 1 , x 2 , ..., x n }, which are then temporally disrupted and eventually recombined together to form the permuted signal x(t) = {x p1 , x p2 , ..., x pn }, where {p1, p2, ..., pn} is a shuffled version of the original order.

• Time-warping: The original signal is divided into n non-overlapping segments x(t) = {x 1 , x 2 , ..., x n }, half of which are randomly selected to be stretched by a linear inter-

polation function F (x i , k),
where k is the stretch factor, and the remaining half of the segments are squeezed by the function F (x i , 1/k), where 1/k is the squeeze factor. The time-warped signal can be concatenated from the transformed segments and finally resized to the original length.

• Cropping: The original signal is divided into n non-overlapping segments x(t) = {x 1 , x 2 , ..., x n }, one of which is randomly selected and resampled to the original length.

By identifying the signal transform types, our model is expected to learn a more robust and generalized representation against disturbances in the magnitude or time domains. For example, Magnitude-warping and Gaussian noise addition can simulate different types of real-world noise, such as measurement errors, signal artefacts caused by the subject's body movements, etc. For time domain transformations, Permutation perturbs the order of time steps to prompt the model for capturing temporal dependencies between data points, Time-warping simulates duration variations in emotional responses by stretching or squeezing time steps, and Cropping allows the model to be more robust to changes in the temporal location of emotional events.

F (t) = k-1 i=0 f (i) • x ′ m (t -d • i) (3.1)
where d is the dilation factor. Following each convolutional layer is a weight normalization layer for the convolution filter, a rectified linear unit (ReLU) layer, and a dropout layer for regularization. In the end, a residual connection is created between the input and output of the block, where a 1 × 1 convolution is introduced to eliminate the mismatch in channel numbers between the input and output. Fig. 3.3 illustrates the detailed structure of the TCN-based backbone.

The dilated causal convolution layers in two residual blocks are equipped with 16 filters with a kernel size of 6, where the dilation factors are 1 and 2, respectively. Zero-padding of 5 and 10 are also introduced to ensure that the input and output sequences are of the same length. Subsequently, a modality-specific projection head (i.e., a linear fully connected layer with 128 units) and a layer normalization are then applied to map the low-level features to a higher dimensional

z m = LayerN orm(M LP (T CN (x ′ m ))) ∈ R N ×d (3.2)
where d is the embedding dimension. Shared encoder: As mentioned in Section 3.2.1, encoding of the coordination and interaction between multimodal signals is essential in order to learn generic representations related to the downstream emotion recognition tasks. This can be done through the transformer in which each modality identifies components of other modalities that are highly correlated with itself through the attention mechanism for better signal transformation classification. To achieve this, the low-level features z m of each modality are first stacked to form a multimodal embedding z multi = [z 1 , . . . , z m , . . . , z M ] ∈ R M N ×d . The scaled dot-product attention proposed in [START_REF] Vaswani | Attention is all you need[END_REF] is then applied to calculate the dependencies between different modalities:

Attn(Q, K, V ) = sof tmax( QK T √ d )V (3.3)
where Q, K, V represent queries, keys and values, respectively. More intuitively, the attention layer acts as a weighted sum of values V , where the attention weight associated with each value is generated by the compatibility of the query with its corresponding key. For our shared encoder E s , queries, keys and values are derived through a linear mapping of multimodal features z multi , and the resulting output of the attention layer is:

z a multi = Attn(z multi W Q , z multi W K , z multi W V ) (3.4)
where W Q , W K , W V ∈ R d×d are the projection matrices. erating attention weights from multimodal embeddings, where cross-modal communications are activated. For our shared encoder, the one-layer vanilla transformer block proposed in [START_REF] Vaswani | Attention is all you need[END_REF] with four-head attention is implemented. The feedforward layer dimension is set to 128. ReLU is selected as the activation function for intermediate layers and a rate of 0.2 is used for Dropout operation. In addition, we did not introduce positional coding information for the stacked multimodal inputs. Since the features of each modality are generated by different encoders, the network performance may not benefit from positional encoding in the context of heterogeneous input. This is further explored in the ablation study (Section 3.5. C m is shown in Fig. 3.5. 1D global average pooling is first applied across all time steps of unimodal features, followed by a fully-connected layer with 64 units. 1D Batch Normalization is placed before the ReLU layer for more efficient learning and a Dropout layer with a rate of 0.1 is applied to avoid over-fitting. The final fully-connected layer is equipped with a softmax activation function, where the unit number is determined by the number of signal transformations n (i.e., n = 6 in our work, 5 transformations plus the original version). In the end, the proposed model is optimized on the pseudo-labeled dataset L through the total loss L total which is a combination of cross-entropy losses of individual modalities (i.e., EDA, BVP, TEMP in our work):

L m = - 1 |L| |L| i=1 y i log(C m (h i m )) (3.5)
L total = M i=1 L m = L eda + L bvp + L temp (3.6)

Multimodal emotion recognition based on physiological signals

After pre-training the proposed SSL model with the pretext task on unlabelled data, only the encoder part E is reserved for extracting efficient multimodal representations in a variety of supervised downstream tasks. In this work, we select emotion recognition as our downstream task. A classification head C emo is applied to the output of the encoder E to generate class probabilities for labeled samples L sup = {(x i m , y i ), , y i ∈ {1, . . . , e}, m ∈ {1, . . . , M }}, where e is the number of the emotion classes. The emotion classification head is constructed in the same way as C m , except that it accepts multimodal features from encoder E, thus the unit number in the first fully-connected layer changes to 192 (i.e., 64 × 3), while the unit number in the last fully-connected layer is equal to the emotion class number. The Dropout rate is set to 0.2 to avoid over-fitting. Finally, the proposed model is optimized through the minimization of cross entropy loss L sup . with the reference number 2022-626-S108 and all trainees were given a consent form prior to training and were required to fully read the form and provide a signature. To allow students to perform normal medical simulation activities under interference-free conditions, Empatica E4 Wristband (Fig. 3.6 (f)), an invasive wearable biometric sensor was adopted to continuously record multimodal physiological signal data of high quality with different frequencies:

L sup = - 1 |L sup | |Lsup| i=1 y i log(C emo (E(x i ))) (3.
3-axis Accelerometer (ACC, 32Hz), Blood Volume Pressure (BVP, 64Hz), Electrodermal Activity (EDA, 4Hz), Skin Temperature (TEMP, 4Hz), Heart Rate (HR, 1Hz), Inter-beat Interval (IBI). In this work, we employed data from three modalities: EDA, BVP and TEMP collected in the five scenarios for self-supervised multimodal representation learning.

WESAD Dataset

The WESAD dataset [START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF] (baseline vs stress vs amusement) can be performed on the WESAD dataset for supervised learning, where the non-stress class is a combination of the baseline and amusement classes.

CASE Dataset

The CASE dataset [START_REF] Sharma | A dataset of continuous affect annotations and physiological signals for emotion analysis[END_REF] is a multimodal emotion recognition dataset with continuous annotations. Eight video clips were employed to stimulate four different emotions: amusing, boring, relaxing and scary from 30 subjects. During the experiment, subjects were required to selfassess their own emotional experiences using an annotation interface based on valence-arousal scores, while six physiological signals were recorded at a frequency of 1000 Hz. In our work, we selected blood volume pressure (BVP), electrodermal activity (EDA) and skin temperature (TEMP) signals as in the self-supervised dataset for the classification task. We adopted the same approach as in the literature [START_REF] Zhang | Corrnet: Fine-grained emotion recognition for video watching using wearable physiological sensors[END_REF][START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF] for the mapping from continuous values of valence and arousal to discrete classes, resulting in a binary (low vs high valence/arousal) and a three-class (low vs medium vs high valence/arousal) classification problem for supervised learning.

K-EmoCon Dataset

The K-EmoCon dataset [START_REF] Park | K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations[END_REF]

Experiments and Results

Data Preprocessing

To eliminate artifacts, we first applied a low-pass Butterworth filter with a cutoff frequency of 0.5 Hz for the EDA and TEMP signals, while the same type of filter with a cutoff frequency of 2 Hz is selected for the BVP signal in PRESAGE, WESAD and K-EmoCon dataset. For the CASE dataset, a low-pass filter with a cutoff frequency of 2 Hz was utilized to clean these three signals. Moreover, we performed z-score normalization as in [START_REF] Sarkar | Self-supervised ecg representation learning for emotion recognition[END_REF] for each signal recording to reduce the variation in physiological responses between different subjects. Since the four datasets involved in the experiments were collected using sensors with different sampling frequencies, we then uniformly downsampled all signals in the different datasets to the most frequently occurring frequency, i.e., 4 Hz. Subsequently, based on previous work [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF][START_REF] Huynh | StressNAS: Affect State and Stress Detection Using Neural Architecture Search[END_REF], we segmented the signal recordings of all datasets into windows of length 60 s with 99.5% overlap for PRESAGE and WESAD, 99% and 95% overlap for CASE and K-EmoCon, respectively. If the data in a window corresponds to multiple labels, we adopt the same strategy as in the previous work [START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF], i.e., choosing the one with the majority as the final label. Table 3.1 concludes the learning tasks corresponding to each dataset and the number of samples created after data segmentation. The last column in the table lists the total size of each dataset, where the first dimension represents the total number of samples, while the second and third dimensions represent the signal length at a frequency of 4 Hz in a 60 s window after segmentation (i.e., 240)

and the number of modalities (i.e., three modalities: BVP, EDA and TEMP), respectively.

Implementation and model training

The training process of our SSL-based approach consists of two main phases. The first phase is to pre-train the proposed model on the PRESAGE dataset using automatically generated pseudo-labels for signal transformation identification. A set of transformation parameter vectors (15, 0.2, 10, 9, 1.05, 4) was chosen based on the experimental results of the previous study [START_REF] Sarkar | Self-supervised ecg representation learning for emotion recognition[END_REF] as SNR, magnitude warping variance coefficient, number of permutation segments, number of time warping segments, time-warping stretching coefficient, and number of cropping segments for each modality to generate the five transformations mentioned in Section 3.3.2.A.

The pre-training process of the proposed model took approximately 26 hours on an NVIDIA RTX 6000 GPU. The second phase retains only the encoder part of the pre-trained model to extract valid, generalized representations for emotion recognition on WESAD, CASE and K-EmoCon datasets. We did not introduce these three public datasets into pre-training stage in order to verify the knowledge transfer ability of the learned features across different datasets.

Ultimately, the proposed model was installed using Pytorch. The optimal models for the pretext and downstream tasks were obtained by the SGD (Stochastic Gradient Descent) optimizer with weight decay parameter of 5e-7 to avoid overfitting. For the first phase (self-supervised pre-training), learning rate, batch size and the number of epochs are set to 5e-3, 32 and 20, respectively. For the second phase (supervised emotion recognition), the learning rate, batch size and number of epochs are set to 1e-4, 128, 20 on WESAD dataset, while for CASE and K-EmoCon datasets, these parameters were set to 1e-3, 64 and 64, respectively.

Evaluation metric and protocol

For a fair comparison, we adopted the same experimental protocol as in [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF][START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF][START_REF] Huynh | StressNAS: Affect State and Stress Detection Using Neural Architecture Search[END_REF][START_REF] Zhang | Corrnet: Fine-grained emotion recognition for video watching using wearable physiological sensors[END_REF][START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF], i.e. Leave-One-Subject-Out cross-validation (LOSO CV), which has the benefit of examining the generalization ability of the model to unpresented subject data. Two metrics, accuracy and F1score applied in [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF][START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF][START_REF] Huynh | StressNAS: Affect State and Stress Detection Using Neural Architecture Search[END_REF][START_REF] Zhang | Corrnet: Fine-grained emotion recognition for video watching using wearable physiological sensors[END_REF][START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF] were selected to evaluate the performance of the proposed approach on the emotion recognition task. Accuracy represents the proportion of correctly classified samples to the total number of samples. F1-score is considered as a harmonic mean of the precision and recall, which is suggested for evaluating imbalanced datasets.

Baseline Models

Since the exploration of wearable emotion recognition based on peripheral physiological signals has not been well established, a series of baseline models based on fully-supervised learning, unsupervised learning, and self-supervised learning were implemented in addition to available state-of-the-art methods to provide a more comprehensive and reliable performance comparison. The followings are brief descriptions of these models:

Supervised learning-based methods:

• SimpDCNN [START_REF] Saeed | Multi-task self-supervised learning for human activity detection[END_REF]: it is a simple convolutional network consisting of three convolutional blocks with kernel sizes of 24, 16 and 8, each followed by a ReLU activation and a dropout layer.

• MulT [START_REF] Tsai | Multimodal transformer for unaligned multimodal language sequences[END_REF]: it is a transformer-based multimodal fusion method applied to video, audio and text. The unimodal data is first passed through a temporal convolutional network to obtain low-level features, then transformers based on cross-modal attention and selfattention mechanisms are applied successively for effective fusion.

• ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]: it is a 1D convolution-based residual network adapted to physiological signals proposed in [START_REF] Jiang | Self-supervised contrastive learning for eeg-based sleep staging[END_REF][START_REF] Hannun | Cardiologist-Level Arrhythmia Detection and Classification in Ambulatory Electrocardiograms Using a Deep Neural Network[END_REF], which is constructed similarly to ResNet-18, consists mainly of 8 residual blocks with batch normalization (BN) operation and ReLU activation function, where each block contains two convolutional layers. The three modalities: BVP, EDA, TEMP are fed into this network as multi-channel signals.

• Ours (Supervised): it is our proposed multimodal network, trained in a fully-supervised manner.

In addition, three additional supervised methods were applied for the performance comparison on the CASE and K-EmoCon datasets since they lacked baseline results compared to the WESAD dataset.

• DCNN [START_REF] Santamaria-Granados | Using deep convolutional neural network for emotion detection on a physiological signals dataset (amigos)[END_REF]: it employs a four-layer 1D convolutional neural network to extract modalityspecific features, and a three-layer fully connected network connected at the bottom of the network for classification.

• Attn-BiLSTM [164]: it applies a multilayer bidirectional LSTM for capturing valid temporal information for multimodal signals. The attention mechanism was applied to select the most relevant multimodal representation of the emotional state as input for a fully connected layer-based classifier.

• MMResLSTM [START_REF] Ma | Emotion recognition using multimodal residual lstm network[END_REF]: it uses separate four-layer LSTM-based models for multimodal signals with residual connections. Moreover, the weights of the LSTM layers of both modalities are shared to activate cross-modal communication.

Unsupervised learning-based methods:

• Autoencoder: it is an autoencoder with the same encoder part as our proposed model, while the decoder part consists of three transposed convolutional blocks for the reconstruction of the BVP, EDA, TEMP signals. Each unimodal decoder consists of four-layer transposed convolution with the same parameters as the convolutional layers in the encoder.

Self-supervised learning-based methods:

• SigRep [START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF]: it adopts a similar model architecture to SimCLR [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], containing an encoder of four inception-inspired blocks and a projection head consisting of fully connected layers, where each inception block consists of 1D convolutional layers with different kernel sizes and a maximum pooling layer in parallel. The model is applied independently to each signal modality for contrastive representation learning.

• BENDR [START_REF] Kostas | Bendr: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data[END_REF]: it is a simpler version of wav2vec 2.0 [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] that was applied to EEG signals. We adapted it for application to peripheral physiological signals at low frequencies. The multi-channel signal consisting of BVP, EDA, TEMP is first passed through a four-layer convolution with kernel sizes of 3, 2, 2, 2, where the GeLU is chosen as the activation function along with GroupNorm and Dropout operations, and the obtained low-level features are randomly masked and fed to the same transformer as our proposed model. The final output features are used to reconstruct the masked features.

For a fair comparison, we followed the parameters provided in these works for the model implementation and applied the same experimental setup. For those models initially designed for non-peripheral physiological signals, the parameters have been slightly adjusted to match the low-frequency wearable data for proper operation.

Experimental Results

3.5.5.A Comparison with state-of-the-art methods

Emotion-related classification tasks were performed on WESAD, CASE, K-EmoCon datasets to evaluate the performance of the proposed SSL model. Tables 3.2, 3.3, 3.4 summarize performance comparisons with state-of-the-art fully-supervised, unsupervised, and self-supervised learning-based methods, which were introduced in Section 3.2 and Section 3.5.4 and hence will not be further described here. For the SSL-based approaches, we report the results under two training modes: Frozen (F) and Fine-Tuned (T). The first mode refers to freezing the pre-trained encoder part and updating only the parameters of the classification head in the downstream tasks, which is designed to investigate the effectiveness of the learned self-supervised multimodal features. The second mode employs the pre-trained encoder parameters for model initialization and updates all parameters normally to examine the performance gain relative to the Frozen mode. From the tables, first, it can be observed that our fully-supervised model obtained better performance than other supervised learning approaches in most emotion recognition tasks, confirming the effectiveness of the proposed architecture. Secondly, regarding our SSL model, the comparison results indicated that, under the Frozen mode, our method achieved superior performance over other fully-supervised, unsupervised, and self-supervised based approaches on 6 out of 10 tasks, demonstrating the generalization and high discrimination of the representation learned through the SSL pretext task. In addition, the performance of our model was improved in the Fine-Tuned mode, further narrowing the gap with supervised baselines and thus achieving state-of-the-art results in 8 out of 10 tasks. Additionally, it is interesting to note that as the number of supervised samples decreases from WESAD to CASE to K-EmoCon, the higher the performance gain obtained by our SSL-based approach with respect to the supervised approaches. This can be attributed to the fact that supervised learning methods are more prone to overfitting than self-supervised learning methods on low data regimes. Further research on the performance comparison of these two types of methods on limited data is presented in Section 3.5.5.B. Thirdly, in comparison with non-supervised learning methods, we improved the performance of SigRep and BENDR, especially on the CASE and K-EmoCon datasets. The source of this performance gap may be related to the deployed fusion strategies, in addition to the selected pretext tasks. SigRep [START_REF] Dissanayake | Sigrep: Toward robust wearable emotion recognition with contrastive representation learning[END_REF] learned effective representations for each modality independently through contrastive learning, whereas BENDR [START_REF] Kostas | Bendr: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data[END_REF] regarded multimodal signals as a whole to reconstruct obscured multimodal features. Thus, these two approaches ignored the encoding of inter-and intra-modal correlations, respectively. The impact of different SSL fusion strategies on downstream performance is later investigated in Section 3.5.6.A. Furthermore, the results of the Autoencoder are inferior to other SSL methods. This may be due to the unsupervised nature of its pre-training process which results in more redundant patterns being captured that are irrelevant to the downstream tasks.

3.5.5.B Self-supervised learning vs Supervised learning on limited labeled data

In the previous section, our self-supervised approach presented state-of-the-art performance on emotion recognition tasks with all labeled data in the dataset. To further investigate the effectiveness of our fine-tuned model on a limited number of labeled samples, we performed a comparison with four supervised learning models: our proposed model with the fully-supervised setting, MulT [START_REF] Tsai | Multimodal transformer for unaligned multimodal language sequences[END_REF], ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] and SimpDCNN [START_REF] Saeed | Multi-task self-supervised learning for human activity detection[END_REF]. MulT and ResNet were selected since they share similar structures to our model and are the best-performing supervised models in addition to ours. Besides, SimpDCNN, as a low-complexity model, is not prone to overfitting in [START_REF] Newell | How useful is self-supervised pretraining for visual tasks?[END_REF] that the advantage of the self-supervised learning-based method is its better regularisation on low data regimes to avoid overfitting problems compared to fully-supervised methods.

As the amount of available labeled data increases, the difference in performance between the two types of models gradually decreases. Overall, the comparison results suggest that the proposed method can produce more meaningful and robust representations for wearable emotion recognition than fully-supervised methods, offering a potential solution to the problem of little labeled data.

Ablation Studies

Different types of ablation experiments were designed and conducted on the WESAD, CASE and K-EmoCon datasets to verify the validity of the proposed method. The encoder part of the models involved was trained in freezing mode and the obtained emotion recognition results are reported in the following sections.

3.5.6.A Ablation study of different fusion strategies

To demonstrate the effectiveness of the selected fusion strategy, we conducted ablation studies on different SSL fusion strategies. all share the same parameter settings as the proposed model. For the Early fusion setup, we treated the multimodal physiological signal as a whole, i.e. a multichannel signal, from which multimodal representations will be learned directly. For the Late fusion setup, separate encoders were applied to individual modalities to extract unimodal features for classification. In addition, the third variant model has the same fusion strategy as ours, where unimodal features were first captured and then concatenated to learn more advanced multimodal features. The difference, however, is that this model performs classification by multimodal features. This is to verify the necessity of conducting modality-specific classification in the proposed method, and we refer to this setup as Intermediate fusion with an overall loss. Consequently, the corresponding evaluation results are listed in Table 3.5. Our model consistently achieved the best performance on all datasets, demonstrating the effectiveness of the selected fusion strategy, i.e., intermediate fusion. In addition, the intermediate fusion-based models performed better than those based on the other two fusions. This can be attributed to the fact that the intermediate fusion simultaneously models the heterogeneity and coordination of multimodal physiological signals, whereas the other two fusion approaches only consider one of these two properties. Furthermore, the third setting Intermediate fusion with an overall loss performs slightly worse than our model, affirming the importance of modality-specific classification. The benefit of applying modality-specific loss functions is that it forces the model to learn, for each modality, generic features that are robust to perturbations in the time or magnitude domain, while the application of an overall loss fails to distinguish each modality's contribution to the learned representation.

3.5.6.B Ablation study of different modalities

We conducted an ablation study of three modalities: EDA, BVP, TEMP and their combinations to explore their performance on emotion recognition tasks. The resulting average accuracies and F1-scores of our model are summarized in Table 3.6. First, for the unimodal performance, the EDA signal performs outstandingly well among all the modalities, especially when detecting stress and arousal states. This is consistent with previous findings that EDA is one of the most relevant indicators of stress [START_REF] Healey | Detecting stress during real-world driving tasks using physiological sensors[END_REF] and has even been adopted as ground truth in some studies [START_REF] Wijsman | Wearable physiological sensors reflect mental stress state in office-like situations[END_REF][START_REF] Hernandez | Under pressure: Sensing stress of computer users[END_REF] for the stress analysis of other signals. In addition, it has been proven to correlate linearly with arousal [START_REF] Hernandez | Under pressure: Sensing stress of computer users[END_REF]. In the bimodal-based classification, we first observed that the BVP+EDA setup performed better on the stress-related tasks (i.e. S-2 and E-3 on the WE-SAD dataset) than the other setups. This suggests that the BVP signal and the EDA signal are highly coordinated and correlated when the stress state is elicited, making their combination more effective for detection. This finding is quite reasonable. The BVP signal contains information on heart rate (HR) and heart rate variability (HRV) thus providing a strong correlation with stress states. In [START_REF] Alberdi | Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review[END_REF], HRV and EDA were identified as the most relevant physiological indicators for the real-time stress detection task. Secondly, the EDA+TEMP setup achieved the best performance on the classification task regarding arousal level. This finding is supported by previous research [START_REF] Sato | Physiological correlates of subjective emotional valence and arousal dynamics while viewing films[END_REF] which indicated that EDA and TEMP had a positive and negative correlation with arousal scores respectively. Lastly, our model achieved performance gains on both bimodal and trimodal data in most cases, confirming again its effectiveness for multimodal fusion.

Table 3.6: Ablation study of different modalities and their combinations: average accuracy and F1-score obtained with different modality combinations in the downstream emotion recognition tasks, where the best performing individual modality and bimodal combinations for each task are underlined. (S-2: Stress-2, E-3: Emotion-3, V-2: Valence-2, A-2: Arousal-2.) The horizontal axis of each subplot represents the name of the missing modality, while the vertical axis represents the drops in model performance compared to the case of complete modalities, where the metrics of the vertical axes in the first and second rows are accuracy and F1-score, respectively. (ns: no significant difference; * : p < 0.05, the more asterisks, the more significant the difference.)

Modality WESAD CASE K-EmoCon S-2 E-3 V-2 A-2 V-2 A-2 Acc F1 Acc F1 Acc F1 Acc F1 Acc F1 Acc

3.5.6.C Ablation study of missing modalities

We investigate the robustness of the proposed SSL model when a signal modality is missing in downstream tasks, which is quite common in real-world scenarios. There exist a variety of experimental setups for incomplete modalities. Based on [START_REF] Ma | Smil: Multimodal learning with severely missing modality[END_REF], we selected the most challenging one, i.e., a modality is missing in both training and testing of the downstream task, where 50% of the multimodal samples were first randomly selected and subsequently the data values of a specific modality were set to 0 to simulate its absence. The robustness of the proposed SSL model was measured by calculating its difference in performance in two cases: one with all modalities present and one with missing modalities. The smaller the difference, the more robust the model is considered to be. The above experimental procedure was repeated 10 times. Additionally, we benchmarked our model against the SSL baseline models: SigRep and BENDR. Fig. 3.9 presents the average degradation in accuracy and F1-score of the compared models when a modality is missing in different downstream tasks. A series of t-tests were further conducted on the performance differences for a more systematic robustness comparison.

From the evaluation results, we can first observe that the performance drops of our model are significantly lower (p < 0.05) than other SSL models on most tasks. This demonstrates the superiority of the proposed method in terms of robustness. Second, we also note that the impact of missing modalities on the robustness of SSL methods is task-dependent. For downstream tasks related to stress and arousal levels, more severe performance declines could be obtained in the absence of the EDA signal, compared to the other two modalities. This result indicates the importance of the EDA signal for identifying these two emotional states. Similarly, missing the TEMP signal also leads to a considerable reduced performance in arousal-based recognition, whereas, in the valence-based tasks, the loss of the BVP signal has the greatest impact on performance. The above results, consistent with those in Section 3.5.6.B, reconfirm the effect of different modalities on specific emotion recognition. proposed in [START_REF] Vaswani | Attention is all you need[END_REF] to the input embedding of the multimodal transformer while With learnable PE adopted the same learnable positional encoding in [START_REF] Zerveas | A transformerbased framework for multivariate time series representation learning[END_REF]. Table 3.7 also show the classification results of the proposed model with different PE setting. We observed that temporal context information injected by two types of PE did not contribute to model performance on all classification tasks as expected. This can be attributed to the fact that the multimodal embeddings generated by the separate encoders already own different structures, hence the additional positional information introduces redundancy into the model. We further explored the impact of using individual transformations and their combinations in the pretext task on downstream emotion recognition performance. As mentioned in Section 3.3.2.A, the five transforms employed can be divided into two classes, i.e., magnitude domain transformations and time domain transformations. Therefore, the types of combinations are arranged accordingly as combinations of transformations within the same domain and combinations of transformations across domains. The evaluation results obtained on different emotion classification tasks are presented in Table 3.8. First, we noticed that Permutation and Time-Warping, which perturbed the temporal order and duration of events within the window, performed best among the individual signal transformations, which is consistent with the results in [START_REF] Sarkar | Self-supervised ecg representation learning for emotion recognition[END_REF][START_REF] Saeed | Multi-task self-supervised learning for human activity detection[END_REF],

demonstrating the necessity to encode the temporal relationships of signals for emotion recognition. Second, the pre-trained models obtained by combining the same domain or cross-domain transformations generally perform better than those based on individual transformations. The performance of these combinations varies depending on the specific task. For the same domain transformation combinations, P+T+C performs better for stress-related tasks, whereas N+M is more appropriate for arousal and valence-based tasks. For the cross-domain combinations, N+T exhibited the best performance on the classification tasks regarding stress, while N+P and M+C performed best in predicting the arousal and valence states. Finally, we found that models based on cross-domain combinations outperformed those based on the same domain combinations in two-thirds of the downstream tasks. Meanwhile, our pre-trained models using the full set of transformations consistently achieved superior performance in the classification tasks. This can be attributed to the fact that different types of transformations inject diverse prior knowledge for multimodal representation learning, thus contributing to the generalizability of the network.

Introduction

From a healthcare point of view, two frequently desired tasks in emotion recognition are stress and pain detection. Stress is a specific emotional state defined by high arousal and negative valence condition according to the dimensional model. When exposed to long-term stressors, a person's mental and physical state can be negatively affected, which can further lead to chronic health problems such as headaches, insomnia or cardiovascular disease [START_REF] Suárez | A tutorial on distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges[END_REF][176][START_REF] Chrousos | The concepts of stress and stress system disorders. overview of physical and behavioral homeostasis[END_REF]. On the other hand, pain is characterized as an unpleasant sensory and emotional experience associated with As complementary information between multimodalities contributes more to the robustness and reliability of the system, therefore, stress/pain detection combining physiological and behavioural indicators has become more attractive. However, the application of multimodal data for automatic identification requires knowledge from different domains, constituting an obstacle for most researchers. In addition, how to effectively fuse multimodal data remains an important challenge for such systems. From the fusion strategy perspective, the few existing efforts are mostly based on early or late fusion and therefore unable to incorporate both intra-modal and inter-modal correlations. From the feature engineering perspective, handcrafted feature-based methods dominate, however, they are typically cumbersome and inefficient for multimodal data.

In this study, we address the above problem by introducing a geometric framework that employs symmetric positive definite (SPD) matrices extracted from physiological and behavioural signals as a joint multimodal feature representation on SPD manifold for stress/pain detection.

Continuous multimodal data recording can first be converted into SPD matrix sequences. The tangent space mapping method is then applied to locally flatten the manifold and approximate the SPD matrix sequences by tangent space vector sequences, where an LSTM-based deep neural network can be implemented to learn the context correlations of input sequences for classification. The overview of the proposed method is illustrated in Fig. 4.1. In summary, the main contributions of this work are:

• We made the first attempt of applying the SPD matrix to fuse behavioural signals and physiological signals for stress/pain detection.

• We employed the SPD-based representation to efficiently capture correlations within and across modalities at different time steps.

• We adopted tagent space mapping for the linearization of manifold data which facilitates high-level temporal feature extraction by the LSTM-based deep neural network.

• We obtained state-of-the-art results on both stress and pain detection tasks using the proposed method. Furthermore, Our integration strategy is proven to be more effective than early fusion or late fusion. Overview of the proposed framework. First, the SPD matrix sequences that incorporate the correlation information between multimodal data (i.e. physiological and behavioural signals) can be extracted from the segmented data records. Subsequently, the tangent space mapping projects the SPD matrix sequences to the vector sequences in the tangent space. Finally, these vectors can be used as input to the LSTM-based classification network for stress/pain recognition.

SPD Representation

Related Work

The multimodal framework is promising to improve the performance of stress/pain detection.

However, few studies can be found using combined data from different fields (i.e. physiological and behavioural).

Multimodal stress/pain detection Some of the multimodal methods presented [5,[START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF][START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF][START_REF] Huynh | StressNAS: Affect State and Stress Detection Using Neural Architecture Search[END_REF][START_REF] Gil-Martin | Human stress detection with wearable sensors using convolutional neural networks[END_REF] in Section 2.5 are targeted at stress and pain detection tasks. Among these methods, multimodal stress detection approaches that combine physiological signals and motion signals (i.e., accelerometers) are relatively popular, as they can be captured by wearable sensors and are well suited to the real-life setting. Alternatively, a few stress detection methods also attempt to fuse physiological modalities with behavioural modalities based on visual data. Aigrain et al. [START_REF] Aigrain | Multimodal stress detection from multiple assessments[END_REF] captured body video, high-resolution facial images and physiological signals from 25 subjects during a mental arithmetic test for stress detection, where 101 features from behavioural and physiological signals were extracted to train an SVM classifier. For pain detection, a majority of methods are based merely on images or videos of facial expressions, thus the effectiveness of multimodality has not been well investigated. In the work of [START_REF] Kächele | Multimodal Data Fusion for Person-Independent, Continuous Estimation of Pain Intensity[END_REF], statistical/time/frequency domain features of the multi-physiological signals (electrocardiogram (ECG), electromyography (EMG), skin conductance level (SCL)) and appearance/geometric features extracted from the facial videos were served as input to a Random Forest (RF) algorithm for continuous estimation of pain intensity. Werner et al. [START_REF] Werner | Automatic pain recognition from video and biomedical signals[END_REF] collected facial distances and gradient-based features from video frames which were combined with the statistical features calculated from biological signals. The resulting multimodal vectors were employed to train a Random Forest (RF) model for pain assessment. The multimodal approach described above shows promising results in terms of stress/pain detection. However, regarding the integration of multimodal data, most of them applied the early fusion strategy, i.e. simply concatenating the features of different modalities as a whole for the corresponding learning task. Although inter-modal interactions could be captured from the obtained joint representation, intra-modal correlations were relatively ignored. In addition, most of these methods employed handcrafted features, their extraction and selection proving to be a challenging task in the multimodal case, as not all features contributed to the learning performance.

Symmetric Positive Definite (SPD) matrices Recently, covariance-based representations have gained great popularity in computer vision and machine learning. This success can be explained by three major advantages. Firstly, several characteristics can be fused into a single tensor and deliver higher-order statistical information. Secondly, the covariance matrices are sym-metric positive definite (SPD) matrices with well-established mathematical theoretical properties [START_REF] Bhatia | Positive Definite Matrices[END_REF][START_REF] Zhang | Cross-covariance matrix: Time-shifted correlations for 3D action recognition[END_REF]. In addition, the SPD matrices have shown impressive accomplishments in many real-world applications such as pedestrian detection [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF], facial expression recognition [START_REF] Otberdout | Automatic analysis of facial expressions based on deep covariance trajectories[END_REF], brain-computer interfaces [START_REF] Barachant | Classification of covariance matrices using a Riemannian-based kernel for BCI applications[END_REF], etc. However, all of these applications concentrate solely on behavioural perspectives or physiological perspectives. Liu et al. [START_REF] Liu | Video modeling and learning on Riemannian manifold for emotion recognition in the wild[END_REF] proposed a multimodal emotion recognition approach using video and audio modalities. Covariance matrix, linear subspace, and Gaussian distribution were built from facial video frames and regarded as points on Riemannian manifolds. Subsequently, the similarity matrix calculated using different Riemann kernels is fed into multiple classifiers (i.e., SVM, partial least squares, and logistic regression).

However, they only constructed the SPD matrix for the video modality, the audio features were extracted using an existing toolkit, and fed into the same type of classifier as the video modality. In the end, the final fusion of the two modalities was established on the decision level.

Thus, the exploration of inter-modal correlations using SPD matrix is still missing here. In the work of [START_REF] Zhang | Cross-covariance matrix: Time-shifted correlations for 3D action recognition[END_REF], they presented a more general covariance-based SPD representation, containing additional cross-covariance information from different time steps for action recognition. Inspired by [START_REF] Zhang | Cross-covariance matrix: Time-shifted correlations for 3D action recognition[END_REF], we migrate this new type of tensor representation to the scenario of multimodal stress/pain detection and show its effectiveness in this paper. Different from the previous work, we fuse behavioural and physiological information into one single SPD matrix-based representation, which not only incorporates intra-modal correlations, but also allows for exchanges across two modalities. To the best of our knowledge, this is the first use of the geometry of SPD manifold matrices to merge physiological and behavioural signals.

Proposed Method

Symmetric positive definite (SPD) matrix for multimodal signal

Let x i = [v 1 , ..., v D ] T ∈ R D , D ≥ 2 represent
a multimodal signal vector comprising the behavioural and physiological signals at the i-th timestamp, where the number of signals is denoted by D. A short-segment can be extracted from the continuous signal recordings of a trial, resulting in a centered matrix X = [x 1 , ..., x i ..., x N ] ∈ R D×N , where N is the number of time instants for each segment and the sample mean vector is µ = 1 N N i=1 x i = 0. The outer product operation (denoted by the symbol ⊗) is then performed on all signal column vector pairs (x i , x j ) for i and j = 1, ..., N in X and consequently produces a partition matrix :

Ω =      x 1 ⊗ x 1 • • • x 1 ⊗ x N x 2 ⊗ x 1 • • • x 2 ⊗ x N . . . . . . . . . x N ⊗ x 1 ... x N ⊗ x N      ∈ R DN ×DN (4.1)
where the element of Ω at position (i, j) is given by:

Ω(i, j) = x i ⊗ x j = x i x j T ∈ R D×D (4.2)
The sample covariance matrix S which is a SPD matrix can be derived from the diagonal elements of Ω:

S = 1 N -1 N i=1 Ω(i, i) = 1 N -1 N i=1 x i x i T ∈ R D×D (4.3)
As can be seen from ( 4.3), the correlation statistics on the pairs of the signal vector (x i , x j ) at the two particular time instant i and j are completely ignored. As such, the second SPD matrix defined as cross-covariance can be extracted from the off-diagonal elements of Ω, which contains the correlation information of the signal vectors at different timestamps and is denoted by C:

C = 1 N 2 -N N,N i=1,j=1,i̸ =j Ω(i, j) = 1 N 2 -N N,N i=1,j=1,i̸ =j x i x j T ∈ R D×D (4.4)
The covariance S and cross-covariance C, are then combined in a symmetric manner to form a more generalised covariance-based representation which remains a SPD matrix and is denoted by P:

P =      S C • • • C C S • • • C . . . . . . . . . . . . C C • • • S      ∈ R (m×D)×(m×D) (4.5)
where m is the dimension of the new SPD matrix P, in other words, P is composed of m blocks of S and m(m -1) blocks of C. A larger m corresponds to a higher computational cost, while the information ratio of the covariance to the cross-covariance decreases [START_REF] Zhang | Cross-covariance matrix: Time-shifted correlations for 3D action recognition[END_REF].

Riemannian Geometry of SPD Matrices

4.3.2.A Mathematical Preliminaires

A manifold M is a topological space that is locally homeomorphic to Euclidean space in a neighbourhood of each point. A differentiable manifold is a topological manifold endowed with a smooth structure, allowing differential calculus to be performed on M. Each point of a differentiable manifold is attached to a tangent space which is a real vector space containing all tangent vectors to M at the point. A differential manifold is referred to as a Riemannian manifold if the tangent space at each point p on the manifold defines a inner product and its value varies smoothly with p. The set of symmetric positive definite (SPD) matrices P(n) = P ∈ S(n), u T Pu > 0, ∀u ∈ R n that we wish to work with, forms a Riemannian manifold, where S(n) = S ∈ M (n), S T = S is the space of all n × n symmetric matrices in the space of square real matrices M (n).

4.3.2.B Riemannian metric and geodesic distance

Given a smooth manifold M, a Riemannian metric g on M is a smooth family of inner products

g p : T p M × T p M → R
L[γ] = b a g γ(t) (γ ′ (t), γ ′ (t)) 1/2 dt (4.6)
The curve γ with the shortest length between two points on M is called the geodesic. Then, the distance between two points p 1 and p 2 on M is defined as

d(p 1 , p 2 ) = inf {L[γ] | γ : [a, b] → M, γ(a) = p 1 , γ(b) = p 2 } (4.7)
Since the space of SPD matrices P(n) is a Riemannian manifold, various efforts have been made to measure the distance between SPD matrices with different metrics. In this study, we consider one of the most popular metrics, the Affine-invariant Riemannian metric (AIRM) metric [START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF] which measures the geodesic distance induced by intrinsic geometry of Riemannian manifold.

By applying this metric, the geodesic connecting any two SPD matrices P i and P j can be easily derived:

γ(t) = P i 1 2 exp(tlog(P i -1 2 P j P i -1 2 ))P i 1 2 (4.8)
and the geodesic distance induced by the Riemannian metric is defined as

δ R (P i , P j ) = ∥log(P i -1 P j )∥ F = n i=1 log 2 λ i 1 2 (4.9)
where ∥•∥ F is the Frobenius norm operator, and λ i , i = 1 • • • n denote the eigenvalues of P i -1 P j .

4.3.2.C Exponential and Logarithm Maps

Given a SPD matrix P ∈ M, a tangent space T p M can be defined, composed by all derivatives of curves through P. Then the exponential map Exp P (•) : T P M → M and its inverse logarithm map, Log P (•) : M → T P M are defined over Riemannian manifolds for exchanging between the manifold and its tangent space at P. More formally, let S ∈ T p M be a tangent vector at P and γ S (t) the unique geodesic such that γ(0) = P and γ ′ (0) = S, then the exponential map is given as:

Exp P (S) = γ S (1) (4.10)
Intuitively, the exponential map travels from P at a constant speed S along the geodesic for one unit of time, arriving at another point on M . For the SPD manifold P(n) equipped with the Riemannian metric (4.9), its exponential map is defined as:

Exp P (S) = P i = P 1 2 exp(P -1 2 SP -1 2 )P 1 2 (4.11)
The logarithm operator, as the inverse of the exponential map, can project the point P i on M to the tangent space T p M [START_REF] Faraki | Log-euclidean bag of words for human action recognition[END_REF] and is defined as:

Log P (P i ) = S = P 1 2 log(P -1 2 P i P -1 2 )P 1 2 (4.12)
An illustration of these two operations is shown in Fig 4 .2.

4.3.2.D Tangent Space Mapping

In current literature, few approaches have been suggested to tackle the non-linearity of the SPD manifold. A common method of dealing with this problem is to estimate the manifold-value data by mapping them into the tangent space of a specific point on the manifold (e.g., the mean of the data) [START_REF] Jayasumana | Kernel methods on riemannian manifolds with gaussian RBF kernels[END_REF]. To achieve this, we first need to construct the minimal representation on the tangent space. Given the expression of the Riemannian metric, we can derive its equivalent definition as δ R (P, P i ) = ∥log P (P i )∥ P = ∥S∥ P = ∥P -1 2 log P (P i )P -1 2 ∥ I (

where I is the identity matrix. Then the minimal representation can be created by the vector operation which maps a n × n matrix into a 1 × n 2 column vector. The coordinates of tangent vector S ∈ T p M at P is defined as:

vec P (S) = vec I (P -1 2 log P (P i )P -1 2 ) (4.14)
Since the tangent space is the space of n × n symmetric matrices, there exist only n×(n+1) 2 independent coefficients (i.e., upper/lower triangular part of the matrix) for the coordinate system. Thus, the vector operation is reformulated by:

vec I (S) = [s 1,1 , √ 2s 1,2 , √ 2s 1,3 , • • • , s 2,2 , √ 2s 2,3 , • • • , s n,n ] ∈ R n(n+1) 2 (4.15)
The off-diagonal elements are multiplied by the weight of √ 2 as they are counted twice when calculating the norm at the identity [START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF]. Now, each point on the SPD manifold is projected into the tangent space through the use of logarithm map Log P (•) which preserves the local structure of the manifold, and can be represented as an n(n+1)/2-dimensional vector for typical machine learning algorithms. In general, the projection M → T P M can occur at any point P on the manifold. However, in the work of [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF], they reported that the best approximation can be obtained using the mean of the data. Therefore, we employ the geometric mean [START_REF] Barachant | Multiclass Brain-Computer Interface Classification by Riemannian Geometry[END_REF] of a set of SPD matrices {P 1 , P 2 , . . . , P I }, I ≥ 1, P i ∈ P(n) to flatten the manifold P ψ = ψ(P 1 , P 2 , . . . , P I ) = arg min

P∈P (n) I i=1 δ R (P, P i ) 2 (4.16)
Since the SPD manifold is a manifold with non-positive cross-sectional curvature, the minimum of the problem defined in (4.16) exists and is unique [START_REF] Karcher | Riemannian center of mass and mollifier smoothing[END_REF]. An effective gradient descent algorithm is adopted for mean computation [START_REF] Fletcher | Principal Geodesic Analysis on Symmetric Spaces: Statistics of Diffusion Tensors[END_REF]. In the end, each SPD matrix P i is converted into the Euclidean vector s i for classification:

s i = vec I (P ψ -1 2 Log P ψ (P i )P ψ -1 2 ) ∈ R n(n+1) 2
(4.17)

Classification of SPD matrix sequences

Let us consider a set L = Q q=1 L q consisting of data from Q subjects. The qth subset L q is represented by L q = {([P i , ..., P i+T ], y i ),

P i ∈ P(n), i ∈ [1, |L q |]} ,where [P i , ..., P i+T ]
is a segmented sequence consisting of T subsequences, P i can be considered as a representation of the correlation information for the corresponding subsequence living on the manifold, and y i is the stress/pain label associated with the entire SPD matrix sequence, such that y i = f ([P i , ..., P i+T ]). For each subset, its corresponding geometric mean P ψ q can be obtained with the equation (4.16). Then each SPD matrix P i in the qth subset L q is mapped into the tangent space and the derived corresponding subset of vector sequences is denoted by

s q = ([s i , ..., s i+T ], y i ), s i ∈ R n(n+1) 2
, i ∈ [1, |s q |] using the equation (4.17). The above process will be repeated for each subject's data. In the end,

L * = Q q=1 s q is considered
Q subjects L = Q q=1 L q = Q q=1 {([P i , ..., P i+T ], y i ), P i ∈ P(n), i ∈ [1, |L q |]} .
Output: ŷ predicted labels of test set L * te . /* Preparation of inputs to the classifier */ 1: L * ← ∅. 2: for each subject data L q do 3:

Compute geometric mean of L q ; P ψ q = ψ(P i , i = 1, . . . , |L q |);

4:
Tangent Space Mapping of SPD matrices in L q ; s i = vec((P ψ q ) -1 2 Log P ψ q (P i )(P ψ q ) -1 2 )

s q = ([s i , ..., s i+T ], y i ), s i ∈ R n(n+1) 2 , i ∈ [1, |s q |] 5:
L * ← L * ∪ s q 6: end for /* LSTM-based classification */ 7: Split L * into L * tr and L * te and Train LSTM using L * tr ; 8: ŷ ← LST M (L * te )

Experiments and Results

To evaluate the validity of the proposed method, we conducted multimodal stress detection experiments on WESAD dataset while multimodal pain detection experiments were carried out on BP4D+ dataset.

Datasets

The WESAD dataset [START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF] is one of the most widely used public datasets for stress and affect recognition. In a restricted laboratory setting, multimodal data consisting of motion and physiological signals from 15 subjects were captured by two wearable devices, a wrist sensor and a chest sensor, respectively, and the experimental protocol was designed to stimulate three different emotional states (baseline, stress, amusement) in the participants. Based on previous work [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF][START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF][START_REF] Huynh | StressNAS: Affect State and Stress Detection Using Neural Architecture Search[END_REF], a binary stress detection problem (stress vs. non-stress) can be formulated on the WESAD dataset by fusing baseline class and amusement class to form the non-stress class.

The BP4D+ dataset [START_REF] Zhang | Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis[END_REF] is a large-scale multimodal spontaneous emotion database. 140 subjects were required to complete 10 tasks to elicit 10 different emotions, during which 2D RGB images, 3D model sequences, thermal videos and 8 physiological signal sequences with 1.5 million frames were captured by different sensors. In addition, the metadata are also provided, including 2D/3D/thermal facial landmarks, hand-labelled FACS codes and auto-tracked head poses. In this work, we focus on the identification of pain. As the dataset only provides the most facially-expressive segments for four emotions: happy, embarrassment, fear and pain, we therefore performed a pain detection task (pain vs. non-pain) by combining happy, embarrassment, fear as the non-pain class as proposed in [START_REF] Hinduja | Multimodal fusion of physiological signals and facial action units for pain recognition[END_REF].

Data Preprocessing and SPD matrix construction

For WESAD dataset, physiological and motion signals captured by the two sensors are filtered and downsampled to the same frequency, followed by a 10-second non-overlapping segmentation as proposed by [START_REF] Sarkar | Self-supervised ecg representation learning for emotion recognition[END_REF]. Finally, the SPD matrix series can be calculated from multimodal data segments. For BP4D+ dataset, we first calculated the Euclidean distance between each of the two facial landmarks for each video frame using the provided 2D/3D/thermal facial landmarks, and automatically selected the 10 most discriminative distances by feature selection based on Anova F-value. Then the distance vector and the synchronized physiological signal vector were concatenated together to form a new augmented vector. Subsequently, all the augmented vectors are sliced into 1-second non-overlapping data segments. In the end, the obtained SPD matrix sequences can be extracted from prepared data segments. automatically selected were all based on the landmarks in the lip region, which is consistent with findings in the literature that lip movements such as oblique lip raising [START_REF] Patrick | Observer judgments of acute pain: facial action determinants[END_REF], horizontal lip stretching [START_REF] Leresche | Facial expression in pain: A study of candid photographs[END_REF], etc. are related to pain. Both intra-modal and inter-modal correlations can be observed from the SPD matrix. Among all physiological signals, there was a strong correlation between electrodermal activity (EDA) and respiration signal (respiration rate (RR) and respiration volt (RV)) when the subject was suffering from pain. Among the selected distances, the 2nd, 3rd and 6th distances were more correlated with each other. Furthermore, the association between physiological and facial indicators can be explored, i.e. EDA, RR and RV were also positively correlated with the 9th and 10th distances.

Implementation and evaluation

All the classification models were implemented using Pytorch. To avoid overfitting, dropout operation was employed after the LSTM layers with a hidden state dimension of 128. The Adam optimizer with a learning rate lr=0.001 was selected to minimize the binary cross-entropy loss function during model training of 50 epoch. Decay coefficients of the first and second moment estimation β 1 and β 2 were set to 0.9 and 0.999, respectively. In the end, the proposed framework is evaluated using Leave-One-Subject-Out cross-validation (LOSO-CV) on WESAD dataset followed by [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF][START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF][START_REF] Huynh | StressNAS: Affect State and Stress Detection Using Neural Architecture Search[END_REF] and Subject independent 10-fold cross-validation on BP4D+ dataset followed by [START_REF] Hinduja | Multimodal fusion of physiological signals and facial action units for pain recognition[END_REF] with two selected metrics: Accuracy and F1-score. Both experimental pro-tocols assess the model's capacity to generalize across previously unseen subject data.

Stress detection results on WESAD

Binary stress detection experiments were performed using the wrist/chest-based data of all subjects in the WESAD dataset. We first verify the necessity of fusing the sample covariance matrix S and cross-covariance matrix C to form the proposed representation P. The evaluation results of these three representations are shown in Fig. 4.5 (a). It can be observed that the proposed SPD matrix optimizes the detection performance for all modality combinations, compared to those using only matrix S and matrix C. Secondly, to investigate the impact of increasing the dimensionality of the SPD matrix defined in the equation (4.5) on the detection results, we performed experiments using representations with different numbers of blocks of S and C (e.g., m=2 meaning that P consists of 2 blocks of S and 2 blocks of C). We found that using more blocks of S and C to compose the proposed representation P slightly enhanced the classification performance. This can be attributed to the increased proportion of cross-covariance information in the high-dimensional SPD matrix, further demonstrating the benefit of correlations between multiple modalities at different instants for the classification task. Besides, we only test up to m=4, since we noticed that sometimes the m=3 case performs best, following the trade-off between classification performance and computational cost. Finally, we also explored whether fusing data from different modalities, i.e., motion signals and physiological signals, could lead to a performance gain. The corresponding results are reported in Table 4.1. For experiments based on wrist/chest sensor data, combining these two types of data for detection yielded better performance, compared to results based on physiological signals only. When fusing data from both devices (wrist + chest), results using all modalities were not improved, which can be attributed to the redundant information generated by the same motion signals from both devices. When combining all physiological-based modalities, the highest performance (96.88% accuracy and 96.44% F1-score) was obtained with P (m = 4).

Comparison with State-of-the-art

To validate the effectiveness of the proposed method for fusing motion and physiological information, Table 4.2 shows the comparison results with 5 state-of-the-art methods using multimodal features. For a fair comparison, only methods that use the same experimental protocol were considered. In the work of Schmidt et al. [START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF], features from the time and frequency domains are used to train a variety of traditional machine learning classifiers, among which the Random Forest (RF) model achieved the best performance. Samyoun et al. [START_REF] Samyoun | Stress Detection via Sensor Translation[END_REF] features from wrist data, and classified emulated features with various machine learning algorithms, among which RF (Random Forest) performed the best. Gil-Martin et al. [START_REF] Gil-Martin | Human stress detection with wearable sensors using convolutional neural networks[END_REF] proposed a CNN-based architecture to extract meaningful features from three transforms: the Fourier transform (F), cube root (C), and constant q spectral transform (Q) of signal windows. Huynh et al. [START_REF] Huynh | StressNAS: Affect State and Stress Detection Using Neural Architecture Search[END_REF] suggested a training scheme using neural architecture search (StressNAS). Filter banks extracted from different modalities were utilised as model input, and the optimal model was selected automatically for each modality from 10,000 deep neural networks for training.

Finally, features of all modalities were concatenated for classification. Lai et al. [START_REF] Lai | Intelligent stress monitoring assistant for first responders[END_REF] employed residual-temporal convolution network (Res-TCN) to process the filtered multimodal signals and proposed various fusion strategies. The above work for comparison simply spliced features from different modalities for prediction and thus ignored the cross-modality correlations.

Overall, our proposed method using the joint SPD representation achieves the state-of-the-art results on wrist sensor data and competitive results on chest/wrist+chest sensor data, respectively, demonstrating its efficiency for integrating multimodal data.

Pain detection results on BP4D+

To further assess the validity of the proposed method, we conducted unimodal and multimodal pain detection experiments on the BP4D+ dataset. Similar to the process performed on WE-SAD, we first consider the results obtained using S or C alone as baseline to explore the im-portance of combining them in the proposed representation. Based on the results in Fig. 4.5

(b), we reach the same conclusion that the joint SPD representation can improve classification performance. In addition, we can infer that increasing the dimension of P can further boost performance for all modalities. In the end, we also explored the performance of different modality combinations. The evaluation results are summarised in state-of-the-art pain recognition methods that accept only visual data were implemented and combined with our physiological signal-based model for comparison. For a fair comparison, only the facial landmark-based methods were considered. We used the code provided by the authors, and if the code was not available, we followed the parameters provided in their article. Szczapa et al. [START_REF] Szczapa | Automatic estimation of self-reported pain by interpretable representations of motion dynamics[END_REF] represented the facial landmark sequences as trajectories on the Riemannian manifold S + (d, n). Each point of the trajectory is a Gram matrix computed from the 2D facial landmarks. Then the Global Alignment Kernel (GAK) was used to calculate the similarity matrix between the trajectories, which was used as feature for SVR-based (Support Vector Regression) pain estimation. To compare the classification performance, we replaced the SVR with an SVM (Support Vector Machine). Huang et al. [START_REF] Huang | Hybnet: a hybrid network structure for pain intensity estimation[END_REF] proposed a hybrid network (HybNet) which combined 1D, 2D and 3D CNN to extract geometric features from facial landmarks and spatio-temporal features for pain recognition. Choo et al. [START_REF] Choo | Pain detection from facial landmarks using spatial-temporal deep neural network[END_REF] employed a dual-layer 3D CNN for capturing the spatial-temporal features of the 2D facial landmark sequences. When comparing the performance of pain recognition based on solely visual information, our model performs better as shown in Table 4.4. To compare the performance based on multimodal information, we used two fusion techniques that are commonly used in the literature, feature level fusion and decision level fusion, respectively. We first note that the performance of all the vision-based models used for comparison is improved when combined with our physiology-based model, providing side evidence that our model learns discriminative physiological features. Secondly, our model outperforms other multimodal approaches, both in terms of feature level fusion and decision level fusion, which confirms that the correlation between two modalities is well captured by the proposed method and that inter-modal communication can further contribute to the classification performance. Overall, our method achieves the state-of-the-art results on both unimodal data and multimodal data, validating again its effectiveness.

Conclusion

In this work, we explore for the first time the feasibility of SPD matrix-based representations for efficiently fusing physiological and behavioural signals, which can capture simultaneously correlation information within and across modalities. Tangent space mapping converts the generated SPD matrix time series into linear vector sequences for its application to the LSTM-based classification. The effectiveness of the proposed method was evaluated on public stress and pain detection datasets. In the end, the proposed framework shows state-of-the-art results on both stress and pain detection tasks. In the next chapter, our practical contribution will be presented, i.e. the real-world application of emotion recognition systems. This chapter mainly reports on practical contributions, where emotion recognition experiments in real-life scenarios (i.e., medical simulation training) are performed to explore its pedagogical implications. We first introduce what is medical simulation, why emotion recognition systems need to be integrated into teaching programs, and the corresponding challenges in Section 5.1. Subsequently, Section 5.2 details the data acquisition performed in the unconstrained environment. In Section 5.3, we conduct emotion recognition experiments on collected real-life data to verify the effectiveness of the proposed algorithms in Chapters 3 and 4. Finally, we demonstrate an intelligent graphical user interface (GUI) incorporating the proposed emotion recognition algorithms and explore the pedagogical impact of its intervention in Sections 5.4 and 5.5, respectively.

Introduction

Simulation is the artificial replication of complex real-world processes or events with sufficient fidelity that is designed to enhance the learning process through immersion, practice, reflection and feedback without encountering the risks that may be present in real-life environments [START_REF] Krishnan | Pros and cons of simulation in medical education: A review[END_REF].

In the medical field, the term medical simulation, as a subset of simulation, refers to modern educational methodologies for training healthcare professionals by recreating clinical situations.

Its benefits are twofold: first, students can acquire and maintain clinical skills better in simulation training than in didactic teaching, considerably increasing self-confidence in practical situations, and second, simulation training can reduce the risk of harm to patients caused by inexperienced personnel during treatment [START_REF] Krishnan | Pros and cons of simulation in medical education: A review[END_REF]. Due to the preceding reasons, simulation has been used extensively as a teaching tool in medical schools. However, immersive learning in sufficiently realistic simulated scenarios tends to induce special emotional experiences in students, with stress being the relatively common emotional state. This stress can arises from a variety of factors or circumstances, for example, complex or high-risk tasks such as cardiac arrest, time constraints in performing tasks, unfamiliar procedures or equipment, or not receiving the desired feedback. Recently, there has been a tendency for teaching programmes to incorporate stress assessment sessions. The benefits are twofold: from the student's perspective, it reduces the negative impact of stressful states on acquisition of new skills, while from the teacher's perspective, stress detection allows training programmes to be adapted to individual needs and promotes teaching effectiveness. The traditional way of assessing emotions such as stress is subjective with the use of a series of psychological scales, however the process of collection and analysis is time-consuming and therefore not suitable for direct application in the pedagogical process.

With the rapid development of Artificial intelligence (AI) technology and sensor devices, intelligent detection algorithms are being developed considerably to provide a solution for the objective identification of students' emotions, especially stress states. However, current stress detection studies are mainly carried out in laboratory settings. In addition, very few researches can be found that have studied the impact of the intervention of emotion recognition tools on medical simulation training. To address these issues, we collected multimodal data from participants in the high-fidelity medical simulation environment and examined the validity of the emotion recognition algorithms proposed in Chapter 3 and 4 on real-life data. Subsequently, a software that integrates the recognition algorithms, i.e. a graphical user interface, was developed and applied during formation to explore the pedagogical impact of the emotion recognition tool. To summarise, the main contributions of this work are as follows:

• We collected a multimodal dataset consisting of peripheral physiological and motion signals via a wearable device in a real-world setting.

• We conducted emotion recognition experiments on real-life data and demonstrated the effectiveness of the previously proposed algorithms.

• We designed and developed a graphical interface that can be equipped with the recognition algorithms, and investigated its feasibility and usefulness as an emotion recognition tool in the simulation training process.

Data Collection

Subjects

The data involved in this chapter were collected during simulation sessions at the PRESAGE centre over the academic years 2020-2022. The subjects were 28 students in their first year of psychiatry internship or second year of healthcare education. All of these students were enrolled at University of Lille and had never participated in the simulation sessions. in the signal recordings. This is consistent with the findings in [START_REF] Ollander | A comparison of wearable and stationary sensors for stress detection[END_REF]. Ultimately, all subjects were required to wear the E4 wristband on their non-dominant hand for data acquisition. Fig. 5.1 illustrates the E4 wristband and its acquired multimodal signals.

Sensors and Multimodal signals

Experimental protocol

The medical simulation training along with the emotional data collection was conducted at the PRESAGE Center, where each simulation session consisted of three phases: pre-simulation phase, simulation training phase, and educational debriefing phase. Fig. 5.2 demonstrates the entire experimental procedure. Details of each phase will be disclosed below.

showing psychotic symptoms and trying to break out of supervision. trainees were assigned the task of acting reasonably to prevent the patient's escape.

• Suicidal risk: the scenario simulated consultation with a patient who was motivated by suicidal ideation, in which a trainee was required to make appropriate decisions based on the assessed level of urgency and risk of suicide.

• Focus: the scenario required students to pass various tests on a surgical simulator. To provoke stress states, they were told that the obtained score on the tests would be recorded in the final grade of the semester.

Once the simulation was complete, the trainees were also required to press the tag button on the E4 wristband to mark the end time. The same psychometric scales as in the first phase were collected from the trainees to annotate their emotional experiences during the simulation training. Each simulation session lasted about 10 minutes.

Educational debriefing phase

Finally, all participants were invited to a collective debriefing with the actors who played patients to review the simulation training process and discuss the trainees' experience, the difficulties they encountered, and potential solutions for improvement. The debriefing session was led by a senior physician with teaching experience. In this session, the physician will apply the emotion detection tool developed (i.e., a graphical interface with embedded AI algorithms) to optimize the teaching process. At the end of the debriefing, trainees were invited to only fill in questionnaires concerning their perceived level of competence and the difficulty of the task, which were then used to study the intervention effects of the emotion detection tool.

Ground truth collection and evaluation

To establish the ground truth of multimodal data for emotion recognition, we collected trainees' subjective emotional experiences before/during/after the simulation training through three types of questionnaires: Self-Assessment Manikins (SAM), State-Trait Anxiety Inventory (STAI) and Likert scale of perceived competence and task difficulty. The collection and analysis of the subjects' self-reports were carried out in collaboration with Thibaut Denis, a graduate student from the Lille Neuroscience and Cognition Research Center. In the following, we describe these questionnaires in detail and present the statistical analysis of the collected ground truth data.

Psychometric scales

• Self-Assessment Manikins: Self-Assessment Manikins (SAM) [6] is a picture-oriented emotion assessment tool, allowing subjects to rate the level of four dimensions: arousal, valence, liking, and dominance associated with emotional responses to diverse stimuli.

Each scale is rated from 1 to 9, with 1 indicating low values for each dimension and 9

indicating high values for each dimension. In our study, we concentrated on the arousal and valence dimensions. The illustration of SAM is shown in Fig. 5.3. We only collected this scale after the first and second phases. not at all, 2. somewhat, 3. moderately so, 4. very much so), the corresponding results of which can be considered as an approximation of a person's acute stress state. We only collected this scale after the first and second phases.

• Likert scale on the level of perceived competence and difficulty of the task: stress can be considered as the result of an interaction between the learner and his or her cognitive appraisal, in accordance with Lazarus and Folkman's model [START_REF] Lazarus | Fifty Years of the Research and theory of R.s. Lazarus: An Analysis of Historical and Perennial Issues[END_REF]. Thus, each trainee in the simulation was repeatedly presented with a measure of his or her perceived level of competence in managing the simulation (primary assessment) and a measure of their perception of the difficulty of the task (secondary evaluation). Based on this, a likert scale of nine items was proposed. In the end, we collected this scale after all the phases. We observed that the agitation scenario differed from the other two scenarios in that the samples were mainly concentrated in the high arousal area and a considerable portion of the samples were distributed in subareas with negative valence. This may be related to the difficulty of the tasks performed by the subjects in this scenario, which required them to cope with the unexpected behavior of the agitated patients and was often accompanied by relatively violent physical conflicts in this case. In contrast, in the focus and suicidal risk scenarios, subjects were exposed to a more calm environment, especially the focus scenario involved only human-machine interaction, thus most of the samples presented with low arousal and positive emotions. Second, to examine the impact of simulation training on trainees' perceived stress from a holistic perspective, we studied samples of STAI scales before and after the simulation. Fig. 5.5 demonstrates the data distribution of STAI scores collected in different scenarios. For the agitation and suicidal risk scenarios, the average stress level of the trainees showed modest increases at the end of the simulation. The Wilcoxon signed-rank tests showed a significant difference between pre-and post-simulation scores in the agitation scenario (p-value= 0.029), whereas there was no significant difference in the suicidal risk (p-value=0.375) and focus scenario (p-value=0.141), this finding is consistent with the analysis in arousal-valence space.

Emotion recognition experiments with real-life data

In Chapter 3 and 4, we validate the proposed emotion recognition methods based on selfsupervised learning and supervised deep learning on data collected in a laboratory setting, respectively. We will continue to explore their performance in unrestricted real-life scenarios in this section.

Data preprocessing

First, the same low-pass filter as in Chapter 3 was used to remove the EDA, BVP and TEMP signal artefacts. In addition, a Butterworth low-pass filter with a cut-off frequency of 5 Hz was used to denoise the ACC signal. Then, z-score normalization was applied to eliminate crosssubject variation in multimodal signals. Finally, all signal modalities were sampled uniformly to 4Hz and segmented into windows of 60 seconds with 90% overlap for emotion classification.

For arousal and valence scores varying from 1 to 9, a threshold of 5 was chosen to generate the labels of the samples for the emotion recognition task, where scores less than 5 were classified as Low and vice versa as High, resulting in two binary classification problems and a quadratic classification problem (i.e., LALV, LAHV, HALV, HAHV) on the arousal and valence dimensions. In addition, stress recognition is a task of particular interest, where HALV states can be categorised as Stress and other categories in the quadratic classification can be regarded as Non-stress. An overview of the pre-processed real-life dataset is shown in Table 5.1.

Methods for comparision

The following methods were implemented for performance comparison:

• Handcrafted-SVM: This is a traditional supervised machine learning-based approach where a series of handcrafted features are extracted from the multimodal data as input Signal Features ACC mean, std of each axis/summation of all axis absolute integral of each axis/summation of all axis peak frequency of each axis BVP std of intervals between heartbeats (SDNN) std of successive differences between adjacent R-R intervals (SDSD) rms of successive differences between adjacent R-R intervals (RMSSD) proportion of differences greater than 50ms / 20ms (PNN50/20) frequency power in 0.15 -0.40Hz (HF) and the 0.04 -0.15Hz (LF) EDA Tonic: mean, std, 20th percentile, 80th percentile, quartile deviation Phasic: peaks prominence, peaks width, peaks per 100 s, strong peaks (peaks that are more than 1 µSiemens) per 100s TEMP mean, std, min, max, slope to the SVM classifier. For ACC and TEMP signals, multiple statistical features presented in [START_REF] Schmidt | Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection[END_REF] were calculated directly from the pre-processed signal data. The EDA signal was first decomposed into two basic components: tonic and phasic using the cvxEDA method proposed by Greco et al. [START_REF] Greco | cvxEDA: a Convex Optimization Approach to Electrodermal Activity Processing[END_REF], from which the corresponding time-domain features were then extracted separately. For the BVP signal, we first detected the peaks to recover the IBI signal and then calculated a series of HRV features for the recognition task. The features employed are listed in Table 5.2.

• SPD-LSTM: The multimodal fusion approach based on Riemannian geometry of the SPD matrices proposed in Chapter 4. An LSTM-based model was used to extract and classify high-level features from the linearised manifold data. For the comparison experiments, we set the dimension of the SPD representation m to 3 and the other parameters to the same settings as described in Chapter 4.

• TCN-TRANS: The self-supervised learning approach based on 1D temporal convolution (TCN) and multimodal transformer (TRANS) proposed in Chapter 3. The experimental

Pratical application of stress analysis tool

In the previous sections, we described the process of collecting real-life data on which we validated the effectiveness of the emotion recognition methods proposed in Chapter 3 and Chapter 4. Next, in order to achieve the main objective of the project, i.e. to verify whether the intervention of emotion detection algorithms, and in particular the stress detection algorithm, could improve the debriefing phase thus facilitating the learning of the trainees, a graphical interface was developed and applied as a stress analysis tool to the debriefing process. In the following, we will show the details of the tool and the evaluation results of its application in realistic scenarios in Sections 5.4.1 and 5.4.1 respectively. on user feedback as shown in Fig. 5.6 (b). At the debriefing phase, the teacher expects to quickly locate and review the most stressful segments of the trainee and provide instruction, thus the probabilities of stress in every 60 s represented by a diverging colormap with values ranging from [0,1] is introduced, where blue and red are biased towards the non-stress and stress states, white corresponds to a neutral state (i.e., a probability of 0.5). In addition, all the physiological signals initially proposed were reduced to the EDA signal for simplification since it is considered to be one of the best real-time relevant indicators of stress [START_REF] Alberdi | Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review[END_REF].

Development of the stress analysis tool

Evaluation of the stress analysis tool

The pedagogical relevance and feasibility of the stress detection method were validated in a simulation training based on suicidal risk scenario in June 2022. To explore the impact of the application of the stress analysis tool on the debriefing phase, particularly in terms of trainees' competence development, subjective variables were first monitored in different phases of the simulation training. As mentioned in Section 5.2.4, a 9-item Likert scale was distributed to the trainees at three moments: pre-simulation, post-simulation and post-debriefing, to measure their perception of task difficulty, as well as their level of ability to manage the task. In addition, a metacognitive dimension of certainty about the answers was added. Fig. 5.7 presents a visualization of the subjective assessment. We can observe a correspondence between a potential decrease in perceived difficulty and an increase in perceived competence, while their corresponding levels of certainty of answers remain either increasing or basically invariant. This is in line with expectations, as the trainers can target the stressful moments of the trainees during the simulation with the aid of the developed tool, which allows them to help the students to better solve their problems and thus promote their personal competence. Secondly, we collected feedback from the trainees on their usage of the stress analysis tool through a 7-item Likert scale. The feedback from trainees generally gave positive feedback on the role of the interface in debriefing and its ability to identify key moments (Fig. 5.8). Moreover, in addition to the the discussion of typical open questions during the debriefing process, the use of the simplified version (i.e. the colormap of stress detection probabilities) made it possible for trainees to rapidly initiate a certain degree of reflective processing based on long-duration video material.

Therefore, the incorporation of the trainee's stress detection in the debriefing session can also enhance their participation and engagement. Furthermore, when reviewing the detected stressful moments with the trainees, we noticed that these segments were usually accompanied by emotional verbal expressions. This further affirms the effectiveness of our proposed algorithm and also inspires the possibility of incorporating audio data in future work to provide more accurate detection.

Conclusion

In this chapter, we investigated the application of an emotion recognition system in a real-life including the facilitation of skill acquisition and increased engagement. In the next chapter we will show the technical contribution on the behavioural aspect, i.e., the landmark-based facial expression recognition approach.

The work presented in this chapter corresponds to the technical contributions on the behavioral aspect, where a non-euclidean metric learning method with application to facial expression recognition is proposed. In Section 6.1 we briefly introduce landmark-based shape analysis and its limitations, followed by Section 6.2 in which we present metric learning methods applied to facial expression recognition. In Section 6.3 we describe in detail the shape representation in the complex projective space, the Fubini-Study metric and its optimization algorithm, in Section 6.4 we focus on facial expression recognition experiments and explore the effectiveness of the proposed method, and finally Section 6.5 presents a summary of the work.

Introduction

Recently landmark detection and tracking methods for human faces and bodies became reliable and accurate. This has greatly encouraged landmark-based shape analysis methods, aiming at shape description and comparison through the location of a pre-defined set of points and the association between them. Since a set of landmarks detected in a video frame or still image is a natural choice for modelling the facial shape or body shape, such methods are widely deployed in the field of computer vision and multimedia such as face verification [START_REF] Hu | Discriminative deep metric learning for face verification in the wild[END_REF], person re-identification [START_REF] Liao | Person re-identification by local maximal occurrence representation and metric learning[END_REF], facial expression recognition [START_REF] Daoudi | Metric learning on the manifold of oriented ellipses: Application to facial expression recognition[END_REF][START_REF] Kacem | Barycentric representation and metric learning for facial expression recognition[END_REF][START_REF] Wan | Spontaneous facial expression recognition: A robust metric learning approach[END_REF] and action recognition [START_REF] Shao | Kernelized Multiview Projection for Robust Action Recognition[END_REF].

Depending on the type of shape space, landmark-based methods can be divided into Euclidean methods and Riemannian methods. The latter are based on the geometry of Riemannian manifolds and are more robust to variation factors in the data, such as affine transformations, thus gaining broader interest compared to the Euclidean methods. In the work of [START_REF] Kacem | A Novel Geometric Framework on Gram Matrix Trajectories for Human Behavior Understanding[END_REF], the face and body landmark sequences were parametrized as trajectories on the Riemannian manifold of symmetric positive definite (SPD) matrices of fixed-rank. Daoudi et al. [START_REF] Daoudi | Gram matrices formulation of body shape motion: An application for depression severity assessment[END_REF] used the Gram matrix of 2D landmarks as the representation of the body shape on a non-linear manifold to assess the depression severity of a patient. Szczapa et al. [START_REF] Szczapa | Automatic estimation of self-reported pain by interpretable representations of motion dynamics[END_REF] encoded the facial movement as trajectories on the manifold of symmetric positive-semidefinite (PSD) matrices for pain intensity estimation. The above approaches always involve metrics defined on the manifold to provide similarity measures for the learning task at hand. However, these Riemannian methods typically employ pre-defined metrics, which may not be the most appropriate for distance measures on a given dataset. In response to this problem, a large number of metric learning algorithms have been proposed in order to obtain more discriminative metrics to further facilitate learning performance. However, most conventional metric learning methods for these vision tasks are designed for the optimization of metrics in the Euclidean space and are therefore not applicable to manifolds.

Considering the above issues, we propose a metric learning method suited to Riemannian geometry. First, we treated the equivalence class of complex landmarks as points in the complex projective space whose shape representation is invariant under a class of affine transformations that consists of translations, rotations, and scaling. Based on the seminal work on shape analysis of Kendall [START_REF] Kendall | Shape manifolds, procrustean metrics, and complex projective spaces[END_REF], a family of metrics -Fubini-Study metrics in this space was selected for distance measurement, where metric learning algorithm based on Large Margin Nearest Neighbors (LMNN) is exploited to improve the discriminative power of the proposed metric. Finally, the learned metric was employed in a similarity-based classification task and its validity was verified in the facial expression recognition scenario. Fig. 6.1 demonstrates the overview of the proposed method. Overall, the main contribution of the paper are:

• We proposed a landmark-based affine-invariant shape representation in complex projective space on the basis of Riemannian geometry.

• We adapted the LMNN algorithm to learn the optimal metric, and the resulting similarity features have smaller intra-class differences and larger inter-class differences.

• We performed a comparison between the Fubini-Study metric and the Euclidean metric on the facial expression recognition task, validating the effectiveness of the proposed metric for shape analysis. Furthermore, our approach also showed competitiveness in comparison with state-of-the-art solutions.

Related work

As presented in Section 2.4.4, most Riemann methods for facial expression recognition tasks rely on full-face feature support, i.e. treating full-face landmark points as a whole and deriving robust shape representations from them. However, such methods ignore the possibility of optimizing the used metric to enhance learning performance. Due to the need for suitable metrics to measure the similarity between a set of landmarks, a series of metric learning-based approaches have been proposed for facial expression recognition tasks. The purpose of metric learning is to find an adequate metric to better capture the potential similarity relationship hidden in the data. The optimal metric obtained from metric learning can bring samples from the same class as close as possible while keeping the differently labeled samples far from each other. In the following, we focus on reporting on previous work that adopted this learning model for facial shape analysis.

Euclidean metric learning

In the work of Wan et al. [START_REF] Wan | Spontaneous facial expression recognition: A robust metric learning approach[END_REF], the facial shape representation was defined as the combination of the shape vector represented by the facial points and the appearance feature vector. Mahalanobis distance-based Euclidean metric learning algorithms was employed to obtain the optimal metric, where similar shapes are relatively closer in the learned feature space. Experimental results demonstrated that the KNN-based classifier with the learned metric achieved superior spontaneous facial expression recognition performance than the methods using the original metric and exhibited good generalization ability on the spontaneous expressions. Kacem et al. [START_REF] Kacem | Barycentric representation and metric learning for facial expression recognition[END_REF] exploited a chart of barycentric coordinates to map affine equivalence classes of facial landmarks to the Euclidean space and then applied metric learning to the Mahalanobis distance for the facial expression recognition task. The DTW technique combined with the optimal metric was used to align facial shape sequences and constructed similarity-based inputs for the pairwise proximity function SVM (ppfSVM) classifier. In the end, the optimal metric yields better recognition accuracy, confirming the necessity of metric learning. However, their method relies on the stability of a reference triangle, which is used to obtain an affine-invariant shape representation for metric learning.

Non-Euclidean metric learning

Some non-linear methods were also proposed to learn a more discriminant metric, which can encode the intrinsic geometry of the manifold. Daoudi et al. [START_REF] Daoudi | Metric learning on the manifold of oriented ellipses: Application to facial expression recognition[END_REF] propose a metric learning over a family of metrics on the space of oriented ellipses centered at the origin in Euclidean n-space and the double cover of the manifold of n × n positive semi-definite (PSD) matrices of rank two. They first adopted the Gram matrices as the facial shape representations. The Average Neighborhood Margin Maximization (ANMM) algorithm was then generalized to learn the optimal quotient Riemannian metric on the manifold data. The proximity measures calculated between facial shapes using the obtained metric were fed into the SVM classifier for static facial expression recognition, obtaining improved performance over that before metric learning.

Proposed Method

Complex projective space

A 2D landmark configuration z consists of n ordered points (x 1 , y 1 ), . . ., (x n , y n ) on the plane. Expressing the points as complex numbers (i.e., writing z j := x j + iy j instead of (x j , y j )), we identify z as a complex vector (z 1 , . . . , z n ) ∈ C n . We will consider two landmark configurations z and w to be equivalent if the points w j (1 ≤ j ≤ n) in the second configuration are obtained from the points z j in the first by means of a common translation, rotation, and scaling. This equivalence is established by the fact that the three transformations only alter the position, orientation or size of the shapes accordingly, without destroying their semantic information.

Therefore, the set of equivalence classes of 2D landmark configurations is the shape space in which we wish to work. This shape space can be considered as a generalization of the original space of landmark configurations, enabling a more realistic measure of the similarity/dissimilarity between shapes, as differences caused by transformations are excluded. Since the landmark points are represented in complex form, the equivalence of two landmark configurations z and w translates into the existence of a nonzero complex number a and a complex number v so that w j = az j + v, ∀j ∈ {1, . . . , n}, where a = λe iθ controls the variation factors of scaling and rotation while v regulates translation. If we consider only centered configurations where z 1 + • • • + z n = 0, then two centered configurations z and w are equivalent if and only if there exists a nonzero complex number a for which w = az. This is precisely the definition of complex projective space (i.e.,the desired shape space). Definition 1. Complex projective space of (complex) dimension n, CP n , is the set of equivalence

Fubini-Study Metrics

A useful feature of complex projective space is that it carries a family of simply-defined and well-studied metrics: let A be a positive-definite Hermitian matrix, the Fubini-Study distance, d A , associated to A between points z = [z 1 : . . . : z n ] and w = [w 1 : . . . :

w n ] in CP n-1 is defined by |⟨Az, w⟩| 2 ⟨Az, z⟩⟨Aw, w⟩ = cos 2 (d A (z, w)) (6.1)
We recall that the expression ⟨z, w⟩ = z 1 w1 + • • • z n wn is the standard complex-valued inner product on C n .

Euclidean Metrics

Given a positive-definite Hermitian matrix, its associated Fubini-Study metric allows us to define the distance between two equivalence classes of landmark configurations. However, there exists also a simple alternative. Assume that for all of our landmark configurations, the i-th landmark is different from the origin. For each landmark configuration z, instead of considering the point in CP n-1 given by z = [z 1 : . . . : z n ], we can consider the vector

z i := z 1 z i , . . . , z i-1 z i , z i+1 z i , . . . , z n z i ∈ C n-1 .
Note that any two landmark configurations z and z ′ are equivalent if and only if z i = z ′ i . In this way, our shape space, or rather the piece of it that interests us, becomes the linear space C n-1 = R 2n-2 . This enables us to use different Euclidean metrics to measure the distance between equivalence classes of landmark configurations.

Recall that each Euclidean metric in R 2n-2 is determined by a positive-definite (2n -2) × (2n -2) real matrix M and that the associated distance is given by the formula

d M (x i , y i ) = (x i -y i ) T M (x i -y i ) (6.2)
where x i and y i are coordinates in C n-1 = R 2n-2 associated to the equivalence classes [x] and [y] in CP n-1 .

Distance function for Fubini-Study metrics

Recall that given a positive-definite Hermitian matrix A, the Fubini-Study distance associated to A between points z = [z 1 : . . . : z n ] and w = [w 1 : . . . : w n ] in CP n-1 is defined by |⟨Az, w⟩| 2 ⟨Az, z⟩⟨Aw, w⟩ = cos 2 (d A (z, w)).

Before performing metric learning using this family of distances, we must consider a cost function that is a real-valued function on the space of positive-definite n × n Hermitian matrices:

C : H n -→ R.
Usually, the cost function takes the following form: there exist points z 1 , . . . , z k in CP n-1 and a real-valued smooth function: c : R

( k 2 ) + -→ R
such that for any positive-definite n × n Hermitian matrices, the value of C(A) only depends on the mutual distances between the points z 1 , . . . , z k :

C(A) = c(d A (z 1 , z 2 ), . . . , d A (z k-1 , z k )).
Sometimes it is easier not to explicitly describe the dependence of the cost function C in terms of the distances d A (z i , z j ), but in terms of the square or other function f of these distances:

C(A) = c(f (d A (z 1 , z 2 )), . . . , f (d A (z k-1 , z k ))).
Since we need to find an optimal metric with the aid of the cost function C, i.e., to find the best positive-definite Hermitian matrix A that assigns small distances to similar points, while assigning relatively large distances to dissimilar points, the function f : R + → R + should be smooth, strictly increasing and satisfy f (0) = 0. Given the formula for the Fubini-Study distance, it will be convenient to work with the function f (t) = 1cos 2 (t) as part of the cost function for metric learning. This function is smooth, satisfies f (0) = 0, and it is strictly increasing in the interval [0, π 2 ]. Notice that π 2 is precisely the diameter of CP n-1 with any of the Fubini-Study metrics, so that whatever happens to f for t > π 2 is irrelevant. As we are particularly interested on the dependency of the positive-definite Hermitian matrix A, the distance function is represented by the term F (A)

F (A) = 1 -cos 2 (d A (z i , z j )) = 1 - |⟨Az i , z j ⟩| 2 ⟨Az i , z i ⟩⟨Az j , z j ⟩ (6.3)
for a number of chosen landmark configurations z 1 , . . . , z k in CP n-1 . To summarize, we will take the cost function C(A) to be a function of F (A). In order to obtain the optimal metric through the cost function, the gradient of F (A) with respect to the variable A can be first given by the formula

∇ A F (A) = - 1
(⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × ⟨Az i , z i ⟩⟨Az j , z j ⟩(⟨Az j , z i ⟩z i z * j + ⟨Az i , z j ⟩z j z * i ) -|⟨Az i , z j ⟩| 2 ⟨Az j , z j ⟩z i z * i + ⟨Az i , z i ⟩z j z * j (6.4)
Proof. Recall that the computation of a gradient of a function F on an inner product space is a two-step process. We first compute the directional derivative of F (A) at point A in the direction of X:

dF (A)(X) := dF (A + tX) dt | t=0 , (6.5)
where X is any positive-definite n × n Hermitian matrices. The resulting derivative is given as

dF (A)(X) = - d dt |⟨(A + tX)z i , z j ⟩| 2 ⟨(A + tX)z i , z i ⟩⟨(A + tX)z j , z j ⟩ | t=0 = - d dt ⟨(A + tX)z i , z j ⟩⟨(A + tX)z i , z j ⟩ ⟨(A + tX)z i , z i ⟩⟨(A + tX)z j , z j ⟩ | t=0 = - d dt ⟨(A + tX)z i , z j ⟩⟨(A + tX)z j , z i ⟩ ⟨(A + tX)z i , z i ⟩⟨(A + tX)z j , z j ⟩ | t=0
For convenience, we can re-write the above equation in the following form:

dF (A)(X) = - d dt f (t) g(t) | t=0 = - f ′ (t)g(t) -f (t)g ′ (t) g(t) 2 | t=0 (6.6)
We can first calculate the derivative f ′ (t) as follows:

f ′ (t) = d dt ⟨(A + tX)z i , z j ⟩⟨(A + tX)z j , z i ⟩| t=0 = d dt ⟨Az i + tXz i , z j ⟩⟨Az j + tXz j , z i ⟩| t=0 = ⟨Xz i , z j ⟩⟨Az j , z i ⟩ + ⟨Xz j , z i ⟩⟨Az i , z j ⟩
We can say that (z + z) = 2ℜ(z), thus f ′ (t) can eventually be written as

f ′ (t) = 2ℜ(⟨Xz i , z j ⟩⟨Az j , z i ⟩) (6.7)
Next, the derivative g ′ (t) can be computed as follows:

g ′ (t) = d dt ⟨(A + tX)z i , z i ⟩⟨(A + tX)z j , z j ⟩| t=0 = d dt ⟨Az i + tXz i , z i ⟩⟨Az j + tXz j , z j ⟩| t=0 = ⟨Xz i , z i ⟩⟨Az j , z j ⟩ + ⟨Xz j , z j ⟩⟨Az i , z i ⟩ (6.8)
Finally, we can bring equations (6.7) and (6.8) back into (6.6) to obtain the final form of dF (A)(X):

dF (A)(X) = - f ′ (t)g(t) -f (t)g ′ (t) g(t) 2 | t=0 = - 1
(⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × 2ℜ(⟨Xz i , z j ⟩⟨Az j , z i ⟩) ⟨Az i , z i ⟩⟨Az j , z j ⟩ -|⟨Az i , z j ⟩| 2 (⟨Xz i , z i ⟩⟨Az j , z j ⟩ + ⟨Xz j , z j ⟩⟨Az i , z i ⟩)
With the directional derivative dF (A)(X) defined in the above equation, we now need to find the unique vector field ∇F (A) (i.e., the gradient on the Riemannian manifold of Hermitian matrices) that satisfies:

⟨∇F (A), X⟩ = dF (A)(X) (6.9)

Since the inner product of two Hermitian matrices is the trace of their product, the equation (6.9) can be written in the following form:

tr(∇F (A)X) = -1 (⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × 2ℜ(⟨Xz i , z j ⟩⟨Az j , z i ⟩) ⟨Az i , z i ⟩⟨Az j , z j ⟩ -|⟨Az i , z j ⟩| 2 (⟨Xz i , z i ⟩⟨Az j , z j ⟩ + ⟨Xz j , z j ⟩⟨Az i , z i ⟩)

With the trick ⟨Xz i , z j ⟩ = tr(X z i z * j ) where z i and z j are considered as n × 1 matrices, we can rewrite the equation as:

tr(∇F (A)X) = -1
(⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × 2ℜ tr(Xz i z * j )⟨Az j , z i ⟩ ⟨Az i , z i ⟩⟨Az j , z j ⟩ -|⟨Az i , z j ⟩| 2 tr(Xz i z * i )⟨Az j , z j ⟩ + tr(Xz j z * j )⟨Az i , z i ⟩ = -1 (⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × tr(Xz i z * j )⟨Az j , z i ⟩ + tr(Xz j z * i )⟨Az i , z j ⟩ ⟨Az i , z i ⟩⟨Az j , z j ⟩ -|⟨Az i , z j ⟩| 2 tr(Xz i z * i )⟨Az j , z j ⟩ + tr(Xz j z * j )⟨Az i , z i ⟩ = -1 (⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × tr(Xz i z * j ⟨Az j , z i ⟩) + tr(Xz j z * i ⟨Az i , z j ⟩) ⟨Az i , z i ⟩⟨Az j , z j ⟩ -|⟨Az i , z j ⟩| 2 tr(Xz i z * i ⟨Az j , z j ⟩) + tr(Xz j z * j ⟨Az i , z i ⟩) = -1 (⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × tr(X(z i z * j ⟨Az j , z i ⟩ + z j z * i ⟨Az i , z j ⟩) ⟨Az i , z i ⟩⟨Az j , z j ⟩ -|⟨Az i , z j ⟩| 2 tr(X(z i z * i ⟨Az j , z j ⟩ + z j z * j ⟨Az i , z i ⟩)

= tr(X(( -1 (⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × ⟨Az i , z i ⟩⟨Az j , z j ⟩(⟨Az j , z i ⟩z i z * j + ⟨Az i , z j ⟩z j z * i )

-|⟨Az i , z j ⟩| 2 ⟨Az j , z j ⟩z i z * i + ⟨Az i , z i ⟩z j z * j

))

In the end, the gradient ∇F (A) is given by:

∇ A F (A) = -1
(⟨Az i , z i ⟩⟨Az j , z j ⟩) 2 × ⟨Az i , z i ⟩⟨Az j , z j ⟩(⟨Az j , z i ⟩z i z * j + ⟨Az i , z j ⟩z j z * i )

-|⟨Az i , z j ⟩| 2 ⟨Az j , z j ⟩z i z * i + ⟨Az i , z i ⟩z j z * j (6.10) which will be used for the optmization of the Fubini-Study metrics. 3: An illustration of the LMNN algorithm modified from [START_REF] Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF]. The proposed Fubinistudy metric will be optimised such that the distances between each sample z i and its k = 3 target neighbours are reduced, while the distances to the impostors are increased.

Metric Learning with Fubini-Study metrics

Large Margin Nearest Neighbors (LMNN) [START_REF] Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF] was selected as the metric learning algorithm. The purpose of this algorithm is to reduce the distance between each sample and its target neighbors, which are the k pre-selected nearest neighbor samples of the same class, while trying to keep it away from its imposters, which are differently labeled samples that invade the margin established by those target neighbor. An illustration of the LMNN algorithm is shown in Fig. 6.3. Assuming that the target neighbor set has been selected (the nearest neighbors of each sample are calculated by the Fubini-Study distance), the loss function of LMNN consists of two terms. The first term is the target neighbors pulling term, given by

ε pull (A) = j⇝i F ij (A) = 1 -cos 2 (d A (z i , z j )) (6.11)
where F ij (A) is the distance function in Eq.(6.3) between complex-valued samples z i and z j corresponding to Hermitian matrix A. j ⇝ i iff jth sample is a target neighbor of ith sample. The second term is the impostors pushing term, given by ε push (A) = i j⇝i l

(1y il )[1 + F ij (A) -F il (A)] + (6.12)

where y il = 0 if ith sample and lth sample are differently labeled and 1 otherwise, the term [•] + is defined as [z] + = max{z, 0}. Finally, the loss function is given by, ε(A) = (1µ)ε pull (A) + µε push (A), µ ∈ [0, 1] (6.13

)
where µ is the weighting parameter for balancing push and pull effects. The gradient of the loss function can be formulated as

∇ A (ε(A)) = (1 -µ)∇ A (ε pull (A)) + µ∇ A (ε push (A)), µ ∈ [0, 1] (6.14)
where,

∇ A (ε pull (A)) = j⇝i ∇ A F ij (A) (6.15)
and,

∇ A (ε push (A)) = i j⇝i l (1 -y il )[∇ A F ij (A) -∇ A F il (A)] (6.16)
The Riemannian Steepest Descent algorithm in the toolbox Pymanopt [START_REF] Townsend | Pymanopt: A python toolbox for optimization on manifolds using automatic differentiation[END_REF] was implemented using the cost function and gradient in Eq.(6.13) and Eq.(6.14), respectively, to seek the optimal solution for positive-definite Hermitian matrix A. The main procedure for our metric learning on complex projective space is given in Algorithm 6.1.

Algorithm 6.1: Metric learning in Complex Projective Space CP n-1 Data: N training samples Z = {(Z i , y i )} N 1 with their associated labels, k is the number of target neighbors. Result: The optimal positive-definite Hermitian matrix A ⋆ 1 A ← I(n), identity complex matrix of dimension n. ; 2 for i = 1 . . . N do 3 Define k target neighbors N A s (i) and their corresponding impostors N A o (i) for each sample Z i using the Fubini-Study metric in Eq.(6.1) ; end 4 Cost ← ε(A, N A s , N A o ), ε(.) is given by Eq.(6.13); 5 Grad ← ∇ A (ε(A)), ∇ A (.) is given by Eq.(6.14); 6 A ⋆ ← SteepestDescent(A, Cost, Grad), SteepestDescent(.) is the Riemannian Steepest Descent optimization algorithm in H n ; based distance metric learning for facial expression recognition task on CK+ and Oulu-CASIA datasets, respectively. From the obtained results, we can observe that the optimal Fubini-Study metric d A ⋆ obtained maximum gains of 4.84% and 9.89% in classification accuracy on CK+ and Oulu-CASIA datasets compared to the original ones, which demonstrates that the use of a suitable metric can enhance facial expression recognition performance. In addition, the SVM classifier generated better classification performance than KNN, both in terms of the original and the optimal metric. Oulu-CASIA datasets using the t-SNE [START_REF] Van Der Maaten | Visualizing data using t-sne[END_REF] with the two metrics. For CK+ dataset, the 2D visualization obtained by the original Euclidean metric in Fig. 6.6 (c) shows an approximately linear relationship, and the points of different expressions are mixed together, while the original Fubini-study metric (Fig. 6.6 (a)) has been able to well distinguish the two expressions: Happy and Surprise. After metric learning, the Fubini-study metric can lead to a more uniform feature distribution for different expressions compared to the Euclidean metric. We reached the same conclusion on the Oulu-CASIA dataset that our proposed metric always yields a more discriminative feature representation both before and after metric learning, with respect to the Euclidean metric. We first compared the proposed method with state-of-the-art approaches for CK+ dataset, with the details reported in Table 6.4. For a fair comparison, only the geometry-based methods that use facial landmarks were considered in the comparative evaluation. We divided these methods into two categories, depending on whether the input is a static image or a dynamic video sequence. The only comparable static method [START_REF] Daoudi | Metric learning on the manifold of oriented ellipses: Application to facial expression recognition[END_REF] used the Gram matrix computed from landmarks as the shape representation and conducted a non-Euclidean metric learning on the manifold of oriented ellipses centered at the origin in Euclidean n-space. The comparison results show that our method performs better, both in terms of the original metric and the optimal metric. For dynamic approaches, the ranked-first approach is the work of [START_REF] Kacem | A Novel Geometric Framework on Gram Matrix Trajectories for Human Behavior Understanding[END_REF], where the facial landmark sequences were modeled as parametrized trajectories on the Riemannian manifold of positive semi-definite matrices of fixed-rank. However, their metric applied to calculate the similarity between trajectories contains a weight parameter, which needs to be carefully selected for each dataset. According to the performance ranking, the following method [START_REF] Tanfous | Sparse Coding of Shape Trajectories for Facial Expression and Action Recognition[END_REF] encoded time-varying facial shapes and used deep neural networks for classification. Compared with these work, our method using solely the last frame achieves competitive results on facial expression recognition. For Oulu-CASIA dataset, few geometry-based methods are available for comparison. Based on this, we implemented the static method [START_REF] Daoudi | Metric learning on the manifold of oriented ellipses: Application to facial expression recognition[END_REF] using the code provided by the author and the comparison results are shown in Table 6.5. Consistent with the results on CK+, our approach always yields higher accuracy rates, regardless of whether metric learning is used, proving its effectiveness once again for facial expression recognition.

Conclusion

In this chapter, a non-Euclidean metric learning method was proposed and applied to facial expression recognition, where the facial shapes were directly encoded by the equivalent class of the landmarks in the complex projective space. The similarity between them can be measured by Fubini-Study metrics. A metric learning algorithm based on Large Margin Nearest Neighbors (LMNN) was adapted to perform metric optimization on the Riemannian manifold, i.e. the space of positive-definite Hermitian matrices. Facial expression recognition experiments were conducted on two public datasets to verify the validy of the proposed method. The experimental results showed that, compared to the metric defined in the Euclidean space, the Fubini-Study metrics are more effective and discriminative for identification of facial expressions. The latter one also achieves comparable performance to the state-of-the-art approaches. In the next chapter, we will discuss the limitations of the methods involved in this thesis and the perspectives for future research on emotion recognition. Emotion recognition, an important branch of affective computing, has received considerable attention for enabling machines to automatically recognise human emotions. With advances in sensor technologies and human-computer interaction systems, there is a growing trend to migrate from laboratory environments to unrestricted scenarios such as medical simulation training. However, there is still a lack of relevant research on the latter. In order to build emotion recognition systems in real-life settings, from a technical point of view, appropriate intelligent algorithms for emotion recognition should be designed at the behavioural, physiological, and even multimodal levels. From a practical point of view, recognition algorithms need to be integrated into the user interface and the effects of the emotional feedback obtained should be investigated. The work we present in this thesis concentrates on exactly these two aspects. At the end of the thesis, we review the main contributions and their corresponding limitations in Sections 7.1 and 7.2, and we discuss the perspectives for future work in Section 7.3. To obtain a generalized model, we adopted a self-supervised learning (SSL) scheme. The proposed model was first pre-trained on a large-scale unlabelled physiological dataset with the pretext objective of signal transformation classification. A series of downstream emotion recognition tasks involving mental stress, emotional state, arousal and valence were then performed to validate the effectiveness of the pre-trained model. Finally, our approach achieved stateof-the-art results in comparison with fully-supervised, unsupervised, and self-supervised-based methods.

Contributions

From the behavioural perspective, We proposed a non-Euclidean metric learning method for the facial expression recognition task. We encoded facial shapes as equivalence classes in the complex projective space using the complex form of detected 2D landmarks, resulting in an affine-invariant shape representation. A metric learning algorithm based on Large Margin Nearest Neighbors (LMNN) was applied to optimize the metrics in the non-linear space, thereby yielding a feature space with smaller intra-class differences and larger inter-class differences. The proposed method finally obtained superior performance to Euclidean metric learning methods and yielded competitive results in comparison with state-of-the-art methods.

From the multimodal perspective, we presented a novel deep geometric framework for multimodal stress and pain detection tasks. To fully explore the potential interactions between different modalities, we fused behavioural and physiological information into a single symmetric positive definite (SPD) matrix-based representation that incorporates both intra-and intermodal correlations. To cope with the non-linearity of the manifold induced by the SPD matrices, we employed a tangent space mapping technique to map the resulting SPD matrix sequences into vector sequences in linear space, enabling the use of an LSTM-based deep neural network for classification. To the best of our knowledge, this is the first use of the geometry of the SPD matrix for fusing physiological and behavioural signals. This novel fusion strategy allows the proposed method to achieve improved performance in comparison with feature-level or decision-level fusion-based methods.

Practical level

We conducted emotion recognition experiments in a real-life setting, i.e., high-fidelity medical simulation, where multimodal data consisting of peripheral physiological and action signals and a series of psychological scales were collected from students. The evaluation results of the experiments confirmed the validity of the proposed algorithms in Chapters 3 and 4, which were then integrated into a graphical user interface to explore its impact on the pedagogical debriefing sessions. Ultimately, the users' feedback showed that emotion-aware teaching can facilitate students' acquisition of competencies and increase their engagement.

Limitations

Technical level

From the physiological perspective, first, we stacked the embedding sequences from the output of modality-specific encoders as inputs to the transformer, expecting to learn inter-modal dependencies through the self-attention mechanism. However, since the complexity of the transformer is O(N 2 ) at an input sequence length of N , this natural concatenation and dense cross-modal attention operation entail high computational costs, especially for large-scale learning [START_REF] Ding | Sparse fusion for multimodal transformers[END_REF]. Secondly, regarding the pretext task of self-supervised learning, we adopted a fixed number of signal transformations to learn generalised representations for downstream tasks.

However, there may be an optimal combination strategy for signal transformations and an optimal set of parameter settings for these transformations. Furthermore, the optimal transformation policy probably varies from one modality to another. Third, the self-supervised pretraining process suffers from an early de-generalisation problem, where the model possibly converges too fast on the discriminative pretext task thereby losing its generalisation in downstream tasks [START_REF] Liu | Self-supervised learning: Generative or contrastive[END_REF].

From the behavioural perspective, our proposed method has two major limitations. First, it is based on static facial landmarks and therefore only takes into account spatial information. However, the execution of facial expressions is a dynamic process. Thus, the method neglects to encode the temporal changes of facial point positions. Secondly, recent methods have attempted to apply 3D facial landmarks for the identification of facial expressions, which can better tackle the distortions introduced by head pose variations and obtain more meaningful distance measurements. However, the proposed method cannot be extended to the 3D case as it is not possible to express the coordinates of 3D points in complex form.

From the multimodal perspective, there are two main limitations in our approach. First, the multimodal signals are temporally inconsistent, which is manifested by dense continuous physiological signals and relatively sparse video images. We adopted the most intuitive approach, a downsampling techinique to match multimodal data sizes for the computation of the proposed SPD-based representation. However, this may distort the correspondence between instances of multimodality [START_REF] Baltrušaitis | Multimodal machine learning: A survey and taxonomy[END_REF]. Second, to address the non-linearity of the SPD manifold, we adopted an intrinsic solution of projecting the manifold data onto a tangent space at some reference point via a logarithmic map. However, this may introduce approximation errors, especially when the projected point lies further away from the reference point.

Practical level

First, in terms of ground truth collection, a series of self-report questionnaires were applied to measure emotional states. However, they inevitably have some limitations. They only record emotions at a particular moment in time, rather than providing a continuous measure, and there exist recall biases in the post hoc collection of subjective experiences. In addition, there may be a delay between the real emotional experience and the emotion perceived by the individual.

All of these factors affect the quality of the labeling. Second, from the perspective of the recognition algorithm, our approach employed a binary classification policy for the target emotion, i.e. stress. However, given the complexity and non-linearity of human emotional responses, differentiating only between stressed and unstressed appears to be ineffective in capturing the subtle evolution of emotional states.

Future work 7.3.1 Technical level

From the physiological perspective, for transformers with stacked multimodal features as input, one of the future research lines is how to reduce their computational complexity. One possible solution is to adopt a sparse attention mechanism, which measures similarity only for certain key positions in a sequence, and the resulting sparse attention matrix allows for reduced computational time and memory requirements. Furthermore, in order to maximise the generalisation of the representations learned by the self-supervised model, a scheme that can automatically learn the optimal set of signal transformations for each modality needs to be designed for its integration into the original framework. Moreover, the same idea could be applied to the design of upstream pretext tasks, as sometimes learned features on the basis of the adopted pretext task show poor transferability on downstream data.

From the behavioural perspective, the proposed method can be extended to dynamic scenarios. Sequences of landmarks will be explored to model the temporal relationship, where successive landmark configurations are connected by geodesic to create a parameterized curve in the complex projective space and the final classification will be executed based on the comparison of similarity between the curves. In addition, we will also explore the application of other types of landmarks, such as body landmarks, for emotion recognition. Body movements involve larger muscle groups with distinct changes, whereas facial motions tend to be more subtle and may be difficult to discern. In such cases, the use of body landmarks as an emotion recognition modality may provide more information than facial landmarks. Moreover, body landmarks are frequently involved in another interesting direction of research, namely action recognition.

From the multimodal perspective, in order to better address the inconsistency on the time scale of multimodal data, an implicit alignment module can be introduced into our proposed model to obtain better performance. Different backbones can first be applied to the physiologi-cal and behavioural modalities respectively to extract low-level features, followed by the calculation of the similarity between the sub-components to align the features from different modalities before proceeding to fusion. Furthermore, future work could develop deep neural networks suitable for non-linear SPD matrices in order to avoid distortion of the manifold data due to tangent space mapping. Recent studies [START_REF] Huang | A riemannian network for spd matrix learning[END_REF][START_REF] Wang | Dreamnet: A deep riemannian network based on spd manifold learning for visual classification[END_REF] have proposed end-to-end deep Riemannian architectures that can be directly applied to SPD matrices. An example is SPDNet [START_REF] Huang | A riemannian network for spd matrix learning[END_REF], which is inspired by classical convolutional neural networks, where a bilinear mapping layer and an eigenvalue rectification layer were built to simulate convolutional and linear rectification operations.

Practical level

First, in order to enhance the reliability of the acquired emotion's ground truth, assessments from a third party such as a trainer or a patient can be incorporated in the data collection procedure. Second, in order to optimise the quality of pedagogy, a multi-level emotion classification system should be developped. A more optimal solution would be performing regression tasks to predict arousal and value scores, allowing to obtain a more comprehensive, continuous emotional evolution of the trainee during the medical simulation. In addition, person-specific emotion recognition models can be also deployed to facilitate individualised teaching programmes for students. Third, during the tests of the developed graphical interface, we revealed correlations between the recognized stress fragments and verbal expressions, which guided us to incorporate speech signals in future recognition frameworks to achieve more accurate and robust performance.

I

  extend my heartfelt thanks to the Presage Simulation Center and I-SITE in Lille, France for their financial support of this thesis project with reference ANR-16-IDEX-0004 ULNE. I would like to thank Dr. Thibaut Denis from the Faculty of Medicine of the University of Lille and all the staff of the Presage Simulation Center for their outstanding efforts and contributions to the acquisition and analysis of real-world data for my research. Through collaboration List of Figures 1.1 The correlation between arousal levels and performance described in Yerkes-Dodson law [1]. Excerpt from [2]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Two types of generic emotion models. . . . . . . . . . . . . . . . . . . . . . . 2.2 Common emotion-related behavioral and physiological modalities and their corresponding data collection devices. . . . . . . . . . . . . . . . . . . . . . . 2.3 Emotion recognition pipelines based on peripheral physiological signals. . . . 2.4 Deep learning-based fusion strategies for physiological emotion recognition. a) Early fusion employs concatenated multimodal data as input and extracts efficient inter-modal correlation information through joint learning. b) Intermediate fusion first learns marginal representations for each modality independently, which are then combined and fed directly into the decision layer or continue to learn advanced joint representations. c) Late fusion aggregates decisions derived from marginal representations of unimodal data to generate final decisions. (NN: Neural Network; NN (opt): optinal neural network for joint leanring.) 2.5 Typical early fusion-based emotion recognition architectures. a) Fully connected neural network can be applied directly to multimodal vectors when the temporal dependencies between data samples are ignored. b) Convolutional neural networks can compute temporal dependencies on local scales in 2D multimodal inputs in a parallel manner. c) Recurrent neural networks capture longterm dependencies in physiological sequences in a sequential manner. . . . . . 2.6 Common intermediate fusion strategies for physiological emotion recognition. a) Integrated marginal representation is applied directly to decision making. b) The marginal representations are first combined to jointly learn a high-level multimodal representation for better exploiting inter-modal correlations, which is later fed into the decision layer. . . . . . . . . . . . . . . . . . . . . . . . . xi 2.7 Common facial landmark configurations in the literature, where the 66 pointsbased configuration has 17 more landmarks (i.e. the blue dots) on the face contour than the 49 points-based configuration, and the 68 points-based configuration is formed by adding 2 additional landmarks (i.e. the green dots) on the inner corner of the mouth to the 66 points-based configuration. . . . . . . . . . 2.8 Landmark-based facial expression recognition pipeline. . . . . . . . . . . . . . 2.9 The cone of SPD matrices. Points A, B and C represent covariance-based shape representations, where the blue and red lines symbolize distance measurements in Euclidean space and Riemannian manifold, respectively. It can be seen that in the manifold the point A is closer to the point B, while in Euclidean space it has the opposite result. Excerpt from [3]. . . . . . . . . . . . . . . . . . . . . . 2.10 Hybrid fusion strategies for multimodal emotion recognition. a) Physiological and behavioural features are first merged within segmented data and their corresponding predictions are aggregated for the final decision [4]. b) Decisions of physiological and behavioural modalities are combined with unimodal features for emotion detection [5]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Overview of our self-supervised multimodal representation learning framework. The proposed SSL model is first pre-trained with signal transform recognition as the pretext task to learn generalized multimodal representation. The encoder part of the resulting pre-trained model is then served as a feature extractor for downstream tasks which is frozen or fine-tuned on the labeled samples to predict emotion classes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 The original EDA signal and the disturbed EDA signals after applying five transformations. For each modality, the raw signal data and the transformed signal data are stacked and fed into the proposed SSL model for multimodal representation learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Modality-specific backbone based on temporal convolutional network (TCN). Each backbone consists of two residual blocks for capturing low-level features for transformed unimodal signals x ′ m . (k: kernel size, f: number of filters, d: dilation factor, p: padding size, s: stride size, weightnorm: weight normalization for convolution filters) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Shared encoder based on the multimodal transformer. (FC: fully-connected layer with 128 units, LN: layer normalization) . . . . . . . . . . . . . . . . . . xii 3.5 Modality-specific classification head C m for signal transformation recognition task. (GAP: 1D global average pooling, FC: fully-connected layer, BatchNorm: batch normalization, num class: number of signal transformations, i.e., 6 in our work.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Images of different scenarios captured by cameras placed in the simulation training room: (a): Doctor consultation, (b): Prevention of escape for patients in an acute agitated state, (c): Second consultation for patients with suicidal tendencies, (d): Management of cardiac arrest/severe head injury/chest trauma, (e): Diagnostic announcement and (f): the wearable sensor Empatica E4 wristband used for physiological data collection during the simulation training. . . . . . . 3.7 Performance comparison with state-of-the-art supervised learning-based methods on limited labeled data sampled from the three emotion recognition datasets.

4. 2

 2 Exponential map and logarithm map between the Riemannian Manifold M and the tangent space T p M at P. . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 An example of 2D texture images/3D model sequences/thermal images from the Pain class and their corresponding facial landmarks provided in BP4D+ dataset.

xiii 4 . 4 6 . 2 3 . 4

 446234 The multimodal SPD representation generated by a pain sample in the BP4D+ dataset, where correlations within and across two modalities (i.e., vision and physiology) can be observed. (D1-D10: 10 distances automatically selected based on Anova F-value, BP: raw blood pressure, BPDia: diastolic blood pressure, LAS: systolic blood pressure, LAM: mean blood pressure, EDA: electrodermal activity, HR: heart rate, RR: respiration rate, and RV: respiration volts.) 4.5 (a) Stress detection performance on WESAD dataset and (b) pain detection performance on BP4D+ dataset using the sample covariance matrix S, the crosscovariance matrix C and the proposed SPD representation P. (Acc: Accuracy, F1: F1-score) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 The wearable device used in this study: E4 wristband and collected raw multimodal signals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Experimental protocol. The grey part with grid lines represents the subject's self-evaluation, which was further used as ground truth for emotion recognition task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Self-Assessment Manikins. Excerpt from [6]. . . . . . . . . . . . . . . . . . . 5.4 Analysis of Self-Assessment Manikin (SAM) data. . . . . . . . . . . . . . . . 5.5 Analysis of State-Trait Anxiety Inventory (STAI) data before and after simulation training. (The green triangle symbol represents the mean value.) . . . . . . 5.6 The graphical interface with the built-in SSL-based stress detection algorithm applied in debriefing phase after simulation training. . . . . . . . . . . . . . . 5.7 Visualization of the subjective assessment of the task difficulty and the level of competence at three moments: pre-simulation, post-simulation and postdebriefing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 Visualisation of trainees' feedback regarding the application of the stress analysis tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Overview of the proposed approach. (a) 2D landmarks detected in images of facial expressions are first converted to vectors in complex space; (b) The variants obtained by affine transformation constitute the equivalence class of landmarks, which can be considered as the same point in the complex projective space; (c) The application of metric learning enables the reduction of intra-class differences while enlarging inter-class distances. Finally, the learned metric is used for similarity-based classification. . . . . . . . . . . . . . . . . . . . . . . . . xiv An illustration of 2D facial landmark configurations and its corresponding shape space in the facial expression recognition scenario. Matching and comparison of facial shapes in the complex projective space is not disturbed by traditional affine transformations: translation, rotation and scaling. . . . . . . . . . . . . . 6.3 An illustration of the LMNN algorithm modified from [7]. The proposed Fubinistudy metric will be optimised such that the distances between each sample z i and its k = 3 target neighbours are reduced, while the distances to the impostors are increased. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Sample images of facial expressions extracted from the CK+ [8] and Oulu-CASIA [9] datasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Average classification accuracy on the CK+ and Oulu-CASIA datasets using the Fubini-Study metric when varying the target neighbor parameter k. . . . . . 6.6 2D visualization of CK+ dataset using t-SNE method with the proposed metrics. (a-b) show the visualization results before and after metric learning with Fubini-Study metric. (c-d) show the visualization results before and after metric learning with Euclidean metric. . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 2D visualization of Oulu-CASIA dataset using t-SNE method with the proposed metrics. (a-b) show the visualization results before and after metric learning with Fubini-Study metric. (c-d) show the visualization results before and after metric learning with Euclidean metric. . . . . . . . . . . . . . . . . . . . . . xv List of Tables 2.1 Overview of supervised machine-learning based methods for physiological emotion recognition. (EF: early fusion; LF: late fusion.) . . . . . . . . . . . . . . . 2.2 Overview of supervised deep learning-based methods for physiological-based emotional recognition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Overview of other learning paradigm-based methods for physiological emotion recognition. (PPS: a series of Peripheral Physiological Signals, SAE: stacked autoencoder; SDAE: stacked denoising autoencoders, C intra/inter : columns used to indicate whether the deep model takes into account intra-and intermodal correlations in the pre-training phase.) . . . . . . . . . . . . . . . . . . . . . . 2.4 Overview of behavioral emotion recognition methods.(FCN: Fully Connected Network) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Overview of behaviroual-physiological emotion recognition methods. (ACC: 3-axis Acceleration, FE: Facial Expressions, PPS: Peripheral Physiological Signals, P+F: Physiology+Face, P+M: Physiology+Body Motion.) . . . . . . . . . 3.1 The learning tasks assigned to each dataset and the corresponding distribution of samples between classes in different datasets. (P: Pretext task, D: Downstream task.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Performance comparison of different emotion recognition tasks with state-ofthe-art methods on the WESAD dataset. (Acc: Accuracy, F1: F1-score, SL: supervised learning methods, UL: unsupervised learning methods, SSL: selfsupervised learning methods, S: supervised, F: frozen, T: fine-tuned.) . . . . . . 3.3 Performance comparison of different emotion recognition tasks with state-ofthe-art methods on the CASE dataset. (Acc: Accuracy, F1: F1-score, SL: supervised learning methods, UL: unsupervised learning methods, SSL: selfsupervised learning methods, S: supervised, F: frozen, T: fine-tuned.) . . . . . . xvi Performance comparison of different emotion recognition tasks with state-ofthe-art methods on the K-EmoCon dataset. (Acc: Accuracy, F1: F1-score, SL: supervised learning methods, UL: unsupervised learning methods, SSL: selfsupervised learning methods, S: supervised, F: frozen, T: fine-tuned.) . . . . . . 3.5 Ablation study of different fusion strategies: average accuracy and F1-score obtained for emotion recognition on WESAD, CASE and K-EmoCon dataset using different variant model. (S-2: Stress-2, E-3: Emotion-3, V-2: Valence-2, A-2: Arousal-2, Inter w/ ol: Intermediate fusion with an overall loss.) . . . . . 3.6 Ablation study of different modalities and their combinations: average accuracy and F1-score obtained with different modality combinations in the downstream emotion recognition tasks, where the best performing individual modality and bimodal combinations for each task are underlined. (S-2: Stress-2, E-3: Emotion-3, V-2: Valence-2, A-2: Arousal-2.) . . . . . . . . . . . . . . . . . . 3.7 Ablation study of different model components: average accuracy and F1-score obtained for emotion recognition on WESAD, CASE and K-EmoCon dataset using different variant model. (S-2: Stress-2, E-3: Emotion-3, V-2: Valence-2, A-2: Arousal-2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Ablation study of individual signal transformations and their combinations: average accuracy and F1score obtained for emotion recognition on WESAD, CASE and K-EmoCon datasets using different transformations in self-supervised pertaining, where the best performing transformations and combinations of transformations in each task are underlined. (N: Noise addition, M: Magnitudewarping, P: Permutation, T: Time-warping, C: Cropping.) . . . . . . . . . . . 4.1 Stress detection performance of uni-modal setting (only physiology) and multimodal setting (physiology + motion) on WESAD dataset using the proposed SPD representation P. (Acc: Accuracy, F1: F1-score, ↑ (↓): multimodal performance is improved (decreased) compared to the unimodal one.) . . . . . . . . . 4.2 Comparison with State-of-the-art Methods on WESAD dataset (Stress vs. Nonstress) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Pain detection performance of uni-modal setting (only physiology) and multimodal setting (physiology + vision) on BP4D+ dataset using the proposed SPD representation P. (2D/3D/Thermal: 2D/3D/Thermal facial landmarks, Physio: all physiological signals, Acc: Accuracy, F1: F1-score, ↑ (↓): multimodal performance is improved (decreased) compared to the unimodal one.) . . . . . . . 4.4 Comparison with State-of-the-art Methods on BP4D+ dataset (pain vs. non-pain).xvii 5.1 A summary of pre-processed real-life data. . . . . . . . . . . . . . . . . . . . . 5.2 List of multimodal features for machine learning-based emotion recognition. . . 5.3 Performance comparison of different emotion recognition tasks with state-ofthe-art methods on the Presage dataset. (Acc: Accuracy, F1: F1-score; B:Behaviour, P:Physiology, B+P:Behaviour+Physiology.) . . . . . . . . . . . . . . . . . . . 6.1 Overall accuracy (Acc %) of of two types of metrics, i.e., the Euclidean metrics and the Fubini-Study metrics, before and after LMNN-based metric learning on the CK+ and Oulu-CASIA datasets. ([•] I : identity matrix; [•]

6. 5 125 xix 1

 51251 Comparison with state-of-the-art geometric methods on the Oulu-CASIA dataset.132 xviii List of Algorithms 4.1 Classification of SPD matrix sequences . . . . . . . . . . . . . . . . . . . . . . 87 6.1 Metric learning in Complex Projective Space CP n-1 . . . . . . . . . . . . . . . Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Thesis contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Technical level Emotion recognition algorithms based on physiological, behavioural and multimodal signals are proposed. Thus, the corresponding contributions are three-fold, depending on the type of modality used: • From the physiological viewpoint, we proposed a self-supervised wearable emotion recognition based on peripheral physiological signals. First, in contrast to most machine learning-based physiological recognition methods, a deep neural network based on residual temporal convolution and transformer was designed to (1) automatically capture more discriminative abstract features, and (2) flexibly fuse multimodal features at different levels, allowing the encoding of intra-and inter-modal correlation information.

  Chapters 3 and 4 correspond to our technical contributions on the physiological and multimodal sides, respectively, where Chapter 3 presents a self-supervised learning approach for emotion recognition based on peripheral physiological signals, and Chapter 4 introduces a deep geometric framework based on SPD representation for multimodal stress and pain detection. Chapter 5 refers to the practical contributions, including the collection of multimodal real-life data, the development and application of the graphical interface, and the pedagogical relevance analysis of emotion recognition. Regarding the technical contributions on the behavioral side, a landmark-based non-Euclidean metric learning approach for facial expression recognition is presented in Chapter 6. Finally, in Chapter 7, we summarise the work presented in this thesis and discuss its corresponding limitations and suggest potential future work.

  Russell's 2D circumplex model based on valence and arousal.

Figure 2 . 1 :

 21 Figure 2.1: Two types of generic emotion models.

Fig. 2 .Figure 2 . 2 :

 222 Figure 2.2: Common emotion-related behavioral and physiological modalities and their corresponding data collection devices.

Figure 2 . 3 :

 23 Figure 2.3: Emotion recognition pipelines based on peripheral physiological signals.

Figure 2 . 5 :

 25 Figure 2.5: Typical early fusion-based emotion recognition architectures. a) Fully connected neural network can be applied directly to multimodal vectors when the temporal dependencies between data samples are ignored. b) Convolutional neural networks can compute temporal dependencies on local scales in 2D multimodal inputs in a parallel manner. c) Recurrent neural networks capture long-term dependencies in physiological sequences in a sequential manner.

  proposed an intelligent remote Internet of Things (IoT) framework for real-time emotion recognition, where wearable devices and IoT technology allowed for wireless data collection and communication, and six peripheral physiological signals were fed into a deep LSTM-based model to classify four categorical emotions, achieving a high performance (F1-score) of 95 %. Zitouni et al. [77] applied a Bidirectional LSTM-based (BiLSTM) architecture for emotion detection in real-life debates using three peripheral signals: EDA, HR and TEMP. In comparison with the baseline models, i.e. Gaussian Naive Bayes (GNB) and eXtreme Gradient Boosting (XGBoost), the proposed model obtained the best performance with an average accuracy of 90.79% for arousal, 90.53% for valence and 86.18% for 4-class classification. Although RNNs have performed well in the above studies, they still exhibit some limitations. Theoretically, RNNs can model arbitrarily long sequences, However in practice, vanilla RNN, including LSTM face vanishing gradient problem when dealing with extra long series. Futhermore, the sequential processing makes the model training extremely time consuming compared to CNNs.

  effectively solve this problem by first learning marginal representations to capture intra-modal correlations, thereby injecting the priori knowledge into the model. Subsequently, these bottom marginal representations are either fed directly into the decision layer (i.e., marginal intermediate fusion, presented in Fig. 2.6 (a)) or jointly learn a higher-level representation for prediction (i.e., joint intermediate fusion, presented in Fig. 2.6 (b)). Later on, we will discuss emotion recognition methods based on these two intermediate fusion strategies. -Marginal In marginal intermediate fusion, representations extracted from different modalities can be directly concatenated as input to the classification. Ma et al. [79] proposed a multimodal residual LSTM (MMResLSTM) network for emotion recognition. The EEG signal and other peripheral signals: EOG (electrooculography) and EMG (electromyography) are fed separately into four-layer LSTM based model interspersed with residual connections and layer

Figure 2 . 6 :

 26 Figure 2.6: Common intermediate fusion strategies for physiological emotion recognition. a) Integrated marginal representation is applied directly to decision making. b) The marginal representations are first combined to jointly learn a high-level multimodal representation for better exploiting inter-modal correlations, which is later fed into the decision layer.

  types, namely appearance-based methods and geometry-based methods. The former encodes the pixel intensity information of a still image or video frame, while the latter analyses the location information of detected facial key points around the main facial components such as eyebrows, eyes, nose and lips in static images or videos. These detected 2D or 3D points are known as facial landmarks and can be considered as a compact representation of a facial shape. Compared with appearance-based methods, facial expression recognition methods based on landmarks have several advantages: first, they are faster and more efficient for long-term emotion recognition in real-world scenes since they only require the tracking of low-dimensional point sets. The number of facial landmarks used in different methods may vary, with the most common being 49 points[START_REF] Xiong | Supervised descent method and its applications to face alignment[END_REF][START_REF] Baltrušaitis | 3d constrained local model for rigid and non-rigid facial tracking[END_REF], 66 points[START_REF] Asthana | Robust discriminative response map fitting with constrained local models[END_REF][START_REF] Yu | Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model[END_REF], or 68 points[START_REF] Tzimiropoulos | Optimization problems for fast aam fitting in-thewild[END_REF][START_REF] Ren | Face alignment at 3000 fps via regressing local binary features[END_REF]. An illustration is shown in Fig.2.7. This is computationally less expensive than analyzing pixel intensity information. Second, they can avoid the privacy issues typically related to appearance-based methods, which require storage or transmission of the original facial images. Thus, landmarkbased methods offer a more privacy-friendly option for practical applications. Third, landmark detection and tracking techniques are relatively insensitive to illumination variations in images, and recent methods are even robust to occlusions. Therefore, we only considered the facial landmark-based expression recognition approach in our work. a) 49 points b) 66 points c) 68 points

Figure 2 . 7 :Fig. 2 .

 272 Figure 2.7: Common facial landmark configurations in the literature, where the 66 pointsbased configuration has 17 more landmarks (i.e. the blue dots) on the face contour than the 49 points-based configuration, and the 68 points-based configuration is formed by adding 2 additional landmarks (i.e. the green dots) on the inner corner of the mouth to the 66 pointsbased configuration.

4 . 1 .Figure 2 . 8 :

 4128 Figure 2.8: Landmark-based facial expression recognition pipeline.

  Zhang et al.[START_REF] Zhang | Facial expression recognition based on deep evolutional spatial-temporal networks[END_REF] proposed a Part-based Hierarchical Bidirectional Recurrent Neural Network (PHRNN) for identifying facial expressions. The facial landmarks were first divided into 4 parts based on facial regions: eyes, eyebrows, nose, and mouth. Then, the landmark sequences of each region were processed by independent Bidirectional Recurrent Neural Network (BRNN) models to extract local features. A cascade fusion strategy was introduced, where the upper facial representation was obtained by concatenating the eyebrow-eye-nose features, while the bottom facial representation was a combination of nose-mouth features. The global representation was finally derived from the upper and lower facial representations for efficient facial expression classification.

  also employed a deep architecture consisting of fully connected layers. Facial movements were encoded by landmark trajectories, which are concatenations of 1D coordinate vectors of all frames in the facial videos and served as input to the model for facial expression recognition. Though the above approaches can provide a more holistic and global understanding of facial expressions, however, they ignore the geometric properties of facial shapes, thus failing to accurately capture the deformations of the face, especially when dealing with complex expressions.

Figure 2 . 9 :

 29 Figure 2.9: The cone of SPD matrices. Points A, B and C represent covariance-based shape representations, where the blue and red lines symbolize distance measurements in Euclidean space and Riemannian manifold, respectively. It can be seen that in the manifold the point A is closer to the point B, while in Euclidean space it has the opposite result. Excerpt from [3].

  To date, most emotion recognition studies have focused on behavioral or physiological responses. Recently, a few studies have focused on fusing explicit behavioral signals such as facial expressions and body motions represented by ACC (3-axis acceleration) signals and implicit physiological signals to leverage the complementary nature of both to provide more robust and enhanced emotion detection. In this section, we classify these multimodal methods into early fusion-based (Section 2.5.1), intermediate fusion-based (Section 2.5.2), late fusion-based (Section 2.5.3), and hybrid fusion-based (Section 2.5.4) according to the integration fashion.

  suggested a facial-EEG based emotion recognition system. The spectral energies of the three frequency bands are first extracted from the EEG signals and transformed into image sequences. Then the facial and EEG image sequences are fed separately into independent fully connected neural networks to extract marginal representations. A hierarchical LSTM model with self-attention was applied to fuse and learn more abstract multimodal representations, which achieved the state-of-the-art results on the emotion classification task. Huynh et al. [69] employed an auto-designed deep neural network based on Neural Architecture Search (NAS) for affective states and stress detection. Filter banks and mixed features of physiological signals and the body motion signal (i.e., ACC signal) were extracted to search for the highest scoring architectures from 10,000 randomly generated neural network candidates for training of individual modalities. The obtained unimodal representations were concatenated for final classification.

Figure 2 . 10 :

 210 Figure 2.10: Hybrid fusion strategies for multimodal emotion recognition. a) Physiological and behavioural features are first merged within segmented data and their corresponding predictions are aggregated for the final decision [4]. b) Decisions of physiological and behavioural modalities are combined with unimodal features for emotion detection [5].

  phy (EEG) signals and a series of peripheral signals. However, the acquisition of EEG data is challenging for implementation in real-life scenarios. With the advance of non-invasive technologies, emotion recognition methods based on multiple peripheral signals captured by smartphones/wearable watches have attracted some attention. Traditional wearable multimodal emotion recognition used hand-crafted features as input of machine learning-based algorithms for classification. Nevertheless, these well-designed features based on comprehensive domain knowledge can be an obstacle for non-domain experts. Most recent researches focus on deep neural networks, which can automatically extract complex patterns from multimodal signals.

  targeted low-frequency wearable peripheral signals, but they ignored the correlation between multimodal signals. In this paper, we propose a self-supervised multimodal representation learning approach for wearable emotion recognition based on peripheral physiological signals. The first stage is model pre-training with the pretext task based on signal transformation recognition, where a large amount of unlabeled multimodal data are automatically assigned labels through a series of transformations. Considering the heterogeneity of multimodal signals, temporal convolution-based modality-specific encoders are first employed separately on the transformed unimodal data to extract low-level features, followed by a transformer-based shared encoder deployed to aggregate unimodal features, enabling the modeling of complementary and collaborative properties between multimodal signals. Finally, modality-specific signal transformation recognition is performed to learn effective multimodal representations for downstream tasks that are robust to perturbations in magnitude or temporal domains. The second stage is supervised emotion recognition, where the SSL pre-trained encoder part is retained

Figure 3 . 1 :

 31 Figure 3.1: Overview of our self-supervised multimodal representation learning framework.The proposed SSL model is first pre-trained with signal transform recognition as the pretext task to learn generalized multimodal representation. The encoder part of the resulting pretrained model is then served as a feature extractor for downstream tasks which is frozen or fine-tuned on the labeled samples to predict emotion classes.

3 . 1 .

 31 In our work, three modalities measured by different sensors are considered: electrodermal activity (EDA), blood volume pressure (BVP) and skin temperature (TEMP).More formally, let x m ∈ R N ×1 represent a 1D time-domain signal from one of the M different modalities (in our work, M = 3), where N is the signal length. Given a set of n transform functions T = {T j (•), j ∈ {1, . . . , n}}, the altered multimodal signal dataset can be generated by applying each transformation to individual modality. Based on this, one can easily build a pseudo-labeled dataset L = {(T j (x i m ), y i ), y i ∈ {1, . . . , n}, m ∈ {1, . . . , M }, i ∈ [1, |L|]} for unlabelled samples through self-supervision enabled by signal transformations. Then, the proposed model consisting of a multimodal encoder E and modality-specific classifiers C is pre-trained to predict the type of transformation applied to samples in L. Ultimately, only the encoder part E of the optimal model obtained after pre-training is retained and is expected to produce generalized multimodal representations in a variety of supervised downstream tasks. Details of the proposed SSL framework are as follows.

3. 3 . 2 Figure 3 . 2 :

 3232 Figure 3.2: The original EDA signal and the disturbed EDA signals after applying five transformations. For each modality, the raw signal data and the transformed signal data are stacked and fed into the proposed SSL model for multimodal representation learning.

3. 3 . 2 .

 32 B Self-supervised multimodal representation learning network architectureThe proposed SSL multimodal deep neural network consists of two key elements, namely the encoder E and the modality-specific transformation classifiers C. The encoder E can be further subdivided into temporal convolution-based modality-specific encoders E p and transformerbased shared encoder E s , where E p models the heterogeneity of multimodal signals and E s activates cross-modal information exchange. Ultimately, the multimodal features obtained from the encoder are used as input to C for identifying transformation types for each modality. The implementation of these key components is described in the following paragraphs. Modality-specific encoder: Considering the heterogeneity of the multimodal signals, separate encoders are first employed for each modality, with a temporal convolution-based network acting as the backbone to capture low-level intra-modal correlation information. The temporal convolutional network (TCN)[START_REF] Bai | An empirical evaluation of generic convolutional and recurrent networks for sequence modeling[END_REF], in a nutshell, is a combination of dilated causal convolution and residual connections, with parallel computational capability and robust gradients at optimization, thus demonstrating better performance than traditional recurrent networks, such as LSTM and GRU. One basic TCN consists of several residual blocks. The most central components of each block are two dilated causal convolution layers. The causality can be easily achieved when the output at the current moment t depends only on the elements of the past historical moments up to t in the previous layer. Meanwhile, the dilation operation injects holes in the standard convolution map, thereby increasing the reception field. More formally, given the transformed 1D signal of modality m: x ′ m = T j (x m ) ∈ R N ×1 with N time steps, and a filter f of size k, the dilated convolution on time step t can be defined as

Figure 3 . 3 :

 33 Figure 3.3: Modality-specific backbone based on temporal convolutional network (TCN). Each backbone consists of two residual blocks for capturing low-level features for transformed unimodal signals x ′ m . (k: kernel size, f: number of filters, d: dilation factor, p: padding size, s: stride size, weightnorm: weight normalization for convolution filters) embedding space. Finally, the output of the modality-specific encoder E p is:

Fig. 3 .Figure 3 . 4 :

 334 Figure 3.4: Shared encoder based on the multimodal transformer. (FC: fully-connected layer with 128 units, LN: layer normalization)

Figure 3 . 5 :

 35 Figure 3.5: Modality-specific classification head C m for signal transformation recognition task. (GAP: 1D global average pooling, FC: fully-connected layer, BatchNorm: batch normalization, num class: number of signal transformations, i.e., 6 in our work.)

The

  PRESAGE dataset is a large-scale multimodal physiological signal dataset for emotion analysis. The data acquisition is done at the Presage training center 2 in Lille, France, whose mission is to ensure the training of medical students and health professionals through immersion in a recreated hospital environment, where the high-tech mannequins or hired actors, take the place of the patients and students act as doctors. In order to analyze the students' emotional state during the simulation training to optimize the educational program, a large amount of unlabeled multimodal physiological data has been collected from 201 trainees (104 males and 97 females) during five different medical simulation scenarios. Fig. 3.6 (a-e) shows the images of different scenarios captured by the cameras installed in the simulation room. The data collection protocol was approved by the Institutional Review Board of University of Lille

Figure 3 . 6 :

 36 Figure 3.6: Images of different scenarios captured by cameras placed in the simulation training room: (a): Doctor consultation, (b): Prevention of escape for patients in an acute agitated state, (c): Second consultation for patients with suicidal tendencies, (d): Management of cardiac arrest/severe head injury/chest trauma, (e): Diagnostic announcement and (f): the wearable sensor Empatica E4 wristband used for physiological data collection during the simulation training.

Figure 3 . 8 :

 38 Figure 3.8: Different architectures used in the ablation studies of fusion strategies. (GAP: 1D global average pooling applied before classification.)

Figure 3 . 9 :

 39 Figure3.9: Evaluation results of the robustness of the SSL methods in the presence of missing modalities. The horizontal axis of each subplot represents the name of the missing modality, while the vertical axis represents the drops in model performance compared to the case of complete modalities, where the metrics of the vertical axes in the first and second rows are accuracy and F1-score, respectively. (ns: no significant difference; * : p < 0.05, the more asterisks, the more significant the difference.)

3. 5 . 6 .

 56 D Ablation study of different model components We also investigate the impact of different model components on the performance of downstream classification tasks. To validate the contributions of the modality-specific encoder and the shared encoder, we designed two alternative models: No TCN and No Transformer. No TCN eliminates the temporal convolution network (TCN) where unimodal data is passed directly through the projection layer (i.e. a fully connected layer with 128 units) in the modalityspecific encoder shown in Fig. 3.3 and the resulting unimodal low-level features are then concatenated as a whole and fed into the transformer. No Transformer removes the multimodal transformer, where unimodal features are first extracted by modality-specific encoders and then averaged along the time dimension by the 1D global average pooling (illustrated in Fig. 3.5) for the final classification tasks. Table 3.7 present the classification results of the above two variant models on the three datasets. Our proposed model enhances both the performance of No TCN and No Transformer models on all classification tasks across different datasets, highlighting the importance of capturing the heterogeneity and cross-modal correlation of multimodal signals simultaneously. Subsequently, we examined whether the addition of positional encoding could lead to better performance for the transformers with heterogeneous embedding as input. We employed two types of positional encoding (PE): With fixed PE and With learnable PE in the transformer and compared their performance with our PE-free model. With fixed PE added the fixed positional encoding obtained from sine and cosine functions of different frequencies as

3. 6 Conclusion

 6 In this paper, we have proposed a self-supervised multimodal representation learning framework for emotion recognition based on physiological signals. Signal transformation recognition is defined as a pretext task, where a large amount of unsupervised data is automatically labeled by the imposed signal transformation category for pre-training of the SSL model. Subsequently, the encoder part of the pre-trained model consisting of a temporal convolution network and transformer is maintained to extract effective multimodal representations for the downstream task, i.e. emotion recognition. Eventually, we executed the pre-training on a large-scale unrestricted emotion dataset PRESAGE and verified the validity of the proposed method on three public multimodal emotion recognition datasets. Experimental results indicated that our approach surpassed fully-supervised or self-supervised learning methods, achieving state-of-the-art results in various emotion-related tasks. Additionally, the proposed method performs better than the fully-supervised learning approaches on limited labeled data, demonstrating its superior generalization ability to avoid overfitting problems. A series of ablation studies have also confirmed the efficiency of the designed model architecture. In the next chapter, we will report on the technical contributions on multimodal aspects, i.e., stress and pain detection based on physiological and behavioral signals.This chapter mainly concerns technical contributions to multimodal aspects, where a deep geometric framework with the symmetric positive definite (SPD) matrix as the joint representation of multimodal signals was proposed to cope with stress and pain detection tasks. In Section 4.1, we shortly introduce stress and pain detection methods and their challenges, Section 4.2 presents relevant multimodal studies and applications of SPD-based representations. A detailed description of the proposed method is given in Section 4.3, followed by Section 4.4 showing details of stress and pain detection experiments and evaluation results of our method, and in Section 4.5, we briefly summarise our work.

  actual or potential tissue damage by the International Association for the Study of Pain (IASP)[START_REF] Raja | The Revised IASP definition of pain: concepts, challenges, and compromises[END_REF]. Its accurate assessment plays a key role in diagnosing the condition, monitoring postoperative progress and optimizing treatment options[179]. Due to the side effects of these two sensations on health conditions, automatic detection methods are greatly encouraged, especially if the patient is alert or uncooperative. A variety of sensors can be applied to collect stress/pain indicators from different perspectives, whereby the corresponding approaches for stress/pain detection can be mainly divided into three categories: 1) physiological based methods via biosignals (e.g., electroencephalogram (EEG), electrodermal activity (EDA), etc.); 2) behaviouralbased methods via physical signals (e.g., facial expressions, speech, body movements, etc.); 3) multimodal-based methods via a combination of physiological and behavioural signals.

Figure 4 . 1 :

 41 Figure 4.1:Overview of the proposed framework. First, the SPD matrix sequences that incorporate the correlation information between multimodal data (i.e. physiological and behavioural signals) can be extracted from the segmented data records. Subsequently, the tangent space mapping projects the SPD matrix sequences to the vector sequences in the tangent space. Finally, these vectors can be used as input to the LSTM-based classification network for stress/pain recognition.

  on the tangent spaces T p M of M. The smoothness condition for g is satisfied if the function p ∈ M → g p (X p , Y p ) ∈ R is smooth for each of the local smooth vector fields X, Y in M. With the definition of the inner product g, we can then define the length of the curve joining any two points on M. Let γ : [a, b] → M be a smooth curve. Then the length L[γ] of γ is given by

Figure 4 . 2 :

 42 Figure 4.2: Exponential map and logarithm map between the Riemannian Manifold M and the tangent space T p M at P.

Algorithm 4 . 1 :

 41 as the input of the LSTM-based deep neural network in Fig. 4.1. During training, the temporal contextual relationships within the tangent space vector sequence are explored by the 2-layer LSTM network, and the output features are then fed into the fully connected layer, followed by the sigmoid function to obtain the predicted probabilities. Algorithm 4.1 summarizes the process of classifying the SPD matrices. Classification of SPD matrix sequences Input: A dataset L consisting of labeled SPD matrix sequence with T data segments collected from

  An example of 2D texture images/3D model sequences/thermal images from the Pain class and their corresponding facial landmarks is shown in Fig. 4.3.

Figure 4 . 3 :

 43 Figure 4.3: An example of 2D texture images/3D model sequences/thermal images from the Pain class and their corresponding facial landmarks provided in BP4D+ dataset.

Fig. 4 .

 4 4 shows an SPD matrix of a pain class sample in BP4D+ dataset that generated from the 10 facial distances between 2D landmarks and 8 physiological signals. During the pain task, the subject was asked to immerse hands in ice water, and her mouth was involuntarily opened, hence the distances

Figure 4 . 4 :

 44 Figure 4.4: The multimodal SPD representation generated by a pain sample in the BP4D+ dataset, where correlations within and across two modalities (i.e., vision and physiology) can be observed. (D1-D10: 10 distances automatically selected based on Anova F-value, BP: raw blood pressure, BPDia: diastolic blood pressure, LAS: systolic blood pressure, LAM: mean blood pressure, EDA: electrodermal activity, HR: heart rate, RR: respiration rate, and RV: respiration volts.)

Figure 4 . 5 :

 45 Figure 4.5: (a) Stress detection performance on WESAD dataset and (b) pain detection performance on BP4D+ dataset using the sample covariance matrix S, the cross-covariance matrix C and the proposed SPD representation P. (Acc: Accuracy, F1: F1-score)

Figure 5 . 1 :

 51 Figure 5.1: The wearable device used in this study: E4 wristband and collected raw multimodal signals.

Figure 5 . 3 :

 53 Figure 5.3: Self-Assessment Manikins. Excerpt from [6].

Figure 5 . 5 :

 55 Figure 5.5: Analysis of State-Trait Anxiety Inventory (STAI) data before and after simulation training. (The green triangle symbol represents the mean value.)

Fig. 5 .

 5 Fig. 5.6 (a) shows a graphical interface developed based on PyQt, allowing to load, synchronize and visualize the trainees' multimodal data collected in the simulation rooms. The graphical interface consists of two parts, the upper part of which allows the display of video recordings from the different cameras in the simulation room, while the lower part shows the physiological signals of the trainees during the entire simulation training, and the parts of the signal curves framed in blue are the 60 s stress segments detected by the SSL emotion recognition algorithm proposed in Chapter 3. Besides, the areas circled by rectangles show the percentage of stress periods during simulation training and the physical activity index (AI) calculated proposed in [208], which represents the intensity of the motion obtained from the 3-axis accelerometry every 60 s. During the development of the interface, we designed a simplified version based

( a )Figure 5 . 6 :

 a56 Figure 5.6: The graphical interface with the built-in SSL-based stress detection algorithm applied in debriefing phase after simulation training.

Figure 5 . 7 :

 57 Figure 5.7: Visualization of the subjective assessment of the task difficulty and the level of competence at three moments: pre-simulation, post-simulation and post-debriefing.

Figure 5 . 8 :

 58 Figure 5.8: Visualisation of trainees' feedback regarding the application of the stress analysis tool.

Figure 6 . 1 :

 61 Figure 6.1: Overview of the proposed approach. (a) 2D landmarks detected in images of facial expressions are first converted to vectors in complex space; (b) The variants obtained by affine transformation constitute the equivalence class of landmarks, which can be considered as the same point in the complex projective space; (c) The application of metric learning enables the reduction of intra-class differences while enlarging inter-class distances. Finally, the learned metric is used for similarity-based classification.

Figure 6 .

 6 Figure 6.3: An illustration of the LMNN algorithm modified from[START_REF] Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF]. The proposed Fubinistudy metric will be optimised such that the distances between each sample z i and its k = 3 target neighbours are reduced, while the distances to the impostors are increased.

Figure 6 . 5 :

 65 Figure 6.5: Average classification accuracy on the CK+ and Oulu-CASIA datasets using the Fubini-Study metric when varying the target neighbor parameter k.

Fig. 6 . 6 ,

 66 Fig. 6.7 show the 2D visualization of the CK+ and

Figure 6 . 6 :

 66 Figure 6.6: 2D visualization of CK+ dataset using t-SNE method with the proposed metrics. (a-b) show the visualization results before and after metric learning with Fubini-Study metric. (c-d) show the visualization results before and after metric learning with Euclidean metric.

Figure 6 . 7 :

 67 Figure 6.7: 2D visualization of Oulu-CASIA dataset using t-SNE method with the proposed metrics. (a-b) show the visualization results before and after metric learning with Fubini-Study metric. (c-d) show the visualization results before and after metric learning with Euclidean metric.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7. 1 . 1

 11 Technical levelFrom the physiological perspective, we proposed a novel emotion recognition method based on peripheral physiological signals. A deep neural network based on the intermediate fusion strategy was designed to optimize feature extraction and fusion of multivariate sequences, where a modality-specific encoder based on residual temporal convolution was first employed to extract low-level unimodal features, and then a transformer-based shared encoder with stacked multimodal features as input was applied to capture both inter-modal correlation information.

  In this chapter, we introduce the basic theoretical background on emotions and review recent emotion recognition methods. Firstly, Section 2.1 introduces two ways of describing emotions: the discrete emotion model and the continuous emotion model. Then, in Section 2.2, we present the common physiological and behavioral indicators associated with emotions and explain the reasons for focusing on peripheral physiological signals and facial landmarks in this thesis. Accordingly, emotion recognition methods can be classified in terms of modality: emotion recognition methods based on physiological signals, landmark-based facial expression recognition methods, and multimodal emotion recognition methods based on physiological and behavioral signals, which are presented in Sections 2.3, 2.4, and 2.5, respectively.

	1.5 Publications
	• Yujin Wu, Mohamed Daoudi, Ali Amad, "Transformer-based Self-supervised Multi-
	modal Representation Learning for Wearable Emotion Recognition", IEEE Trans. On
	Affective Computing, Accepted for publication.
	• Yujin Wu, Mohamed Daoudi, Ali Amad, Laurent Sparrow, Fabien D'Hondt, "Fusion
	of Physiological and Behavioural Signals on SPD Manifolds with Application to Stress
	and Pain Detection". IEEE International Conference on Systems, Man, and Cybernetics
	(IEEE SMC 2022): 2949-2955.

• Yujin Wu, Mohamed Daoudi, "Metric Learning on Complex Projective Spaces", International Conference on Smart Multimedia (ICSM 2022):116-127. • Eman A. Abdel-Ghaffar, Yujin Wu, Mohamed Daoudi, "Subject-Dependent Emotion Recognition System Based on Multidimensional Electroencephalographic Signals: A Riemannian Geometry Approach", IEEE Access 10: 14993-15006 (2022) • Yujin Wu, Mohamed Daoudi, Ali Amad, Laurent Sparrow, Fabien D'Hondt, "Unsupervised Learning Method for Exploring Students' Mental Stress in Medical Simulation Training", ACM International Conference on Multimodal Interaction Companion (ICMI 2020): 165-170. 2 Related Work Contents 2.1 Emotion models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Emotion related modalities . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Emotion recognition based on peripheral physiological signals . . . . . . 2.4 Emotion recognition based on facial expressions . . . . . . . . . . . . . . 2.5 Emotion recognition based on multimodal signals . . . . . . . . . . . . .

Table 2 . 1 :

 21 Overview of supervised machine-learning based methods for physiological emotion recognition. (EF: early fusion; LF: late fusion.)

	Modality	Type	Selection Fusion	Classification	Paper
	ECG	Time	-	-	KNN, SVM	[57]
	ECG	Freq	-	-	KNN	[60]
	EDA	Time	-	-	SVM	[56]
	ECG	Mix	-	-	RF, PA, GB, DT, RC, SVM, KNN, LR [63]
	ECG	Mix	-	-	SVM, RF, DT, KNN, GBDT	[67]
	ECG, EMG, EDA, RSPs	Mix	SBS	EF	pLDA	[58]
	ECG, EDA, RSP	Non-linear	PCA	EF	QRC	[52]
	ECG, EMG, EDA, BVP	Mix	KPCA	EF	GBDT, SVM, KNN, GBN	[59]
	ECG, EMG, EDA, RSP Time-Freq	-	EF	SVM	[62]
	EDA, EKG, RSP	Mix	SBL-PCA	EF	SVM, ELM	[65]
	ECG, EMG, EDA	Time-Freq	SFS	LF	SVM, RF, NB, DT, KNN, GBDT	[61]
	ECG, EDA, TEMP	Mix	-	LF	SVM, KNN, QDA, LDA	[66]
	BVP, RSP, TEMP	Time	mRMR	LF	RF	[64]
	ECG, EMG, EDA, BVP	Mix	MI	LF	SVM	[54]
	2.3.3 Supervised deep learning-based methods
	Deep learning-based methods have recently gained extensive attention in the field of emotion
	recognition and have been shown to outperform machine learning methods in several studies
	[15,53,68-70]. Its major advantage is the automatic mining of complex and abstract features in
	the raw input, through multi-level non-linear transformations introduced by successive layers,
	thus drastically simplifying the process of feature extraction and selection compared to machine
	learning methods [28]. Due to their automatic representation learning properties, deep learn-
	ing approaches for physiological emotion recognition typically employ multiple modalities to

Table 2 . 2 :

 22 Overview of supervised deep learning-based methods for physiological-based emotional recognition.

	Fusion	Modality	Real-life?	Taxonomy	Papers
				Architecture
		EDA, HR	Yes	Fully connected	[72]
		ECG, EDA	Yes	Fully connected	[73]
		ECG, EDA	Yes	Convolutional	[68]
	Early	EDA, BR, HR, HRV	Yes	Convolutional	[75]
		BVP, ECG, EDA, EMG, RESP, TEMP	Yes	Recurrent	[76]
		EDA, HR, TEMP	Yes	Recurrent	[77]
		ECG, EDA HR	Yes	Hybrid	[78]
				Representation
		EEG, EMG , EOG	No	Marginal	[79]
	Intermediate	EEG, EMG, EOG EEG, EMG, EOG, EDA, BVP, RESP, TEMP	No No	Marginal Marginal	[80] [81]
		EEG, EMG, EOG, ECG, EDA, RESP	No	Joint	[82]
				Aggregation
		EEG, ECG, EDA	No	Majority	[83]
	Late	ECG, EMG, EDA, RESP	Yes	Weighted	[85]
		EEG, ECG, EDA	No	Weighted	[86]

Table 2 . 3 :

 23 Overview of other learning paradigm-based methods for physiological emotion recognition. (PPS: a series of Peripheral Physiological Signals, SAE: stacked autoencoder; SDAE: stacked denoising autoencoders, C intra/inter : columns used to indicate whether the deep model takes into account intra-and intermodal correlations in the pre-training phase.) Learning scheme Modality Real-life? C intra C inter Architecture Papers

		PPS	Yes	-	-	Hand-crafted	[89]
	Unsupervised	EEG, PPS PPS	No Yes	-✓	-✗	Hand-crafted SAE	[90] [91]
		PPS	Yes	✓	✓	CorrNet	[70]
	Semi-supervised	EEG, PPS PPS	No Yes	✓ ✗	✗ ✓	SDAE LSTM-AE	[94] [95]

Table 2 . 4 :

 24 Overview of behavioral emotion recognition methods.(FCN: Fully Connected Network)

	Feature Support	Shape Space	Architecture Papers
		Euclidean	Hand-crafted [107]
		Euclidean	Hand-crafted [108]
	Point-based	Euclidean Euclidean	1DCNN LSTM	[109] [110]
		Euclidean	Hand-crafted [111]
		Euclidean	Hand-crafted [112]
		Euclidean	Attn-LSTM	[113]
	Local region-based	Euclidean	PHRNN	[114]
		Euclidean	Hand-crafted [115]
		Euclidean	Hand-crafted [116]
		Euclidean	Hand-crafted [117]
		Euclidean	FCN	[118]
	Full face-based	Euclidean Non-linear Manifold Hand-crafted FCN	[119]

  .5. From the table, we can first notice that half of the multimodal methods are on the basis of traditional machine learning algorithms. Second, half of the solutions employed either the early fusion or late fusion strategy, neither of which could capture simultaneously intra-modal and inter-modal correlation information. Third, facial expressions are considered a key aspect in recognizing emotions, however, few studies have investigated the usefulness of combining facial expressions with peripheral physiological signals for predicting emotions, especially in practical environments. These studies that have been conducted

	Fusion	Modality	Domain Real-life?	Architecture	Papers
		EEG, FE	P+F	No	Hand-crafted	[130]
	Early	PPS, FE PPS, ACC	P+F P+M	Yes Yes	Hand-crafted Hand-crafted	[131] [53]
		PPS, ACC	P+M	Yes	2DCNN	[132]
		EEG, FE	P+F	No	Hier-SA-LSTM [133]
	Intermediate	PPS, ACC	P+M	Yes	NAS	[69]
		PPS, ACC	P+M	Yes	Res-TCN	[15]
	Late	EEG, PPS, FE	P+F	No	3DCNN-LSTM [134]
	Hybrid	PPS, FE PPS, ACC	P+F P+M	Yes Yes	Hand-crafted Hand-crafted	[4] [5]

typically involve EEG signals, which are not suitable for use outside of a controlled laboratory setting. Additionally, the lower focus on integrating physiological signals with facial expressions compared to body motions from the ACC signal is likely due to the technical difficulties in combining visual and physiological data, which have distinct structures and characteristics Table 2.5: Overview of behaviroual-physiological emotion recognition methods. (ACC: 3-axis Acceleration, FE: Facial Expressions, PPS: Peripheral Physiological Signals, P+F: Physiol-ogy+Face, P+M: Physiology+Body Motion.)

Table 3 . 1 :

 31 The learning tasks assigned to each dataset and the corresponding distribution of samples between classes in different datasets. (P: Pretext task, D: Downstream task.)

	Dataset	Type	Task	Category (no. of samples)	Total Size
	PRESAGE	P	Transformation Recognition	Original version and five transformations (681641)	(4089846, 240, 3)
	WESAD	D	Stress-2 Emotion-3	stress (36279), non-stress (85574) baseline (66859), stress (36279), amusement (18715)	(12185, 240, 3)
			Arousal-2	low (33211), high (61919)
	CASE	D	Valence-2 Arousal-3	negative(32017), positive (63113) low (4847), medium (26898), high (63385)	(95130, 240, 3)
			Valence-3	negative(9312), neutral (56870), positive (28948)
			Arousal-2	low (3729), high (1488)
	K-EmoCon	D	Valence-2 Arousal-3	negative(4050), positive (1167) low (1783), medium (1904), high (1530)	(5217, 240, 3)
			Valence-3	negative(1783), neutral (1904), positive (1530)

Table 3

 3

	.2: Performance comparison of different emotion recognition tasks with state-of-the-art
	methods on the WESAD dataset. (Acc: Accuracy, F1: F1-score, SL: supervised learning meth-
	ods, UL: unsupervised learning methods, SSL: self-supervised learning methods, S: supervised,
	F: frozen, T: fine-tuned.)
	Type Methods	Stress-2 Acc F1	Emotion-3 Acc F1
		LDA [53]	86.46 83.77 68.85 58.18
		RF [53]	88.33 86.10 76.17 66.33
		SimpDCNN [152] 90.12 88.22 78.30 74.59
	SL	MulT [160] ResNet [161]	91.76 91.17 81.09 78.27 91.93 90.97 80.85 79.63
		StressNAS [69]	92.87	-	81.78	-
		Res-TCN [15]	94.16 93.62 83.69 81.61
		Ours (S)	93.83 92.55 84.81 83.76
	UL Autoencoder	91.51 90.33 80.39 79.13
		SigRep [139] (F)	92.71 91.99 81.11 78.92
		SigRep [139] (T)	94.91 93.09 84.27 82.35
	SSL	BENDR [166] (F) 92.53 91.72 81.98 79.71 BENDR [166] (T) 93.19 92.55 82.44 80.69
		Ours (F)	94.81 93.69 83.81 82.01
		Ours (T)	96.29 95.11 84.94 82.60

Table 3 . 3 :

 33 Performance comparison of different emotion recognition tasks with state-of-the-art methods on the CASE dataset. (Acc: Accuracy, F1: F1-score, SL: supervised learning methods,

	UL: unsupervised learning methods, SSL: self-supervised learning methods, S: supervised, F:
	frozen, T: fine-tuned.)
	Type Methods	Valence-2 Acc F1	Valence-3 Acc F1	Arousal-2 Acc F1	Arousal-3 Acc F1
		SimpDCNN [152]	71.33 68.74 59.20 51.95 67.16 61.60 56.80 53.85
		DCNN [145]	72.35 69.96 59.78 52.80 69.63 63.43 56.09 53.51
		MMResLSTM [165] 73.34 70.96 60.78 53.09 71.12 68.06 57.41 54.69
	SL	Attn-BiLSTM [164] 74.25 71.27 61.97 53.64 70.40 66.52 58.27 54.76
		MulT [160]	74.81 73.17 63.14 62.50 71.28 70.44 62.15 58.48
		ResNet [161]	75.29 74.62 62.89 62.18 72.35 72.19 65.46 59.69
		Ours (S)	76.94 75.06 64.58 63.29 74.15 72.86 66.32 61.78
	UL	Autoencoder CorrNet [70]	73.23 72.05 60.77 57.32 69.16 67.13 60.08 55.12 76.37 76.00 60.15 53.00 74.03 72.00 58.22 55.00
		SigRep [139] (F)	71.74 64.78 63.85 54.97 70.79 67.28 63.09 56.99
		SigRep [139] (T)	73.29 69.84 64.63 55.68 72.08 69.45 64.88 58.81
	SSL	BENDR [166] (F) BENDR [166] (T)	72.94 68.48 61.56 50.86 72.04 67.43 62.37 55.63 72.33 67.62 62.15 53.03 71.51 67.32 63.52 57.01
		Ours (F)	77.49 75.85 65.51 64.07 73.67 70.76 65.09 59.64
		Ours (T)	78.57 77.74 66.64 64.85 74.98 73.10 66.19 60.56

Table 3 . 4

 34

	Type Methods	Valence-2 Acc F1	Valence-3 Acc F1	Arousal-2 Acc F1	Arousal-3 Acc F1
		SimpDCNN [152]	77.14 70.06 59.67 48.98 72.48 61.21 46.49 38.34
		DCNN [145]	78.72 72.09 61.97 51.39 73.67 65.53 49.91 39.24
		Attn-BiLSTM [164] 79.76 72.19 62.56 54.35 73.30 66.23 46.95 46.77
	SL	MMResLSTM [165] 78.79 72.76 61.25 51.65 74.31 67.88 44.68 37.19
		MulT [160]	80.13 76.72 63.95 59.07 74.19 72.49 49.25 47.86
		ResNet [161]	80.53 78.04 64.60 62.22 74.35 73.20 50.09 46.77
		Ours (S)	81.51 78.60 64.07 60.83 75.17 73.62 50.42 47.52
	UL Autoencoder	80.58 75.58 63.65 58.32 71.56 69.10 48.83 47.10
		SigRep [139] (F)	78.98 73.15 63.00 54.07 73.36 67.80 47.85 42.58
		SigRep [139] (T)	79.14 73.55 61.74 52.69 73.94 68.71 48.56 43.90
	SSL	BENDR [166] (F) BENDR [166] (T)	79.83 72.62 61.38 53.20 72.86 66.16 50.68 48.03 78.73 72.15 61.85 54.47 73.82 69.46 52.88 51.24
		Ours (F)	82.95 80.07 66.97 61.28 74.79 73.40 50.76 48.66
		Ours (T)	84.14 81.08 68.37 63.10 76.40 74.29 54.60 52.34

: Performance comparison of different emotion recognition tasks with state-of-the-art methods on the K-EmoCon dataset. (Acc: Accuracy, F1: F1-score, SL: supervised learning methods, UL: unsupervised learning methods, SSL: self-supervised learning methods, S: supervised, F: frozen, T: fine-tuned.)

  Three variants of the proposed model based on early fusion, late fusion, and intermediate fusion strategies were implemented for comparison. The corresponding model architectures used for comparison are illustrated in Fig 3.8. In all variant models, the TCN-based encoder E p , transformer E s and signal transformation classification head C

Table 3 .

 3

	5: Ablation study of different fusion strategies: average accuracy and F1-score ob-
	tained for emotion recognition on WESAD, CASE and K-EmoCon dataset using different vari-
	ant model. (S-2: Stress-2, E-3: Emotion-3, V-2: Valence-2, A-2: Arousal-2, Inter w/ ol: Inter-
	mediate fusion with an overall loss.)
			WESAD			CASE			K-EmoCon
	Type	S-2		E-3		V-2		A-2		V-2		A-2
		Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
	Early	91.22 89.94 80.82 78.79 73.01 72.20 69.13 67.06 79.20 74.12 71.57 69.60
	Late	93.02 91.73 81.48 80.91 75.58 72.27 71.96 68.52 80.94 76.43 72.81 70.87
	Inter w/ ol 93.53 92.77 82.82 81.62 76.69 73.52 72.24 69.11 81.48 77.22 73.06 71.29
	Ours	94.81 93.69 83.81 82.01 77.49 75.58 73.67 70.76 82.95 80.07 74.79 73.40

  F1 EDA 92.36 90.58 78.72 75.90 75.21 74.80 72.15 70.13 80.65 74.60 73.09 72.69 BVP 87.82 86.35 75.20 68.80 75.90 75.15 69.23 65.07 80.76 74.13 72.88 70.67 TEMP 78.15 76.91 69.86 65.37 71.64 68.66 68.97 62.16 79.02 72.78 72.52 70.42 EDA + BVP 93.73 92.38 82.32 80.61 76.26 75.13 72.13 70.27 80.87 75.48 73.14 71.78 EDA + TEMP 90.95 89.62 79.74 76.09 76.03 74.97 72.92 70.54 81.70 77.77 74.61 72.93 BVP + TEMP 84.82 80.45 72.88 66.16 72.35 71.31 71.14 68.04 80.12 75.05 72.43 69.86 All 94.81 93.69 83.81 82.01 77.49 75.85 73.67 70.76 82.95 80.07 74.79 73.40

Table 3 . 7 :

 37 Ablation study of different model components: average accuracy and F1-score obtained for emotion recognition on WESAD, CASE and K-EmoCon dataset using different variant model. (S-2: Stress-2, E-3: Emotion-3, V-2: Valence-2, A-2: Arousal-2) .[START_REF] Piana | Real-time automatic emotion recognition from body gestures[END_REF] 79.86 78.77 64.15 62.20 66.87 57.87 79.39 75.34 71.24 66.59 No Transformer 92.18 91.55 81.08 79.30 74.65 71.53 70.04 68.40 80.67 76.17 72.09 70.66 w/ fixed PE 93.49 91.63 82.38 80.47 76.37 74.45 72.59 70.05 80.32 77.43 73.35 71.20 w/ learnable PE 92.68 91.32 82.42 81.24 76.46 75.35 73.16 70.33 81.59 78.62 74.08 72.09 Our Model 94.81 93.69 83.81 82.01 77.49 75.58 73.67 70.76 82.95 80.07 74.79 73.40 3.5.6.E Ablation study of different signal transformation task

			WESAD			CASE			K-EmoCon
	Model Variants	S-2		E-3		V-2		A-2		V-2		A-2
		Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
	No TCN	91.09 90

Table 3 . 8 :

 38 Ablation study of individual signal transformations and their combinations: average accuracy and F1score obtained for emotion recognition on WESAD, CASE and K-EmoCon datasets using different transformations in self-supervised pertaining, where the best performing transformations and combinations of transformations in each task are underlined. (N: Noise addition, M: Magnitude-warping, P: Permutation, T: Time-warping, C: Cropping.) .[START_REF] Silva | In-vehicle driver recognition based on hand ecg signals[END_REF] 78.04 76.24 73.53 70.42 69.16 61.48 79.69 74.30 70.23 67.38 M 89.74 87.86 76.90 73.77 68.75 67.46 68.28 66.94 80.13 75.44 71.11 70.54 P 91.20 89.33 78.38 75.67 73.61 71.17 71.46 69.82 80.22 78.12 71.69 70.82 T 91.34 90.87 81.15 80.44 71.17 69.53 70.81 69.01 80.39 78.31 70.59 67.90 C 89.48 88.06 79.21 77.38 69.35 62.46 69.17 66.60 79.68 74.07 70.87 69.05 .12 80.02 78.93 75.77 73.57 72.85 70.09 80.26 78.54 73.09 71.73 M+P 91.75 90.47 79.24 77.31 74.01 72.92 72.39 69.87 80.34 79.15 71.89 70.76 N+T 92.95 91.69 83.11 81.14 73.51 71.21 70.03 68.91 81.57 79.43 72.04 69.75 M+T 91.51 90.18 79.85 77.52 71.45 70.07 71.61 69.14 80.29 79.04 71.39 69.61 N+C 91.28 90.15 78.81 76.66 73.96 72.60 70.57 68.44 79.52 77.83 73.13 71.52 M+C 90.08 89.62 78.27 76.40 71.90 69.45 69.53 67.89 82.14 79.61 73.89 72.51 All 94.81 93.69 83.81 82.01 77.49 75.85 73.67 70.76 82.95 80.07 74.79 73.40

	Type	S-2	WESAD	E-3	V-2	CASE	A-2	V-2	K-EmoCon	A-2
	Single	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
	N 90.18 89Same Domain Acc F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
	N+M	89.69 88.43 77.52 75.59 75.87 74.22 71.83 69.62 81.46 78.95 73.14 72.08
	P+T+C	93.67 92.88 82.31 80.47 73.71 71.95 70.92 69.05 80.77 78.58 72.66 71.11
	Cross Domain Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
	N+P	92.15 91

  presented GAN/RNN-based deep model to generate gold standard chest sensor

	Wrist Sensor ~ (Physio + Motion)	Chest Sensor ~ (Physio + Motion)	Wrist +Chest ~ (Physio + Motion)
	Physio + 2D landmarks	Physio + 3D landmarks	Physio + Thermal landmarks
	Acc	F1	Acc	F1	Acc	F1
	Metrics		Metrics		Metrics

Table 4 .

 4

	Sensor Type	Modalities	P(m = 2) Acc F1	P(m = 3) Acc F1	P(m = 4) Acc F1
	Wrist	Physio Physio + Motion(↑) 94.08 93.81 93.10 92.78 94.65 93.99 92.15 92.20 93.00 92.38 93.47 93.25
	Chest	Physio Physio + Motion(↑) 94.55 94.33 95.54 94.76 94.25 93.97 92.76 92.61 92.96 92.64 92.12 91.75
	Wrist + Chest	Physio Physio + Motion(↓) 94.23 94.19 94.71 93.81 96.35 95.76 95.33 95.29 95.09 94.15 96.88 96.44

1: Stress detection performance of uni-modal setting (only physiology) and multimodal setting (physiology + motion) on WESAD dataset using the proposed SPD representation P. (Acc: Accuracy, F1: F1-score, ↑ (↓): multimodal performance is improved (decreased) compared to the unimodal one.)

Table 4 . 2 :

 42 Comparison with State-of-the-art Methods on WESAD dataset (Stress vs. Nonstress)

	Sensor Type Methods	Accuracy F1-score
		RF [53]	87.12	84.11
		GAN/RNN-RF [198]	92.1	89.7
		FCQ-CNN [199]	92.7	92.55
	Wrist	StressNAS [69] Res-TCN [15]	93.14 94.16	-93.62
		Ours with P(m = 2)	94.08	93.81
		Ours with P(m = 3)	93.10	92.78
		Ours with P(m = 4) 94.65	93.99
		GAN/RNN-RF [198]	91.1	90.2
		RF [53]	92.83	91.07
		FCQ-CNN [199]	93.10	93.01
	Chest	Res-TCN [15]	96.69	96.61
		Ours with P(m = 2)	94.55	94.33
		Ours with P(m = 3)	95.54	94.76
		Ours with P(m = 4)	94.25	93.97
		RF [53]	92.28	90.74
		GAN/RNN-RF [198]	94.7	93.4
		FCQ-CNN [199]	96.62	96.63
	Wrist + Chest	Res-TCN [15]	97.75	97.74
		Ours with P(m = 2)	94.23	94.19
		Ours with P(m = 3)	94.71	93.81
		Ours with P(m = 4)	96.35	95.76

Table 4

 4

	.3. We noticed that all the mul-

Table 4 . 4 :

 44 Comparison with State-of-the-art Methods on BP4D+ dataset (pain vs. non-pain).

	Modality	Methods	Accuracy F1-score
		RF [195]	77.7	30.0
	Physiology	Ours with P(m = 2) Ours with P(m = 3)	81.82 83.24	81.72 82.42
		Ours with P(m = 4)	82.81	81.42
		β G (t) on S + (d, n) [200] Dual-layer 3DCNN [201]	81.86 84.03	77.34 83.98
	Vision	HybNet [202] Ours with P(m = 2)	86.43 91.44	85.71 89.24
		Ours with P(m = 3)	91.59	89.46
		Ours with P(m = 4)	90.94	88.56
		(L) β G (t) on S + (d, n) [200] 82.77 (E) β G (t) on S + (d, n) [200] 84.32 (E) HybNet [202] 87.94	76.32 78.83 87.16
	Vision + Physiology	(L) Dual-layer 3DCNN [201] 89.08	88.68
		(L) HybNet [202]	89.36	89.13
		(E) Dual-layer 3DCNN [201] 89.80	89.46
		Ours with P(m = 2)	92.86	90.03
		Ours with P(m = 3)	93.04	90.85
		Ours with P(m = 4)	93.45	91.37
	(E) : Early fusion, (L) : Late fusion.

Table 5 .

 5 1: A summary of pre-processed real-life data.

	Task	Category (number of samples) Total
	Arousal-2	low (3417), high (4857)
	Valence-2	low(2591), high (5683)
	Stress-2	non-stress (6687), stress (1587)	8274
	Four-class	LALV(1004), LAHV(2413) HALV(1587), HAHV(3270)

Table 5 . 2 :

 52 List of multimodal features for machine learning-based emotion recognition.

Table 5 . 3 :

 53 Performance comparison of different emotion recognition tasks with state-of-the-art methods on the Presage dataset. (Acc: Accuracy, F1: F1-score; B:Behaviour, P:Physiology, B+P:Behaviour+Physiology.)

	Type	Methods	Arousal-2 Acc F1	Valence-2 Acc F1	Stress-2 Acc F1	4-class Acc F1
		Handcrafted-SVM	57.39 52.09 63.07 59.62 64.41 61.83 44.55 41.56
	B	SPD-LSTM	72.15 70.45 75.26 71.07 76.57 72.55 61.39 54.06
		TCN-TRANS	74.54 71.79 77.92 74.43 79.31 75.47 63.47 58.75
		Handcrafted-SVM	54.35 49.80 65.82 61.13 66.15 62.44 48.27 43.93
	P	SPD-LSTM	71.44 69.82 77.27 73.57 80.42 75.17 66.08 61.57
		TCN-TRANS	73.62 71.64 80.27 75.61 83.59 79.42 67.29 63.11
		Handcrafted-SVM (↓) 55.68 50.76 66.87 65.25 61.94 59.93 46.86 41.96
	B+P	SPD-LSTM (↑)	73.69 71.81 78.52 75.72 82.66 78.70 67.21 62.06
		TCN-TRANS (↑)	76.96 74.24 82.14 78.06 86.81 83.58 70.98 65.05

  situation, i.e., medical simulation training. Students' multimodal signals were recorded during the simulation via a wearable sensor, while a series of psychometric scales were applied to establish the ground truth of emotion. Subsequently, emotion recognition experiments involving arousal, valence and stress were conducted on the collected data, proving again the validity of the recognition algorithms presented in Chapters 3 and 4. In order to explore the feasibility and usefulness of the intervention of emotion recognition in the pedagogical process, a graphical interface incorporating the emotion recognition algorithm was developed and used in the post-simulation debriefing session. Based on feedback from debriefing participants, the positive effects of the emotion recognition tool in educational practice were initially confirmed,

	(a) Usefulness in identifying key moments	(b) Level of usefulness of the interface
		Fair		Very Good
		20.0%		20.0%
	Very Good 60.0%	20.0%	Good		80.0%
			Exceptional Very Good Good Fair Poor Bad Detrimental	Good

Table 6 . 1 :

 61 Overall accuracy (Acc %) of of two types of metrics, i.e., the Euclidean metrics and the Fubini-Study metrics, before and after LMNN-based metric learning on the CK+ and Oulu-CASIA datasets. ([•] I : identity matrix; [•] ⋆ : optimal matrix.) Study metric vs Euclidean metric A comparison with the Euclidean metric in Eq.(6.2) in measuring the similarity between different facial expressions is also executed on CK+ and Oulu-CASIA datasets to validate the tempt, Fear, Sad, Surprise expressions is enhanced by up to 11.1%. For Oulu-CASIA dataset, the two metrics can both better identify Happy and Surprise than other expressions. Among all expressions, Angry, Disgust, Fear, Happy can be better classified by the Fubini-Study metric with a maximum gain of 11.2%.

	Dataset	Type	Metric	Classifier KNN SVM
	CK+	Euclidean Fubini-Study	Original Metric d M Optimal Metric d M Original Metric d A Optimal Metric d A ⋆ I ⋆ I	79.84 86.41 85.81 90.65	79.97 89.15 87.53 91.05
	Oulu-CASIA	Euclidean Fubini-Study	Original Metric d M Optimal Metric d M Original Metric d A Optimal Metric d A ⋆ I ⋆ I	51.75 61.66 53.01 62.91	50.49 61.84 55.62 63.75
	6.4.3.A Fubini-

Table 6 . 4 :

 64 Comparison with state-of-the-art geometric methods on the CK+ dataset. Shape trajectories on S + (d, n) [127] 96.87

	Input	Methods	Accuracy
	Video sequence Video sequence	Shape velocity on G [222] LDCRFs [223]	82.80 85.84
	Video sequence	ITBN [224]	86.30
	Video sequence	Intrinsic SCDL -SVM [220]	89.43
	Video sequence Extrinsic SCDL -Bi-LSTM [220]	95.73
	Video sequence Static image Static image	original metric on S + c (2, n) [211] original metric on CP n-1 (ours)	85.83 87.53
	Static image Static image	optimal metric on S + c (2, n) [211] optimal metric on CP n-1 (ours)	90.53 91.05

Table 6 . 5 :

 65 Comparison with state-of-the-art geometric methods on the Oulu-CASIA dataset. Input Methods Accuracy Static image original metric on S + c (2, n) [211] 53.33 Static image original metric on CP n-1 (ours) 55.62 Static image optimal metric on S + c (2, n) [211] 62.08 Static image optimal metric on CP n-1 (ours) 63.75

Dans cette thèse, du point de vue physiologique, nous proposons une méthode d'apprentissage de représentation multimodale auto-supervisée pour la reconnaissance d'émotion portable basée sur des signaux physiologiques périphériques pour faire face au problème du surapprentissage posé par des données étiquetées limitées et le biais d'étiquetage inexact dans le monde réel. Du point de vue comportemental, nous suggérons une méthode d'apprentissage métrique non euclidienne pour la reconnaissance d'expressions basées sur des points de repère faciaux en 2D afin de résoudre des problèmes tels que la variation de pose et la variation intra-classe. D'un point de vue multimodal, nous présentons un cadre géométrique profond basé sur une représentation matricielle symétrique définie positive pour la détection multimodale du stress et de la douleur afin de résoudre le problème de la fusion des modalités physiologiques et comportementales. En outre, nous réalisons le déploiement de la reconnaissance des émotions dans un scénario réel, à savoir la formation par simulation médicale, où l'algorithme proposé précédemment est

https://medecine.univ-lille.fr/presage.

Scaling multiplies time series values by a random scalar whereas Magnitude-warping distorts the signal values by a smooth curve.

https://medecine.univ-lille.fr/presage.

Acknowledgments

The horizontal axis of each subplot is the number of randomly selected samples from each class, varying from 1 to 1000, while the vertical axis is the corresponding average accuracy.

on limited data, allowing for a more comprehensive performance comparison. We implemented a similar sampling procedure reported in [START_REF] Banville | Uncovering the structure of clinical EEG signals with self-supervised learning[END_REF][START_REF] Jiang | Self-supervised contrastive learning for eeg-based sleep staging[END_REF], i.e., 1, 50, 100, 500, and 1000 samples were randomly selected for each class in the three datasets for training the classification model.

This process was executed 50 times independently for different numbers of samples. The resulting average accuracy and the corresponding standard deviation of all compared models are illustrated in Fig. 3.7. First, our fine-tuned model consistently outperforms other supervised learning-based models for sample sizes varying from 1 to 1000 on the emotion recognition tasks of all three datasets. Among supervised learning-based methods, SimpDCNN exhibited the poorest results, over which our SSL model could achieve significant performance gains of 6.84% -21.19% for different downstream tasks. Our fully-supervised model yields the highest results compared to other supervised models, whereas the fine-tuned model initialized by selfsupervised learning parameters continues to enhance performance by 5.24% -13.63%. Second, for all downstream tasks, the standard deviation obtained by our fine-tuned model is narrower with respect to the supervised learning-based deep models, demonstrating its superior generalization ability across different samples. The above findings are consistent with those reported

Pre-simulation phase

Before entering the simulation room, the trainer first introduced the simulation scenario and the teaching objectives to the trainees, i.e. the participants of the experiment. Then, participants received an information letter, an image rights form, a consent form describing the entire experiment, and a set of psychometric scales. Meanwhile, a staff assisted participants in wearing the E4 bracelet and perform a short device test to ensure the quality of the data collected. Upon completion and signature of all relevant documents, participants were requested to press the tag button on their E4 wristbands to mark the start time of the simulation training and then enter the simulation room to perform the appropriate medical tasks. The above process lasted about 5 to 10 minutes

Simulation training phase

Multimodal data was collected from three different simulation scenarios: Agitation, Suicidal risk, Focus for the analysis of the trainee's emotional state. In each scene, highly realistic mannequins or hired actors, took on the role of patients, while trainees acted as doctors. The following are short descriptions of these scenarios:

• Agitation: the scenario simulated a patient who was apparently in an acute agitated state, parameters were configured as described in Chapter 3. For the comparison experiments, the frozen setting was applied, i.e. the encoder part of the pre-trained model was frozen in the downstream emotion recognition tasks.

Evaluation metric and protocol

To validate the generalizability of the model on different subject data, we adopted Leave-One-Subject-Out cross-validation protocol that is frequently used in real-life scenarios. Two common metrics: Accuracy and F1-score were selected for the performance evaluation.

Experimental Results

We performed the four emotion-related classification tasks mentioned in Section 6.3.1 on the collected data and investigated the performance of three modality combinations:

• Behaviour: all behavioral modalities, i.e., 3-axis accelerometer.

• Physiology: all physiological modalities, i.e., EDA, BVP and TEMP.

• Behaviour+Physiology: all modalities. i.e., ACC, EDA, BVP and TEMP.

The corresponding results are summarised in Table 5. Hence, the altered landmark configurations can be all regarded as the same point in the complex projective space, providing a better measure of the similarity between different facial expressions.

Experiments

To investigate the effectiveness of the proposed metric in the complex domain, we conducted several experiments for the facial expression recognition task. First, we introduced the CK+ and Oulu-CASIA datasets used in the experiments. All details of the experimental setup are then reported in Section 6.4.2. In the end, classification results using the proposed metric and comparative results with the Euclidean metric and other state-of-the-art approaches in facial expression recognition are presented in Section 6.4.3.

Datasets

The proposed non-Euclidean metric learning algorithm was evaluated on two public facial expression datatsets. The Cohn-Kanade Extended (CK+) dataset [START_REF] Lucey | The extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression[END_REF] consists of 327 annotated frontal video sequences performed by 118 subjects, wherein each subject is required to exhibit seven facial expressions during the experiment, namely -anger, contempt, disgust, fear, happy, sad and surprise. One sequence contains images from neutral expression (first frame) to peak expression (last frame). For the classification experiments, we exploit only the last frame, where the intensity of facial expression attains its peak. The Oulu-CASIA dataset [START_REF] Zhao | Facial expression recognition from near-infrared videos[END_REF] consists of frontal facial videos from 80 subjects captured under three illumination conditions: dark, normal and weak normal, during which the subjects were required to imitate six classic facial expressions: anger, disgust, fear, happy, sad and surprise. Similar to the CK+ dataset, the last frame of each sequence has the highest expression intensity. Therefore, we use the last frame of the 480 video sequences under normal illumination for the classification task. Randomly selected images of facial expressions from the two datasets are presented in Fig. 6.4.

Experimental setting

For each landmark configuration consisting of 68 points in CK+ and Oulu-CASIA datasets, we first excluded the 17 points of face contour. The 2D centered landmarks are then written in complex form to represent the facial shape z ∈ CP 51 which is invariant to translation, scaling and rotation. Inspired by previous work [START_REF] Daoudi | Metric learning on the manifold of oriented ellipses: Application to facial expression recognition[END_REF][START_REF] Kacem | Barycentric representation and metric learning for facial expression recognition[END_REF], proximity data were constructed to serve as input to classifiers by using the optimal Fubini-study metric obtained from the LMNN algorithm, where each sample z i is represented by its similarity with all samples

T in the dataset. In the end, K-Nearest Neighbors (KNN) and Support Vector Machines (SVM) were selected as classifiers which are most frequently used in metric learning-based methods [START_REF] Suárez | A tutorial on distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges[END_REF]. The hyperparameters of KNN: number of neighbors n For evaluation, we performed the same experimental protocol in the literature [START_REF] Jung | Joint fine-tuning in deep neural networks for facial expression recognition[END_REF], [START_REF] Kacem | A Novel Geometric Framework on Gram Matrix Trajectories for Human Behavior Understanding[END_REF], [START_REF] Tanfous | Sparse Coding of Shape Trajectories for Facial Expression and Action Recognition[END_REF], [START_REF] Daoudi | Metric learning on the manifold of oriented ellipses: Application to facial expression recognition[END_REF], namely 10-fold subject-independent cross-validation on CK+ dataset and Oulu-CASIA dataset with two metrics: accuracy and confusion matrix. All subjects were divided into 10 groups in ascending order of ID, 9 groups were used for training, and the remaining group was used for testing. In the end, the average classification performance is reported in the following sections.

Results and Discussion

First, we investigated the effect of target neighbors parameter k in the LMNN algorithm on the classification performance. The evolutions of accuracy when varying k are shown in Fig. 6.5.

The optimal performance was achieved on CK+ and Oulu-CASIA datasets when k = 8 and k = 5, respectively, for the KNN classifier, while k = 7 and k = 5 give the best results for the SVM classifier. The above settings of parameter k were retained for subsequent experiments. We then verified the effectiveness of the LMNN algorithm based on the proposed metric. Table 6.1

presents the average classification accuracy of the Fubini-Study metric before and after LMNN- effectiveness of the proposed metric. From the results of the Euclidean metric in Table 6.1, we can observe that the performance of KNN and SVM is improved by the proposed metric, regardless of whether metric learning is employed or not. These results demonstrate that facial deformations can be better modeled in complex projective space rather than in Euclidean space. To further investigate the validity of the proposed metric for identifying each facial expression, we provide also the confusion matrix obtained using the two learned metrics, i.e., Fubini-Study metric and Euclidean metric obtained after metric learning, for the CK+ and Oulu-CASIA datasets in Table 6.2 and Table 6.3, respectively. For CK+ dataset, the Disgust, Happy, and Surprise expressions can be well recognized using the Fubini-study metric with an accuracy rate of over 95%, while the main confusions happened in the Contempt expression. This can be explained by its slight facial changes and its smaller sample size compared to other expressions. When compared with the Euclidean metric, the recognition performance of the Con-