Jonathan Aigrain

Michel Spodenkiewicz

Severine Dubuisson

Marcin Detyniecki

David Cohen

Mohamed Chetouani

Séverine Dubuisson

Multimodal stress detection from multiple assessments

Keywords: Stress, assessment, behaviour, physiology, multimodal, classification. F

published or not. The documents may come   L'archive ouverte pluridisciplinaire

INTRODUCTION

R ECENT progress in computer vision and social signal processing have helped to understand the impact of affective and mental states on human behaviour and body. For instance, frameworks for automated analysis and detection of stress [START_REF] Giakoumis | Using activity-related behavioural features towards more effective automatic stress detection[END_REF], [START_REF] Lefter | Recognizing stress using semantics and modulation of speech and gestures[END_REF] provide valuable information about the predictive performance of certain features in stressful contexts. However, these results greatly depend on the way stress is assessed. Lutchyn et al. suggest that, regarding automatic stress detection, "inconsistent results reported in some areas of research can be partially explained by the choice of measurements that capture different manifestations of affective phenomena, or focus on different elements of affective processes" [START_REF] Lutchyn | Stress is in the eye of the beholder[END_REF]. In this paper, we propose a multi-assessment methodogy to analyze stress detection results.

Background 1.Stress definition and measure

Although researchers have studied the topic for more than a century, the stress definition is still debated [START_REF] Koolhaas | Stress revisited: a critical evaluation of the stress concept[END_REF] and can be studied from different perspectives. In this work, we focus on 3 perspectives: the biological perspective, the phenomenological perspective and the behavioural perspective. The biological perspective aims at understanding how the body responds to a stressful stimulus. It was pioneered by Hans Selye [START_REF] Selye | A syndrome produced by diverse nocuous agents[END_REF], who defined stress as the non-specific neuroendocrine response of the body to a demand placed on it, Manuscript received April 19, 2005; revised August 26, 2015. such as extreme temperatures [START_REF] Selye | A syndrome produced by diverse nocuous agents[END_REF], [START_REF] Szabo | The legacy of Hans Selye and the origins of stress research: A retrospective 75 years after his landmark brief "Letter" to the Editor # of Nature[END_REF]. The body responds to a stressful stimulus by the activation of the hypothalamopituitary-adrenal (HPA) pathway and the autonomic nervous system (ANS) that mediates the general adaptation syndrome [START_REF] Selye | Stress without Distress[END_REF]. After the stimulus, a neuroendocrine chain reaction begins in the brain. Recent neuroimaging studies support evidence of the major implication of some cerebral structures with a multiroad processing system of stress [START_REF] Pessoa | Emotion processing and the amygdala: from a 'low road' to 'many roads' of evaluating biological significance[END_REF], [START_REF] Ledoux | Rethinking the Emotional Brain[END_REF]. At a peripheral level, adrenal glands respond by the release of epinephrin and cortisol into the bloodstream with an effect on cardiovascular, musculoskeletal, gastrointestinal, nervous and endocrine systems. This physiological cascade can be measured through salivary or blood sampling with biomarkers such as cortisol. It can also be measured with wearable sensors [START_REF] Elenko | Defining digital medicine[END_REF] via valuable signals, such as skin conductance [START_REF] Jacobs | Use of skin conductance changes during mental stress testing as an index of autonomic arousal in cardiovascular research[END_REF] or heart-rate variability (HRV) measures [START_REF] Taelman | Influence of Mental Stress on Heart Rate and Heart Rate Variability[END_REF], [START_REF] Traina | Effects. of anxiety due to mental stress on heart rate variability in healthy subjects[END_REF]. Using filtering techniques, the sympathovagal balance can be directly calculated as the ratio of low and high frequencies of HRV [START_REF] Shaffer | A healthy heart is not a metronome: an integrative review of the heart's anatomy and heart rate variability[END_REF]. Previous experimental studies suggest that the stress response is linked to a modulation in spectral density of the low frequency band of HRV (HRV-LF) [START_REF] Moriguchi | Spectral change in heart rate variability in response to mental arithmetic before and after the beta-adrenoceptor blocker, carteolol[END_REF]. The phenomenological perspective considers that selfperception is the key aspect. This vision has been supported within the Cannon-Bard theory: the authors state that stress can occur even when the body changes are not present because the physiologic response of the body is more slowly recognized by the brain compared to its function to release an emotional response [START_REF] Walter B Cannon | The James-Lange Theory of Emotions: A Critical Examination and an Alternative Theory[END_REF], [START_REF] Dalgleish | The emotional brain[END_REF]. A major contribution to the field of research on stress was described within Lazarus' theory of cognitive appraisal: stress is a two-way process which includes both the stressor and the individual assessment of ressources required to minimize, tolerate or eradicate the stressor and the stress it produces. Experimental studies confirmed recently that stress experience is moderated by the ability of a human subject to feel his body signals such as the heartbeat [START_REF] Schulz | Interoception and stress[END_REF]. Lazarus states that "Stress occurs when an individual perceives that the demands of an external situations are beyond his or her ability to cope with them" [START_REF]Psychological stress and the coping process[END_REF], [START_REF]From psychological stress to the emotions: a history of changing outlooks[END_REF]. Since this definition focuses mainly on individual perception, stress can be measured by questionnaires [START_REF] Cohen | A Global Measure of Perceived Stress[END_REF], [START_REF] Kenneth B Matheny | The coping resources inventory for stress: A measure of perceived resourcefulness[END_REF], [START_REF] Mcdowell | Measuring health: a guide to rating scales and questionnaires[END_REF], Likert and visual analogue scales [START_REF] F X Lesage | Clinical stress assessment using a visual analogue scale[END_REF]. Finally, the behavioural perspective investigates the impact of stress on human and animal behaviour both at individual and group levels [START_REF] Troisi | Displacement Activities as a Behavioral Measure of Stress in Nonhuman Primates and Human Subjects[END_REF], [START_REF] Mohiyeddini | Displacement behaviour regulates the experience of stress in men[END_REF]. Both transfer of ethological research to human behaviour and social signal processing lead researchers to a promising approach of behavioral measure of stress in non human primates [START_REF] Troisi | Displacement Activities as a Behavioral Measure of Stress in Nonhuman Primates and Human Subjects[END_REF] and more recently in human subjects [START_REF] Mohiyeddini | Displacement behaviour regulates the experience of stress in men[END_REF], [START_REF] Weisman | Dynamics of non-verbal vocalizations and hormones during father-infant interaction[END_REF]. Engaging in displacement behaviors such as scratching, face touching and lip biting have been associated with stressfull experiences and may give more valuable information about the subjects emotional state than verbal statements and verbal expressions [START_REF] Troisi | Displacement Activities as a Behavioral Measure of Stress in Nonhuman Primates and Human Subjects[END_REF]. Authors suggest that these behaviors could impair cognitive performance by cutting-off attention temporarily from stressful or threatening stimuli. This short term diversion of attention could reduce the ability to deal with a mentally challenging or stressful task [START_REF] Chance | An interpretation of some agonistic postures; the role of "cut-off" acts and postures[END_REF], [START_REF] Mohiyeddini | Displacement Behaviour Is Associated with Reduced Stress Levels among Men but Not Women[END_REF]. In this perspective, behavior characteristics do not infer internal subjective feelings but are used as external marker for behavior adaptation (as in the ethological principle). Therefore, stress can be assessed on the basis of ones behavior modifications when a subject is exposed to a stressor.

Automatic stress detection

During the last decade, several authors studied the feasibility of automatic stress detection. Table 1 presents the stimuli, the signals from which features are extracted and the stress annotations considered in several works. Most stimuli used are based on cognitive tasks (Stroop test, mental arithmetic task, logical puzzle task, etc.). Dickerson and Kemeny state that this type of stimulus is the most efficient for stress induction when combined with social-evaluative threat [START_REF] Sally | Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research[END_REF]. Two works focus on automatic stress detection during driving tasks [START_REF] Fernandez | Modeling drivers' speech under stress[END_REF], [START_REF] Healey | Detecting Stress During Real-World Driving Tasks Using Physiological Sensors[END_REF] and three other works also study the impact of physical stimulus such as cold pressor [START_REF] Plarre | Continuous Inference of Psychological Stress from Sensory Measurements Collected in the Natural Environment[END_REF], [START_REF] Shi | Personalized Stress Detection from Physiological Measurements[END_REF] and riding a roller-coaster [START_REF] Zhou | Nonlinear Feature Based Classification of Speech Under Stress[END_REF]. Regarding the signals collected, 2 categories are usually employed: Regarding stress annotations, there is a wide variety of methods: self-assessment, assessment from biomarkers (cortisol, GSR, etc.), assessment from external observers, inference from experimental conditions and inference from acting instructions. Most of the works presented use only one of these methods.

Objective of study

In this work, we propose to study the stress phenomenon in a more comprehensive way by considering the results Fig. 1: Data is annotated in 3 different ways. First, following the phenomenological perspective, we ask the subject to provide her Self-Assessment (SA). Then, following the behavioural perspective, we ask external observers (recruited using a crowdsourcing platform) assessments (EOA). Finally, following the biological perspective, a physiology expert assesses the presence of stress from the percentage of low frequencies in the heart rate variability (PEA).

obtained from different annotations. We introduce original behavioural features expected to be relevant for automatic stress detection. We evaluate the predictive power of 101 behavioural and physiological features for each annotation. Overall, we argue that stress detection should be tackled with a multiple assessment approach because of the complexity of stress. Figure 1 is an extension of Brunswik Lens [START_REF] Brunswik | Perception and the representative design of psychological experiments[END_REF] and summarizes how we address the issue of the annotation choice. The Brunswik Lens is used in the affective computing literature to illustrate the difference between self-assessment and external assessment for phenomena such as personality [START_REF] Vinciarelli | A Survey of Personality Computing[END_REF] and stress [START_REF] Lefter | Recognizing stress using semantics and modulation of speech and gestures[END_REF]. We extend it by adding the assessment provided by a physiology expert. Thus, we annotate stress in 3 different ways:

• Following the behavioural perspective, we gather external observer assessments (EOA) using a crowdsourcing platform.

• Following the phenomenological perspective, we ask the subject to provide her self-assessment (SA).

• Following the biological perspective, a physiology expert assesses the presence of stress from the percentage of low frequencies in the heart rate variability (PEA).

Using these 3 annotations, we evaluate the predictive power of behavioural (or distal) and physiological (or internal) cues.

This paper is organized as follows: Section 2 presents the experiment we used for data collection. Section 3 describes the 3 collected assessments. Section 4 introduces our framework for automatic stress detection: the extracted features, the preprocessing transformations and the feature subset selection methods. Section 5 presents the different results obtained. Section 6 discusses the results and the limitations and strengths of this study. Section 7 concludes and gives some perspectives for future works.

ACQUIRED DATASET

In this section, we present the experiment we designed to obtain behavioural and physiological data in a stressful situation. Designing this new protocol was necessary to collect the required data for a multi-perspective analysis of stress.

The experimental stressor

In [START_REF] Sally | Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research[END_REF], Dickerson and Kemeny state that the best way to increase the cortisol level of a subject is to have her experience cognitive load while being socially evaluated. Based on this work, we designed a time-constrained mental arithmetic test as the stress-induction stimulus of the experiment (Figure 2). Participants were told that the objective of the experiment is to estimate their developmental age and to correlate it with their academic and professional careers. It made them believe that they were socially evaluated while keeping hidden the stress induction aim of the experiment. This way, stress induction occured as naturally as possible.

Fig. 2: Screenshot of the test software used for the study.

The question asked is shown in the middle of the screen. The two possible answers are below the question. At the bottom, the remaining time is displayed using a progress bar. On the right, the color of the score bar provides a feedback regarding the performance of the participant: green means "above average", yellow means "average" and red means "below average".

In our protocol, participants are first briefed about the fake objective of the experiment and asked to sign the consent form and the release waiver accordingly to the ethical committees rules. We also informed the participants that they could stop the experiment at anytime. Then, the physiological sensors are installed, and the participant starts taking the test. The test is composed of 6 steps of increasing difficulty.

There is a break period of 5 seconds between 2 steps. The participant is told that both quickness and correctness of her answers are taken into account to compute her score.

In reality, the values of the score bar are set in advance. It displays an "above average" score at the beginning, so that the participant finds the test easy enough and feels like she should suceed. Then, the score drops to "average" and "below average" levels, giving the participant the feeling she is actually failing. Overall, the score bar and the fake objective induce the feeling of social evaluation while the questions and the time bar induce cognitive load. This combination follows the recommandations of Dickerson and Kemeny [START_REF] Sally | Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research[END_REF]. Once the test is finished, the real objective is revealed and the experiment is debriefed.

Participants

Participants were recruited among medical students of the Université Pierre et Marie Curie in Paris after written informed consent was given. 25 people participated in the experiment. However, because of acquisition problems for the physiological signals, the data of 4 participants was dismissed. Thus, the data of 21 participants has been used: 15 women and 6 men. Their average age is 26.3 ± 4.6 years old.

Hardware setup

Video and skeleton data were recorded using a Microsoft Kinect. Since the resolution (640 ⇥ 480 pixels) of the video recorded by the Kinect is too low for an accurate facial expression analysis, we also recorded video data of the participant's face using the optic zoom of a high definition (1440 ⇥ 1080 pixels) camera. Physiological data were recorded with a Nexus-10 portable device (MindMedia B.V., The Netherlands) with a measurement of EMG, GSR, skin temperature, respiration, BVP and HR.

Acquired data

For each of the 21 participants and for each of the 6 steps of the mental arithmetic task, we acquired:

• the video of the whole body from the Kinect;

• the skeleton from the Kinect;

• physiological measures from the Nexus-10;

• the video of the face from the HD camera;

In total, we have recorded 6 ⇥ 21 = 126 steps.

ASSESSMENT OF STRESS

Each step was annotated in 3 differents ways (Figure 1), one for each perspective we presented in Section 1.1.1:

• External Observers Assessment (EOA) • Self-Assessment (SA) • Physiology Expert Assessment (PEA)
In this section, we describe in detail these 3 annotations and study their correlations.

Description of External Observers Assessment (EOA)

We used the crowdsourcing platform CrowdFlower 1 to obtain annotations from external observers. It allows to easily obtain a large amout of annotations while providing some quality control mechanisms.

Crowdsourcing acquisition procedure

We presented the video of the body recorded by the Kinect for all 126 steps. Three questions were asked for each video (Figure 3):

• Do you think this person is stressed? Answers: not stressed/stressed (Q1)

• How stressed is the person in this video? Answers: Likert scale 1-5 (Q2)

• How confident are you on your ratings? Answers: Likert scale 1-5 (Q3)

Regarding the instructions given to the annotators, we told them that they were shown videos of people taking a cognitive test. This was done so that they would have just enough knowledge about the context to provide accurate ratings.

To obtain acceptable statistical significance, we requested 10 annotations per video.

1. www.crowdflower.com Fig. 3: Screenshot of the CrowdFlower platform.

Crowdsourcing annotation quality control

We used 3 mechanisms to ensure the annotation quality. First, as CrowdFlower proposes 3 categories of contributors according to their experience and their accuracy on the platform, we have chosen the highest ranked category. Second, it is possible to set a minimum amount of time that an annotator should take to provide her answers for one video. We can then discard annotators that do not actually watch the videos until the end before answering. Since the shortest video lasts 50 seconds, we set that duration as the threshold.

Finally, CrowdFlower randomly inserts "Test Questions" among the videos. Test Questions are videos for which we provided an expected set of answers. If an annotator misses too many Test Questions, she is not allowed to provide new ratings and her previous ratings are marked as unreliable.

In order to select the videos for the Test Questions, we preselected a set of videos (N = 14) that we considered prototypical. These videos were presented in an online questionnaire where people had to answer questions Q1, Q2 and Q3. We obtained 28 answers for each video. We discarded the videos that achieved an agreement rate lower than 90% for Q1, decreasing the number of Test Questions to 11.

Annotators

248 people annotated an average of 6.45 ± 5.22 videos. Their repartition over the continents is presented in Table 2. We can see that most of them are from western culture, as the subjects of the experiment are. This is important since stress may be expressed and perceived differently depending on one's culture. Analysis on the impact of culture on stress expression and perception is important but is out of the scope of this paper. TABLE 2: Repartition of the annotators over the continents (EU = Europe, SA = South America, AS = Asia, NA = North America, AF = Africa, OC = Oceania).

Annotations aggregation

Since we have several annotations per video, we have to use an aggregation method in order to assign a single label to each video. To do so, we use the Honeypot method [START_REF] Quoc | An evaluation of aggregation techniques in crowdsourcing[END_REF]. First, we remove untrustworthy annotators using answers to Test Questions. Then, we assign the majority decision to each video: if more than half of the remaining annotations are Stress answers, we assign the Stress label, otherwise we assign the Non-Stress label.

Description of Self-Assessment (SA)

Self-assessment of stress was conducted during the debriefing of the experiment described in Section 2.1. The subjects answered a Likert-scaled (1-5) question about how stressed they felt during each step. To limit memory bias, they watched their own videos before providing their answers. Then, in order to obtain binary labels, we use a threshold on the stress level: Non-Stress = {1, 2} and Stress = {3, 4, 5}. This threshold has been chosen regarding the repartiton of the answers to Q2 according to the answer given for Q1. As shown in Figure 4, it appears that stress levels 1 and 2 are associated with Non-Stress, while stress levels 3, 4 and 5 are associated with Stress.

Description of Physiology Expert Assessment (PEA)

For PEA, presence of stress was assessed based on psychiatric expertise on the physiological impact of stress. We used the percentage of low frequency in the heart-rate variability (HRV-LF%) measure provided by the Nexus-10. HRV is considered to be a reliable indicator to assess the presence of physiological stress [START_REF] Taelman | Influence of Mental Stress on Heart Rate and Heart Rate Variability[END_REF], [START_REF] Traina | Effects. of anxiety due to mental stress on heart rate variability in healthy subjects[END_REF] and its percentage of low frequency is seen as a valuable marker [START_REF] Moriguchi | Spectral change in heart rate variability in response to mental arithmetic before and after the beta-adrenoceptor blocker, carteolol[END_REF], [START_REF] Cohen | Autonomic dysregulation in panic disorder and in post-traumatic stress disorder: Application of power spectrum analysis of heart rate variability at rest and in response to recollection of trauma or panic attacks[END_REF], [START_REF] Boonnithi | Comparison of heart rate variability measures for mental stress detection[END_REF]. It also has the advantage to be a fast physiological marker of the activation of the HPA pathway and the ANS. Thus, it gives a fast image of the impact made by the stressor, unlike cortisol which is realeased with a 5 to 20 minutes delay [START_REF] Bozovic | Salivary cortisol levels as a biological marker of stress reaction[END_REF]. In order to obtain binary labels, we compare the values obtained with the average of HRV-LF% over all the steps. For each step, if the HRV-LF% observed is above the computed average, then the step is associated with the Stress label. Otherwise, it is associated with the Non-Stress label. Consequently, we obtain the distribution shown in Table 5.

Are PEA, SA, and EOA significantly associated ?

To assess how PEA, SA, and EOA were associated, we performed two analyses. First, we calculated Cohens Kappa to assess their agreement based on binary labels. Second we used correlation analysis based on non-binary values. Table 6 shows the Cohen's kappa scores for each combination of 2 assessment sets. The only score which is considered as fair by the guidelines given by Landis et al. in [START_REF] Landis | The measurement of observer agreement for categorical data[END_REF] is obtained by the pair SA⇥EOA. This could be explained by the fact that the experiment subjects are asked to watch their own videos before providing their self-assessment. Thus, they looked at the same distal cues as the external observers before judging whether they felt stressed or not. The low kappa scores obtained by PEA with SA and EOA may be explained by the differences in distribution: PEA is more balanced (Table 5) than SA (Table 4) and EOA (Table 3). Since the kappa scores are impacted by the choice of a specific threshold for each assessment, we also used correlation analysis on non-binary values. Table 7 presents the correlation coefficient between the non-binary values associated with each assessment:

• EOA: the proportion of annotators that answered "Stressed" for Q1.

• SA: the self-reported answers to the Likert-scaled (1-5) question asked during the debriefing of the experiment.

• PEA: the HRV-LF% values.

The correlation coefficients are very similar to the kappa scores: the only significant correlation is obtained by the pair SA⇥EOA. The correlation coefficient -0.41 -indicates a modest correlation. There is no significant correlations between PEA and EOA and between PEA and SA.

Overall, the kappa scores and the correlation coefcients are going in the same direction. The lack or limited correlation found supports : (1) the idea that stress is a complex phenomenon which can be expressed through ones body, behaviour and/or mind. ( 2) physiological parameters may differ in timing for stress induction compared to behavioral cues; (3) despite the correlation between EOA and SA, it appears that the two phenomena have both common basis and separate cues. Thus, it is important to assess stress in several ways because of the diversity of its expression.

FRAMEWORK FOR AUTOMATIC STRESS DETEC-

TION

In this section, we describe how we use the data and annotations described in Sections 2 and 3 to perform automatic stress detection.

Feature extraction

In this work, we extract 101 features from 3 sources: body features from the Kinect data, facial features from the HD video and physiological features from the signals provided by the Nexus-10. Body and facial features are presented in Table A1 gathered as behavioural features. Physiological features are presented in Table A2.

Body features

In the literature, several sets of body features have been used to recognize or to synthesize someone's affective state [START_REF] Caridakis | Synthesizing Gesture Expressivity Based on Real Sequences[END_REF], [START_REF] Giraud | Assessing postural control for affect recognition using video and force plates[END_REF], [START_REF] Glowinski | Toward a Minimal Representation of Affective Gestures[END_REF]. Regarding automatic stress detection systems, only few of them actually use body features. In [START_REF] Giakoumis | Using activity-related behavioural features towards more effective automatic stress detection[END_REF],

Giakoumis et al. state that behavioural features such as body movement, head position and posture enhance the performances of "standard" systems based on physiological features. In this work, we extract 15 original features from body activity, posture and occurence of specific gestures. We describe them in the following paragraphs.

Quantity of Movement

The main body activity feature extracted is the Quantity of Movement (QoM). We compute it in two ways: using the RGB video (IQoM), and using the skeleton joints (SQoM) (Figure 5). IQoM is the number of pixels that changed between two successive frames and SQoM is the sum of the displacements of the skeleton joints. Each method has its advantages and drawbacks. SQoM enables us to detect slight movements in the camera axis. However, as the Kinect skeleton can be unstable during the recordings, IQoM is also used in order to extract a less noisy quantity of movement. For both IQoM and SQoM, we compute their mean value over all the Kinect video frames of the protocol step. Then, in order to make these features invariant to the size of a person and to the distance between her and the camera, we normalize them with respect to the surface of the box bounding the person. We also compute the SQoM only for the head joint (HeM) and isolate its movement along the camera-axis (HeMZ).

Periods of high body activity

We make the hypothesis that periods of high body activity characterize an increasing uncomfortability. These periods are extracted by detecting the peaks in the IQoM signal (Figure 6). We use the number of periods extracted (HAPC), their average duration (HAPMD) and their average intensity (HAPMV) as features.

Posture changes

As for periods of high body activity, posture changes may reveal uncomfortability. We use the number of posture changes (PCC) that occur during the video as a feature.

Because of the skeleton stability issues in the recordings, especially when the participant crosses her arms, we use the periods of high body activity described previously to extract the posture changes. For each period, we compare the first frame and the last frame by computing their difference (Figure 7). If the number of pixels divided by the surface of the bounding box is above a given threshold, we consider that there is a posture change. Thus, we can consider that this feature is an additional information for periods of high activity.

Detection of self-touching

Harrigan suggests in [START_REF] Harrigan | Self-touching as an indicator of underlying affect and language processes[END_REF] that self-touching can be an indicator of negative affect. We detect two types of self-touching: face touching, which is part of the displacement behaviours described by Troisi [START_REF] Troisi | Displacement Activities as a Behavioral Measure of Stress in Nonhuman Primates and Human Subjects[END_REF], and rubbing fingers together. Since detecting self-touching requires a precise tracking of the hand, we use skin detection to refine the hand joint location provided by the Kinect. Starting from the position given by the Kinect, we look for the closest skin pixel detected. This becomes the new position of the hand. To determine if a person is touching her face, we compute the hand-head and the hand-neck distances. If one of these distances is below a given threshold, we consider that the person is face touching (Figure 8). The number of occurences (FTC) and the average duration (FTMD) are used as features. Similar features are extracted when the person is self-touching her head with two hands (FT2HC and FT2HMD). To detect gestures such as rubbing fingers together, the Kinect skeleton is not sufficient since it does not provide joints for the fingers. Thus, using the hand positions, we first extract the sub-image of each hand region. Then, we compute the IQoM between successive extracted subframes. We only compute it when the person is not moving her hand since the IQoM can be affected by changes in the background. This feature is computed for each hand separately (LHM for the left hand, RHM for the right one) and for both (HM).

Facial features

For facial expressions, we extract the activation levels of 12 Actions Units (AU). Paul Ekman presented AUs as part of the Facial Action Coding System (FACS) [START_REF] Ekman | Facial action coding system[END_REF]. This system has become one of the standards for systematic categorization of facial expressions. To extract these activation levels, we use the method presented in [START_REF] Nicolle | Facial Action Unit Intensity Prediction via Hard Multi-Task Metric Learning for Kernel Regression[END_REF], which proposes a multi-task extension for a subspace learning algorithm called Metric Learning for Kernel Regression. Once we have extracted the activation level of each AU for each HD video frame, we compute the average and the standard deviation and use them as features.

Physiological features

The Nexus-10 device is used to extract physiological features classically associated in the literature with stress: cardiac functions (blood volume pulse, heart rate and heart rate variability) [START_REF] Koolhaas | Stress revisited: a critical evaluation of the stress concept[END_REF], [START_REF] Finsen | Muscle activity and cardiovascular response during computermouse work with and without memory demands[END_REF], respiratory system (chest and abdominal respiration) [START_REF] Widjaja | Cardiorespiratory dynamic response to mental stress: a multivariate time-frequency analysis[END_REF], galvanic skin response [START_REF] Healey | Detecting Stress During Real-World Driving Tasks Using Physiological Sensors[END_REF], [START_REF] Shi | Galvanic skin response (GSR) as an index of cognitive load[END_REF], skin temperature [START_REF] Herborn | Skin temperature reveals the intensity of acute stress[END_REF] and electromyographic activity of the sternocleidomastoid and upper trapezius [START_REF] Wijsman | Trapezius muscle emg as predictor of mental stress[END_REF]. For most of the signals described in Table A2, the mean, the variance, the minimum and the maximum values are used as features. In total, 62 physiological features are extracted.

Feature transformation

One issue when working with humans is interindividual differences. Indeed, as said in [START_REF] Bernhardt | Detecting affect from non-stylised body motions[END_REF], "Different people tend to display the same emotion in very different ways". These differences may impact the feature distributions and prevent the machine learning algorithm from finding the most adequate model. We use the Box-Cox transformation [START_REF] Sakia | The Box-Cox Transformation Technique: A Review[END_REF] to normalize the feature distributions in a systematic way. The Box-Cox transformation is defined as:

x 0 = x 1
The aim is to find the value of that maximizes the correlation between the transformed feature x 0 and the normal distribution N (µ(x 0 ), (x 0 ) 2 ). We only compute the correlation for specific values of that can be found in Table 8. Tables A1 andA2 show which transformation is used for each behavioural and physiological feature.

Feature subset selection

We perform feature subset selection in order to avoid overfitting and better understand the predictive power of each feature. Each result presented in Section 5 corresponds to the best one obtained among the 3 following feature subset selection methods.

Forward selection wrapper (FSW)

Wrappers evaluate a subset of features by using the same machine learning algorithm as in the final application [START_REF] Kohavi | Wrappers for feature subset selection[END_REF].

In our case, we use a SVM with a linear kernel function.

Since training SVMs is computationally expensive, exploring the space of feature subsets is usually done using greedy methods [START_REF] Guyon | An Introduction to Variable and Feature Selection[END_REF]. With forward selection, starting from using only the feature with the best accuracy, we iteratively add the best feature among the remaining ones. Once all features have been added, we keep the subset that gives the best classification performances.

Backward elimination wrapper (BEW)

This method also uses a SVM to evaluate subsets. Backward elimination is also a greedy search strategy: starting from the complete set of features, we iteratively remove the worst feature of the remaining set. Once all features have been removed, we keep the subset that gives the best classification performances.

Simulated annealing with Hall correlation (SAHC)

For this method, we use the simulated annealing metaheuristic [START_REF] Kirkpatrick | Optimization by simmulated annealing[END_REF] to explore the space of feature subsets. Because of the computational cost of this space search strategy, we use the Hall correlation [START_REF] Mark | Correlation-based Feature Selection for Machine Learning[END_REF] to evaluate feature subsets.

We then get a good approximation of the subset that both maximizes the correlation between features and labels and minimizes the inter-feature correlation.

Evaluation Process

We use a classification task to evaluate the predictive value of our framework. The objective is to predict the binary stress label -Stress or Non-Stress -of each of the 126 collected steps. To do so, we use SVMs with three different kernel functions: linear, polynomial and radial basis. We use a 10 fold subject independant cross validation strategy to compute the results: steps from 2 or 3 people are used as the testing set. The steps of the remaining people are used as the training set. This cross validation is also used with the training set to determine the SVM and kernel function parameters. Since our dataset is unbalanced for 2 assessment sets -SA and EOA -we have chosen the average of the F1 score for both Stress and Non-Stress classes as the performance metric. This metric allows us to consider the recall and the precision of both classes, unlike the usual F1 score that considers the recall and the precision only for the positive class and ignores the true negative rate. We use the Student's t-test to compare two average F1 scores.

It is important to note that we use all the data for the feature transformation and feature subset selection steps. It has been done in order to facilitate the interpretation of the results and of the revelance of each feature. Consequently, we also present the average F1 score when the parameters for the feature transformation and the feature subset selection are selected using only data from the training set and are then applied on the testing set.

EVALUATION OF OUR FRAMEWORK

In this section, we present the predictive power of our framework for the 3 assessment sets described in Section 3.

We present the results obtained by each modality -behaviour and physiology -and by each feature.

Evaluation of the predictive power of each modality

Figures 9, 10 and 11 show the classification results obtained by the best selected feature subset for all 3 assessment sets. Feature are selected from the whole set of features (All*), only behavourial ones (Behaviour*) or only physiological ones (Physio*). Regarding PEA, we compute the results in 2 different conditions. First, we compute the classification results after having dismissed all the features which are theoretically too correlated to the heart-rate variability: HRV-LF%, HRV-SDNN, HRV-RMSSD and RSP+HR. Including all the functionals applied to each feature (i.e. mean, standard deviation, min and max), we dismissed a total of 10 features that we refer to as Physiology Label Related (PLR) features.

In the second condition,we dismiss only the features directly related to the low frequency in the heart-rate variability (HRV-LF%) that we used as our assessment. Including all the functionnals, we dismissed 4 features.

In general, we can see that, for most of the subsets, the linear kernel outperforms both RBF and polynomial kernels. It is due to the fact that most of the best subsets were provided by one of the 2 wrappers, which are optimized for the linear kernel. This may artificially boost its performances, thus inducing a bias when comparing modalities. We take this into account in the following discussion: when necessary, we also compare the average F1-scores of two modalities without including the linear kernel. Overall, the results show that depending on the assessment set considered, the effectiveness of each modality and of their combination varies.

All Regarding EOA (Figure 9), we can see that both modalities achieve good mean F1 scores: 0.829 (± 0.025) for behavioural features and 0.758 (± 0.033) for physiological ones. It is not surprising that behavioural features significantly outperform physiological ones (p < 0.0001) since annotators based their judgement solely on the behaviour of the person in each video. It is however interesting to see that physiological features can predict how stress is assessed by external observers. It could have been explained by the fact that some of the physiological features selected can be visually perceived: features related to the respiration rate and EMG features, which can reflect the upper body activity. But the results obtained after having dimissed these features are similar with a mean F1 score of 0.751 (± 0.028). This feature subset is composed of 15 features: 7 related to HRV, 4 related to HR, 2 related to BVP, one related to GSR and one to skin temperature. The best results are obtained when we combine both modalities with a mean F1 score of 0.855 (± 0.020). The subset All* is composed of 24 features: 15 behavioural features and 9 physiological ones. It is however interesting to note that the difference in mean F1 score between the subsets All* and Behaviour* is statistically significant if we consider the 3 kernel functions, but is not if we do not consider the linear kernel. Overall, it seems that using only behavioural features is sufficient for the prediction of EOA. When we use all features and we include the feature transformation and the feature subset selection in the training phase, we obtain a mean F1 score of 0.739 (± 0.023).

Regarding SA, Figure 10 shows that the combination of physiological and behavioural features outperforms the results obtained when using only one modality. standable since the subjects of the experiment described in Section 2.1 watch their own videos before annotating them. Thus, their answers are the result of both their personal experiences and their behaviour analysis. The subset All* obtains a mean F1 score of 0.795 (± 0.028) and is composed of 32 features: 21 physiological features and 11 behavioural ones. When we use all features and we include the feature transformation and the feature subset selection in the training phase, we obtain a mean F1 score of 0.691 (± 0.034). Figure 11 displays the results obtained for the prediction of PEA. As expected, physiological features obtain a good classification performance with an average F1 score of 0.777 (± 0.021) for the first condition (i.e. no features related to the heart-rate variability) and of 0.810 (± 0.026) for the second condition (i.e. no features related only to the low frequency in the heart-rate variability). As expected, using the features related to the heart-rate variability significantly improve the results (p = 0.0059) since these features are related to the one we used to compute PEA labels. Overall, both conditions significantly outperforms behavioural features (p < 0.05). It is however surprising to see that using only behavioural features also provides a good average F1 score of 0.740 (± 0.020). The selected subset Behaviour* is composed of 10 features: 7 features related to action units, 2 features related to body movement and the mean duration of face touching (FTMD). Regarding the combination of behavioural and physiological features, there is no significant difference between the results obtained by both conditions: 0.831 (± 0.020) and 0.833 (± 0.021) for the first and second condition respectively. However, it is noteworthy that the best subset selected for the second condition is smaller -17 features (12 physiological and 5 behavioural ones) against 25 features (16 physiological and 9 behavioural ones) for the first condition -and contains 3 features related to HRV: Features selected in All*: AU1-mean, AU2-std, AU15-mean, AU17-std, AU25-mean, AU25-std, AU26-mean, BVPAmean, BVPA-min, BVPA-var, EMGA-mean, EMGMF-max, EMGMF-mean, EMG-min, GRS-max, HR-max, HR-mean, HRVA-max, HRVA-var, RSPA-max, RSPA-min, RSPA-var, RSPR-var, FTC, FTMD HRV-RMSSD, HRV-SDNN and RSP+HR-Mean. When we use the features of the first condition and we include the feature transformation and the feature subset selection in the training phase, we obtain a mean F1 score of 0.705 (± 0.020).

Evaluation of the predictive power of each feature

We use the evaluation process described in Section 4.4 using only one feature at a time in order to better understand the classification performance of each feature for each assessment set. We compute the F1 score obtained by the three kernel functions. The average F1 score is used to rank features. In order not to overload the charts, we present only the five best features of each assessment set. The average F1 scores for each feature are presented in Tables A1 andA2.

Regarding EOA, the results obtained by the 5 best features are shown in Table 9. We can see that these features achieve good classification performances even when used alone. Among these 5 features, there are 4 behavioural features and one physiological one, which is not surprising since the external observers had only access to the participants' behaviour. The 4 behavioural features are all related to movement: 2 features are related to head movement (HeM and HeMZ), one to hand movement (HM) and one to body movement (IQoM). The best physiological feature is the minimum of the Blood Volume Pulse (BVP -Min). It is noteworthy that the 5 best physiological features for the prediction of EOA are all related to the BVP: 3 are related to the raw BVP signal (BVP-Min, Mean and Var) and 2 are related to the amplitude of the BVP signal (BVPA-Var and Max). Table 10 displays the 5 best features for the prediction of SA. We can notice that these features achieve lower F1 scores than the best features for EOA and PEA. Added to the fact that, as shown in Figure 10, only a combination of

DISCUSSION

In this section, we interpret and discuss the results obtained and discuss the strengths and limitations of the current study. We look at both the composition of the best selected feature subsets and the classification performances of each feature in order to better evaluate the relevance of each modality/feature for automatic stress detection.

Results interpretation

It was expected to see behavioural features perform better than physiological ones for the prediction of the assessment of external observers (EOA) and to see physiological features achieve better performances than behavioural ones for the prediction of the assessment of a physiology expert (PEA). However, it is interesting to notice that behavioural features still achieved good performances for PEA prediction (Figure 11, mean F1 score = 0.74), and that physiological features also provided good performances for EOA prediction (Figure 9, mean F1 score = 0.78). The fact that we can predict both the assessment of a physiology expert from behavioural features and the assessment of external observers from physiological features shows that there is some coherence between behavioural and physiological cues when one is experiencing stress despite the lack of agreement between EOA and PEA annotations (Tables 6 and7). This interplay between physiology and behaviour that we observe in our results is coherent with several works on facial expressions [START_REF] Davidson | Approach-withdrawal and cerebral asymmetry: emotional expression and brain physiology[END_REF], [START_REF] Ekman | The Duchenne smile: emotional expression and brain physiology[END_REF], [START_REF] R W Levenson | Emotion, physiology, and expression in old age[END_REF], emotions [START_REF] Scherer | On the nature and function of emotion: A component process approach[END_REF], and stress [START_REF] Weisman | Dynamics of non-verbal vocalizations and hormones during father-infant interaction[END_REF], [START_REF] Megan R Gunnar | Psychobiological studies of stress and coping: An introduction[END_REF].

However, for all 3 assessment sets considered, the combination of behavioural and physiological features provided the best results. It is especially true for SA, for which multimodal features outperform behavioural features by +31% and physiological features by +27%, as shown in Figure 10. Overall, we think that when the obtrusiveness of physiological sensors is acceptable, it is preferable to use a combination of behavioural and physiological features for automatic stress detection. Nonetheless, when unobtrusiveness is required, using only behavioural features still provide good classification performances. These results are coherent with those presented by Giakoumis et al. in [START_REF] Giakoumis | Using activity-related behavioural features towards more effective automatic stress detection[END_REF]. We also investigate whether some features provide relevant information for more than one assessment. The results obtained for SA and EOA are similar in some aspects (Tables 9 and10). Indeed, the 5 best features for these 2 assessment sets are mainly related to body or body part movement (HeM, HeMZ, HM, IQoM, SQoM and HAPMV). The 5 best features for the prediction of PEA are all related to the heart and belong to 2 categories: heart rate (HR -max and HR -var) and amplitude of the heart rate variability (HRVAmax, HRVA -mean, HRVA -var).

If we look at both the composition of the best subset for each assessment set and the predictive power of each feature (cf. Tables A1 andA2), it appears that some features provide relevant information for a multi-perspective stress detection.

In particular, IQoM achieves good F1 scores for the 3 assess-ment sets and is present in the best subsets selected for SA and EOA. In general, features related to body or body part movement (IQoM, HeM, HeMZ, HM) provide good results for both SA and EOA prediction. Then, features related to the amplitude of blood volume pulse are present in all of the 3 best select subsets, and features related to the raw signal of blood volume pulse provide good classification performances for EOA prediction. Finally, the maximum of the heart rate achieves reasonably good performances and is present in the subsets selected for SA and PEA. These results and the fact that several works report the effect of stress on BVP [START_REF] Finsen | Muscle activity and cardiovascular response during computermouse work with and without memory demands[END_REF], [START_REF] Hjortskov | The effect of mental stress on heart rate variability and blood pressure during computer work[END_REF] and heart-rate [START_REF] Koolhaas | Stress revisited: a critical evaluation of the stress concept[END_REF], [START_REF] Taelman | Influence of Mental Stress on Heart Rate and Heart Rate Variability[END_REF] lead us to conclude that these 2 categories of features, along with body movement, provide valuable information when designing automatic stress detection systems.

Strengths and limitations

This work should be interpreted within the context of its strengths and limitations. The strengths include (i) the multi-perspective approach of the stress phenomenon; (ii) the introduction of new behavioural features for stress detection: face-touching and fingers-rubbing, (iii) the high number of features evaluated and (iv) the high classification scores that allow us to interpret predictive power of some features. The limitations include (i) the sample size; (ii) the bias regarding participants being all volunteer medical students; and (iii) the experimental stimulus that elicits stress in a specific context.

CONCLUSION AND PERSPECTIVES

In this paper, we have studied the classification performances of modalities and features for the automatic detection of a multi-perspective stress. The definition of stress is still debated and can be studied from different perspectives.

We have looked at this phenomenon from 3 perspectives: the phenomenological, the behavioural and the biological ones. For the classification task, we have collected 3 different assessments:

• external oberservers assessment from annotators recruited on a crowdsourcing platform for the behavioural perspective;

• self-assessment from the subjects for the phenomenological perspective;

• assessment from a physiology expert for the biological perspective;

The low agreement between the different annotations displayed in Tables 6 and7 show that it is important to consider the 3 annotations in order to get a broader view on the performances of stress detection systems. We have extracted 3 categories of features from different sources: body features from Kinect data, facial features from HD video and physiological features from signals provided by the Nexus-10. In total, 101 features have been automatically extracted. We performed binary classification on the 3 assessment sets to evaluate our features. Overall, it appears that it is preferable to use a combination of behavioural and physiological features for automatic stress detection and that there is an interplay between physiology and behaviour in stressful situations that could be used to enhance the performance of stress detection. By investigating the composition of the best selected feature subsets and the individual feature classification performances, we show that features related to body movement, blood volume pulse and heartrate provide valuable information for the classification of the 3 assessments. Future works include studying how we can use the interplay between physiology and behaviour to enhance the performance of automatic stress detection, investigating the feasibility of aggregating the introduced assessments, using a multi-label classification task and applying this methodology to other mental states or disorders such as anxiety disorders.

APPENDIX

Fig. 4 :

 4 Fig. 4: Answers to Q2 according to the answer given for Q1. Best viewed in color.

Fig. 5 :

 5 Fig. 5: The skeleton joints extracted by the Kinect 2

Fig. 6 :

 6 Fig. 6: Extraction of periods of high body activity from the IQoM. Blue line: IQoM values per frame. Red dashed line: periods of high activity extracted. Best viewed in color.

Fig. 7 :

 7 Fig. 7: Example of detection of a posture change. From left to right: first frame of the period, last frame and their thresholded absolute difference.

Fig. 8 :

 8 Fig. 8: Examples of detections of face touching. Left image: face touching with one hand. Right image: face touching with two hands.

2 TABLE 8 :

 28 Tested values of for the Box-Cox transformation and their associated transformation functions.

(Fig. 11 :

 11 Fig.11: Performances of each kernel for each modality for the prediction of PEA without using features related to HRV. The baseline average F1 score obtained by a random classifier is 0.422 (± 0.080). Features selected in All*: AU1-mean, AU2-std, AU15-mean, AU17-std, AU25-mean, AU25-std, AU26-mean, BVPAmean, BVPA-min, BVPA-var, EMGA-mean, EMGMF-max, EMGMF-mean, EMG-min, GRS-max, HR-max, HR-mean, HRVA-max, HRVA-var, RSPA-max, RSPA-min, RSPA-var, RSPR-var, FTC, FTMD

TABLE 1 :

 1 Stimuli, signals and stress annotations for some automatic stress detection systems.

	Study	Stimulus	Signals	Stress annotation
	Barreto et al. [4]	Stroop test	BVP, GSR, skin temperature and pupil	Task complexity
			diameter
	Chen et al. [5]	Trier Social Stress Test	Tissue oxygen saturation	Assessement from cortisol
	Fernandez et al. [6]	Mental arithmetic task while driv-	Speech	Experimental conditions
		ing
	Gaggioli et al. [7]	Naturalistic settings	ECG and 3D acceleration	Self-assessment
	Giakoumis et al. [1] Stroop test	GSR, ECG, body movement, head posi-	Assessment from GSR +
			tion, posture and occurence of specific	self-assessment
			gestures
	Healey et al. [8]	Driving in several conditions	ECG, EMG, GSR and respiration	Assessment from external
				observers
	Lefter et al. [2]	None	Speech, movement and gestures valence	Acted
			and arousal
	Plarre et al. [9]	Public speaking task + mental arith-	ECG, respiration and 3D acceleration	Experimental conditions +
		metic task + cold pressor		self-assessment
	Shi et al. [10]	Public speaking task + Mental arith-	HR, ECG, respiration, GSR, temperature Self-assessment
		metic task + cold pressor
	Wijsman et al. [11]	Mental arithmetic + logical puzzle +	HR, ECG, EMG and respiration	Self-assessment
		memory tasks
	Zhou et al. [12]	Riding a roller-coaster	Speech	Experimental conditions +
				acted
	Our approach	Mental arithmetic task	HR, GSR, EMG, respiration, skin tem-	Self-assessment + assess-
			perature, body movement, posture, oc-	ment from external ob-
			curence of specific gestures and actions	servers + assessment from
			units	HRV

TABLE 3 :

 3 EOA label distribution.

TABLE 4 :

 4 SA label distribution.

TABLE 5 :

 5 PEA label distribution.

TABLE 6 :

 6 Cohen's Kappa for each combination of 2 assessment sets.

	Assessment sets	SA⇥EOA SA⇥PEA EOA⇥PEA
	Correlation coefficient	0.41*	-0.11	-0.06

TABLE 7 :

 7

Correlation coefficients for each combination of 2 assessment sets. The only significant correlation (p < 0.05) is marked with *.

TABLE 9 :

 9 Five best features according to their average F1 score for the prediction of EOA. The Stdev column represents the standard deviation over 10 runs. physiological and behavioural features achieved good classification performances, it tends to show that the prediction of SA is more complex, is based on both behavioural and physiological cues, and requires more information than the prediction of EOA and PEA. Among the 5 best features, 3 are behavioural and 2 are physiological. 2 features are related to body movement (IQoM and HeM), 2 to blood volume pulse (BVP -Min and Var) and one to periods of high activity (HAPC).

	Feature	Average F1 Stdev
	HeM	0.780	0.016
	IQoM	0.723	0.025
	HeMZ	0.716	0.021
	BVP-Min	0.705	0.029
	HM	0.696	0.024
	Feature	Average F1 Stdev
	IQoM	0.621	0.028
	HAPMV	0.617	0.028
	SQoM	0.616	0.025
	HeM	0.614	0.038
	RSP-Min	0.609	0.031

TABLE 10 :

 10 Five best features according to their average F1 score for the prediction of SA. The Stdev column represents the standard deviation over 10 runs.

	The 5 best features for the prediction of PEA are all related
	to the heart: 3 are related to the heart rate (HR-Mean,
	Max and Var) and 2 are related to the amplitude of the
	heart rate variablity (HRVA -Max, Mean). Their average
	F1 scores range from 0.711 ± 0.047 for HR-Mean to 0.652 ± 0.024. These results are coherent with what we described in Section 1.1.1 for the biological perspective: activating the
	HPA pathway and the ANS leads to an increased heart
	rate, which impacts the heart rate variability. Thus, it is not
	surprising to see these features perform well.
	However, it is also interesting to see which non cardiac
	features are relevant for the prediction of a heart-related
	annotation. Thus, Table 11 presents the 5 best non heart-
	related features for the prediction of PEA. 3 features are
	related to respiration (RSP-Mean, RSPR-Max and Min) and
	2 are related to action units (AU4-Mean, AU2-Std). It is also
	noteworthy that 4 of the 5 best behavioural features for the
	prediction of PEA are facial features (AU4-Mean and Std,
	AU2-Std and AU9-Std).

TABLE 11 :

 11 Five best non heart-related features according to their mean F1 score for the prediction of PEA. The Stdev column represents the standard deviation over 10 runs.

	Feature	Average F1 Stdev
	RSP-Mean	0.621	0.028
	RSPR-Max	0.617	0.033
	AU4-Mean	0.600	0.031
	RSPR-Min	0.590	0.043
	AU2-Std	0.590	0.030

  Marcin Detyniecki is professor at the Polish Academy of Science (IBS PAN) and associate researcher at the computer science laboratory LIP6 of the University Pierre and Marie Curie (UPMC). He has worked on the usage of new media, with challenges ranging from multimedia information retrieval to image understanding. Several of the developed applications have not only been deployed in the market, but they have also been singled out in international competitions such as TrecVid, ImageClef, MediaEval. This applicative success is the results of a dialogue with more theoretical works on topics such as new challenges in approximate reasoning, information aggregation and fusion, and machine learning from a computational intelligence perspective. Marcin Detyniecki studied mathematics, physics and computer science at the University Pierre and Marie Curie (UPMC) in Paris. In 2000 he obtained his Ph.D. in Artificial Intelligence from the same university. Between 2001 and 2014, he was a research scientist of the French National Center for Scientific Research (CNRS). Today he is member of the research and academic council of UPMC University, member of the executive board of laboratory SMART, elected member of the LIP6 laboratory council, and member of the editorial board of the International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems (IJFUKS). He also funded and animated until 2014 the UPMC Sorbonne Universit és Computer Science Colloquium. Dr. Detyniecki has over 90 publications in journals and conference proceedings, including 6 keynotes. David Cohen received a M.S. in neurosciences from the University Pierre and Marie Curie (UPMC) and the Ecole Normale Sup érieure in 1987, and a M.D. from Necker School of Medicine in 1992. He specialized in child and adolescent psychiatry and certified in 1993. His first field of research was severe mood disorders in adolescent, topic of his PhD in neurosciences(2002). He is Professor at the UPMC and head of the department of Child and Adolescent Psychiatry at La Salp étri ère hospital in Paris. He is also member of the lab Institut des syst èmes Intelligents et Robotiques (CNRS UMR 7222). His group runs research programs in the field of pervasive developmental disorder (autism) and learning disabilities, childhood onset schizophrenia, catatonia and severe mood disorder. He supports a developmental and plastic view of child psychopathology, at the level of both understanding and treatment. His team proposes a multidisciplinary approach and therefore collaborates with molecular biologist, methodologist, experimental psychologist, sociologist and engineer. He has published numerous research papers (more than 100) including some in high impact journals such as the American Journal of Psychiatry, Biological Psychiatry, Nature, Nature Genetics, PlosOne, Psychotherapy & Psychosomatics, World Psychiatry and the Journal of the American Academy of Child and Adolescent Psychiatry (see http://speapsl.aphp.fr).MohamedChetouani is the head of the IMI2S (Interaction, Multimodal Integration and Social Signal) research group at the Institute for Intelligent Systems and Robotics (CNRS UMR 7222), University Pierre and Marie Curie-Paris 6. He received the M.S. degree in Robotics and Intelligent Systems from the UPMC, Paris, 2001. He received the PhD degree in Speech Signal Processing from the same university in 2004. In 2005, he was an invited Visiting Research Fellow at the Department of Computer Science and Mathematics of the University of Stirling (UK). Prof. Chetouani was also an invited researcher at the Signal Processing Group of Escola Universitaria Politecnica de Mataro, Barcelona (Spain). He is currently a Full Professor in Signal Processing, Pattern Recognition and Machine Learning at the UPMC. His research activities, performed at the Institute for Intelligent Systems and Robotics, cover the areas of social signal processing and personal robotics through nonlinear signal processing, feature extraction, pattern classification and machine learning. He is also the co-chairman of the French Working Group on Human-Robots/Systems Interaction (GDR Robotique CNRS) and a Deputy Coordinator of the Topic Group on Natural Interaction with Social Robots (euRobotics). He is the Deputy Director of the Laboratory of Excellence SMART Human/Machine/Human Interactions In The Digital Society.

	Jonathan Aigrain is a PhD student at the Institute for Intelligent Sys-
	tems and Robotics (CNRS UMR 7222), University Pierre and Marie
	Curie-Paris 6. He also obtained the Engineering degree from the EPITA
	Engineering School in 2012 and the Masters degree from University of
	Paris 6 in 2013. His PhD work focuses on multimodal stres detection.
	His interests include CAPTCHA solving methods, affective computing,
	behaviour recognition and machine learning.
	Michel Spodenkiewicz received a MSc in research in psychopathology
	from the University Paris Descartes -Sorbonne Universit és in 2011 and
	a MD from the University Paris Diderot -Sorbonne Universit és in 2013.
	He specialized in child and adolescent psychiatry and certified in 2014.
	His first field of research was methodological innovation and multimodal
	assessment of mental states and psychiatric disorders during adoles-
	cence. He works as a physician in child and adolescent psychiatry and
	as a researcher in methodology at the Centre Hospitalier Universitaire
	Sud R éunion and its Centre dInvestigation Clinique (CIC-EC 1410) in
	Saint Pierre on the Reunion Island, France. He is also member of the
	Institute for Intelligent Systems and Robotics (CNRS UMR 7222) and
	Inserm U1178.

S éverine

Dubuisson was born in 1975. She received the Ph.D. degree in system control from the Compi ègne University of Technology, France, in 2001. From 2002 to 2013, she has been an Associate Professor with the Laboratory of Computer Sciences (LIP6), UPMC Sorbonne Universit és, France. She is now associate professor in Institut for Intelligent Systems and Robotics (ISIR), UPMC Sorbonne Universit és, France. Her research interests include computer vision, visual tracking, probabilistic models for video sequence analysis and human interaction.

JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

ACKNOWLEDGMENTS

This work was partially supported by the Labex SMART (ANR-11-LABX-65) under French state funds managed by the ANR within the Investissements d'Avenir programme under reference ANR-11-IDEX-0004-02.

x' represents the transformation given by the Box-Cox transformation for each function applied to the signal. F1 score displays the results obtained by the each feature when used alone for each assessment set. The 5 best features of each assessment set are in bold. In best subset shows whether the feature is present in the best subset selected for each assessment set.


M Mikikian
email: maxime.mikikian@univ-orleans.fr

S Labidi

E Von Wahl

J F Lagrange

T Lecas

V Massereau-Guilbaud

I Géraud-Grenier

E Kovacevic

J Berndt

H Kersten

T Gibert

Optical diagnostics of dusty plasmas during nanoparticle growth

Carbon-based thin films deposited on surfaces exposed to a typical capacitively-coupled RF plasma are sources of molecular precursors at the origin of nanoparticle growth. This growth leads to drastic changes of the plasma characteristics. Thus, a precise understanding of the dusty plasma structure and dynamics is required to control the plasma evolution and the nanoparticle growth. Optical diagnostics can reveal some particular features occurring in these kinds of plasmas. High-speed imaging of the plasma glow shows that instabilities induced by nanoparticle growth can be constituted of small brighter plasma regions (plasmoids) that rotate around the electrodes. A single bigger region of enhanced emission is also of particular interest: the void, a main central dust-free region, has very distinct plasma properties than the surrounding dusty region. This particularity is emphasized using optical emission spectroscopy with spatiotemporal resolution. Emission profiles are obtained for the buffer gas and the carbonaceous molecules giving insights on the changes of the electron energy distribution function during dust particle growth. Dense clouds of nanoparticles are shown to be easily formed from two different thin films, one constituted of polymer and the other one created by the plasma decomposition of ethanol.

Introduction

Dusty plasmas are fascinating media revealing very complex interactions between dust particles and between dust particles and the plasma. The presence of nanoparticles or dust particles in a plasma is a quite common situation. These large particles (with respect to ions and electrons) can be deliberately produced by using appropriate plasma chemistries and conditions or unintentionally obtained due to an uncontrolled nucleation in the gas phase or the injection of wall or electrode fragments. These multicomponent plasmas are then called dusty (or complex) plasmas [1][2][3] and the presence of this additional massive species induces the existence of many new phenomena. Dusty plasmas are easily encountered in many different fields as in astrophysics [4][5][6][7], thermonuclear fusion plasmas [8][9][10] and industrial plasmas [11,12]. The knowledge of chemical and physical processes at the origin of nanoparticle formation, growth and dynamics is thus of major interest and needs to be investigated both theoretically and experimentally.

Capacitively-coupled radio-frequency (ccrf) discharges are convenient setups to implement precise insitu diagnostics that can reveal the fundamental processes involved in dusty plasmas. In order to form nanoparticles, two main approaches are used, either through the injection of a molecular precursor or by sputtering a thin film deposited on the electrodes. Several molecular precursors can be used, originally in the gas state (Silane [11,13], HMDSO [14,15], acetylene [16][17][18][19][20], or methane [6,[21][22][23]. When dust particles are obtained by sputtering a thin film, the plasma ions impinging the surfaces exposed to the plasma are injecting reactive species in the gas phase. Then, these molecules behave in a rather similar way than an injected reactive gas and enter in reaction pathways leading to nanoparticle growth. Many materials can be easily sputtered in this way: graphite [24][25][26], tungsten [25,27], polymers [3,28] or other materials [25,29,30].

In the present study, thin film sputtering is the chosen method to grow dense nanoparticle clouds. As nanoparticles are growing in the plasma, they get electrically charged and strongly disturb the electron distribution function. Thus, changes in plasma and discharge characteristics contain a footprint of the growth process. These modifications can be detected using non-intrusive optical diagnostics. In this paper, we show that normal or high-speed imaging and optical emission spectroscopy with spatiotemporal resolution are interesting techniques to reveal some impressive phenomena occurring in dusty plasmas. A focus is proposed on particular instabilities (rotating plasmoids) occurring during dust particle growth and on the complex dust cloud structure containing a central dust-free region (the void) with different plasma properties. More generally, this work shows that dense dust particle clouds can be easily obtained from thin film sputtering.

High-speed imaging of plasma emission instabilities during dust particle growth

Due to the presence of dust particles affecting the plasma equilibrium, dusty plasmas are media where many different types of instabilities can appear. Usually, these instabilities can be detected easily by any plasma or discharge diagnostics. In some particular cases, high-speed imaging is able to reveal unstable phenomena that are not detectable with other diagnostics. It is for example the case of a particular behavior that we called the carousel instability [31] and consisting of millimeter size regions of enhanced emission (plasmoids) rotating around the electrode circumference.

These experiments are performed in the PKE-Nefedov reactor, a ccrf discharge in push-pull mode (both electrodes are powered), where nanoparticles are grown by sputtering a thin film composed of polymer microparticles (melamine formaldehyde) and grown nanoparticles from previous experiments. This film present on the electrodes is sputtered by an Ar plasma created at ∼ 4 W and ∼ 1.5 mbar (without gas flow).

The carousel instability is not well-detected by the diagnostics integrating global changes of plasma or discharge properties. Indeed, the plasma is not globally changing during the rotation of these regions, it is just the places of enhanced emission that are moved. Plasmoids have been observed in several typical laboratory discharges [31][32][33][34][35][36][37][38] but their origin is still unclear. They seem to be favored by electric field disturbances as they are observed in grid holes [37] or preferentially close to guard rings encircling electrodes [31][32][33]. In this last case, the number of plasmoids can vary and their regular rotation around the circumference of the electrodes has been reported [31][32][33]. Other kinds of rotating instabilities are known to occur in dusty or electronegative plasmas that are related to the void rotation [25,39,40], to rotating plasmoids in the middle of the interelectrode region [38] or to a more general rotation of the entire plasma [41]. In the present experiments, the regular rotation of plasmoids has been obtained during dust particle growth instabilities (DPGI) [25,28,39,42,43] and detected thanks to high-speed imaging at a few thousands frames per second (fps). An example is given in figure 1 where 8 plasmoids in the vicinity of each electrode are rotating clockwise. This figure shows an extracted raw image from the movie taken at 5000 fps and the filtered image where the background plasma emission has been removed. As it can be seen, plasmoids are difficult to detect in the raw image as their light is superimposed on the bright presheath emission. The 8 plasmoids have a typical size of about 3 mm and they take the path around the electrodes in about 25 ms corresponding to a typical speed ∼ 5 m/s. This relatively high speed confirms the difficulty to detect them using classical 25 fps cameras.

This carousel instability is quite tricky to obtain as the occurrence and the number of plasmoids is highly fluctuating during DPGI. In addition, they are not always rotating so clearly and more irregular behaviors can be observed [33].

During an experiment, a single plasmoid rotating around the bottom electrode has been obtained. A slight change in the applied RF power changed its rotation speed. A change from 3.3 W to 4 W with steps of about 0.1 W, leads to a slight gradual increase of the speed from 7 m/s up to 7.4 m/s. Even if this change is relatively small, it is significant and further experiments are required to better characterize the dynamics and structure of these plasmoids. Indeed, several questions arise from these observations. They are related to the origin of the plasmoids, the physical phenomenon inducing their rotation (as no magnetic field is voluntarily applied), the presence or not of nanoparticles inside the plasmoids (are they similar to usual voids observed in dusty plasmas?) and finally to the plasma properties inside them. In this context, future experiments will be conducted using different laser diagnostics to improve our understanding of this surprising phenomenon.

Imaging of the dust cloud: the void

In many discharges, especially in ccrf discharges, a main dust-free region appears in the center of the plasma [3,20,25,[44][45][46][47][48][49][50][51]. This void is the consequence of an equilibrium between an outward ion drag force pushing away the negatively charged dust particles and an inward electric force [45]. This region has been studied with micrometer size dust particles under microgravity conditions [47,52] or using an additional force like thermophoresis in on-ground experiments [53]. In dust forming plasmas, due to the submicron size of the dust particles, the gravity is not the predominant force and this region can be observed more easily. This region is of particular interest as it is involved in fundamental aspects of dust particle growth and plasma stability. Indeed, it has been shown that this dust-free space enables the growth of successive generations of dust particles while a new nucleation cannot occur in regions already filled with dust particles [14,17,21,46,48,[54][55][56]. The abrupt transition between these two regions creates a relatively unstable equilibrium characterized by the occurrence of a wide variety of self-excited instabilities of the void [45,[57][58][59][60].

The plasma properties inside the void are drastically different from the dust containing plasma surrounding it. This difference is easily observed to the naked eye by just looking at the plasma glow. When the grown dust particle density is relatively high, the void appears to be a region of enhanced emission [39,51,61] as shown in figure 2. The light emission properties are connected to the plasma parameters, especially to the electron density and temperature, and to the species created in the volume. In sputtering discharges as in the PKE-Nefedov reactor, the emission from produced molecules has also to be taken into account in addition to the buffer gas emission. Optical Emission Spectroscopy (OES) is then a very useful non intrusive diagnostics to get insights on the plasma properties in the dusty region and inside the void.

Optical emission spectroscopy during dust particle growth

Optical emission spectroscopy is a relatively simple diagnostics to implement in many plasma discharges. Nevertheless, the analysis of the obtained spectra is always relatively challenging. Its application to dusty plasmas can provide interesting information on the plasma changes induced by the growing dust particles [61]. As the structure of a growing dust particle cloud is changing continuously, a spatiotemporal analysis of the plasma emission is required. For this purpose, two optical lenses concentrate the light of the 3 cm high plasma on a bundle of 32 optical fibers mounted on a 1 cm vertical line. The plasma emission is then analyzed by a spectrometer with a focal length of 500 mm and a CCD detector of 400x1340 pixels. A 600 lines/mm grating is used as a good compromise between wavelength resolution and observed range.

Before analyzing directly the plasma emission, the spectrometer is used to follow dust particle growth by looking at the signal at 685 nm corresponding to the light scattered by the nanoparticle cloud illuminated by a laser sheet. As shown in figure 3, a Laser Light Scattering (LLS) signal starts to be detected at around 80 s after the plasma ignition. It is observed nearly simultaneously in the plasma center (fiber 20) and bottom edge (the laser light sheet was not optimized to observe the upper part of the interelectrode space). As dust particles are growing, they start moving downwards (from ∼ 200 s to ∼ 1000 s) due to gravity leaving a huge central dust-free region. Then, they accumulate in the discharge bottom part between the sheath and the plasma center.

Knowing the dust cloud spatial distribution in the discharge, the atomic and molecular species present in the plasma can be considered. As dust particle growth is obtained from the sputtering of a melamine formaldehyde film, several carbonaceous molecules are observed as C 2 (Swan band system around 516.52 nm), CN (violet system around 388.34 nm) and CH (Gerö band around 431. 42 cannot provide information on higher mass molecules.

nm). OES in the visible range

Those can be investigated using mass spectrometry [2,16,23] or absorption techniques [5,17,23,62], the latter being also able to give absolute species concentration. While the spatiotemporal evolution of CH is more difficult to connect to dust particle growth, CN and C 2 behave similarly in a way that can be connected to dust particle growth kinetics. As shown in figure 4, the strong C 2 emission is mainly located in the plasma center region and not in the presheath regions.

The emission intensity rapidly increases in the plasma center as soon as the plasma is ignited. After 50 s, the intensity starts to decrease until 300 s also corresponding to the time where the high emission region has the smallest size. Then, a continuous increase in the emission intensity is observed. In first approximation, the line intensity is connected to the species density. More precisely, to the population of the upper level of the observed transition, which is function of the electron density and energy. Self-absorption is considered as constant during the experiment. As carbon is involved in dust particle growth, its density is highly changing during this process and the evolution in figure 4 can be due: first to an important increase of the carbon species density due to plasma sputtering, then to an important consumption of C 2 due to dust particle growth, and finally to an equilibrium between carbon molecule production and a smaller consumption in the slow surface deposition phase of the growth [2]. Two hypotheses could be suggested to explain the fact that the C 2 emission is mainly observed in the center. First, growing dust particles present in the rest of the plasma can trap C 2 molecules in their vicinity leading to a smaller C 2 emission in these dusty regions. Secondly, the void can be a region of smaller electron energy while presheaths are known to be the places of electron heating in particular when dust particles accumulate in these regions. As the excitation energy of C 2 is small (∼ 2.4 eV) compared to argon, C 2 could be more easily excited in the regions of colder electrons. Indeed, by comparing figures 3 and 4, it clearly appears that regions of low C 2 emission are in accordance with dusty regions evidenced by LLS. The fact that the void region contains smaller energy electrons is in accordance with previous experimental [61] and simulation works [63].

To better understand the evolution of the carbon molecule lines, several typical argon lines have been analyzed. The lines well-known to be mainly populated from the ground state by high energy electrons (as for example the 750.59 nm line requires at least 13.48 eV) have a higher emission in the presheath regions and a much weaker one in the plasma center, as expected. In addition, these lines have an increasing emission as dust particles are growing, especially in the bottom part of the interelectrode space where they accumulate. This is consistent with the considered higher electron temperature in plasmas containing dust particles [63][64][65][66][67][68][69][70]. The above observation and the results presented in figure 4 tend to show that the dust particle presence in the plasma and the formation of the void, drastically affect the electron energy distribution function. It shows that the electron population in the void seems to be colder than in dusty regions.

In order to confirm this assumption, the ratio between two argon lines is calculated in figure 5. This ratio has been shown to be an easy method for providing information on the electron energy distribution function [71][72][73]. These two lines at 419.95 nm and 420.19 nm (419.83 nm and 420.07 nm in air) have roughly the same ground-state optical emission cross-sections and thus their excitation due to high-energy electrons is similar.

In contrast, their excitation through metastable levels is clearly different, the 420.19 line being highly favored by this process. As shown in figure 5, this ratio is close to 1 in the presheath region where dust particles tend to accumulate while it can be as strong as 3.5 in the void region. The fact that this ratio evolves during dust particle growth and its spatial distribution, confirm that higher energy electrons are mainly in the dusty regions and low energy electrons able to induce emission through metastables, are more present in the dust-free region. While a complete understanding of the time evolution of this ratio during dust particle growth has to be further improved, it is a clear confirmation of the presence of colder electrons in the void (compared to electrons in the dusty regions).

Dust particle growth in the "soot" experiment

A new experimental setup has been developed in GREMI to study dust particle growth in various chemistries.

This setup has been built in the framework of a project related to soot formation from biofuels.

The objective is to study in controlled conditions, the possible formation of soot particles from a simple and typical liquid like ethanol (C 2 H 5 OH). The new chamber is 40 cm in diameter and the electrodes are 20 cm in diameter. As in usual ccrf discharge, one electrode (the upper one) is powered with a rf generator 0-300 W, while the other one is grounded. The distance between the two electrodes can be adjusted up to a distance of 10 cm. Several reactive gases can be injected, like acetylene, and the reactor is also equipped with a Controlled Evaporator Mixer allowing the injection of argon (or other gases) mixed with a vaporized liquid.

Currently, only several discharge conditions have been explored as many parameters can be tuned: rf power, total pressure, argon flow, ethanol percentage, distance between the electrodes.

The formation of nanoparticles directly in the gas phase has not been obtained up to now. Nevertheless, during the Ar/ethanol plasma, the ethanol decomposition leads to the formation of a dense brown film on the electrodes. Then, a pure Ar plasma is created and the argon ion bombardment on this carbon based film induces the injection of precursors in the gas phase. As in the PKE-Nefedov reactor these molecules lead to the formation of nanoparticles easily detected through the change in the electrical characteristics of the discharge. Once they have grown to a few tens of nanometers, LLS using a laser sheet (at 532 nm) is able to reveal their presence in between the electrodes as shown in figure 6. A dense cloud is clearly visible even outside the interelectrode region. It contains different dust particle generations that are not mixed and shows two symmetric lobes. A void region cannot be clearly evidenced in this cloud, suggesting that either the discharge parameters or the reactor geometry do not favor the void appearance. It is noted that, in conditions that need to be confirmed, two symmetric voids (in each lobes) have been observed.

As in the PKE-Nefedov reactor, different types of complex instabilities have been observed, including rotating instabilities. Figure 7 shows the central part of the dust cloud containing dust particles of different sizes that are moved during a rotating instability. The very preliminary results obtained in this new reactor are promising for studying different gas chemistries leading to direct (in the gas phase) or indirect (through the sputtering of a layer formed by decomposing a gas or a vaporized liquid) nanoparticle growth. New inputs in complex plasma instabilities are also expected. Ethanol mixtures will be first tested and many diagnostics have been installed in order to characterize the physics and the chemistry of this particular mixture plasma: an electrical probe to measure all rf parameters, a Langmuir probe with an automated linear drive, an energy resolved mass spectrometer, an optical emission spectrometer, a illuminating system with a green laser at 532 nm and both a standard CCD camera and a high-speed camera. Ex-situ diagnostics of the deposited films and grown nanoparticles will be also conducted.

Conclusions

These experiments show that nanoparticles can be easily obtained in typical ccrf discharges as soon as a thin film is deposited on the electrodes. These results emphasize that possible nanoparticle formation has to be considered in plasma processes using gases favoring thin film deposition or with easily sputtered plasma facing components. Unwanted dust particle growth can take place affecting considerably the plasma properties and structure, and in some conditions giving birth to uncontrolled plasma instabilities. Dusty plasmas are media revealing a rich variety of complex phenomena with a strong spatiotemporal dependence. These aspects can be detected and analyzed using appropriate optical diagnostics taking into account this dependency.

Figure 1 .

 1 Figure 1. Rotating plasmoids during DPGI. Eight plasmoids in front of each electrode are moving with a clockwise rotation. Raw image and the filtered one after background subtraction. The background is obtained by averaging plasma emission over time (at least during a complete plasmoid rotation).

Figure 2 .

 2 Figure 2. Picture of the plasma glow emission in the presence of grown dust particles except in the central part of the plasma where a very bright dust-free region (void) is formed.

Figure 3 .

 3 Figure 3. Laser Light scattering signal at 685 nm collected by the 32 fibers and the optical emission spectrometer. Dust particles start to be detected at about 100 s close to the plasma center. As they grow, the gravity drags them downwards close to the presheath region where they accumulate.

Figure 4 .

 4 Figure 4. Intensity of the head of the C 2 Swan band system at 516.52 nm as a function of time for the 32 fibers.

Figure 5 .

 5 Figure 5. Ratio of the 420.19 nm and 419.95 nm argon lines as a function of time for the 32 fibers.

Figure 6 .

 6 Figure 6. Dust cloud illuminated by a green laser at 532 nm. Dust particles are grown by sputtering a carbon-based deposited layer with an argon plasma. The thin film was the result of ethanol decomposition in an argon/ethanol plasma.

Figure 7 .

 7 Figure 7. Zoom in a grown dust particle cloud during plasma instabilities. Several dust particle generations are observed giving a complex cloud shape. The electrode center is localized by the maintaining tube seen in the upper part of the image. The distance between the electrodes is ∼ 4 cm.

Acknowledgments

The authors would like to thank A. Petit, H. Rabat, S. Dozias, J. Simonneau and E. Jordano for technical assistance. The PKE-Nefedov chamber has been made available by the Max-Planck-Institute for Extraterrestrial Physics, Germany, under the funding of DLR/BMBF under Grant No. 50WM9852. The work on the PKE-Nefedov reactor was partly supported by the French National Research Agency (ANR) project INDIGO n o ANR-11-JS09-010-01 and by the PHC PROCOPE project n o 30790RL from Ministères des Affaires Etrangères et du Développement International (MAEDI) et de l'Education Nationale de l'Enseignement Supérieur et de la Recherche (MENESR). The soot experiment was partly funded by Region Centre project SuB-PPM n o 2013-00085493, by the European Union FEDER project n o 40321 and by the GREMI laboratory.