MD Eric Ettore
email: ettore.e@chu-nice.fr

MD Philipp Müller

PhD Jonas Hinze

MSc Matthias Riemenschneider

PhD Michel Benoit

PhD Bruno Giordana

MD Danilo Postin

MSc Rene Hurlemann

PhD Amandine Lecomte

MSc Michel Musiol

PhD Hali Lindsay

MSc Philippe Robert

MD, PhD Alexandra König

Digital Phenotyping for Differential Diagnosis of Major Depressive Episode: Narrative Review

Keywords: depression, bipolar disorder, posttraumatic stress disorder, differential diagnosis, digital phenotyping, speech analysis, nonverbal behavior, physiological measures, posttraumatic stress disorder, mental health, clinical interview, diagnosis, mental disorder, interview, digital health, psychotrauma, digital, information cepstral coefficient Hamilton Depression Rating Scale. g NVB: nonverbal behavior. h AU: action unit. i ECG: electrocardiogram -square surface distance skin conductance response skin conductance response

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Major depression is a frequent syndrome affecting more than 264 million people worldwide [START_REF] Gbd | DiseaseInjury Incidence Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017[END_REF]. Major depressive episode (MDE), as defined by the current criteria [START_REF]Diagnostic and statistical manual of mental disorders (DSM)[END_REF], describes a large heterogeneous clinical syndrome comprising more than 1490 combinations of symptoms [START_REF] Ostergaard | The heterogeneity of the depressive syndrome: when numbers get serious[END_REF]. The possibility of fulfilling MDE criteria with opposite symptoms, such as insomnia and hypersomnia, decrease/increase in appetite or agitation, and psychomotor retardation, highlights this heterogeneity. This heterogeneity could explain why studies with large samples of patients with major depression show that only 30% of them remitted with a first-line antidepressant and another 30% did not remit after 4 consecutive treatment trials [START_REF] Gaynes | What did STAR*D teach us? Results from a large-scale, practical, clinical trial for patients with depression[END_REF].

These various "profiles" of MDE may require different therapeutic approaches. In fact, according to international classifications [START_REF]Diagnostic and statistical manual of mental disorders (DSM)[END_REF], MDE symptoms are the same whether in major depressive disorder (MDD) or in bipolar disorder (BD). In clinical practice, hypomanic or manic episodes have been the most recognizable characteristics of BD, while depression seems to be most frequent clinical manifestation [START_REF] Hirschfeld | Differential diagnosis of bipolar disorder and major depressive disorder[END_REF][START_REF] Judd | The long-term natural history of the weekly symptomatic status of bipolar I disorder[END_REF]. Performing differential diagnosis between MDD and BD during an MDE is challenging. Approximately 20% of people with MDE would be misdiagnosed as MDD when the correct diagnosis is BD [START_REF] Angst | Bipolar spectrum in major depressive disorders[END_REF]. Thereby, the delay for a correct BD diagnosis can vary from almost 7 to 10 years after the first mood symptoms [START_REF] Hirschfeld | Perceptions and Impact of Bipolar Disorder[END_REF]. These misdiagnoses can have several damaging consequences, for example, induction of manic, hypomanic, or mixed states; development of treatment resistance; or cycle acceleration [START_REF] Galvão | Clinical differences between unipolar and bipolar depression: interest of BDRS (Bipolar Depression Rating Scale)[END_REF]. Furthermore, Rytwinski et al [START_REF] Rytwinski | The co-occurrence of major depressive disorder among individuals with posttraumatic stress disorder: a meta-analysis[END_REF] showed in their meta-analysis that almost 52% of patients with posttraumatic stress disorder (PTSD) have comorbidity with MDD. In fact, in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), several symptoms are shared between MDD and PTSD [START_REF] Gootzeit | Factors of PTSD: differential specificity and external correlates[END_REF] and a study by Gros et al [START_REF] Gros | Symptom overlap in posttraumatic stress disorder and major depression[END_REF] showed that in patients classified by DSM-5 as having MDD and PTSD, only the presence of trauma (criterion A of DSM-5) could differentiate MDD from PTSD efficiently. Moreover, patients classified as comorbid between MDD and PTSD showed the highest severity of symptoms, more cognitive deterioration, higher suicidality, and worse prognosis compared with noncomorbid patients [START_REF] Gros | Symptom overlap in posttraumatic stress disorder and major depression[END_REF]. One possible explanation could be the time-dependent fluctuation of PTSD symptoms [START_REF] Mcfarlane | Multiple diagnoses in posttraumatic stress disorder in the victims of a natural disaster[END_REF]. In a dynamic model, MDD could be a progression or an evolution of psychotraumatism [START_REF] Alarcon | The cascade model: an alternative to comorbidity in the pathogenesis of posttraumatic stress disorder[END_REF]. Although there is a frequent association between MDD and PTSD, the link between them remains unclear. However, it points to the importance of exploring the presence of psychological trauma and PTSD in patients with MDE because it carries the potential to drastically impact further therapeutic care. Misdiagnosing PTSD or the presence of psychological trauma could lead to a worse prognosis due to inappropriate drug treatment or psychotherapy.

In this article, we focus on the use of new digital markers in MDE, given their potential to serve as an additional and objective diagnostic support. For this, we explore studies in MDE, especially when it is presented in the context of either MDD or BD, as well as when it is presented alongside the presence of psychological trauma or PTSD.

Digital phenotyping refers to the moment-to-moment quantification of human behavior in everyday life using data from digital devices [START_REF] Insel | Digital Phenotyping: Technology for a New Science of Behavior[END_REF]. It suggests the possibility of revealing clinically relevant information by a continuous and nonintrusive monitoring of behavioral and mental states. Two categories of data can be captured by digital phenotyping. First, active data, which require the input of the participants being studied. It includes, for example, recordings, responses to surveys, and social media activity. Passive data, however, do not require the individual's participation to be captured. They include, for example, accelerometer-derived data or GPS coordinates [START_REF] Waring | Introduction to Digital Phenotyping for Global Health[END_REF]. Digital markers correspond to disease indicators obtained digitally, which can be used to define a digital phenotype [START_REF] Andrea | Digital biomarkers and digital phenotyping in mental health care and prevention[END_REF]. It refers to the possibility of capturing, through computerized measurement tools, certain symptoms or behavior specific to a psychiatric disorder.

There is a specific interest in psychiatry in which symptoms and clinical states are mainly measured using question-based scales and without biological markers. Thus, the interest of digital phenotyping would be to obtain objective and quantifiable measurements of these symptoms or behaviors [START_REF] Waring | Introduction to Digital Phenotyping for Global Health[END_REF]. Moreover, diagnostic categories do not capture the heterogeneity of symptoms, and variability between patients can lead to misdiagnosis and incorrect treatments [START_REF] Brietzke | Integrating digital phenotyping in clinical characterization of individuals with mood disorders[END_REF]. Identifying objective markers of clinical states, including trans-diagnostic symptoms, could improve disease classification and treatment [START_REF] Schilbach | Using interaction-based phenotyping to assess the behavioral and neural mechanisms of transdiagnostic social impairments in psychiatry[END_REF]. Thus, digital phenotyping should play a role in routine clinical practice, especially by improving clinical diagnosis and treatment by an early detection of condition onset, by assessing treatment response, or even by detecting relapse [START_REF] Huckvale | Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety[END_REF]. Therefore, we assume that new digital measures may provide indicators for the heterogeneous characteristics of MDE and could help to better distinguish between its potential different clinical profiles. Based on this rationale, we performed a narrative review including studies on several technologies, such as speech and video analysis and physiological measures, namely, heart rate variability (HRV) and electrodermal activity (EDA) for the assessment of MDE.

Methods

Search Strategy

We conducted this narrative review from June to September 2021, which mainly concerned the current application of digital tools for MDE diagnosis. The following electronic databases were searched: PubMed/MEDLINE, Web of Science, and Google Scholar. The review was limited to articles in English or French and because we aimed to establish an overview of the most recent advances in these domains, we restricted our search to studies published after January 2010. For this paper, we decided to concentrate on digital tools that are easy to use during a clinical interview, minimally invasive, and least dependent on compliance. Thus, we focused on 4 types of digital markers: (1) automated voice analysis, behavior analysis by [START_REF]Diagnostic and statistical manual of mental disorders (DSM)[END_REF] video and physiological measures, (3) HRV, and (4) EDA. We then used broad search terms to capture as many studies as possible that are specifically related to these technologies and we associated them with terms related to specific psychiatric issue. We concentrated on MDE and on several psychiatric conditions where it may occur, especially on (1) MDD, (2) BD, (3) PTSD, and (4) psychological trauma. The search strings are included in Multimedia Appendix 1.

Inclusion and Exclusion Criteria

The inclusion criteria were as follows: primary articles or reviews dealing with depression, BD, bipolar depression, PTSD, or psychological trauma using digital tools for speech or video analysis or analysis of physiological parameters such as HRV or skin conductance. The exclusion criteria were as follows: articles related to activity monitoring on social networks or by SMS text messages or phone and articles related to ecological momentary assessment. There is an abundance of literature on these topics, but we wanted to focus here on digital tools that can be used specifically during a social interaction such as an interview with the clinician. Unrelated and redundant articles or studies using technologies for therapeutic purposes were also excluded. Theoretical papers, study protocols, letters, books or book chapters, statistical reviews, and dissertations were also excluded.

Major Depressive Disorder

Speech Analysis

Prosodic abnormalities in patients with depression are well known and a monotonous speech or reduced prosody can be easily and frequently heard. It is assumed that cognitive and physiological changes in depression could affect speech production, and changes in the automatic and somatic nervous system cause disturbances in muscle tension and respiratory rate. These changes will not only influence vocal folds and vocal tract dynamics, but also constrain articulatory movement [START_REF] Cummins | A review of depression and suicide risk assessment using speech analysis[END_REF].

Speech analysis, which consists of automatically extracting vocal and linguistic features from audio signals, offers the possibility to detect and measure these changes and serves as an additional objective assessment of depression. Traditionally, acoustic features can be divided into 5 types: temporal, prosodic, spectral, source, and filter features. Temporal features, defined as speech prosodic timing measures, appear to be a promising measure of depression and its severity [START_REF] Low | Automated assessment of psychiatric disorders using speech: A systematic review[END_REF].

Yamamoto et al [START_REF] Yamamoto | Using speech recognition technology to investigate the association between timing-related speech features and depression severity[END_REF] used 3 temporal features in a prospective follow-up study: speech rate, pause time, and response time.

They found a correlation with scores on the Hamilton Rating Scale for Depression (HAM-D). Mundt et al [START_REF] Mundt | Vocal acoustic biomarkers of depression severity and treatment response[END_REF] found 5 prosodic timing measures to be significantly correlated with depression severity: total speech time, total pause time, percentage pause time, speech pause ratio, and speaking rate. Xu et al [START_REF] Xu | Automated Verbal and Non-verbal Speech Analysis of Interviews of Individuals with Schizophrenia and Depression[END_REF] found that people with depression have a longer response time and interrupt their communication partner more often. To summarize, people with depression used a much larger percentage of words related to the past and sadness emotions. Similarly, Shinohara et al [START_REF] Shinohara | Depressive Mood Assessment Method Based on Emotion Level Derived from Voice: Comparison of Voice Features of Individuals with Major Depressive Disorders and Healthy Controls[END_REF] used a "vitality" score based on emotional elements of speech (ie, joy, anger) and showed a negative correlation between the vitality score and HAM-D scores. Arevian et al [START_REF] Arevian | Clinical state tracking in serious mental illness through computational analysis of speech[END_REF] analyzed the types of word used and found, among others, more negative emotion and less complex word use in patients with depression.

Finally, Alghowinem et al [START_REF] Alghowinem | Detecting depression: A comparison between spontaneous and read speech[END_REF] and Hönig et al [START_REF] Hönig | Automatic modelling of depressed speech: relevant features and relevance of gender[END_REF] found that the average syllable duration could be positively correlated with depression severity.

With prosodic features in people with depression, reduced fundamental frequency (F0) range and average are frequently found [START_REF] Hönig | Automatic modelling of depressed speech: relevant features and relevance of gender[END_REF][START_REF] Horwitz | On the relative importance of vocal source, system,prosody in human depression[END_REF][START_REF] Quatieri | Vocal-Source Biomarkers for Depression: A Link to Psychomotor Activity[END_REF][START_REF] Kiss | Mono-and multi-lingual depression prediction based on speech processing[END_REF]. F0 corresponds to the lowest frequency of the speech signal perceived as pitch. It could explain the depressive monotonous speech and could be a consequence of disturbances in laryngeal muscle tension due to psychomotor retardation. The latter may also explain reduction in F0 variability [START_REF] Cummins | A review of depression and suicide risk assessment using speech analysis[END_REF][START_REF] Low | Automated assessment of psychiatric disorders using speech: A systematic review[END_REF]. However, some studies also report no significant correlation between F0 variables and depression [START_REF] Mundt | Vocal acoustic biomarkers of depression severity and treatment response[END_REF][START_REF] Taguchi | Major depressive disorder discrimination using vocal acoustic features[END_REF].

Several studies on depression showed a shift in spectral energy, and mel-frequency cepstral coefficients (MFCCs) are often used as spectral features. Spectral features characterize the spectrum of speech, which at a given time correspond to the frequency distribution of the speech signal. Taguchi et al [START_REF] Taguchi | Major depressive disorder discrimination using vocal acoustic features[END_REF] showed that the second dimension of MFCC allowed a significant discrimination between patients with depression and controls. Further, Wang et al [START_REF] Wang | Acoustic differences between healthy and depressed people: a cross-situation study[END_REF] found that MFCC 5, MFCC 7, and loudness were consistently lower in people with depression.

According to the task designed during speech recording, MFCC 3 and F0 were significantly lower than in people with depression than in healthy people.

Finally, source features such as harmonic-to-noise ratio (HNR), jitter, and shimmer also tend to increase with depression severity, supporting the hypothesis of a more breathy phonation in depressed speech [START_REF] Quatieri | Vocal-Source Biomarkers for Depression: A Link to Psychomotor Activity[END_REF]. This could be explained by a more open and turbulent glottis, linked to a reduction in laryngeal muscle tension. The study by Hönig et al [START_REF] Hönig | Automatic modelling of depressed speech: relevant features and relevance of gender[END_REF] also supports this hypothesis. Alghowinem et al [START_REF] Alghowinem | Detecting depression: A comparison between spontaneous and read speech[END_REF] showed that not only shimmer and jitter but also loudness and MFCC features are part of the general characteristics of depressive speech.

Nonverbal Behavior Analysis

Traditionally, evaluation of clinical depression is based on patients' verbal information from psychological interviews and rating scales or self-report questionnaires. However, verbal analysis has several limitations. First, patients may underreport or overreport depressive symptoms due to different reasons (eg, social stigma). Second, it can be assumed that most human communication is taking place through nonverbal behavior. Indeed, a large part of nonverbal behavior is mostly outside of conscious control, and these cues and signals may differ from verbal reports. Facial expression, gestures, and body postures are mainly involuntary, and therefore represent a privileged way of expressing feelings and emotions [START_REF] Fiquer | Talking bodies: nonverbal behavior in the assessment of depression severity[END_REF][START_REF] Geerts | Ethological approaches to psychiatric disorders: focus on depression and schizophrenia[END_REF]. Girard et al [START_REF] Girard | Social risk and depression: Evidence from manual and automatic facial expression analysis[END_REF][START_REF] Girard | Nonverbal Social Withdrawal in Depression: Evidence from manual and automatic analysis[END_REF] showed that when the severity of depressive symptoms was high, participants made fewer affiliative facial expression such as smiling, and more nonaffiliative facial expression such as contempt. They also showed that patients exhibited diminished head motion (amplitude and velocity). These results support the social withdrawal hypothesis in depression. In fact, rather than affective valence with an increase or decrease in facial expression, it is the social-communicative value with affiliative expression that seems to be affected in depression. Thus, nonverbal behavior may serve to maintain interpersonal distance while facilitating social withdrawal.

Fiquer et al [START_REF] Fiquer | Talking bodies: nonverbal behavior in the assessment of depression severity[END_REF] supported the assumption that depression involves behaviors related to social withdrawal and negative feelings. Crying, asymmetric smiling, and motionless head/eyebrows were associated with higher levels of depression severity, while patients' illustrative gestures, defined as "the hand and arm movements used to support the speech," were associated with a lower level of depression severity.

Moreover, decrease of negative facial and head indicators and increase of eye contact and yes nodding accompanied clinical improvement, whereas illustrative gestures did not [START_REF] Fiquer | Talking bodies: nonverbal behavior in the assessment of depression severity[END_REF]. In another study, Fiquer et al [START_REF] Fiquer | What is the nonverbal communication of depression? Assessing expressive differences between depressive patients and healthy volunteers during clinical interviews[END_REF] compared nonverbal behavior in patients with depression versus controls. They showed that patients with depression displayed higher levels of negative nonverbal behaviors (shrug, head and lips down, adaptative hand gesture, frown, and cry) and lower levels of positive nonverbal behaviors (eye contact and smile). The authors raise the important role of these behaviors in social withdrawal [START_REF] Fiquer | What is the nonverbal communication of depression? Assessing expressive differences between depressive patients and healthy volunteers during clinical interviews[END_REF]. Nevertheless, the aforementioned nonverbal behaviors was not associated with depression severity before treatment, which suggests its independence.

The authors concluded that certain typically found nonverbal behaviors could represent a predisposition to depression possibly influenced by personality features. These social symptoms expressed through nonverbal behavior may be the last ones to vanish [START_REF] Fiquer | What is the nonverbal communication of depression? Assessing expressive differences between depressive patients and healthy volunteers during clinical interviews[END_REF].

On the contrary, Fiquer et al [START_REF] Fiquer | Is nonverbal behavior in patients and interviewers relevant to the assessment of depression and its recovery? A study with Dutch and Brazilian patients[END_REF] studied nonverbal variables such as speaking effort, encouragement, attunement, and change in attunement, and found no association between these behaviors and the severity of depression. Finally, several recent studies achieved classifying depression with high accuracy using video analysis with head and facial movements [START_REF] Dibeklioglu | Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding[END_REF][START_REF] Kacem | Detecting Depression Severity by Interpretable Representations of Motion Dynamics[END_REF], only head movement [START_REF] Bhatia | Automated Measurement of Head Movement Synchrony during Dyadic Depression Severity Interviews[END_REF], facial expressivity [START_REF] Jiang | Classifying Major Depressive Disorder and Response to Deep Brain Stimulation Over Time by Analyzing Facial Expressions[END_REF], or with head pose and eye activity [START_REF] Alghowinem | Cross-cultural detection of depression from nonverbal behaviour[END_REF]. Guo et al [START_REF] Guo | Deep Neural Networks for Depression Recognition Based on 2D and 3D Facial Expressions Under Emotional Stimulus Tasks[END_REF] found that the highest accuracy of depression recognition was when patients were watching film clips of positive and negative emotional stimuli compared with other tasks.

In a recent paper, Alghowinem et al [START_REF] Alghowinem | Interpretation of Depression Detection Models via Feature Selection Methods[END_REF] used feature selection methods and found that some features have a high capability for distinguishing between depression severities. For speech analysis, the strongest features were the temporal features, F0, HNR, formants, and MFCC; and for behavioral analysis, it was the left-right eye movement and gaze direction and the yaw head movement. The authors showed that these features outperformed all other features in depression detection.

The studies mentioned so far demonstrate the usefulness of behavioral analysis in understanding the mechanisms that underlie the onset and course of depression. These analyses are less influenced by conscious control compared with verbal communication [START_REF] Geerts | Ethological approaches to psychiatric disorders: focus on depression and schizophrenia[END_REF], which could make them a reliable objective measure.

Physiological Measures

Heart Rate Variability

HRV corresponds to the beat-to-beat variations in the heart rate over a given period. HRV appears to be a good reflection of the heart's ability to modulate its rhythm in response to external and internal stimuli. HRV is regulated by the autonomic nervous system and, as a result, can inform of its functioning. HRV is related to the influence of the parasympathetic nervous system (PNS) and the sympathetic nervous system (SNS) [START_REF] Servant | Heart rate variability. Applications in psychiatry][END_REF]. Three parameters are commonly used: high-frequency HRV (HF-HRV; 0.15-0.40 Hz), low frequency HRV (LF-HRV; 0.04-0.15 Hz), and the LF-to-HF ratio. HF-HRV is mainly under the control of the PNS, whereas LF-HRV is under the influence of both PNS and SNS, but mostly the SNS. The LF-to-HF ratio is considered an index of sympathovagal balance [START_REF] Servant | Heart rate variability. Applications in psychiatry][END_REF].

In addition, time-domain parameters such as the root-mean-square of successive differences (RMSSD) and the SD of normal-to-normal intervals (SDNN) are frequently used [START_REF] Malik | Heart rate variability: Standards of measurement, physiological interpretation, and clinical use[END_REF]. Normal-to-normal is the interval between 2 heartbeats. HRV can be an indicator of the ability to regulate stress and emotions, which is particularly relevant in psychiatric disorders [START_REF] Appelhans | Heart rate variability and pain: associations of two interrelated homeostatic processes[END_REF]. Furthermore, a good cardiovascular adaptability can be reflected by a high HRV [START_REF] Schwerdtfeger | Heart rate variability (HRV): From brain death to resonance breathing at 6 breaths per minute[END_REF], while LF can be a risk factor of cardiovascular diseases [START_REF] Hillebrand | Heart rate variability and first cardiovascular event in populations without known cardiovascular disease: meta-analysis and dose-response meta-regression[END_REF]. A meta-analysis in depression [START_REF] Koch | A meta-analysis of heart rate variability in major depression[END_REF] showed significant reductions in frequency domain parameters such as HF, LF, and very-low frequency (VLF) HRV as well as a significantly higher LF-to-HF ratio compared with healthy controls. Significant reductions in time domain parameters were further found with reduced RMSSD, SDNN, and interbeat intervals.

Some recent studies have focused on different clinical subdimensions of MDD. Lee et al [START_REF] Lee | Heart Rate Variability after Treatment for Depression in North Korean Defectors[END_REF] showed a positive correlation between fatigue-related items of the Hamilton Depression Rating Scale (HDRS) and LF-HRV. The latter was positively correlated with the items "midnight insomnia" and "slower thinking." In the same line, Adolph et al [START_REF] Adolph | High frequency heart rate variability: Evidence for a transdiagnostic association with suicide ideation[END_REF] found that HF-HRV in reaction to watching a sad film was predictive of higher scores on suicidal ideation, whereas HF-HRV at rest was not predictive of suicidal ideation. Further studies investigated the relationship between HRV and treatment response. First, Hartmann et al [START_REF] Hartmann | Heart Rate Variability as Indicator of Clinical State in Depression[END_REF] found that patients with MDD normalized their HRV after treatment, especially for HF-HRV and LF-HRV. Further, Giurgi-Oncu et al [START_REF] Giurgi-Oncu | Evolution of Heart Rate Variability and Heart Rate Turbulence in Patients with Depressive Illness Treated with Selective Serotonin Reuptake Inhibitors[END_REF] showed a normalization of HRV parameters (including HF, RMSSD, and SDNN) with the depression treatment by therapy and medication. Kircanski et al [START_REF] Kircanski | Heart rate variability as a biomarker of anxious depression response to antidepressant medication[END_REF] predicted treatment success among patients with depression with or without anxiety. The authors found that a higher HF-HRV in pretreatment had better outcomes than lower HF-HRV in anxious depression, whereas lower HF-HRV had better outcomes than higher HF-HRV in nonanxious depression.

By contrast, Sarlon et al [START_REF] Sarlon | Heart Rate Variability Changes in Patients With Major Depressive Disorder: Related to Confounding Factors, Not to Symptom Severity?[END_REF] found no association between HRV and depressive symptoms severity. Neyer et al [START_REF] Neyer | Reduction of depressive symptoms during inpatient treatment is not associated with changes in heart rate variability[END_REF] showed in a prospective study with HRV measures before and after treatment that depressive symptoms improved without change in HRV, suggesting a more complex relationship.

Finally, Fernandes et al [START_REF] Fernandes | Nonverbal behaviors are associated with increased vagal activity in major depressive disorder: Implications for the polyvagal theory[END_REF] explored the association between HRV and nonverbal behavior using a validated ethogram.

Results showed a positive correlation among HRV (HF and RMSSD) and positive nonverbal behaviors, but no association between HRV and negative affect. The authors suggested that HRV and nonverbal behaviors could be regulated by vagal activity, which may be related to the social engagement system.

Electrodermal Activity

The most basic indicators of the state of the autonomic nervous system are heart rate and EDA. EDA can measure electrical conductance of the skin, which depends on the quantity of sweat secreted by glands in the hypodermis and reflects sympathetic nervous activity.

EDA has a tonic and a phasic component. The former is measured in the skin conductance level (SCL) and the latter with the skin conductance response (SCR) [START_REF] Sarchiapone | The association between electrodermal activity (EDA), depression and suicidal behaviour: A systematic review and narrative synthesis[END_REF]. EDA can be used as an indicator for emotional reactivity [START_REF] Bloch | New aspects of the psychogalvanic or electrodermographic method as a criterion of emotional tensions[END_REF]. In fact, rather than the valence of emotion, it is the intensity that seems related to EDA [START_REF] Sequeira | Electrical autonomic correlates of emotion[END_REF] and negative stimulation induces more extensive reactions than positive [START_REF] Balconi | Appetitive vs. defensive responses to emotional cues. Autonomic measures and brain oscillation modulation[END_REF]. Concerning emotional reactivity, we distinguish labile and stable patients for EDA. EDA labile patients could be described as calm and deliberative, whereas EDA stable patients tend to be more irritable, emotionally expressive, and active. EDA labile patients would be dependent on anxiety traits and better control for potentially threatening stimuli [START_REF] Crider | Personality and electrodermal response lability: an interpretation[END_REF].

Several studies show that EDA could be an indication of nonconscious emotional processes [START_REF] Hamm | Affective blindsight: intact fear conditioning to a visual cue in a cortically blind patient[END_REF][START_REF] Bauer | Electrodermal discrimination of familiar but not unfamiliar faces in prosopagnosia[END_REF][START_REF] Silvert | Autonomic responding to aversive words without conscious valence discrimination[END_REF]. A recent systematic review [START_REF] Sarchiapone | The association between electrodermal activity (EDA), depression and suicidal behaviour: A systematic review and narrative synthesis[END_REF] showed lower EDA, especially lower SCL and SCR, in patients with depression compared with healthy controls. Others studies report that, compared with patients with depression also experiencing agitation, those with psychomotor retardation or symptoms of inhibition have lower EDA levels [START_REF] Bernstein | Differing patterns of electrodermal and finger pulse responsivity in schizophrenia and depression[END_REF][START_REF] Lader | Physiological measures in agitated and retarded depressed patients[END_REF][START_REF] Thorell | Electrodermal activity in suicidal and nonsuicidal depressive patients and in matched healthy subjects[END_REF]. Moreover, patients with "psychotic " and "endogenous" depression could have lower EDA than "nonendogenous" depression [START_REF] Thorell | Electrodermal activity in suicidal and nonsuicidal depressive patients and in matched healthy subjects[END_REF][START_REF] Byrne | A Psychophysiological Distinction between Types of Depressive States[END_REF][START_REF] Mirkin | Electrodermal Activity in Depression: Clinical and Biochemical Correlates[END_REF].

Other recent studies showed moderate to high accuracy in classifying patients with MDD and healthy controls [START_REF] Kim | Automatic detection of major depressive disorder using electrodermal activity[END_REF][START_REF] Kim | Skin conductance responses in Major Depressive Disorder (MDD) under mental arithmetic stress[END_REF], such as that by Smith et al [START_REF] Smith | Using Resting State Heart Rate Variability and Skin Conductance Response to Detect Depression in Adults[END_REF], but the latter used HRV and SCR for depression detection and found no benefit in including SCR to improve accuracy. In addition, Pedrelli et al [START_REF] Pedrelli | Monitoring Changes in Depression Severity Using Wearable and Mobile Sensors[END_REF] showed that EDA and HRV features from wearables as well as activity level and sleep parameters could provide an estimate of changes in severity of depressive symptoms.

In a cohort of patients with MDD at 1-year follow-up, Litwińska-Bołtuć et al [START_REF] Litwińska-Bołtuć | Clinical effectiveness of the electrodermal orienting reactivity test for evaluating relapse and recurrence risk in patients hospitalized for depression[END_REF] concluded that SCR hyporeactivity was associated with recurrent episodes and relapse, and recurrence of depression was almost 5 times higher than that in the reactive patients.

Concluding Remarks

Based on these aforementioned studies, the digital phenotype of MDD could be identified as follows: a reduction in certain speech temporal features (ie, speech rate or speech time), an increase in others (ie, pause time or response time) [START_REF] Mundt | Vocal acoustic biomarkers of depression severity and treatment response[END_REF][START_REF] Yamamoto | Using speech recognition technology to investigate the association between timing-related speech features and depression severity[END_REF], and a change in the type of words used (ie, less complex and more emotionally negative) [START_REF] Arevian | Clinical state tracking in serious mental illness through computational analysis of speech[END_REF][START_REF] Shinohara | Depressive Mood Assessment Method Based on Emotion Level Derived from Voice: Comparison of Voice Features of Individuals with Major Depressive Disorders and Healthy Controls[END_REF]. In addition, prosodic features (ie, F0 range and average) [START_REF] Hönig | Automatic modelling of depressed speech: relevant features and relevance of gender[END_REF][START_REF] Horwitz | On the relative importance of vocal source, system,prosody in human depression[END_REF][START_REF] Quatieri | Vocal-Source Biomarkers for Depression: A Link to Psychomotor Activity[END_REF][START_REF] Kiss | Mono-and multi-lingual depression prediction based on speech processing[END_REF], variability [START_REF] Cummins | A review of depression and suicide risk assessment using speech analysis[END_REF][START_REF] Low | Automated assessment of psychiatric disorders using speech: A systematic review[END_REF], spectral features [START_REF] Taguchi | Major depressive disorder discrimination using vocal acoustic features[END_REF][START_REF] Wang | Acoustic differences between healthy and depressed people: a cross-situation study[END_REF], and source features (ie, HNR, jitter, shimmer [START_REF] Alghowinem | Detecting depression: A comparison between spontaneous and read speech[END_REF][START_REF] Quatieri | Vocal-Source Biomarkers for Depression: A Link to Psychomotor Activity[END_REF]) are changed.

The behavior of patients with depression could be characterized by a more negative nonverbal behavior (ie, head motion, facial expressivity, and hand or body gestures) [START_REF] Fiquer | Talking bodies: nonverbal behavior in the assessment of depression severity[END_REF][START_REF] Girard | Social risk and depression: Evidence from manual and automatic facial expression analysis[END_REF][START_REF] Girard | Nonverbal Social Withdrawal in Depression: Evidence from manual and automatic analysis[END_REF]. But more than the valence, it seems that it is the affiliative behaviors that are impacted, supporting a social withdrawal [START_REF] Girard | Social risk and depression: Evidence from manual and automatic facial expression analysis[END_REF][START_REF] Girard | Nonverbal Social Withdrawal in Depression: Evidence from manual and automatic analysis[END_REF]. In fact, certain patterns of behavior when interacting with others may predict the severity of symptoms or treatment response [START_REF] Fiquer | Is nonverbal behavior in patients and interviewers relevant to the assessment of depression and its recovery? A study with Dutch and Brazilian patients[END_REF][START_REF] Geerts | Ethological approaches to psychiatric disorders: focus on depression and schizophrenia[END_REF][START_REF] Bouhuys | Analysis of observed behaviors displayed by depressed patients during a clinical interview: relationships between behavioral factors and clinical concepts of activation[END_REF][START_REF] Bouhuys | A longitudinal study of interaction patterns of a psychiatrist and severely depressed patients based on observed behaviour: an ethological approach of interpersonal theories of depression[END_REF][START_REF] Geerts | Nonverbal support giving induces nonverbal support seeking in depressed patients[END_REF][START_REF] Bouhuys | Do interactional capacities based on observed behaviour interfere with improvement in severely depressed patients?[END_REF]. Finally, physiological measures are mainly lowered for both HRV (ie, HF, LF, VLF, and time domains) [START_REF] Koch | A meta-analysis of heart rate variability in major depression[END_REF] and EDA (ie, SCR and SCL) [START_REF] Sarchiapone | The association between electrodermal activity (EDA), depression and suicidal behaviour: A systematic review and narrative synthesis[END_REF].

Bipolar Disorder

Speech Analysis

Several studies managed to classify the course of mood episodes or relapses with high confidence. For instance, speech pause duration [START_REF] Maxhuni | Classification of bipolar disorder episodes based on analysis of voice and motor activity of patients[END_REF] and number of longer pauses [START_REF] Guidi | Features of vocal frequency contour and speech rhythm in bipolar disorder[END_REF] are significantly higher in depressive states than in a hypomanic or euthymic state. Concerning prosodic and source features, Guidi et al [START_REF] Guidi | Features of vocal frequency contour and speech rhythm in bipolar disorder[END_REF][START_REF] Guidi | Voice quality in patients suffering from bipolar disease[END_REF][START_REF] Guidi | Analysis of running speech for the characterization of mood state in bipolar patients[END_REF] repeatedly found an increase of F0 in hypomanic states compared with euthymic or depressive states. Similar results were also obtained by Vanello et al [START_REF] Vanello | Speech analysis for mood state characterization in bipolar patients[END_REF]. Likewise, filter features seem to increase with manic mood [START_REF] Zhang | Analysis on speech signal features of manic patients[END_REF], especially F1 and F2 formants, which correspond to the first and the second peak in the spectrum that results from a resonance of the vocal tract.

Finally, many studies extracted several speech features (ie, the openSMILE toolkit), achieving good classification accuracies [START_REF] Gideon | Mood state prediction from speech of varying acoustic quality for individuals with bipolar disorder[END_REF][START_REF] Karam | Ecologically valid long-term mood monitoring of individuals with bipolar disorder using speech[END_REF]. Classification of depressive states could be obtained with an area under the curve (AUC) value of 0.78 and manic and mixed states with an AUC value of 0.89 [START_REF] Faurholt-Jepsen | Voice analysis as an objective state marker in bipolar disorder[END_REF].

Physiological Measures

Heart Rate Variability A meta-analysis [START_REF] Faurholt-Jepsen | Heart rate variability in bipolar disorder: A systematic review and meta-analysis[END_REF] found reduced HRV compared with healthy controls. In particular, LF-HRV was reduced but no differences in either HF or LF-to-HF ratio were found. However, these analyses did not consider different mood phases. Recent studies were interested in the exploration of HRV during a specific phase of BD. First, Wazen et al [START_REF] Wazen | Heart rate variability in patients with bipolar disorder: From mania to euthymia[END_REF] showed that several HRV parameters (HF-HRV, time-domain, and non-linear domain measures) could increase when patients go from the mania phase to the euthymic phase. Conversely, Faurholt-Jepsen et al [START_REF] Faurholt-Jepsen | Heart rate variability in bipolar disorder: A systematic review and meta-analysis[END_REF] reported an increased HRV in manic states compared with depressive and euthymic states, but no difference between the depressive and euthymic states. Hage et al [START_REF] Hage | Diminution of Heart Rate Variability in Bipolar Depression[END_REF] found that LF-HRV and heart period were significantly lower in patients with BD compared with healthy controls. However, after 8 weeks of treatment, there were no significant changes in HRV parameters.

The link between disease severity and HRV has also been explored. Benjamin et al [START_REF] Benjamin | Heart rate variability is associated with disease severity in psychosis spectrum disorders[END_REF] found a significant association between HF-HRV and disease severity. In the same way, Ortiz et al [START_REF] Ortiz | Reduced heart rate variability is associated with higher illness burden in bipolar disorder[END_REF] found that longer illness duration, higher number of depressive episodes, longer duration of most severe manic/hypomanic episode, comorbid anxiety disorders, and a family history of suicide were associated with a reduced HRV. Moreover, the severity during a depressive episode was associated with lower HRV. Finally, Freyberg et al [START_REF] Freyberg | Differences in psychomotor activity and heart rate variability in patients with newly diagnosed bipolar disorder, unaffected relatives, and healthy individuals[END_REF] compared 20 newly diagnosed BD cases, 20 unaffected first-degree relatives, and 20 healthy controls, and demonstrated that HRV did not differ in any measures between the 3 groups.

Lastly, 2 studies have tried to distinguish bipolar depression and MDD, in terms of HRV. First, Chang et al [START_REF] Chang | Distinguishing bipolar II depression from unipolar major depressive disorder: Differences in heart rate variability[END_REF] studied a population of patients with depression and BD type II and found that, compared with patients with MDD, they had lower total HRV, with lower HF and LF-HRV but higher LF-to-HF ratio.

The authors concluded that HRV may aid in the differential diagnosis of bipolar depression type II and unipolar depression. Second, Hage et al [100] supported these findings by comparing patients with MDD with those with BD depression including BD type I, type II, and "not otherwise specified." Indeed, patients with MDD had significantly higher baseline sinus arrythmia (ie, HF-HRV) and LF-HRV in comparison to patients with BD.

Electrodermal Activity

Greco et al [101] showed that EDA levels could differentiate the phases of BD (depressive, euthymic, and mixed states). Indeed, electrodermal hypoactivity could be a rather stable trait of patients with depression and may reflect euthymia or remission when EDA increases [101]. Lemaire et al [102] measured the intensity and duration of physiological responses to affective stimuli. For this purpose, they used EDA and the heart interbeat intervals. The authors concluded that affective dysregulation, a key dimension of BD, could be measured as an increased affective reactivity to neutral pictures and reduced maintenance of subjective affective responses to all type of pictures, irrespective of the clinical state.

Concluding Remarks

For the digital phenotype of BD, we found that, in the depressed phase, speech changed in temporal (ie, decreased speech pause) and prosodic features (ie, decreased F0), but both increased when patients go into the hypomanic phase [START_REF] Maxhuni | Classification of bipolar disorder episodes based on analysis of voice and motor activity of patients[END_REF][START_REF] Guidi | Features of vocal frequency contour and speech rhythm in bipolar disorder[END_REF][START_REF] Guidi | Voice quality in patients suffering from bipolar disease[END_REF][START_REF] Guidi | Analysis of running speech for the characterization of mood state in bipolar patients[END_REF][START_REF] Vanello | Speech analysis for mood state characterization in bipolar patients[END_REF].

Independent of the phase, HRV appears reduced (ie, LF) [START_REF] Faurholt-Jepsen | Heart rate variability in bipolar disorder: A systematic review and meta-analysis[END_REF], but the change in HRV between phases remains unclear [START_REF] Wazen | Heart rate variability in patients with bipolar disorder: From mania to euthymia[END_REF][START_REF] Hage | Diminution of Heart Rate Variability in Bipolar Depression[END_REF]103]. For EDA, an electrodermal hypoactivity in a depressive state is reported, which increases when patients move toward euthymia [101].

PTSD and Psychological Trauma

Speech Analysis

The link between trauma and voice has been likewise explored.

Nonverbal Behavior Analysis

A core symptom of PTSD is an exaggerated startle response [START_REF]Diagnostic and statistical manual of mental disorders (DSM)[END_REF]. In their study on patients with PTSD, McTeague et al [114] focused on the eye-blink startle response to narrative texts corresponding to traumatic stories. The authors found a heightened startle reflex response during trauma imagery, which was also evident in reactivity to anger, panic, or physical danger compared with controls. Surprisingly, participants with multiple traumas showed a blunted reactivity compared with participants with a single trauma. Furthermore, participants with multiple traumas had more chronic and severe PTSD with more anxiety and mood comorbidity [114].

In the same paradigm, Blechert et al [115] studied the bodily startle response to electric stimuli in individuals with trauma, with and without PTSD. The results implied a primary response characterized by reflexlike facial and torso movement and a secondary response characterized by an emotional facial response. Besides, participants with PTSD had a stronger initial startle response and a more negatively valenced secondary facial expression compared with controls with trauma. Beyond the startle response to threat, several studies with eye tracking showed that higher levels of PTSD symptoms were associated with an increased attentional bias toward threat [116]

Physiological Measures

Heart Rate Variability

Several studies have investigated HRV in PTSD and suggested reduced parasympathetic activity in PTSD. Indeed, 2 meta-analyses found that HRV is lower both at rest and during stress in PTSD compared with controls. Particularly, RMSSD and HF-HRV were reduced, while SDNN and LF-HRV were also reduced, but with a smaller effect. Furthermore, a higher LF-to-HF ratio was found in PTSD [119,120]. Another meta-analysis [121] investigated the relationship between early life maltreatment (ELM) and resting-state HRV. Although no evidence for a relationship was found in comparative and correlational meta-analyses, the meta-regression analysis disclosed interesting results regarding both age and presence of psychopathology. In fact, studies including older-aged samples reported greater reductions in resting-state HRV in association with more severe ELM. Moreover, in clinical samples, compared with nonclinical samples, the authors found stronger reductions in resting-state HRV in ELM-exposed individuals compared with those nonexposed. The authors assumed that ELM could lead to alterations in the development of the autonomic nervous system, especially to an insufficient increase in vagal activity during childhood and adolescence, leading particularly to deficiencies in affective regulation that could increase the risk for psychopathology [121].

Finally, Stone et al [122] compared HF-HRV in women with depression with and without a history of childhood emotional abuse (CEA) and in controls without CEA, and reported that women with depression with CEA exhibited lower HF-HRV than both groups without CEA.

Electrodermal Activity

In patients with PTSD, higher subjective levels of intrusive reexperiencing have been linked to physiological reactivity during trauma recall, while acute dissociative symptoms could show reduced physiological reactivity to trauma memories. Actually, increased physiological reactivity was not associated with the self-report of memory avoidance or dissociation but rather with the degree of memory intrusiveness felt [123,124]. Thus, it can be hypothesized that persistently high physiological reactivity may facilitate chronic arousal dysregulation that disrupts neural systems and could lead to maintain the intrusiveness of remembering [125,126].

Patients with PTSD showed an increased physiological reactivity during trauma recall, while patients with depression had a low physiological response to the memory of an event that they associated with the onset of their depression [126]. Although some researchers suggest a possible attenuation of physiological reactivity in patients with PTSD by depressive symptoms [114], when comparing trauma recall between PTSD with and without depression, no blunting effect was found by depressive mood on physiological reactivity [126]. D 'Andrea et al [127] studied individuals exposed to trauma and showed different autonomic reactivities in response to startling sounds according to the type and symptomatology of trauma. Indeed, blunted autonomic reactivity was found in participants with significant symptoms and early exposure to multiple traumas. Conversely, more attenuated trauma exposure had large heart rate acceleration and SCRs.

Concluding Remarks

In PTSD, modifications in speech prosodic and source features are found as in depression [104,110-113], but specific changes are reported in the speech content, especially during trauma narrative or trauma recall, that is, use of first-person singular pronouns [105,106], and changes in narrative coherence [108,109].

In the same way, behavior analysis shows a specific startle response during trauma recall (ie, eye blink or bodily startle) [114,115]. Besides, facial expressivity with higher fear or anger could be predictive of the presence of PTSD [106].

In physiological measures, for HRV, PTSD studies indicated a reduced parasympathetic activity (ie, HF, LF, and time domains) [119,120] as in depression. However, the presence of trauma in people with depression could further decrease HRV (ie, HF) [122]. For EDA, the presence of PTSD, especially with memory intrusiveness felt, leads to an increased EDA reactivity during trauma recall while acute dissociative symptoms could have the opposite effect [123,124,126].

Discussion

Principal Expectations

In this review, we were interested in the state-of-the-art application of new digital measurements to assess symptoms in patients with several psychiatric conditions. In particular, we explored digital markers through speech and behavioral analysis, as well as physiological measures such as HRV and EDA. We focused our research on depression and its different clinical characteristics.

Limitations

Although several databases have been included, relevant papers may have been missed due to the choice of keywords. This paper is not intended to be a systematic review and a broader review could explore other interesting areas. We included a wide range of study designs due to the exploratory nature of our review but the design of the included studies also limited the review. Other reviewers have attempted to reduce the study selection bias but they have acknowledged the possibility of subjectivity in their analyses of the findings.

In addition, we noticed that overall, most studies only investigated the use of 1 digital measure, except for a few that tried to combine, for instance, the measure of EDA with HRV or speech with behavioral analysis.

Apart from a few ethological analyses, most studies analyzed the patient alone, without being in interaction. However, social interaction, especially during clinical interviews, seems to be the most important source of information for clinical assessment.

RenderX

Moreover, we frequently found that each study used different measures that raise a problem of harmonization. For example, in speech analysis, some studies assessed the type of word used, prosodic or spectral features, or even analyzed a "toolkit" with hundreds of features. Further, certain studies investigated features extracted from free speech, whereas others focused on more constrained speech tasks.

Video analysis can be heterogeneous as well, in that some studies are interested in eyes, head, or torso movements at rest or during an evaluation. In HRV assessment, several studies used electrocardiogram but others used a finger clip sensor. In the same way, EDA can be measured with or without a concurrent task and with different types of technological tools, with variable degrees of precision.

Finally, we did not take into consideration articles dealing with depression with comorbid anxiety. Future research on this topic would be relevant to complete our findings.

Implications for Practice

Assuming that digital markers would help in the differential diagnosis for an MDE, clinicians will be able to make an early diagnosis. Thus, psychotherapy and drug treatment could be introduced early, and diagnostic and therapeutic mistakes could be limited. Despite the lack of comparative studies, promising digital markers appear to emerge from this review, particularly within physiological measures such as HRV. In fact, patients with bipolar depression had more severe decrease than those with unipolar depression (ie, HF and LF-HRV) [START_REF] Chang | Distinguishing bipolar II depression from unipolar major depressive disorder: Differences in heart rate variability[END_REF]100], whereas the presence of comorbid PTSD in MDD seems to induce a greater reactivity of physiological measures, especially when the clinician touched upon psychological trauma [126]. This reactivity with PTSD also seems to be reflected in speech and nonverbal behavioral analysis, which suggests other candidates for digital markers [108,109,114,115].

Conclusion and Future Directions

Despite many limitations, the studies detailed in our review showed promising results on the usefulness of digital phenotyping for the differential diagnosis of MDE. The authors believe that additional research is needed to better understand the potential value of digital phenotyping in clinical practice. Indeed, what should be relevant is a prospective and observational study in which different digital tools will be combined during recordings of social interaction. At least three distinct groups would be required: MDD, bipolar depression, and depression comorbid with PTSD. For each group, it would be interesting to merge speech, behavior, and physiological measures extracted from recordings of clinical interaction between patients and clinicians. By designing larger and comparative studies, a distinct digital phenotype could be defined for each of the aforesaid 3 groups. Thus, digital markers to help differential diagnosis could emerge. Moreover, if large prospective studies are conducted, validated digital markers could be used for follow-up and for predicting treatment response or risk of relapse. Digital phenotyping could therefore contribute to personalized and precision medicine.

JMIR

  Ment Health 2023 | vol. 10 | e37225 | p. 15 https://mental.jmir.org/2023/1/e37225 (page number not for citation purposes) Ettore et al JMIR MENTAL HEALTH XSL • FO

Table 1 .

 1 Summary of speech analysis, nonverbal behavioral analysis, heart rate variability (HRV), and electrodermal activity studies (EDA) in patients with major depressive disorder (MDD).

	JMIR MENTAL HEALTH						Ettore et al
	Study	Participants	Recording setting	Principal findings	Principal features
	Speech analysis studies
	Al-ghowinem et al [21]	•	30 with MDD and 30 in HC a	• •	Reading tasks with negative and positive meaning Free speech by answering questions, where the patients describe events that had aroused significant emotions	•	MFCC b , jitter, shimmer, ener-gy, and loudness features were robust in getting the general characteristic of de-pressive speech	•	Prosodic, spectral, and source features
	Arevian	•	15 with MDD,	•	Free speech (answering ques-	•	Correlation with providers'	•	Type of words used
	et al [22]		14 with BD c , 14 with schizophre-nia, and 14 with affective schizophrenia		tions from an app with phone call)	• •	global assessment: ↓ Negative emotional lan-guage ↑ Positive emotional lan-guage
						•	↑ Complex word use
	Hönig et	•	219 participants	•	Read speech, spontaneous	•	Depression severity:	•	Prosodic, spectral, source,
	al [23]		with BDI (BD type I) assess-ment		speech, telling an imagined story	• • • • • •	↓ Average and SD MFCC 2 ↓ SD MFCC 3 ↓ Average pitch ↓ Shimmer ↓ Spectral harmonicity ↓ Speech rate (↑ Average		and filter features
							syllables duration)
	Horwitz	•	35 participants	•	Free speech, reading, and dis-	•	Depression severity:	•	Prosodic, spectral, source,
	et al [24]		with Hamilton Rating Scale for Depression and Quick Inventory of Depressive Symptomatology assessment		tinct vowels	• • • • •	↑ Jitter and shimmer ↓ Harmonic-to-noise ratio ↓ First and third formants ↓ Speech rate ↓ (Read) and (free speech) pitch-based features		and filter features
	Mundt et	•	105 with MDD	•	Free and with tasks: reciting	•	Correlation with depression	•	Temporal and prosodic fea-
	al [25]				the alphabet, counting, reading,		severity:		tures
					and sustained vowels	• • • • • •	↑ Total speech time ↑ Total pause time ↑ Variable pause length ↑ Percentage pause time ↓ Speech pause ratio ↓ Speaking rate
	Quatieri	•	35 with MDD	•	Free speech for prosodic mea-	•	Depression severity and psy-	•	Prosodic, source, and filter
	et al [26]				surements and distinct vowels		chomotor retardation:		features
					for shimmer, jitter, and aspira-tion measurements	• • • • • • •	↑ Shimmer ↑ Aspiration ↓ Harmonic-to-noise ratio Depression severity ↑ Jitter ↓ Pitch variance ↓ Average velocity
	Shino-	•	30 with MDD	•	Free speech and reading	•	MDD correlated with:	•	Temporal and "emotional"
	hara et al [27]		and 14 in HC	•	phrases aloud Detection of emotional ele-ments (ie, "anger" and "joy")	•	↓ Vitality scores (effect size 1.03 and AUC d 0.76, sensitiv-		prosodic features
				•	2 indices extracted: "vitality"		ity 0.93, and specificity 0.55)
					and "mental activity"
	https://mental.jmir.org/2023/1/e37225				JMIR Ment Health 2023 | vol. 10 | e37225 | p. 4
	XSL • FO								(page number not for citation purposes)
	RenderX

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 6 https://mental.jmir.org/2023/1/e37225 (page number not for citation purposes) JMIR Ment Health 2023 | vol. 10 | e37225 | p. 7 https://mental.jmir.org/2023/1/e37225 (page number not for citation purposes) JMIR Ment Health 2023 | vol. 10 | e37225 | p. 9 https://mental.jmir.org/2023/1/e37225 (page number not for citation purposes)

  . Some authors assumed that negative affect in PTSD could have an important role in maintaining this attention bias[117].

	JMIR MENTAL HEALTH	Ettore et al
	singular pronouns, lower audio intensity, and reduced pitch
	were important features for the PTSD classification, whereas a
	reduced frequency of positive words was predictive of
	depression.
		More generally, Katz et al [118] emphasized the importance of
		trauma disclosure. The authors analyzed 40 video interviews
		of children who have suffered an abuse, but only half of them
		disclosed it. Children who did not disclose the abuse showed
		specific nonverbal cues, such as more physical disengagement,
		than children who disclosed. Schultebraucks et al [106]
		performed deep learning audio-and video-based classification
		of depression versus PTSD for 81 participants, 1 month after
		their admission to a trauma center following a traumatic
		exposure. The authors successfully classified between PTSD
		and MDD status with a high discriminatory accuracy. Likewise,
		several voice and video markers have been identified as
		predictive of either PTSD or MDD. In a video analysis, higher
		fear and anger expressivity were predictive of PTSD, while
		higher contempt expressivity was important for the classification
	XSL • FO
	RenderX

of MDD. In an audio analysis, an increased use of first-person JMIR Ment Health 2023 | vol. 10 | e37225 | p. 14 https://mental.jmir.org/2023/1/e37225

(page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 8 https://mental.jmir.org/2023/1/e37225(page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 11 https://mental.jmir.org/2023/1/e37225(page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 12 https://mental.jmir.org/2023/1/e37225(page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 13 https://mental.jmir.org/2023/1/e37225(page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 16 https://mental.jmir.org/2023/1/e37225 (page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 17 https://mental.jmir.org/2023/1/e37225 (page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 18 https://mental.jmir.org/2023/1/e37225(page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 19 https://mental.jmir.org/2023/1/e37225 (page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 20 https://mental.jmir.org/2023/1/e37225(page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 21 https://mental.jmir.org/2023/1/e37225(page number not for citation purposes)

JMIR Ment Health 2023 | vol. 10 | e37225 | p. 23 https://mental.jmir.org/2023/1/e37225(page number not for citation purposes)

Acknowledgments

This research is funded by the MEPHESTO project (BMBF grant number 01IS20075; DRI-0120105/291). We thank the "Innovation Alzheimer" Association for its financial support to the publication.

Authors' Contributions

EE, AK, PM, JH, BG, DP, HL, MB, AL, and PR conceptualized the study. EE and AK wrote and prepared the original draft. AK, PM, JH, and PR provided visualization and supervision support. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest

None declared. ©Eric Ettore, Philipp Müller, Jonas Hinze, Matthias Riemenschneider, Michel Benoit, Bruno Giordana, Danilo Postin, Rene Hurlemann, Amandine Lecomte, Michel Musiol, Hali Lindsay, Philippe Robert, Alexandra König. Originally published in JMIR Mental Health (https://mental.jmir.org), 23.01.2023. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.

Multimedia Appendix 1

Search strings. [DOCX File , 72 KB-Multimedia Appendix 1]

Multimedia Appendix 2


Othman Lakhal

Abdelkader Belarouci

Xinrui Yang

Taha Chettibi

Rochdi Merzouki

CoboShell Robot for Automatic Scallop Shelling Process: Concepts and Applications

Keywords: Automatic scallop shelling, Collaborative robot, Artificial intelligence, Force-compliance control

Shelling scallops can be a labor-intensive and repetitive task that can lead to musculoskeletal disorders for the operators. The repetitive nature of the task and the force required to open the scallop shells can significant strain the hands, wrists, and arms of the operators. In this study, we explored the automation of the shelling process in the seafood industry. We introduced a new mechatronic system called CoboShell, which automates the opening of scallops and the cutting of the nut muscle using a collaborative robot (cobot). CoboShell ensures the preservation of the quality of the fresh sea product. Initially, we addressed a technological challenge by proposing an integrated solution based on the Venturi effect. We used appropriately sized suction cups to handle the non-uniform surfaces of the shells. Subsequently, we focused on a scientific inquiry involving the control of the knife blade's trajectory using a cobot. The goal was to guide a flexible knife blade along the inner flat part of the scallop shell. This process aimed to cut the nut muscle and separate the two parts of the scallop. For that, the entry point of the knife is determined automatically in the centre of the scallop opening using an Artificial Intelligence (AI) based technique. Finally, experimental results as well as the repetitive tests have shown that the cobot provided better quality on the muscle cutting with high accuracy and cadence, comparing to conventional industrial arm in the automation of the scallop shelling process.

I. INTRODUCTION

S CALLOPS are commercially important shellfish world- wide, with over 40 commercially exploited species of bivalve shellfish. These shellfish are frequently harvested or cultivated to meet the high demand [START_REF] Dupaul | An evaluation of atsea handling practices: Effects on sea scallop meat quality, volume and integrity[END_REF]. Scallops are usually processed into frozen adductor muscles for sale and export. Manual shelling is the most reliable method for removing the meat from the shell while retaining its quality for raw consumption.

Bivalve molluscs like black scallops, scallops, and oysters have two shells or valves connected by a ligament along the hinge, held closed by an adductor muscle called a nut. When these molluscs are taken out of their natural environment, the adductor muscles contract, tightly linking the two valves together. Extracting the flesh from inside the shell requires separating the adductor muscles from the valve, traditionally done by inserting a knife. Manual shelling ensures the physical integrity, sensory quality, and sanitation of the meat, but it demands skilled individuals with strength and experience Othman Lakhal, Abdelkader Belarouci, Xinrui Yang , Taha Chettibi and Rochdi Merzouki are with University of Lille, CRIStAL, CNRS-UMR 9189, Avenue Paul Langevin, 59655 Villeneuve d'Ascq, France. e-mail: othman.lakhal@univ-lille.fr [START_REF] Lu | Application of response surface methodology in scallop shelling[END_REF]. Manual shelling is performed by inserting a knife blade near the hinge of the shells and carefully separating the nut without causing damage. This technique offers more control and precision compared to mechanical or automated methods, resulting in a superior quality product. The process of manual shelling follows the steps shown in Figure 1. The quality of the nut recovered and the amount of adductor muscle remaining attached to the shell vary from person to person, posing challenges when dealing with a large number of shells. If not treated promptly, the freshness of the flesh decreases, particularly with manual shelling. Moreover, the manual shelling process often leads to wrist problems, microtraumas, and musculoskeletal disorders due to the required force to hold the shell open during muscle separation. The scallop shelling sector faces declining attractiveness in shelling activities and the need to enhance and ensure the quality of landed scallop products.

Addressing the problem of separating the shell from the mollusc meat, regardless of the processed products, is crucial. Automation or robotisation of scallop shelling is of particular interest in the food processing industry, as it can process large quantities of scallops in a short time. Various shelling methods have been developed to assist manual sheller, with a significant increase in inventions and innovations in this field. Martin [START_REF] Martin | Oyster shucking technologies: past and present[END_REF] provides a review comparing the advantages and disadvantages of different scallop shelling technologies, including mechanical, thermal, and pressure-based processes.

Steam machines [START_REF] Zhang | Research on control system of steam scallop shelling machine[END_REF] and high-pressure (HP) shelling [START_REF] Yi | Shucking of bay scallop (argopecten irradians) using high hydrostatic pressure and its effect on microbiological and physical quality of adductor muscle[END_REF] have been proposed, but they can significantly alter the moisture, pH, color, and texture of the adductors. High hydrostatic pressure (HHP) has been explored as an alternative for scallop shelling [START_REF] Yi | Shucking of bay scallop (argopecten irradians) using high hydrostatic pressure and its effect on microbiological and physical quality of adductor muscle[END_REF], but it must be applied thoroughly to avoid scale appearance in the muscles. Heating and cooling processes have also been used for scallop shelling [START_REF] Martin | Development and testing of a heat-cool methodology to automate oyster shucking[END_REF], [START_REF] Martin | Effectiveness of a heat/cool technique for shucking oysters[END_REF], but they compromise freshness by cooking the product.

Other methods, such as laser technology, have been applied to sever the adductor muscle precisely [START_REF] Singh | Laser modernizes oyster shucking[END_REF], but their use is limited to certain mollusc species. Numerous patents exist, reflecting the ongoing efforts to automate scallop shelling. Mechanical, cold, electric discharge, hot or steam, and highpressure processes have been explored [START_REF] Kiczek | Method for opening a mollusk[END_REF], [START_REF] Hanks | Method and apparatus for removing meat from the shells of bivalve mollusks[END_REF], [START_REF] Brown | Apparatus for removing scallops from their shells[END_REF], [START_REF] Yoshida | Method and apparatus for peeling-off bivalve, as well as bivalve before-separation heating device[END_REF], [START_REF] Rodman | Method of shucking scallops and an apparatus therefor[END_REF], [START_REF] Earnshaw | Shellfish processing apparatus[END_REF]. However, these methods often result in mutilation or cooking of the nut, hindering the live recovery of scallops in good condition. Despite the progress made so far, there is still a need to develop an affordable and efficient shelling method that preserves quality and allows for the live recovery of scallops in good condition [START_REF] Martin | Optimization and automation of a thermal oyster shucking process[END_REF].

Developing an automatic scallop shelling device that minimises impurities and maintains product quality poses a significant challenge. Replicating the manual shelling process is crucial to preserve the raw consumption characteristics and production conditions. The device must separate the shell surfaces without breakage, minimise waste, and ensure the adductor muscle remains uncooked and alive. Additionally, the scallop fishing season is limited to a few months per year. The compact and easily installable CoboShell device presented in this article addresses these requirements and can be tailored to optimise the working area. Real-time control of the knife robot-guide's kinematics during the shell opening process is the scientific problem addressed. The paper's structure includes sections on the CoboShell system and shell opening detection methodology (Section II), modelling and control of the shelling system (Section III), experimental results and discussions (Section IV), and concluding remarks (Section V).

II. SYSTEM AND PROCESS DESCRIPTION

In this section, we will outline our concept for processing freshly caught scallops, obtaining shelled scallops while keeping the animal alive. Then, we will also provide a detailed description of the components of our scallop shelling system. The proposed process in this article is used to treat freshly caught shellfish, as shown in Fig. 2. The shells are placed manually or automatically by another system on a conveyor in a vertical position on a support. After fixing the shell holder with stops installed on the conveyor, the system opens the shells by separating the two valves held by the muscle. This separation, known as opening, is detected by a vision system based on a learning algorithm. The latter, by image processing, locates and returns the coordinates of the entry point of the knife tip. Finally, the system cuts the muscle to recover the curved shell containing the live animal. This will then be processed by an operator or another system. The table I shows the dimensions of the scallops during the different fishing seasons. This information was used to design a specific support by rapid prototyping, capable of adapting to different sizes of scallops shells and keeping them in vertical positions. The system comprises a scallop opening system, a camera for detecting the opening and locating the insertion point, and a knife guided by a cobot for muscle cutting, as depicted in Figure 3. Unlike mechanical methods, the proposed system is designed to minimise contact with the live animal and reduce the risk of shell breakage. This ensures that the bivalve mollusc remains alive and uncooked throughout the process, resulting in shelling with minimal impurities and without compromising product quality. Fig. 3: CoboShell description : The shell is on a conveyor (1) in a vertical position on a support (2), the opening system (3) applies compliance to different types of valve surfaces using a vacuum system. The detection of the opening and the location of the entry point of the knife tip is achieved by a vision system (4). Finally, the adductor muscle is cut using a cobot (5) that guides the knife blade (6) along the flat shell for minimal material loss.

A. Shell opening system

The opening system consists of two rigid suction cups adapted to optimise contact with the shell whether it is rough or smooth, dirty or clean, flat or domed, so as to obtain a good airtight seal. Indeed, the shape of the scallop shell is jagged and not regular. In addition, the valves are ridged and are not symmetrical: one side is flat and the other curved. Thus, a small suction cup is used for the curved part, and a large suction cup for the flat part, which guarantees optimal contact regardless of the size of the shell, as shown in Fig. 4. In order to allow a good softness of the suction pads, an universal joint has been added linking the suction cup to the cylinder. During the opening procedure, it is essential to adapt the orientation of the suction cup to the inclination of the valves. Three pneumatic cylinders are used to grip and open the shells. These actuators are operated simultaneously to secure the shell without causing any impact. One specific actuator (3) is activated to achieve a partial opening of at least 10 mm in the flat valve, allowing the knife to pass between the valves and slide along the flat valve for muscle cutting. After the muscle is cut, the guided cylinder ( 1) is engaged to break the hinge and remove the flat valve. The vacuum is then released from the suction cups to separate the shell parts. The guided cylinder [START_REF] Hanks | Method and apparatus for removing meat from the shells of bivalve mollusks[END_REF] is retracted to its initial position, enabling the operator to retrieve the assembly. An air-jet pump generates the vacuum, and vacuum sensors ensure a vacuum level between 60 and 70%. Positioning the vacuum sensor and air-jet pump close to the suction cup is crucial for optimal operation and maximum vacuum efficiency. The recommended maximum distances are 200 mm for the air-jet pump and 100 mm for the sensor based on testing. Due to the external force (ambient air) being greater than the internal force, the suction cup and the part remain connected throughout the procedure. It is important to note that opening a freshly caught shell may require a significant force approximately 70 N.

B. Detection and location of opening

A deep learning-based approach has been implemented for the detection and localization of shell opening area, as reported in [START_REF] Yang | Integrated design of an aerial soft-continuum manipulator for predictive maintenance[END_REF]. The approach uses images captured by a visual sensor (camera) located above the opening system to detect the opening area of the shell, as illustrated in Fig. 5-a. The model's output provides a binary mask that covers the opening area between the two walls of the shell, as shown in Fig. 6. From this binary mask, the contour of the opening is determined by calculating the central moment of the binary mask, as shown in Fig. 5-b. Furthermore, by applying the morphological transformation of the median axis, a central line (i.e., a skeleton) can be extracted from the opening area. The entry point is then calculated by determining the central point of the central line of the opening area. The location of the entry point in the industrial arm frame is subsequently determined by applying a transformation. To achieve this, a QR Code is placed in the image frame for precise localization (Fig. 5b). The instance segmentation model that we choose is Mask RCNN [START_REF] He | Mask r-cnn[END_REF], it has been implemented in other single-targetsingle-class instance segmentation tasks in multiple fields and has shown better performance compared to other models in term of efficiency and precision. The model is constructed with a backbone consisting of a ResNet50 and a Feature Pyramid Network (FPN), as shown in Fig. 6. A shell-opening

C. Collaborative Robot (cobot)

The cobot is utilized to guide the knife blade such that it passes between the two shells and cuts the muscle, after the opening. The specific robot used is the P-Rob 2R developed by F&P Robotics AG. This robot is lightweight and its computer is integrated within the structure, simplifying the installation and manual movement. The P-Rob 2 robotic arm is a 6 degree of freedom (DoF) manipulator. Additionally, it is force-limited and detects collisions, preventing damage to the shell in the event of impact in our context. This is because the blade of the knife must be in contact with the flat surface of the shell throughout the cutting of the muscle. In addition, it is important to note that the shelling trajectory varies according to the size of the shell and the opening distance. The effort required to open each shell is different. Although the opening cylinder's range is limited to 2 cm, it rarely reaches this limit with freshly caught shells. In order to insert a knife and cut the muscle, the opening distance must be greater than 10 mm. The vision system described in the previous section returns the coordinates of the entry point. The cobot must guide the knife blade towards the entry point. Once the entry point is reached by the blade, it becomes a constraint on the robot's kinematics. As illustrated in the Fig. 7 , the orientation of the knife must be relative to the entry point while maintaining contact with the flat surface to prevent material loss, by adjusting the compliance accordingly.

III. MODELLING AND CONTROL OF THE SYSTEM

In this section, we describe the steps for the trajectory control of the knife blade tip based on the cobot.

A. kinematics transformation

Let's affect the frame ℜ 0 to the base of the cobot, the frame ℜ TCP to the knife blade tip, the frame ℜ Qr to the center of the QR Code, the frame ℜ C to the camera and the frame ℜ SO to the scallop opening machanism, as presented in the Fig. 8. 0 TCP T is the transformation of the knife blade tip with respect to the basis frame of the cobot. This transformation is obtained Fig. 8: Kinematic frames of cobot and vision systems from the Forward Kinematic Model (FKM) based on the joint positions. Transformation SO 0 T between the cobot frame and that of the scallop opening mechanism is measured during the preparation stage. The TCP has been calibrated according to the cobot frame. The QR Code is attached to the frame of the scallop opening mechanism, so the transformation Qr SO T . Through a vision system based on a camera, the transformation C Qr T of the QR Code position is detected. This method uses image analysis to identify the contours of the QR Code and determine its position. Finally, EP C T is obtained from the RCNN mask described in the previous section. Thus, the entry point (EP) is calculated in the knife blade frame, which is the target point for introducing the knife into the scallop. The total kinematics transformation EP TCP T is calculated using Eq. ( 1).

B. CoboShell cobot Kinematics

Fig. 9 represent the kinematics chain of the 6-DoF Cobot. The forward kinematic equations are obtained from the modified Denavit-Hartemberg (D-H) parameters, which describes the homogeneous transformation matrix i-1 T i , where i is the joint number. Finally, the position of the tool frame with respect to the cobot frame is obtained as follows: Fig. 9: CoboShell cobot kinematics chain. 0

TCP T = 0 1 T 1 2 T 2 3 T 3 4 T 4 5 T 4 5 T 5 6 T 6 TCP T (2)
0 TCP T represents the position and orientation of knife blade tip as a function of the six joint variables q i where i = [1..6]. 6 TCP T describes the transformation between the end joint frame and the knife blade TCP frame. This changes based on the type of compliance exhibited by the blade and can be estimated by measuring the applied force obtained from the cobot. The Inverse Kinematic Model (IKM) calculates the joint variables θ i based on the pose (position and orientation) of the knife tip. Paul's method [START_REF] Paul | Robot manipulators: mathematics, programming, and control[END_REF], a technique for solving the inverse kinematic problem in robots with simple geometry, as seen in previous works [START_REF] Lakhal | Hybrid approach for modeling and solving of kinematics of a compact bionic handling assistant manipulator[END_REF]. The IKM is also utilised for trajectory planning, as reaching a specific target point can be achieved in various ways depending on the knife's orientation. The kinematic model is essential for controlling the knife tip and verifying the knife tip's accessibility based on constraints.

Let EP a entry point be defined by 0 EP T expressed in frame ℜ 0 of the robot as shown in Fig. 8. As previously mentioned in previous sections, the target point varies from one shell to another. It is dependent on both the size of the shell and the opening distance. For this purpose, we assume that the opening forms an isosceles triangle as shown in Fig. 10, with a the opening distance obtained by the vision system and h the height which is the distance from the entry point to the shell hinge, can be considered constant as it depends on the fishing season for which the size is knowledge. Thus from the opening distance and the height , the opening angle α can be calculated and the trajectory of the knife tip can be deduced. Based on the coordinates of the entry point {x EP , y EP , z EP } and the trajectory points {x pi , y pi , z pi } ,with i the number of the point , the vector w of the matrix 0 EP T can be obtained as follows:

w = [x pi -x EP , y pi -y EP , z pi -z EP ] T (x pi -x EP ) 2 + (y pi -y EP ) 2 + (z pi -z EP ) 2 (3)
The vector u is normal to the vector w and is parallel to the plane ZX

The vector v is perpendicular to the plane formed by the vectors u and w

v = w ∧ u (5)
Thus, the rotation matrix of 0 EP T can therefore be deduced for each target point.

IV. EXPERIMENTAL RESULTS

The current average rate of manual scallop shelling is 5 shells/minute/person. It should be noted that the tests were carried out under real conditions in humid environment and a temperature of between 2 and 8 • C. In this study, among the 67 shells that were shelled on the day of fishing, 5 shells could not be opened by the system due to their surface degrading during fishing. Indeed, the vacuum could not be made correctly. Moreover, tests on about thirty shells made it possible to determine the effort required to open a fresh shell, the force required being between 40 and 70 N. With an opening rate of over 90%, this proves the robustness of our system, especially on freshly caught shellfish while preserving their quality. Vacuum adsorption is a technique widely used in many robotic fields. The size of the suction cups depends on the type of contact surfaces and the applied efforts for the grasping [START_REF] Zhu | Autonomous pose detection and alignment of suction modules of a biped wall-climbing robot[END_REF] or the loading hold. The main difficulty for the scallop surface is its irregularities, where even if the soft material of the suction cups, such as thin-film polymer, poly (3, 4-ethylenedioxythiophene) [START_REF] Aoyagi | Bellows suction cup equipped with force sensing ability by direct coating thinfilm resistor for vacuum type robotic hand[END_REF] allows a perfect compliance to the surface of the scallop, it generates the air leaks due to the contact with the surface irregularities under air pressure equivalent to 70N of effort, compared to regular surfaces of fruits as an example. Several types and sizes of suction cups were tested to obtain the type of suction cup that would have a perfect grip by adapting to irregular surfaces. In the end, flat and round suction cups with an aluminium support plate were chosen, as shown in the Fig. 11. This vacuum has been adapted by removing the internal part in order to optimise contact with the scallop shell and better manage depression whether rough or smooth, dirty or clean, flat or convex. Fig. 12 illustrates the knife trajectories for a set of shells from different views. The generation of the knife trajectory is observed to adapt to the shell size but also to the opening distance through the insertion point. The measured trajectory is obtained by FKM from the coordinates of the joints. The trajectory performed by the robot is smoothed using a smoothing parameter in the robot language. We observe that the knife deviates +/-5mm from its desired trajectory. However, this is not a problem because the blade of the knife is flexible, on the contrary, it favours the cutting of the muscle. Fig. 12 also shows the accuracy of our detection and localisation based on deep learning. As the robot is a cobot, the speed of its joints is limited to between 50 and 100 deg/s. The peeling rate obtained is about 5 seconds/shell. Other tests were carried out with an industrial robot and we were able to reach 2 seconds/shell. Finally, the laboratory tests allows to identify the stiffness of the knife blade and establish the desired force profile to ensure a stable compliance during the cutting task. This enables more effective and optimal contact with the scallop, thereby enhancing the quality of the shelling process. In summary, this approach represents a noteworthy improvement over existing scallop shelling methods employed in the industry.

V. CONCLUSION

This article deals with a new integrated concept of automatic scallop shelling, called CoboShell. The latter consists of a system for the automatic opening of the scallop shells by the venturi effect, caused by two compliant suction cups, sized in relation to the profiled surface on each side of the scallop shell. A vision system was coupled with artificial intelligence tools for deep learning to accurately determine the shape and width of the opening as well as the position of the entry point to reach the nut muscle. A collaborative robot (cobot) was used to guide the knife blade on a curvilinear trajectory that could reach the scallop muscle. The cobot is controlled in force and compliance to manage interaction with the flat shell surface, following a parametric curve. CoboShell has been designed to replicate the manual shelling process and maintain the quality of the fresh sea product. Experimental results show conformance control performance with 90% material removed and a rate of approximately 5 seconds/scallop.

Fig. 1 :

 1 Fig. 1: Manual process of scallop shelling : (a) Holding the scallop with the hollow shell, inserting the knife into a small opening. (b) Leveraging the knife to lift the flat part and open the scallop. (c) Securing the flat part with the thumb and sliding the knife to cut the adductor muscle. (d) Breaking the hinge to separate the flat part from the hollow part of the scallop.

Fig. 2 :

 2 Fig. 2: System flowchart describing the process steps

Fig. 4 :

 4 Fig. 4: Description of the automatic scallop opening mechanism : (1) and (10) Guided pneumatic cylinder. (3) Pneumatic cylinder. (6) Flexible suction cup mounted on curved surfaces. (7) and (8) Flexible suction cup mounted on flat surfaces. (9) Free-control joint to facilitate the suction cup compliance to curved surfaces. (11) Scallop shell support.

Fig. 5 :

 5 Fig. 5: (a) Considered scallop parts for the deep learning (b) Segmentation model for automatic detection of scallop opening.

Fig. 6 :

 6 Fig. 6: Architecture of Mask RCNN for automatic detection of the scallop opening.

Fig. 7 :

 7 Fig. 7: Tool Centre Point (TCP) of the Knife blade's trajectory under compliance interaction

Fig. 10 :

 10 Fig. 10: Assumption: the opening of the shell forms an isosceles triangle

3 )

 3 , 0 ,w(1)] T (w(3)) 2 + (-w(1)) 2

Fig. 11 :

 11 Fig. 11: Description of the modified suction cup : (a) Removing the rubber disc from the existing suction cup. (b) Large suction cup for the flat part (80 mm) (c) Small suction cup for the curved part (55 mm).

Fig. 12 :

 12 Fig. 12: Trajectories of knife tip performed for the shelling of 15 scallops : (a) Parameters representation (b) 3D view. (c) top view. (d) view on the zy plane.

Fig. 13 :

 13 Fig. 13: Deformation of the knife as a function of the reaction force.

TABLE I

 I

		Dimensions of scallop shells
	Length [mm]	Width [mm]	Depth [mm]	Weight [g]
	Min	Max	Min	Max	Min	Max	Min	Max
	110	140	110	140	20	30	130	240

ACKNOWLEDGMENT

This work was funded by the Regional Funds for Research and Innovation Hauts-de-France (2015-2020) under the name COBOFISH with the reference: DRESS-2018-006407 and the regional network S-MART Hauts-de-France.