CFS Correlation-based Feature Selection CHMM Coupled Hidden Markov Model CLM Constraint Local Models method

Keywords:

introducing me to the academic world. My gratitude to all my professors in the Master Computer Vision that taught me how to extend my limits, accelerate, and improve. More importantly I want to

Next, I would like to thank the Greek State Scholarship Foundation, which funded me through the bequest "in the memory of Maria Zaousi", may God rest her soul.

Continuing I would like to thank my supervisors each one and separately. I will forever be grateful to Prof. Marias who matched me to this topic I came to adore, it was an insightful matching that kept my motivation high during these years. My sincere gratitude goes also to Prof. Tsiknakis, it was inspiring to see you working so hard and being active all the time. Prof. Simos I would like to thank you for teaching me so much and helping me gain some background in terms of the clinical aspects of my work, and for always being close to me and supporting me. Prof. Meriaudeau also for being supportive and close to me, as well as for giving me the opportunity to enter the PhD programme, and for guiding me and offering insights in terms of the machine learning and image processing fields. My sincere thank you to Prof. Yang, for always being supportive too.

Besides my supervisors I had support from fellow researchers, my postdocfriends! Dr M.Pediaditis, Mat I will never say thank you enough, for the xi discussions, for helping me in every step, for the brainstormings, for always being there willing and available any time I dropped by your office, once again thank you! My gratitude also goes to Dr S.Sfakianakis, who was also willing to discuss with me about issues I had doubt, especially regarding the field of statistics. Thank you both for your mentoring.

Continuing I would like to thank the people who supported me during the long term endeavor of the data collection project, it wouldn't have been possible, or at least would have been even more difficult without your help. Mpourou. Thank you all for the co-operation, I also learned from you too. I am also thankful for an unexpected co-operation, this with my old friend Dr A.Fazlollahi, it was great to have the chance to work together again! V.Maniadi and A.Vergetaki thank you for your support in any administration related issues, but also for your valuable friendship. My friends K.Spanakis, G.Christodoulakis, and K.Nikiforaki, with whom we were the "early" birds in the office, thank you for sharing mornings and thoughts! C.Hernadez Matas, I would like to thank you for sharing the same concerns during the PhD struggle! My dearest friend Koula thank you for always being there for me! My academic journey has never been a straight line, and there were many people whom I feel the need to thank, as I believe they all had some contribution to me reaching to this point. My highschool teachers Argyris Nikolaidis, who saw my talent in programming and encouraged me to follow computer science when I was going for English literature, but also R.Tsaknaki for giving me strong programming skills. I would also like to thank Prof. Papadourakis

Introduction

The present chapter presents concepts and facts related to depression, in order to acquaint the reader with the disorder. Methods for depression assessment are summarized, including diagnostic procedures and self-reports. Non-verbal signs of depression, along with the motivation of the work described herein, are also presented. Parts of the contents of this chapter have been published in [START_REF] Pampouchidou | Automatic assessment of depression based on visual cues: A systematic review[END_REF].

Depression Facts

Depression is the most common mood disorder characterized by persistent negative affect of functioning (...)" [START_REF]Diagnostic and Statistical Manual of Mental Disorders (DSM-5 R )[END_REF].

MDD is reported to be the fourth most prominent cause of disability and is expected to become the second by 2020 due to its increasing prevalence [START_REF] Kessler | The Costs of Depression[END_REF]. The "Survey of Health, Ageing and Retirement in Europe" [START_REF]Ageing and Retirement in Europe[END_REF] documents a consistent rise of depression among adults with increasing age, which is associated with significantly elevated risk for suicidal behavior [START_REF] Van De Ven | User-friendly ICT Tools to Enhance Self-management and Effective Treatment of Depression in the EU[END_REF]. The ongoing economic crisis in Europe resulting in high unemployment is implicated as a trigger, since 70-76% of unemployed people have been reported to display significant depressive symptomatology [START_REF] Fountoulakis | Relationship of Suicide Rates to Economic Variables in Europe: 2000-2011[END_REF]. Further studies have shown that the economic burden of MDD has increased during the 2005-2010 period by 21.5% in the US, while in Europe the cost is estimated at 1% of Gross Domestic Product [START_REF] Sobocki | Cost of depression in europe[END_REF]. The total cost of MDD in 2010 in 30 European countries was estimated at 91.9

billion euros [START_REF] Olesen | on behalf of the CDBE2010 study group, and the European Brain Council. The Economic Cost of Brain Disorders in Europe[END_REF].

Depression Assessment

Depression Assessment

Various procedures for depression assessment are presented in this section, while nonverbal signs of depression are also described, emphasizing visual signs.

Clinical diagnosis & Self-Reports

A structured clinical interview assessing the presence of DSM-5 criteria is the standard procedure for depression diagnosis [START_REF] First | Structured Clinical Interview for DSM-IV-TR Axis I Disorders: Patient Edition[END_REF]. Quantification of the presence and severity of depressive symptomatology is often aided by rating scales completed by a specially trained mental health professional in the context of the clinical interview. The Hamilton Depression Rating Scale (HAM-D) is one of the most popular scales in clinical settings.

HAM-D assesses the severity of 17 symptoms, such as depressed mood, suicidal ideation, insomnia, work and interests, psychomotor retardation, agitation, anxiety, and somatic symptoms [START_REF] Hamilton | A rating scale for depression[END_REF]. Both HAM-D and DSM-5 clinical criteria have been criticized regarding their reliability [30] [START_REF] Chmielewski | Method matters: Understanding diagnostic reliability in DSM-IV and DSM-5[END_REF], as diagnosis of MDD is not as consistent as other common medical conditions [START_REF] Kraemer | DSM-5: How Reliable Is Reliable Enough[END_REF]. In general, "there is no blood test" for depression [START_REF] Thomas-Maclean | Diagnosing Depression: There is no Blood Test[END_REF], as the disorder lacks biological gold standards [START_REF] Kapur | Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it[END_REF].

Clinical diagnosis of depression may also be supported by scores on self-report scales and inventories (Self-RIs). Most often used Self-RIs in affective computing research are the various forms of Patient Health Questionnaire (PHQ)-2/8/9, comprised of 2, 8, or 9 items, respectively and Beck's Depression Inventory (BDI). The Depression and Somatic Symptoms Scale (DSSS) was also used in one study. Self-RIs are convenient and economical, with reported sensitivity and specificity approaching 80-90% (e.g., PHQ-9 [START_REF] Williams | Performance of the PHQ-9 as a Screening Tool for Depression After Stroke[END_REF]), but bear certain disadvantages. Importantly, they do not take into account the clinical significance of reported symptoms, and do not permit adjustments for individual trait characteristics, other psychiatric and medical comorbidities, and potentially important life events, as opposed to a clinical interview [START_REF] Pichot | New Results in Depression Research, chapter Self-report inventories in the study of depression[END_REF]. Additionally Self-RIs are vulnerable to intentional (such as norm defiance) or unintentional reporting bias (e.g., subjective, central tendency [i.e., avoiding extreme responses], social desirability, and acquiescence) [START_REF] Ben-Porath | Handbook of Psychology, chapter Assessing Personality and Psychopathology With Self-Report Inventories[END_REF]. In summary, although Self-RIs alone are insufficient to support the diagnosis of depression [171] [193], they are widely used for screening purposes in various settings, including primary health care. While the cost-effectiveness of widespread screening practices for improving the quality of depression care is debated [START_REF] Mitchell | Screening for depression in clinical practice: an evidence-based guide[END_REF], practical issues related

INTRODUCTION

to the aforementioned limitations of Self-RIs raise questions regarding the overall utility and effectiveness of this practice for population-based mental health.

Nonverbal Signs of Depression

It is well known that depression manifests through a variety of nonverbal signs [76] [222].

Involuntary changes in the tonic activity of facial muscles, as well as changes in peripheral blood pressure and skin electrodermal response, often mirror the frequent and persistent negative thoughts and feelings of sadness that characterize depression. Preliminary findings suggest that electroencephalographic recordings may contain features related to depression [START_REF] Hosseinifard | Classifying depression patients and normal subjects using machine learning techniques and nonlinear features from EEG signal[END_REF]. Functional Near-Infrared Spectroscopy has also attracted interest [19] [199]. Additionally, speech conveys non-verbal information on the mental state of the speaker; prosodic, source, and acoustic features, as well as vocal tract dynamics, are speech-related features affected by depression [START_REF] Cummins | A Review of Depression and Suicide Risk Assessment Using Speech Analysis[END_REF]. Furthermore, depression as a mood disorder, is portrayed on the individual's appearance, in terms of facial expression, as well as body posture [76] [222]. Face as a whole, and individual facial features, such as eyes, eyebrows or mouth, are of particular interest when it comes to depression assessment.

Some of the visual signs identified in the literature are briefly described in the paragraphs that follow.

Specific facial expressions, have been widely examined for depression assessment, in terms of frequency of occurrence, variability, and intensity of a specific expression.

Typically, the facial expression classification system proposed by Ekman [START_REF] Ekman | Handbook of Cognition and Emotion, chapter Basic Emotions[END_REF] is employed, which includes a set of six basic emotions (joy, surprise, anger, sadness, disgust, fear).

Measuring the frequency of occurrence of each of the six emotional expressions [START_REF] Zhou | Tackling Mental Health by Integrating Unobtrusive Multimodal Sensing[END_REF] [84] [195] [176] relies on the premise that depressed individuals tend to show reduced expressivity [START_REF] Ellgring | Non-verbal Communication in Depression[END_REF]. Other studies focused on specific facial features, such as the eyes and mouth. These include gaze direction [START_REF] Scherer | Audiovisual Behavior Descriptors for Depression Assessment[END_REF] [178], reduced eye contact [START_REF] Lucas | Towards an affective interface for assessment of psychological distress[END_REF], eyelid activity [START_REF] Alghowinem | Eye Movement Analysis for Depression Detection[END_REF], eye openings/blinking [START_REF] Zhou | Tackling Mental Health by Integrating Unobtrusive Multimodal Sensing[END_REF] [96] and iris movement [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF]. Smile intensity [START_REF] Scherer | Audiovisual Behavior Descriptors for Depression Assessment[END_REF] [178], smile duration [START_REF] Scherer | Audiovisual Behavior Descriptors for Depression Assessment[END_REF] [178], mouth animation [START_REF] Gupta | Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions Categories and Subject Descriptors[END_REF], listening smiles (smiles while not speaking) [START_REF] Gratch | The Distress Analysis Interview Corpus of Human and Computer Interviews[END_REF], and lack of smiles [START_REF] Lucas | Towards an affective interface for assessment of psychological distress[END_REF] also constitute potentially useful facial signs for assessing depression.

Head pose, orientation, and movement have been used extensively for depression assessment [START_REF] Zhou | Tackling Mental Health by Integrating Unobtrusive Multimodal Sensing[END_REF] [84] [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: the Effect of Gender[END_REF] [90] [START_REF] Scherer | Automatic Behavior Descriptors for Psychological Disorder Analysis[END_REF] [239] [START_REF] Gupta | Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions Categories and Subject Descriptors[END_REF] [23] [START_REF] Joshi | Can Body Expressions Contribute to Automatic Depression Analysis[END_REF], along with motor variability and general facial animation [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: the Effect of Gender[END_REF] [176] [START_REF] Gupta | Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions Categories and Subject Descriptors[END_REF]. Another visual sign that has drawn considerable

Depression Assessment

attention by clinicians in relation to depression assessment is pupil dilation. Siegle et al. [START_REF] Siegle | Remission Prognosis for Cognitive Therapy for Recurrent Depression Using the Pupil: Utility and Neural Correlates[END_REF] reported faster pupillary responses in non-depressed individuals to positive as compared to negative stimuli. More recently Price et al. [START_REF] Price | From Anxious Youth to REFERENCES Depressed Adolescents: Prospective Prediction of 2-Year Depression Symptoms via Attentional Bias Measures[END_REF] investigated attentional bias, including pupil bias and diameter, to predict depression symptoms over a two year follow up period in a sample of adolescents displaying high ratings of anxiety.

Additionally, body gestures [114] [116] involving the entire body, upper body, or separate body parts can also contribute to the assessment. Finally, shaking and/or fidgeting behavior, self-adaptors, and foot tapping [START_REF] Gratch | The Distress Analysis Interview Corpus of Human and Computer Interviews[END_REF] have also been considered as signs of depression. Table 1.1, taken from Pampouchidou et al. [START_REF] Pampouchidou | Automatic assessment of depression based on visual cues: A systematic review[END_REF], is summarizing signs and signals related to depression assessment as found in the literature to date.

Research Motivation

Recent classification schemes (e.g., DSM-5) run the risk of confusing normal sadness (e.g., bereavement) with depression, raising the likelihood of false positive diagnoses [START_REF] Wakefield | Sadness or Depression? International Perspectives on the Depression Epidemic and Its Meaning, chapter Introduction: Depression, One and Many[END_REF]. Depression assessment is a complex process and diagnosis is associated with a significant degree of uncertainty, given the lack of objective boundaries, and the need to evaluate symptoms within the person's current psychosocial context and past history [START_REF] Schumann | Physicians' attitudes, diagnostic process and barriers regarding depression diagnosis in primary care: a systematic review of qualitative studies[END_REF]. Diagnostic accuracy typically improves when results from successive clinical assessments, performed over several months, are taken into account [START_REF] Mitchell | Clinical Diagnosis of Depression in Primary Care: a Meta-analysis[END_REF]. Importantly, a simple "symptom checklist" approach is severely limited and diagnosis requires considerable time investment in order to develop rapport with the patient [START_REF] Thomas-Maclean | Diagnosing Depression: There is no Blood Test[END_REF]. The validity and clinical significance of strict classification schemes has also been questioned [START_REF] Markon | The reliability and validity of discrete and continuous measures of psychopathology: a quantitative review[END_REF].

For instance, MDD has been questioned as a "homogeneous categorical entity" [START_REF] Goldberg | Sadness or Depression? International Perspectives on the Depression Epidemic and Its Meaning, chapter The Current Status of the Diagnosis of Depression[END_REF] and the notion of a "continuum of depressive disorders" is often advocated [START_REF] Maj | Sadness or Depression? International Perspectives on the Depression Epidemic and Its Meaning, chapter The Continuum of Depressive States in the Population and the Differential Diagnosis Between "Normal[END_REF]. These reasonable concerns go beyond the scope of the present work, given that currently affective computing research relies heavily upon established clinical practice tools and procedures.

Objective measures of psychoemotional state, which are implicitly desirable in clinical and research applications alike [START_REF] Hollis | Technological innovations in mental healthcare: harnessing the digital revolution[END_REF] [88], could complement Self-RIs and help overcome some of their shortcomings. Certain Self-RIs are sufficiently brief and can be completed on a regular basis (e.g., monthly or weekly) as part of electronic platforms designed to support long-term monitoring of persons at risk. As suggested by Girard and Cohn [START_REF] Girard | Automated Audiovisual Depression Analysis[END_REF],

technological advances in the field have paved the way for viable automated methods for measuring signs of depression with multiple potential clinical applications. Thus, decision support systems capable of capturing and interpreting nonverbal depressionrelated cues, combined with verbal reports (Self-RIs), could be valuable in both clinical and research applications. In principle, such measures may reduce or even eliminate report bias. In addition, such measures are minimally invasive and do not require extra effort on the part of the respondent, thus likely to increase long-term compliance.

Apart from the high prevalence of MDD, and the complexity of the diagnosis, an additional motivation to pursue the development of a methodology for such a decision support system, is the high number of underdiagnosed depressive episodes. In the overall 85% of depressed individuals are underdiagnosed [START_REF] Falagas | Under-diagnosis of common chronic diseases: prevalence and impact on human health[END_REF]. Relevant research [START_REF] Arbabzadeh-Bouchez | A European perspective on depression in the community: the DEPRES study[END_REF] also showed that about 30% of patients suffering from an episode of major depression do not

Research Motivation

seek treatment, with eventually only 10% of them being adequately treated. In addition, applying technological innovations could enhance accessibility to mental healthcare by overcoming traditional barriers [START_REF] Comer | Introduction to the special series: Applying new technologies to extend the scope and accessibility of mental health care[END_REF]. Current technological means can provide the infrastructure for monitoring psychoemotional state in high-risk individuals as part of early detection and/or relapse prevention programs. A system devoted to the assessment of depressive symptomatology based on visual cues could provide reliable indices, partly based on facial expression analysis, in an unobtrusive manner.

Furthermore, the widespread and relatively low-cost accessibility to computer and internet technologies, webcams, and smart phones, renders an efficient system for depression assessment viable. Practical issues involved in developing such a system, like the storage of sensitive data, could become an issue if not handled properly, but there are ways to tackle such challenges; encryption, protection by password, or even an authorization procedure could be implemented to regulate access to sensitive personal data.

In parallel, a great number of Web-based tools for depression management have been developed and used clinically displaying a high degree of acceptance and patient adherence [START_REF] Brown | Effectiveness of Web-Delivered Acceptance and Commitment Therapy in Relation to Mental Health and Well-Being: A Systematic Review and Meta-Analysis[END_REF]. Early attempts for internet-based interventions for prevention and treatment of depression have shown promising outcomes [START_REF] Martínez | Internet-based interventions for the prevention and treatment of depression in people living in developing countries: A systematic review[END_REF], as well as mobile-based interventions that help in reducing relevant symptomatology [START_REF] Josephine | Internet-and mobile-based depression interventions for people with diagnosed depression: a systematic review and meta-analysis[END_REF]. In the overall telepsychiatry promotes patient-centered care [START_REF] Hilty | An update on telepsychiatry and how it can leverage collaborative, stepped, and integrated services to primary care[END_REF].

Currently, video-based systems for depression assessment have only been found in research-related projects, and have not been applied in the general population to evaluate their feasibility. Although currently limited to research applications, the field has been very popular, with a dedicated section within the Audio/Visual Emotion Challenge (AVEC). AVEC'13 had three papers accepted for the Depression Recognition Subchallenge (DSC) [START_REF] Valstar | AVEC 2013: The Continuous Audio/Visual Emotion and Depression Recognition Challenge[END_REF] [208], while AVEC'14 [START_REF] Valstar | AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge[END_REF] respectively attracted 44 submissions by 13 teams worldwide; the AVEC'16 attracted submissions from 7 teams for the DSC [START_REF] Valstar | Summary for AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge[END_REF], while the latest DSC in terms of AVEC'17 attracted submissions by 10 teams.

Besides from being an active field drawing broad interest, AVEC submissions document the sheer number of research groups working towards the development of such methods.

This fact implies that the idea of developing automated depression assessment methods is not only promising, but is continuously progressing towards more robust and reliable measures.

INTRODUCTION

However, given the current state of the art, video-based systems for depression assessment are not intended as standalone tools, but mainly to function as a decision support systems assisting mental health professionals in the monitoring of persons at risk. "Behaviormedics" is the term Valstar [START_REF] Valstar | Automatic Behaviour Understanding in Medicine[END_REF] introduced for applications designed for the automatic analysis of affective and social signals, among others, to support objective diagnosis. Finally, the development of tools to assist clinicians in the diagnosis and monitoring response to treatment and illness progression is gradually being supported by clinical studies [START_REF] Clark | Mental Health Practice in a Digital World: A Clinicians Guide, chapter Technology Tools Supportive of DSM-5: An Overview[END_REF]. For the purpose of the present thesis, the term "depression assessment" will refer to the process of detecting and assessing the severity of signs of and/or presence of depression.

Thesis Outline

The current thesis is organized in six Chapters. Chapter 2 exhaustively reviews the stateof-the-art, presenting relevant methods employed in the literature, as well as a metaanalysis of existing approaches. Methodology developed in terms of this PhD is described in Chapter 3, with preliminary evaluation of the different methods being presented in Chapter 2

Literature Review

The present chapter is a systematic review of existing methods for automatic detection and/or severity assessment of depression. In the current exhaustive review of more than

Review Design

The technical report entitled "Procedures for Performing Systematic Reviews" by Kitchenham [START_REF] Kitchenham | Procedures for performing systematic reviews[END_REF] was used as a guide for the present review. The keywords used to search electronic databases and related resources are listed in Table 2.1. Inclusion criteria for the review involved a) adequate description of an algorithm for automatic depression assessment utilizing visual cues and b) presentation of systematically derived data, producing concrete results. Fig. 2.1 illustrates the rapid increase of relevant studies during the past few years proving that automatic depression assessment based on visual cues is a rapidly growing research domain. The small drop in 2015 can be attributed to the

LITERATURE REVIEW

Depression Datasets

The availability of empirical data is of paramount importance for the evaluation of methods for automatic assessment of depression. Such data are critical during algorithm development and testing. Due to the sensitive nature of clinical data, availability is neither wide nor free, and this is the reason that most research groups resort to generating their own data sets. This section describes procedures used for data collection and derived datasets found in the reviewed studies (c.f. Table 2.2).

Depression Datasets

Data Collection Procedure

Recruitment of participants is perhaps the most challenging step in this line of research.

Patients with MDD can be recruited from the community, in many cases by clinical psychologists or social workers, assessed using DSM-IV [START_REF]Diagnostic and Statistical Manual of Mental Disorders -4th edition (DSM-IV T M )[END_REF] criteria [185] [186] [219] [START_REF] Cohn | Detecting Depression from Facial Actions and Vocal Prosody[END_REF] [90] [START_REF] Joshi | Can Body Expressions Contribute to Automatic Depression Analysis[END_REF] and/or HAM-D scores [START_REF] Cohn | Detecting Depression from Facial Actions and Vocal Prosody[END_REF] [89] [118] [116]; patients may be medicated, unmedicated, or in remission. The Mini International Neuropsychiatric Interview (MINI)

was employed in the data collection for the dataset reported in [START_REF] Alghowinem | Multimodal Depression Detection:Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors[END_REF] in order to obtain diagnosis, and the Quick Inventory of Depressive Symptomatology-Self Report (QIDS-SR)

for defining the symptom severity. BDI has also been used in [START_REF] Siegle | Remission Prognosis for Cognitive Therapy for Recurrent Depression Using the Pupil: Utility and Neural Correlates[END_REF] to establish whether a given patient was in remission. Comparison data were obtained from individuals who had never been diagnosed with depression or other mood disorder. Data collection from non-clinical samples, employed Self-RIs such as PHQ-9 [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences[END_REF] [84] [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: the Effect of Gender[END_REF] [176] [START_REF] Scherer | Audiovisual Behavior Descriptors for Depression Assessment[END_REF] [93],

and BDI [START_REF] Valstar | AVEC 2013: The Continuous Audio/Visual Emotion and Depression Recognition Challenge[END_REF] [210], assessing the severity of (sub-clinical) depression-related symptomatology. Recruitment methods further included flyers, posters, institutional mailing lists, social networks, and personal contacts.

Establishing conditions which enable the collection of signs related to depression is by far the most important step, as also discussed in [START_REF] Cummins | A Review of Depression and Suicide Risk Assessment Using Speech Analysis[END_REF]. Emotion elicitation is The amount of visual data that is necessary for a reliable assessment depends heavily upon the temporal nature of MDD. The specificity of the assessment method may benefit from multiple recording sessions, such as that of the data reported in [START_REF] Cohn | Detecting Depression from Facial Actions and Vocal Prosody[END_REF] [90] [START_REF] Girard | Social Risk and Depression: Evidence from Manual and Automatic Facial Expression Analysis[END_REF].

Recording length depends on the elicitation method, with structured interviews being considerably longer in comparison with recordings based on emotion elicitation through films.

Studies vary widely depending on the types of equipment utilized and particular signs monitored. For instance, studies focusing on the pupillary response may only use a pupilometer [START_REF] Siegle | Remission Prognosis for Cognitive Therapy for Recurrent Depression Using the Pupil: Utility and Neural Correlates[END_REF] [186] [START_REF] Jones | Motivational and Emotional Influences on Cognitive Control in Depression: A Pupillometry Study[END_REF] [219] and pay special attention to ambient illumination in order to optimize sensitivity. Again, depending on the approach, one or more cameras, typically color, are simultaneously used to cover more than one viewing angles and fields of view (e.g., both face and body separately [START_REF] Cohn | Detecting Depression from Facial Actions and Vocal Prosody[END_REF]). Depth sensors (e.g., Microsoft Kinect)

have also been utilized in some cases [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences[END_REF] [177].

Reported Datasets

The various datasets reported in relevant work are summarized in

LITERATURE REVIEW

State-of-the-Art Methods

The generic processing flow of an automatic system for depression assessment, combining the standard structure for automated audiovisual behavior analysis proposed by Girard and Cohn [START_REF] Girard | Automated Audiovisual Depression Analysis[END_REF], with fusion methods presented in Alghowinem's thesis [START_REF] Alghowinem | Multimodal Analysis of Verbal and Nonverbal Behaviour on the Example of Clinical Depression[END_REF], is illustrated in Fig.

LITERATURE REVIEW

Preprocessing

Given a visual input (video), illumination normalization, registration and alignment between the image sequences, and face detection are typical required preprocessing steps.

Other types of signals, such as speech or physiological recordings, may also need preprocessing, such as segmentation. The most popular algorithm for face detection has been proposed by Viola and Jones [START_REF] Viola | Robust Real-Time Face Detection[END_REF]. Some off-the-shelf facial expression analysis applications have also been used widely as preprocessing tools, enabling researchers to focus on deriving high level information. An example of such a tool is the OpenFace freeware application 4 [31]. The Computer Expression Recognition Toolbox (CERT) [START_REF] Littlewort | The computer expression recognition toolbox (CERT)[END_REF] has been quite popular, but has now become commercialized. Z-Face [START_REF] Jeni | Dense 3D Face Alignment From 2D Videos in Real-time[END_REF] has also been employed for alignment and landmark detection.

Feature Extraction / Manipulation

This Subsection describes processes involved in feature extraction, dimensionality reduction, and fusion. The output of this processing stage generates the input to the machine learning stage, where no further manipulation of features is taking place.

Feature Extraction

Feature extraction is an important step in the processing workflow, since subsequent steps entirely depend on it. The approaches reviewed employ a wide range of feature extraction algorithms which, according to the well-established taxonomy in [START_REF] Corneanu | Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications[END_REF], can be classified as a) geometry-based, or b) appearance-based. In the field of depression assessment, several features are derived from the time-series of both (a) and (b) in the form of dynamic features. Close inspection of depression manifestations, listed in Deep learning, a subfield in machine learning, has become quite popular during recent years, presenting winning approaches in many contests [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF]. Deep learning methods can be employed all the way from the beginning of the pipeline, incorporating feature extraction and machine learning level, or they can be utilized in a transfer-learning manner as pre-trained networks just for feature extraction, combined with other machine learning algorithms. DCNN, DNN, SDA, DTL, ResNet, VGG, and Alex-Net are some of the networks that can be found in the relevant literature.

Facial Landmarks Facial landmarks have been very popular in addressing problems related to facial expression analysis, and have been applied to depression assessment. Such algorithms localize fiducial points of the face and facial features, which are very useful in extracting high level traits directly associated with signs of depression, e.g. smiling. The Constraint Local Models method (CLM) introduced by Saragih et al. [START_REF] Saragih | Face Alignment Through Subspace Constrained Mean-shifts[END_REF] is displayed in Fig. 2.3 to illustrate its application to the modeling of facial geometry.

Active Shape Models (ASM) as well as Active Appearance Models (AAM) have also been utilized for depression assessment methods. Facial landmark data have also been analyzed as time series. Displacement, velocity, acceleration, as well as the landmark coordinates alone, have been used as features. In addition, polynomial fitting, as well as statistics derived from shape eigenvector velocities have been utilized. Landmark Motion History Images (LMHI) is a low level feature which, instead of the actual intensities, Action Units Action Units (AU) encode the coordinated activity of groups of facial muscles in correspondence to specific actions, including specific emotional expressions.

State-of-the-Art Methods

They can be employed for measuring the Variability of Facial Expression (VFE), as depressed individuals tend to be less expressive. Other approaches apply AU as high-level features. AU occurrence by itself is meaningful as there are specific facial actions that are directly linked to the presence of depression (reduced smiling, mouth corners angled down, etc.). Additionally, although several approaches implement AU dynamically (e.g. duration, base rate, ratio of onset/offset), AU are essentially static signs.

Mouth & Eyes

Apart from the face as a whole, features extracted separately from the mouth and eyes have also been found in the reviewed literature. Smile intensity and duration is a mouth-based feature which has been employed for automatic depression assessment, consistent with the clinical literature, as depressed individuals tend to smile less often. Mouth deformations, velocity and acceleration of horizontal and vertical movements have also been proposed. For the eye region, average vertical gaze, blinking 2. LITERATURE REVIEW rate, and pupil dilation have been reported. Additionally, functionals from velocity and acceleration of horizontal and vertical eyelid movement have been used. Additional features include saccade latency, peak velocity of initial saccade, saccade duration, mean and standard deviation (SD) of intersaccadic intervals.

Body

Although body signs in general have been shown to convey manifestations of depression, few approaches have exploited their utility. Existing applications can be classified as relying on either upper body or relative body part movements. Features for upper body movements have been extracted through the STIP and DCS algorithms. Relative body part movements, on the other hand, have been exploited via the parts algorithm that represents orientation and distance from the torso center expressed in polar coordinates.

Dimensionality Reduction

Many feature extraction algorithms produce vectors of high dimensionality. The goal of dimensionality reduction algorithms is to reduce the number of features in a meaningful manner, in order to avoid corrupting the classifier. Dimensionality reduction algorithms can be classified in two groups: (a) Feature Transformation, and (b) Feature Selection [START_REF] Lemaître | Computer-Aided Detection and Diagnosis for Prostate Cancer Based on Mono and Multi-Parametric MRI: A Rreview[END_REF]. In the first group features are transformed/combined by being projected from a high-dimensional space to low-dimensional space, to increase separability. On the other hand, in the second group, as the name implies, a selection procedure takes place, and the most discriminative/significant features are selected. Below, examples from both groups, as retrieved in reported approaches, are being described.

Feature Transformation Principal Components Analysis (PCA) is the most popular algorithm in this category [53] [225] [108] [182] [189] [107] [126] [100] [221] [162] [163] [141] [119] and has been used

to generate new features based on a linear transformation of the original features. Another set of approaches for reducing dimensionality involves codebooks. Bag-of-Words (BoW) [START_REF] Joshi | An Automated Framework for Depression Analysis[END_REF] [118] [START_REF] Joshi | Neural-net Classification for Spatio-temporal Descriptor Based Depression Analysis[END_REF] [59] [117] [121] [36] [START_REF] Bhatia | A Video-Based Facial Behaviour Analysis Approach to Melancholia[END_REF] [100] [START_REF] Dang | Investigating Word Affect Features and Fusion of Probabilistic Predictions Incorporating Uncertainty in AVEC[END_REF], initially intended for document classification, has been applied to image processing problems. A histogram-based approach was presented in [START_REF] Jan | Automatic Depression Scale Prediction using Facial Expression Dynamics and Regression[END_REF] which entails maintaining the highest-scoring bins based on a predefined threshold adjusted to the total samples size.

State-of-the-Art Methods

Visual Features

Face

Full Face -Variance of basic expressions [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences[END_REF][84] [195][176][239] -Neutral expression [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences[END_REF][84] [195][176] [239] -Coordinates difference from 1st to last frame [START_REF] Espinoza | Fusing Affective Dimensions and Audio-Visual Features from Segmented Video for Depression Recognition[END_REF] -Average min max coordinates and velocity [START_REF] Espinoza | Fusing Affective Dimensions and Audio-Visual Features from Segmented Video for Depression Recognition[END_REF] -Functionals from velocity and acceleration of roll pitch and yaw [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF][23] [START_REF] Alghowinem | Multimodal Depression Detection:Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors[END_REF][70] -Eigenvector velocities [START_REF] Cohn | Detecting Depression from Facial Actions and Vocal Prosody[END_REF] -Angular amplitude and velocity [START_REF] Girard | Nonverbal Social Withdrawal in Depression: Evidence from Manual and Automatic Analyses[END_REF] -Average pixel difference of two successive frames with its variance and quantiles [START_REF] Kächele | Inferring Depression and Affect from Application Dependent Meta Knowledge[END_REF] -Rotation angles' frequencies [114] [118] -Average pitch roll and yaw [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences[END_REF][84] [195][176][239] [START_REF] Scherer | Audiovisual Behavior Descriptors for Depression Assessment[END_REF][178] -Head motion from feature points time-series [START_REF] Zhou | Tackling Mental Health by Integrating Unobtrusive Multimodal Sensing[END_REF] -Multi-scale entropy on eigenvalues [START_REF] Harati | Discriminating Clinical Phases of Recovery from Major Depressive Disorder Using the Dynamics of Facial Expression[END_REF] -Mean/median/S.D. of appearance eigenvectors' velocities [START_REF] Smailis | Fusing Active Orientation Models and Mid-term Audio Features for Automatic Depression Estimation[END_REF] -Gabor wavelet [START_REF] Girard | Nonverbal Social Withdrawal in Depression: Evidence from Manual and Automatic Analyses[END_REF] -Mean distance of upper to lower landmarks of the eyelids [START_REF] Gupta | Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions Categories and Subject Descriptors[END_REF] -Mean squared distance of all mouth landmarks to mouth centroid [START_REF] Gupta | Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions Categories and Subject Descriptors[END_REF] -Mean/median/SD of shape eigenvectors' velocities [START_REF] Smailis | Fusing Active Orientation Models and Mid-term Audio Features for Automatic Depression Estimation[END_REF] -LMHI [START_REF] Pampouchidou | Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text[END_REF][162] -LMM [START_REF] Pampouchidou | Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text[END_REF] -Polynomial fitting [START_REF] Nasir | Multimodal and Multiresolution Depression Detection from Speech and Facial Landmark Features[END_REF] -Velocity of distance and area features [START_REF] Nasir | Multimodal and Multiresolution Depression Detection from Speech and Facial Landmark Features[END_REF] -Eccentricity [START_REF] Pampouchidou | Détection de la dépression par lŠanalyse de la géométrie faciale et de la parole[END_REF][214] -HDR [START_REF] Yang | Multimodal measurement of depression using deep learning models[END_REF] AUs -Region Units [START_REF] Mcintyre | An Approach for Automatically Measuring Facial Activity in Depressed Subjects[END_REF] -Likelihood [START_REF] Hamm | Automated Facial Action Coding System for Dynamic Analysis of Facial Expressions in Neuropsychiatric Disorders[END_REF] -Intensity [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences[END_REF]

[89] [108][137] -Eigenfaces [157] [156] -Fisherfaces [157] [156] -LBP-TOP [96][114][115][117][107] [36][37] -LGBP-TOP [96][184][182][189][126] -1-D MHH [108] -BW-LBP-TOP [65] -LCBP-TOP [159] -LCBP-POP [160] -LPQ [120][126][125][162][141] -HOG [231][162][141][123][136] -LPQ-TOP [223] -DCS [100] -MHH [148] [167] -LBP [96][161][162][141][123] -MHI [162][141] -GMHI [162][141] -MHI+LMHI+GMHI [174] -FDHH [109] -MBH [136] -VGG [162][109] -AlexNet [174][109] -ResNet [238] -CNN [48] -STIP [100][118][59][

LITERATURE REVIEW

Feature Selection

In [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF], [START_REF] Alghowinem | Multimodal Depression Detection:Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors[END_REF], and [START_REF] Ghosh | A Multimodal Context-based Approach for Distress Assessment[END_REF] distributional statistics (e.g. t-tests) were employed to select only those features that met a predefined statistical threshold. In [START_REF] Dibeklioglu | Multimodal Detection of Depression in Clinical Interviews[END_REF] [71] [START_REF] Kacem | Detecting depression severity by interpretable representations of motion dynamics[END_REF] the authors implemented the minimum Redundancy Maximum Relevance (mRMR) feature selection, which considers statistical dependency between features. Several feature selection approaches were evaluated in [START_REF] Gupta | Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions Categories and Subject Descriptors[END_REF]: supervised feature selection, brute-force selection, and a backward selection scheme using bivariate correlations. Greedy forward feature selection [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences[END_REF] [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: the Effect of Gender[END_REF], relief from WEKA 1 [START_REF] Espinoza | Fusing Affective Dimensions and Audio-Visual Features from Segmented Video for Depression Recognition[END_REF], and Mutual Information Maximization (MIM) [START_REF] Nasir | Multimodal and Multiresolution Depression Detection from Speech and Facial Landmark Features[END_REF], are additional algorithms used for feature selection in the reviewed studies.

Correlation-based Feature Selection (CFS) has also been employed in some studies [START_REF] Li | Classification study on eye movement data: Towards a new approach in depression detection[END_REF] [92].

Machine Learning

The next step following feature extraction and manipulation, in all methods reviewed, is the machine learning stage. Depending on the particular research goals, different types of decision methods may be applied. Classification methods are appropriate to address categorical questions (e.g., "depressed" versus "non-depressed" and low versus high depression severity). When the research question concerns the concurrent prediction of depression severity through video-derived indices in a continuous manner, regression approaches are predominantly employed. Cross validation methods are typically applied before classification / regression step.

Cross Validation Methods

Cross validation methods are employed to establish algorithm reliability, namely its capacity to generalize well with newly introduced data. To establish reliability, a given data set is divided in two parts, one used to train the proposed algorithm, and another (left-out) to test its performance. Specific procedures used for dataset splitting include the leave-one-out [START_REF] Cohn | Detecting Depression from Facial Actions and Vocal Prosody[END_REF] Gaussian Mixture Models (GMM) [START_REF] Alghowinem | Eye Movement Analysis for Depression Detection[END_REF] [23] [START_REF] Maddage | Videobased Detection of the Clinical Depression in Adolescents[END_REF] Naïve Bayes (NB) [194] [195] [START_REF] Li | Classification study on eye movement data: Towards a new approach in depression detection[END_REF] Random Forest (RF) [START_REF] Yang | Decision Tree Based Depression Classification from Audio Video and Language Information[END_REF] [161] [START_REF] Li | Classification study on eye movement data: Towards a new approach in depression detection[END_REF] Logistic Regression (LogR) [START_REF] Dibeklioglu | Multimodal Detection of Depression in Clinical Interviews[END_REF] [133] [START_REF] Ma | Cost-sensitive two-stage depression prediction using dynamic visual clues[END_REF] Neural Networks (NNet) [START_REF] Joshi | Can Body Expressions Contribute to Automatic Depression Analysis[END_REF] [START_REF] Joshi | Neural-net Classification for Spatio-temporal Descriptor Based Depression Analysis[END_REF] Maximum Entropy Model (MaxEnt) [START_REF] Ghosh | A Multimodal Context-based Approach for Distress Assessment[END_REF] [START_REF] Zhou | Multimodal Prediction of Psychological Disorders: Learning Verbal and Nonverbal Commonalities in Adjacency Pairs[END_REF] Relevance Vector Machines (RVM) [START_REF] Senoussaoui | Model Fusion for Multimodal Depression Classification and Level Detection[END_REF] [START_REF] Dang | Investigating Word Affect Features and Fusion of Probabilistic Predictions Incorporating Uncertainty in AVEC[END_REF] Hidden Conditional Random Fields (HCRF) [START_REF] Zhou | Multimodal Prediction of Psychological Disorders: Learning Verbal and Nonverbal Commonalities in Adjacency Pairs[END_REF] Hidden Markov Model (HMM) [START_REF] Yang | Coupled HMM-based Multimodal Fusion for Mood Disorder Detection Through Elicited Audio-Visual Signals[END_REF] Coupled Hidden Markov Model (CHMM) [START_REF] Yang | Coupled HMM-based Multimodal Fusion for Mood Disorder Detection Through Elicited Audio-Visual Signals[END_REF] Stacked Denoising Autoencoders (SDA) [START_REF] Dibeklioglu | Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding[END_REF] set. Samples could be several for one subject, and therefore the leave-one-out could also be implemented in a leave-one-subject-out manner, where all samples from a specific subject are excluded each time. In the k-fold procedure the dataset is randomly split into k partitions, with one partition kept each time for testing and the remaining used for training the algorithm. This procedure is repeated for k times. In the context of the AVEC challenges, partitioning of the dataset into training and test sections was performed by the organizers, to permit direct comparisons between the algorithms used by participating groups.

Classification

An exhaustive list of classifiers employed in the reviewed studies can be found in Table 2.3 and were ranked in Fig. 2.5. Support Vector Machines (SVM) is by far the most popular method for categorical assessment of depression. This can be justified by the fact that SVMs are well suited for binary problems of high dimensionality [START_REF] Bartlett | Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior[END_REF], such as the distinction of low symptom severity/absent depression from high symptom severity/present depression.

LITERATURE REVIEW

SVM

Regression

The continuous nature of depressive symptomatology is well supported by the clinical literature, as discussed in Subsection 1.2.2. As a result, relevant approaches have recently been gaining momentum, including the AVEC challenges aimed at predicting scores on self-report depression scales as a continuous variable using speech and video cues. As it can be observed in Table 2.4, the most popular regression algorithm, similarly to the classification-based approaches, is Support Vector Regression. Again the ranking of algorithms is illustrated in Fig. 2.6.

Selected Approaches and Meta-Analysis

In this section different approaches, either classification-or regression-based, are compared in a quantitative manner. To be included in the analysis, studies must have reported results on automatic assessment of depression using visual features. Direct comparison is not the case for the various approaches included in this analysis and summarized in Tables 2.5, 2.6, and 2.7, since they were typically evaluated on different datasets (or subsets from the same corpus of data). Even in the case of AVEC participations, direct comparisons across the three challenges are not possible given that different data sets were used in each. The specific objective of our quantitative meta-analysis is

Selected Approaches and Meta-Analysis Table 2.4: Regression algorithms employed in relevant studies

Support Vector Regression (SVR) [96] [59] [120] [121] [182] [231] [223] [226] [92] [200] [63] [69]

Linear Regression (LR) [START_REF] Williamson | Vocal and Facial Biomarkers of Depression Based on Motor Incoordination and Timing[END_REF] [108] [231] [START_REF] Jan | Artificial intelligent system for automatic depression level analysis through visual and vocal expressions[END_REF] Partial Least Square Regression (PLS) [START_REF] Meng | Depression Recognition Based on Dynamic Facial and Vocal Expression Features Using Partial Least Square Regression[END_REF] [108] [START_REF] Syed | Depression severity prediction based on biomarkers of psychomotor retardation[END_REF] [START_REF] Jan | Artificial intelligent system for automatic depression level analysis through visual and vocal expressions[END_REF] Gaussian Staircase Regression (GSR) [START_REF] Williamson | Detecting Depression Using Vocal, Facial and Semantic Communication Cues[END_REF] [105] [START_REF] Dang | Investigating Word Affect Features and Fusion of Probabilistic Predictions Incorporating Uncertainty in AVEC[END_REF] Canonical Correlation Analysis (CCA) [START_REF] Kaya | Ensemble CCA for Continuous Emotion Prediction[END_REF] [START_REF] Kaya | Eyes Whisper Depression: A CCA based Multimodal Approach[END_REF] Relevant Vector Machines (RVM) [START_REF] Senoussaoui | Model Fusion for Multimodal Depression Classification and Level Detection[END_REF] [START_REF] Huang | Staircase Regression in OA RVM, Data Selection and Gender Dependency in AVEC 2016[END_REF] Random Forest Regressor (RFR) [START_REF] Valstar | AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge[END_REF] [START_REF] Ringeval | Avec 2017: Real-life depression, and affect recognition workshop and challenge[END_REF] Deep Convolutional Neural Networks (DCNN) [START_REF] Yang | Decision Tree Based Depression Classification from Audio Video and Language Information[END_REF] [START_REF] Song | Human behaviour-based automatic depression analysis using hand-crafted statistics and deep learned spectral features[END_REF] Deep Convolutional Neural Netowork -Deep Neural Network (DCNN-DNN) [START_REF] Yang | Hybrid depression classification and estimation from audio video and text information[END_REF] [START_REF] Yang | Multimodal measurement of depression using deep learning models[END_REF] Deep Transformation Learning (DTL) [START_REF] Kang | Deep transformation learning for depression diagnosis from facial images[END_REF] [START_REF] Stepanov | Depression severity estimation from multiple modalities[END_REF] Cascade Random Forest (CRF) [START_REF] Sun | A random forest regression method with selected-text feature for depression assessment[END_REF] Ordinary Least Squares (OLS) [START_REF] Ma | Cost-sensitive two-stage depression prediction using dynamic visual clues[END_REF] Discriminative Mapping (DM) [START_REF] Wen | Automated Depression Diagnosis Based on Facial Dynamic Analysis and Sparse Coding[END_REF] Moore-Penrose Generalized Inverse (MPGI) [START_REF] Kaya | Ensemble CCA for Continuous Emotion Prediction[END_REF] Extreme Learning Machines (ELM) [START_REF] Williamson | Vocal and Facial Biomarkers of Depression Based on Motor Incoordination and Timing[END_REF] Long Short Term Memory Recurrent Neural Networks (LSTM-RNN) [START_REF] Chao | Multi task sequence learning for depression scale prediction from video[END_REF] Stochastic Gradient Descent (SGD) [

LITERATURE REVIEW

to identify general trends, key and strong points, to be considered in future studies of automatic depression assessment, given that a direct comparison of results is not viable.

Categorical Depression Assessment

Approaches for categorical depression assessment presented in this subsection are grouped and compared in terms of the employed dataset, in accordance with Table 2.2. Further, the results are considered with regard to the evaluated features, in reference to the taxonomy presented in Fig. 2.4. The various approaches, apart from reporting different performance metrics, were tested on datasets or particular subsets of varying sizes.

Performance metrics in each report are explained next.

Accuracy, which was reported in the majority of studies, is computed according to Equation (2.2) based on the following confusion matrix:

C = T P F N F P T N (2.1)
where T P is the number of true positives, T N the number of true negatives, F P the number of false positives, and F N the number of false negatives. Certain studies report "depressed accuracy", which implies sensitivity (or recall) given by (2.4).

Accuracy = T P + T N T P + T N +

F P + F N (2.2)
Where precision is given by:

precision = T P T P + F P (2.3)
and recall by:

recall = T P T P + F N (2.4)
The F1 score, also reported in several studies, is given by:

F 1 = 2 • precision • recall precision + recall (2.5)
The aforementioned performance metrics, however, fail to take chance agreement into consideration, which varies across different studies. To address this limitation and to permit direct comparisons between classification approaches, Cohen's Kappa statistic

Selected Approaches and Meta-Analysis

[52], a chance and skew robust metric, was computed whenever possible. It is based on the confusion matrix (2.1) and given by:

κ = p 0 -p e 1 -p e (2.6)
where p 0 is the proportion of accurately predicted decisions given by the accuracy formula as defined in (2.2), and p e the proportion of expected chance agreement, given by:

p e = M a + M b T P + F N + F P + T N (2.7)
where M a and M b are defined as follows:

M a = (T P + F N ) * (T P + F P ) T P + F N + F P + T N (2.8) M b = (T N + F P ) * (T N + F N ) T P + F N + F P + T N (2.9)
Whenever confusion matrices for user-/gender-independent depression assessment (based on one or more visual cues) were not included in the original publication, they were requested from the study authors. For the cases that the requested information was not provided, and if at least two performance metrics where reported in the original publication, along with the total number of subjects per class (depressed/non-depressed, as defined in (2.10) and (2.11)), a 4 × 4 linear system of equations was solved in order to derive the confusion matrix.

#depressed = T P + F N (2.10) #not -depressed = T N + F P (2.11)
In the case of [START_REF] Scherer | Automatic Behavior Descriptors for Psychological Disorder Analysis[END_REF] a quadratic system was solved, since the reported metrics were averaged for the two classes (depressed/non-depressed). Finally, the computed confusion matrices were cross-checked to reproduce the originally reported performance metrics. If the estimated confusion matrices for a given study could not be verified by the reported performance metrics, the relevant study was not considered any further. It should be noted that Dibeklioglu et al. [START_REF] Dibeklioglu | Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding[END_REF], Pampouchidou et al. [START_REF] Pampouchidou | Quantitative comparison of motion history image variants for video-based depression assessment[END_REF], and Kacem et al. [START_REF] Kacem | Detecting depression severity by interpretable representations of motion dynamics[END_REF] 2. LITERATURE REVIEW are the only reviewed publications which originally included Kappa statistics in their published report. Table 2.5 groups the reviewed studies according to the dataset used.

Studies are ranked by decreasing κ value within each dataset-specific group. Table 2.6

presents similar information on studies using datasets or dataset combinations reported in single studies, precluding direct across-study comparisons.

Pittsburgh

The first report involving the Pittsburgh dataset is that of Cohn et al. [START_REF] Cohn | Detecting Depression from Facial Actions and Vocal Prosody[END_REF]. Girard et al. [START_REF] Girard | Nonverbal Social Withdrawal in Depression: Evidence from Manual and Automatic Analyses[END_REF] extended this work by investigating the correlation of changes in patient clinical status with corresponding changes in facial expression and head motion patterns [START_REF] Girard | Social Risk and Depression: Evidence from Manual and Automatic Facial Expression Analysis[END_REF].

The cross-cultural study of Alghowinem et al. [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF] was also tested on the Pittsburgh dataset among others, reporting an average recall of 94.7%. Dibeklioglu et al. [START_REF] Dibeklioglu | Multimodal Detection of Depression in Clinical Interviews[END_REF] tested several feature settings on the Pittsburgh dataset. More recently Dibeklioglu et al. [START_REF] Dibeklioglu | Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding[END_REF] presented a deep learning approach for detecting three levels of depression. Finally, Joshi et al. [START_REF] Joshi | Relative Body Parts Movement for Automatic Depression Analysis[END_REF], and Joshi [START_REF] Joshi | An Automated Framework for Depression Analysis[END_REF], reported 97.2% accuracy for assessing depression severity based on the Pittsburgh data.

BlackDog

McIntyre et al. [START_REF] Mcintyre | An Approach for Automatically Measuring Facial Activity in Depressed Subjects[END_REF] were the first to report on the BlackDog dataset. They reported identifying two clusters of patients, those who showed psychomotor agitation and those who showed motor slowing [147] [146]. In a subsequent study, Joshi et al. [START_REF] Joshi | Neural-net Classification for Spatio-temporal Descriptor Based Depression Analysis[END_REF] achieved depression detection accuracy as high as 88.3%. Higher performance, up to 91.7%, was achieved when additional modalities were included (speech, independent and relative movement of body parts) in Joshi et al. [START_REF] Joshi | Can Body Expressions Contribute to Automatic Depression Analysis[END_REF] and Joshi et al. [START_REF] Joshi | Multimodal Assistive Technologies for Depression Diagnosis and Monitoring[END_REF]. Alghowinem et al. studied depression detection based on the analysis of either eye movements in [START_REF] Alghowinem | Eye Movement Analysis for Depression Detection[END_REF],

or head pose and head movements in [START_REF] Alghowinem | Head Pose and Movement Analysis as an Indicator of Depression[END_REF]. The maximum reported recall rate was 80% for the eye-based approach, and 82.6% for the head-based approach among women. In the same cross-cultural study mentioned in subsection 2.4.1.1, Alghowinem et al. [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF] combined the two approaches (eye-based and head-based) achieving an average recall of 85%. Recently, Alghowinem et al. [START_REF] Alghowinem | Multimodal Depression Detection:Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors[END_REF] reported improved recall performance of 88.3%, by extending their approach in terms of feature extraction, as well as machine learning methods.

Selected Approaches and Meta-Analysis

DAIC-WOZ

Several approaches have been tested on this dataset, mainly from the primary research group, but also as part of the AVEC'16 depression sub-challenge. Scherer et al. [START_REF] Scherer | Audiovisual Behavior Descriptors for Depression Assessment[END_REF] examined the value of the audiovisual approach, achieving 89.74% accuracy. Stratou et al.

[194] corroborated Alghowinem's findings [START_REF] Alghowinem | Eye Movement Analysis for Depression Detection[END_REF] [23] on gender differences in classification accuracy, reporting F1 scores of 0.858 for women and 0.808 for men in detecting presence of depression and PTSD. Finally, DAIC-WOZ served as the benchmark dataset in AVEC'17 and AVEC'16. Raw data were provided for audio recordings and transcripts, while for videos only features extracted with OpenFace were provided. Despite this limitation several interesting approaches were presented.

AVEC

The AVEC dataset, although intended for continuous assessment of depression, was also used for categorical assessment using case groupings based on BDI scores. For instance, Senoussaoui et al. [START_REF] Senoussaoui | Model Fusion for Multimodal Depression Classification and Level Detection[END_REF] achieved classification accuracy of 82% for categorical assessment of depression by using a cutoff score of 13/14 points on BDI. In their crosscultural study, Alghowinem et al. [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF] tested their algorithm on a subset of the AVEC data and reported an average recall of 68.8% for the fixed set of features across the three datasets. Pampouchidou et al. [START_REF] Pampouchidou | Quantitative comparison of motion history image variants for video-based depression assessment[END_REF] achieved 89% accuracy, which is the highest reported performance for categorical assessment on the AVEC dataset.

Other Datasets

Datasets or dataset combinations used in a single study are summarized in Table 2.6.

Alghowinem et al. [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF] attempted to merge several datasets (e.g., Pittsburgh and AVEC).

This resulted in an improvement of classification performance as compared to relying solely on the AVEC dataset, reporting an average recall of 85.7%. In one of the earliest studies, the corpus constructed by the Oregon Research Institute (ORI), was used to test the approach of Maddage et al. [START_REF] Maddage | Videobased Detection of the Clinical Depression in Adolescents[END_REF] for video-based depression detection in adolescents.

The ORYGEN dataset was employed for a more challenging endeavor undertaken by Ooi et al. [START_REF] Ooi Kuan Ee | Prediction of Clinical Depression in Adolescents Using Facial Image Analaysis[END_REF], in order to predict whether initially non-depressed adolescents would develop depression at the end of a two-year follow-up period. Zhou et al. [START_REF] Zhou | Tackling Mental Health by Integrating Unobtrusive Multimodal Sensing[END_REF] at the University of Rochester departed from the traditional laboratory settings and obtained

LITERATURE REVIEW

data in realistic conditions. They reported 0.817 precision and 0.739 recall for classifying patients versus healthy volunteers reporting high levels of negative mood. Finally, recent results from the CHI-MEI dataset [START_REF] Yang | Coupled HMM-based Multimodal Fusion for Mood Disorder Detection Through Elicited Audio-Visual Signals[END_REF], attempting to distinguish unipolar depression (MDD) from bipolar disorder, reached 65.38% classification accuracy when combining AUs and audio features.

Continuous Depression Assessment

The majority of the approaches reviewed here are based on AVEC datasets, either as participations to the actual challenge or as independently published studies. The depression challenge of AVEC 2013 and AVEC 2014 required prediction of individual BDI scores based on corresponding video recordings (both visual and speech data are available).

Video recordings were divided into three subsets (training, development and testing), with labels being released only for the first two. AVEC 2013 included the complete recordings of the participants executing 12 tasks, while for AVEC 2014 only two tasks were kept: the Northwind (reading a novel passage) and Freeform (answering a series of questions-both neutral and potentially emotionally challenging). AVEC 2016, although focused on categorical assessment, encouraged participants to address prediction of selfreported scores on the PHQ-8 scale, employed by DAIC-WOZ. Table 2.7 summarizes relevant proposed approaches. In all cases, performance metrics were the Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) given by equations (2.12) and (2.13), where n is the number of samples, p the predicted value, and a the actual value.

RM SE = 1 n n i=1 (p i -a i ) 2
(2.12)

M AE = 1 n n i=1 |p i -a i | (2.13)

Scope of the Thesis

The scope of this thesis is based on the motivation to deliver an unobtrusive depression assessment methodology based on visual cues, as described in Section 1. the audience effect (human-computer interaction setup), eliminating cues that could only occur in a social context.

Relying on self-reported symptoms for data annotation, as in AVEC and DAIC-WOZ, is far more complex in comparison to the other datasets. Self-RIs scores depend on a variety of biasing factors (subjective, social, etc.), and although depression may be accurately portrayed by facial expressions, this type of ground-truth may not accurately measure depression severity. It appears that an optimal dataset should be comprised of a number of patients diagnosed by experienced psychiatrists, using largely uniform diagnostic criteria. Consequently, the development of an algorithmic tool for depression assessment, would most definitely require direct supervision by clinicians.

With respect to video acquisition parameters, average image resolution and frame rate are typically reported. However, it is not clear whether image acquisition with higher-level specifications could improve assessment accuracy. A quantitative comparison of different resolutions and frame rates employing the same algorithm(s) may be required to conclusively address this question.

Scope of the Thesis

Given the temporal variability of depressive manifestations, the majority of facial signs utilized in the reviewed studies are dynamic, while the occurrence of static signs is typically considered over time. Therefore it is clear that video recordings are of interest, as opposed to static images.

Treating depression as a continuous variable (i.e., reflecting depression severity) is gaining ground over categorical decision systems. This may be due to the fact that, despite the apparent simplicity of categorical assessment, it does not represent neither properly nor reliably the complex nature of mood disorders.

Consequently, the scope of this thesis can be summarized as:

• Constructing a clinically valid dataset, comprising diagnosed patients as well as healthy control individuals

• Test different types of stimuli, in order to compare the effect of interpersonal vs non-social context

• Evaluate several annotation methods to test their accuracy; specifically:

1. Diagnosis 2. Self-report

Expert annotation

• Experiment with different video acquisition parameters

• Develop video-based methodology to extract features correlating with signs of depression

• Investigate categorical vs continuous depression assessment

Methodology

The work-flow for the proposed methodology was based on the one presented in Fig. 2.2, which was synthesized based on the literature review. Different methods corresponding to the preprocessing, feature extraction, dimensionality reduction, and machine learning stages are described in the respective sections. The algorithms involved in the different stages are described in some level of detail, while not all of them were implemented together. Later on in the thesis, and more specifically in Ch.4 Preliminary Experimental Evaluation different experiments are described, while the main study is described in Ch.5. Thus the experiment that involves each algorithm described in this Chapter is mentioned accordingly.

Preprocessing

Conditions during data collection are not always conveniently established, and therefore preprocessing can help in obtaining meaningful information for further analysis. The

Illumination Normalization

Illumination normalization is employed in order to establish the appropriate contrast, brightness and other attributes of the images related to light. It is important in order

METHODOLOGY

to make sure that the illumination level is adjusted to a level which can contribute to further analysis. Illumination normalization was employed for the methodology and experimental tests carried in terms of Pampouchidou et al. [START_REF] Pampouchidou | Designing a Framework for Assisting Depression Severity Assessment from Facial Image Analysis[END_REF]. The algorithm utilized for the illumination normalization was the one provided by the INFace Matlab Toolbox1

[197] [START_REF] Štruc | Gabor-based kernel partial-least-squares discrimination features for face recognition[END_REF]. During this procedure the original images (video frames) were first passed through homomorphic filtering to obtain an illumination normalized image, followed by contrast enhancement. Homomorphic filtering is an algorithm employed to improve the image quality by compressing the intensity range while enhancing contrast.

Face Detection

Although

Facial Landmarks Detection

The accurate detection of the facial region was a rather complex albeit important first

METHODOLOGY

Motion Representation

The GNDPM model requires accurate face detection, which was implemented based on the Tanaka face parts detection [START_REF] Tanaka | Face Parts Detection[END_REF], and the detected landmarks were further used for aligning the face regions based on the eyes. GNDPM algorithm is pretrained and minimizes the reconstruction error between the original image and a Generative Parts Model iteratively, with the use of Gauss-Newton optimization.

Motion Representation

The ultimate goal of the proposed thesis was to detect signs of depression based on visual input. Based on the clinical background and the signs reviewed in Table 1.1, it is evident that most of the signs are of temporal nature, thus video based analysis is of focus rather than mere static image processing. This constitutes the motion representation as the most important step of the pipeline, and therefore most of the work has been done within this context.

Local Curvelet Binary Patterns

LBP was initially proposed for extracting texture-based information from a static image, however it was later extended in order to extract spatiotemporal information. Work presented in Zhao and Pietikainen [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF] proposed the Volume Local Binary Pattern the spatial coordinates, and Z the frame index (time). Volume textons are interpreted in histograms, in the same manner as in LBP, for all three planes, as illustrated in Fig.

3.3.

Another notable extension of this approach, was from Almaev and Valstar [START_REF] Almaev | Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition[END_REF],

who considered Gabor filtered image frames rather than the original intensity-based images. Furthermore, inspired by Pampouchidou [START_REF] Pampouchidou | Facial-gesture based interaction with expressive virtual characters[END_REF], where Curvelet transform was used for facial expression recognition, another extension was proposed in the context of this thesis, which considered the Curvelet transformed frames for applying the Three Orthogonal Planes process. Next the Curvelet transform is being explained in short.

Curvelet Transform

Curvelets belong to the family of geometrical wavelets, and were proposed by Candès et al. [START_REF] Candès | Curvelets: A Surprisingly Effective Nonadaptive Representation for Objects with Edges[END_REF]. Their origin can be found in ridgelets [START_REF] Candès | Ridgelets: Theory and Applications[END_REF], and thus it is preliminary to begin with the ridgelet definition. A characteristic ridgelet waveform can be seen in Figure 3.4, while ridgelet transform for an image function f (x, y) is given by:

R f (a, b, θ) = ψ(x, y)f (x, y)dxdy
where a > 0, b ∈ R, and θ are respectively the scale, the translation, and the orientation. ψ is the ridgelet function given by:

ψ a,b,θ (x, y) = a -1 2 ψ( x cos θ + y sin θ -b a ) (3.1)

METHODOLOGY

Just like Gabor, ridgelets have the property of being tuned in different scales and orientations [START_REF] Dyla | Content Based Images Retrieval Based on PDE Model: use of Curvelet Transform and Gabor Wavelets[END_REF]. Accordingly, the digital curvelets transform for an image f (n, m) of dimensions M by N is given by [START_REF] Islam | Rotation invariant curvelet features for texture image retrieval[END_REF]:

C f (a, b, θ) = 0≤m<M 0≤n<N f (m, n)ψ a,b,θ (m, n) (3.2)
Curvelets, that have been characterized as an "optimal representation of objects with piecewise C2 singularities" [START_REF] Candès | New tight frames of curvelets and optimal representations of objects with piecewise C2 singularities[END_REF], can be expressed in frequency domain by [START_REF] Islam | Rotation invariant curvelet features for texture image retrieval[END_REF]:

C a,b,θ = F(F(f (m, n) × F(ψ a,b,θ (m, n))) (3.3)
Zhang et al. in [START_REF] Zhang | Face recognition based on curvefaces[END_REF] introduced the term "curvefaces" in an effort to reduce the dimensionality of the facial image [START_REF] Mandal | Face recognition using curvelet based PCA[END_REF]. Motivation for further exploration of curvelets lies into the fact that the facial expression is characterized by the geometry of the facial features. Numerous examples demonstrate how the geometry of the face gives information about the expression. For instance something that people learn in a young age when drawing: on a happy face the mouth curve is facing up, while on a sad face the opposite happens.

The method introduced by Candès et al. in [START_REF] Candès | Fast Discrete Curvelet Transforms[END_REF] is being followed, according to the available implementation in the Curvelab toolbox1 . As already explained previously, curvelets originate in ridgelets, and therefore inherit and extend their properties. Extensions are such as that any arbitrary function can be expanded as series of curvelets, but also the parabolic scaling of the geometrical wavelet. Parabolic scaling means that the curvelet is elongated more in certain directions, and does not keep the same profile as ridgelets.

Curvelets are also discrete in terms of scale, location, and angle. Scale is doubled with each discretization level; that is at each scale, resolution is doubled as it moves further from the center in frequency domain, as shown in Figure 3.5. Curvelet algorithm in simple steps is given by first computing the 2D Fast Fourier Transform (FFT) of the image, then fixing the scales to be considered by the parabolic scaling in a resulting f .

Finally the curvelet coefficients are obtained by the inverse FFT applied on the product of f and ψ function as given in Equation 3.1. In Fig. 3.6 curvelet coefficients are wrapped

Motion Representation

for display purposes; the coarse level for scale one and original orientation is placed in the center, then moving towards outside, coefficient for different scales and rotations are collocated. Computational complexity of curvelets for an n × n image is O(n 2 log n).

The curveface image computed in this step consists of complex values, therefore before further processing it should be converted to real. The justification applied in order to obtain real values, is to take the Euclidean distance between the real part and the imaginary, given by taking the magnitude:

x -y = (y 1 -x 1 ) 2 + (y 2 -x 2 ) 2 + ... + (y n -x n ) 2 = n i=1 (y i -x i ) 2 (3.4)

Three Orthogonal Planes vs Pairwise Orthogonal Planes

The first integrated methodology implemented in terms of this PhD was the one published in Pampouchidou et al. [START_REF] Pampouchidou | Designing a Framework for Assisting Depression Severity Assessment from Facial Image Analysis[END_REF] and involved the LCBP-TOP. As mentioned previously, the idea of this preliminary version derived from the LGBP-TOP proposed by [START_REF] Almaev | Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition[END_REF], with the difference that in the present Gabor wavelets are replaced with Curvelet transform [START_REF] Candes | Fast discrete curvelet transforms[END_REF].

The norm for approaches based on Orthogonal Planes is to take all descriptors from each of the three planes (XY, XZ, YZ), where Z refers to the time, and concatenate them all together, producing a vast vector of thousands of features. In the work published in Pampouchidou et al. [START_REF] Pampouchidou | Video-Based Depression Detection Using Local Curvelet Binary Patterns in Pairwise Orthogonal Planes[END_REF] an effort was made to reduce this high dimensionality, by introducing the Pairwise Orthogonal Planes. Thus, in [START_REF] Pampouchidou | Video-Based Depression Detection Using Local Curvelet Binary Patterns in Pairwise Orthogonal Planes[END_REF] two approaches were tested: a) Frame-based Classification, which considered only XY planes for each frame separately, and b) Video-based Classification for which XZ and YZ planes are considered in pairs of two. This way, for the Video-based approach the plane corresponding to the first row is combined with the plane corresponding to the first column, second row with second column, (...), and the last row with the last column.

The LCBP-POP modification has the advantage of preserving the motion information in both axis, with a considerably shorter feature vector. A further improvement in compare to the previous approach was that overlapping window was used instead of the previous that employed sequential windowing. In addition, apart from facial expression, curvature contains information on person-specific biometrics (e.g. different shapes of facial features, varying symmetry), as well as occlusions (e.g. facial hair, eye-glasses). Video-based Classification was implemented in order to overcome this limitation. The LBP descriptor was again computed on each plane.

METHODOLOGY

Motion History Image

It is already established that most of the non-verbal signs of depression are dynamic by nature [START_REF] Ellgring | Non-verbal Communication in Depression[END_REF][START_REF] Waxer | Therapist Training in Nonverbal Communication. I: Nonverbal Cues for Depression[END_REF]. Therefore, the use of video-based methods (dynamic), as opposed to frame-based (static) is preferable. In addition to the previously described motion representation methods, three different motion history images were also implemented: The algorithm produces a grayscale image, in which the white pixels correspond to the most recent movements and the darkest gray correspond to the earliest motion elements.

Black pixels indicate absence of movement. It is a popular algorithm for motion analysis

[20], and has been extensively used in the field of human action recognition [START_REF] Bobick | Real-time recognition of activity using temporal templates[END_REF].

An early approach of MHI to facial image analysis was that of Valstar et al. [START_REF] Valstar | Motion history for facial action detection in video[END_REF],

who employed MHI in facial action recognition from videos. Meng et al. [START_REF] Meng | Depression Recognition Based on Dynamic Facial and Vocal Expression Features Using Partial Least Square Regression[END_REF] published a continuous depression assessment approach in their participation to the DSC of AVEC'13; they proposed an extension of MHI, the MHH, which considers patterns of movement. In the DSC of AVEC'14 Pérez Espinoza et al. [START_REF] Espinoza | Fusing Affective Dimensions and Audio-Visual Features from Segmented Video for Depression Recognition[END_REF] employed MHI, and for the same challenge Jan et al. [START_REF] Jan | Automatic Depression Scale Prediction using Facial Expression Dynamics and Regression[END_REF] proposed the 1-D MHH, an extension of MHI, which is computed on the feature vector sequence instead of the intensity image.

Original MHI

The MHI is a gray scale image, where white pixels correspond to the most recent movement in the video, intermediate gray scale values to corresponding less recent movements, and black pixels to the absence of movement. The MHI algorithm, with slight variations as explained next, is applied on the aligned face image sequences derived from the preprocessed data using OpenFace.

The MHI H, with a resolution equal to the one of the aligned faces, is computed based on an update function Ψ(x, y) as follows:

H i (x, y) =        0 i = 1 i • s Ψ i (x, y) = 1 H (i-1) (x, y) otherwise (3.5)
where s = 255/N , N the total number of video frames, (x, y) the position of the corresponding pixel, and i the frame number. Ψ i (x, y) represents the presence of movement, derived from the comparison of consecutive frames, using a threshold ξ:

Ψ i (x, y) = 1 D i (x, y) ≥ ξ 0 otherwise (3.6)
where D i (x, y) is defined as a difference distance:

D i (x, y) = I i (x, y) -I (i-1) (x, y) (3.7) I i (x, y)
is the pixel intensity value in (x, y) at the ith frame. The final MHI is the

H N (x, y).

Landmark Motion History Image

LMHI encodes the motion of the facial landmarks into a grayscale image, with the most recent movement corresponding to white pixels, the earliest corresponding to the darkest gray, and temporally intermediate movements indexed by corresponding gray values.

The extension of the proposed work in comparison to that of Ptucha and Savakis [START_REF] Ptucha | Towards the Usage of Optical Flow Temporal Features for Facial Expression Classification[END_REF] is that the in-between motion is also preserved with the use of respective gray-scales, which is important for the descriptors applied later on the LMHI.

The landmarks considered are the ones that correspond to the facial features (eyes, eyebrows, nose-tip, and mouth), while the face outline is excluded. This step was taken in order to emphasize inner-facial movements, and ignore the overall head movements. This is achieved by co-registering the involved landmarks using affine transformation before computing the LMHI, through alignment of the points corresponding to the temples, chin, inner and outer corners of the eyes (landmarks {1, 9, 17, 37, 40, 43, 46}).

LMHI differs from the conventional MHI in that image intensities are not considered, but only the facial landmarks, which are detected in each frame. The adopted LMHI algorithm is similar to MHI, by maintaining the same H i as in (3.5), and modifying Ψ i as follows:

Ψ i (x, y) = 1 (x, y) ∈ L i 0 otherwise (3.8)
where L i corresponds to the selected landmarks as detected in the ith frame. 52

Motion Representation

Gabor Motion History Image

Gabor filters have been frequently used in both facial expression analysis and emotion recognition [78,[START_REF] Tian | Handbook of Face Recognition, chapter Facial Expression Analysis[END_REF]. In relevant approaches the feature vector is extracted from the convolution of the original image with a 2D Gabor wavelet function at different orientations and wavelengths. This describes the spatial frequency structure around each pixel. In [START_REF] Cruz | Facial emotion recognition with anisotropic inhibited gabor energy histograms[END_REF] the Gabor energy was used for facial emotion recognition, which gives a smoother response to an edge or a line of appropriate width with a local maximum exactly at the edge or in the center of the line. The authors also applied background texture suppression on the response of the filter, by removing an image filtered by the difference of Gaussians (DoG) from the original response for each orientation. This approach, also known as anisotropic inhibition [START_REF] Grigorescu | Contour detection based on nonclassical receptive field inhibition[END_REF], removes noise and provides a sharper representation of facial features.

GMHI is another variant of MHI, where Gabor inhibited images substitute original image intensities. The motivation for implementing this variant is that it focuses on the important details of the facial features, and thus extracts the most relevant information.

The motion representation algorithm is identical to the one described in subsection 3.2.2.1, but the input image I is the result of the Gabor inhibition. The process of obtaining the Gabor inhibited image is explained in detail below.

The Gabor wavelet at position (x, y) is given by:

Ψ λ,θ,φ,σ,γ (x, y) = exp(-x ′ 2 +γ 2 y ′ 2 2σ 2 )cos(2π x ′ λ + φ) (3.9) with x ′ = xcosθ + ysinθ y ′ = -xsinθ + ycosθ (3.10)
where λ stands for the wavelength, θ for the orientation, φ for the phase offset, σ for the standard deviation of the Gaussian, and γ for the spatial aspect ratio [START_REF] Dahmane | Continuous Emotion Recognition Using Gabor Energy Filters[END_REF].

The input image is usually filtered with many wavelets for multiple orientations and wavelengths. The energy filter response is obtained by combining the convolutions obtained from two different phase offsets (φ 0 = 0 and φ 1 = π/2) using the L2-norm. Background texture suppression is applied on the filter response, by removing a Difference of Gaussians (DoG) filtered image from the original response for each orientation [START_REF] Grigorescu | Contour detection based on nonclassical receptive field inhibition[END_REF].

Finally, the mean response of Gabor filtering is used to combine the responses across

Motion Representation

the different orientations, resulting in the pseudo-image used to compute the GMHI.

An example of applying the common Gabor and the Gabor inhibited algorithms to an aligned face image is illustrated in Fig. 3.9, where the Gabor inhibited image appears to be sharper and with less texture in uniform regions than the original Gabor response.

Time-series of Geometrical Features

Geometric features are quite popular in the field of facial expression recognition, while relevant approaches using these features in depression assessment have also been found.

Given that smiling tends to be reduced in individuals suffering from depression [START_REF] Ellgring | Non-verbal Communication in Depression[END_REF], we focused on certain distances between facial landmarks, which are affected by smiling [START_REF] Vonikakis | Group happiness assessment using geometric features and dataset balancing[END_REF]. Landmarks that correspond to the Veraguth fold were also considered, as well as blinks. Right and left eye width, as well as mouth width and height shown in figure 3.10, were also considered in time-series manner for further feature extraction. Taking all these into account we defined the final set of distances, illustrated in Fig. 3.10.a, considered as time-varying parameters. In order to account for changes of the distances due to external movements (not intra-facial), all distances were normalized based on the distance between the side temples (see Fig. 3.10.a landmarks 1 and 17).

An additional video based feature set was subsequently constructed. This feature set is based on landmarks' activity, in terms of displacement, velocity, and acceleration, that were estimated for all 68 landmarks for a specified window of frames. The landmarks were grouped according to facial features, as illustrated in Figure 3.10.b: right eyebrow, left eyebrow, right eye, left eye, mouth, and face as a whole. The values of displacement, velocity, and acceleration, were averaged, resulting in three corresponding time-series for each region. Statistical measures extracted are the same as with the distance time-series.

Distances, both in space and time, were computed based on Euclidean distance:

D = (x 1 -x 2 ) 2 + (y 1 + y 2 ) 2 (3.11)
Distance in time implies considering the coordinates of one landmark at a given time frame and the coordinates of the same landmark for a next time frame; in this manner the displacement of a specific landmark is estimated. While for distance in space the coordinates of two different landmarks are considered in the same time frame.

Appearance-based Descriptors

Appearance descriptors employed hereby exploit intensity and more specifically texture

Local Binary Patterns

LBP were introduced by Ojala et al. in [START_REF] Ojala | A comparative study of texture measures with classification based on featured distributions[END_REF], who extended Wang and He's work [START_REF] Wang | Texture classification using texture spectrum[END_REF] in texture classification. At their first attempt [START_REF] Ojala | A comparative study of texture measures with classification based on featured distributions[END_REF] they obtained gray-scale invariance, while later [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] they also achieved rotation invariance, yet keeping the algorithm computationally simple and efficient.

LBP [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] entails dividing the image into partially overlapping cells. Each pixel of the cell is compared to its neighbors to produce a binary value (pattern). The resulting descriptor is a histogram which represents the occurrence of different patterns. LBP for two sets of {radius, neighbourhood} results to feature vectors of size 1×59 for {1,8} and size 1×243 for {2,16}. The example of the LBP pipeline is illustrated in Fig. 3.11.

When LBP is applied the image is divided into cells of a given size, e.g. 16×16. For each pixel in the cell LBP descriptor is computed according to the example in Fig. 3.12.a as follows. The central Pixel is compared to all of its neighbours, depending on the neighbourhood and radius defined, and each of the neighbours are assigned a binary Ojala et al. in [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] proved that for neighbourhood 8 and radius 1 there exist 36 unique different orientations for the LBP descriptor as illustrated in Fig. 3.12.b, with the first 9 being uniform. Uniform in the case of LBP is interpreted as at most two bitwise transitions. For instance the descriptor "00000000" has zero transitions, "00000111" has just one transition, 00111000 has two transitions, while 01001101 has five transitions. The different implementations were employed based on the current availability of the respective toolboxes.

The first nine descriptors in

Local Phase Quantization

LPQ [START_REF] Ojansivu | Rotation invariant local phase quantization for blur insensitive texture analysis[END_REF] is computed in the frequency domain, based on the Fourier transform, for each pixel. Local Fourier coefficients are computed, while their phase information results in binary coefficients after scalar quantization. The final descriptor corresponds to the histogram of the binary coefficients. More specifically, for an N × N image f (x, y) the local phase is computed using the short-term Fourier transform (STFT) based on the following equation:

fu i (x) = (f * φ u i )(x) (3.12)
φ u i is a complex valued m × m mask defined in the discrete domain by:

φ u i = e -j2πu T i y |y ∈ Z 2 ; y ∞ ≤ r (3.13)
where r = (m -1)/2, and u i a 2-D frequency vector. STFT for the specific implementation of LPQ [START_REF] Heikkila | Improved blur insensitivity for decorrelated local phase quantization[END_REF] is computed at four frequencie points: for the LPQ pipeline is shown in Fig. 3.13 in similar manner with [START_REF] Belahcene | Local descriptors and tensor local preserving projection in face recognition[END_REF].

u 1 = [α, 0] T , u 2 = [0, α] T , u 3 = [α, α] T ,

Histogram of Oriented Gradients

HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] entails counting gradient orientations in a dense grid. Each image is divided into uniform and non-overlapping cells, the weighted histogram of binned gradient orientations for each cell is computed, and subsequently combined to form the final feature

Transfer Learning from Pretrained Networks

Deep learning, which has become increasingly popular during recent years, is a selflearning tool designed to identify patterns in several sets of data samples, extracted from multiple processing layers. Each layer is composed by representation-learning methods, and is processed in a higher and more abstract level [START_REF] Lecun | Deep learning[END_REF]. Convolutional Neural In order to achieve optimal outcome from deep neural networks training and fine tuning is required. However in order to have accurate training a high number of training samples is needed, in the order of more than 100,000. In clinical applications, and espe-

METHODOLOGY

Machine Learning

The machine learning level is concerned with exploiting the extracted features in order to train models that generalize well and are able to recognize signs of depression. Given the high dimensionality of the features there is the need to employ a reduction algorithm.

Further, before the training/testing of the models there is another issue to address, this of the cross validation which is used so as to deal with less biased measures. Finally, in the proposed work two approaches were investigated, the categorical (classification) and the continuous (regression). Next, the different steps are described.

Dimensionality Reduction with PCA

In the present work, Principal Component Analysis (PCA) was employed to achieve dimensionality reduction. PCA is one of the most popular methods for this purpose, and is based on the linear transformation of the original feature vector, into a set of uncorrelated principal components. For a data set of size N × M (i.e., N samples and M features) PCA identifies a M × M coefficient matrix (component loadings) that maps each data vector from the original space to a new space of M principal components.

However, by properly selecting a smaller set of K < M components, the dimensionality of the data can be reduced while still retaining much of the information (i.e., variance)

in the original dataset.

Cross Validation

Cross validation is a technique employed in order to establish the experimental conditions is such manner that minimum possible bias is introduced to the models. For this reason the data samples are partitioned to train and test sets, so that samples that are used for training are not included in the testing process. The performance of a model can be objectively evaluated when it is introduced with unseen data. Furthermore, by experimenting with several configurations of train/test sets the outlier effect on the performance evaluation is reduced.

Some techniques employed by the proposed work for performing cross validation

involve the k-folds, Leave-One-Out (LOO), and Leave-One-Subject-Out (LOSO). Implementation of k-folds partitions the data samples in k sets, e.g. for k = 10 the dataset is divided in 10 different non-overlapping subsets, each time one subset is kept out, while LOSO is a particular type of LOO, and applied in cases where more than one samples of a given subject/participant are included in the dataset. This process is repeated as many times as the number of subjects included in the dataset (rather than samples as in LOO) each time all samples of the given subject are held out from the training set, and then used for testing the performance.

Machine Learning

Gender Dependency

Gender-based classification for depression has been reported to substantially improve performance [23] [194]. In the present work, in addition to gender-independent classification/regression, a gender-based model was also implemented by building two separate classifiers, one for male and another for female participants. The classifier for male subjects was trained on feature-sets extracted from data of male participants and the female classifier with feature-sets extracted from data of female participants.

kNN

kNN is considered to be one of the fastest and simplest algorithms commonly used for supervised learning. kNN considers the known feature vectors, and for every new vector computes the Euclidean Distance to find the nearest known class, based on the Equation 3.11. The class is computed according to the nearest k neighbours, that is to which it has the smallest Euclidean distance. k, the number of neighbours has to be specified.

Random Forest

Random Forest is a flexible, popular, easy to use supervised machine learning algorithm.

The idea behind RF is creating an ensemble of decision trees in a random manner, as its name suggests, based on the "bagging" method, which is nothing more than combining learning models in order to increase the overall performance. More specifically, RF builds a series of decision trees and combine their outputs for a more accurate prediction2 (c.f. Fig. 3.21). The implementation employed in the present work is the one from MATLAB3 .

Machine Learning

SVM

SVM is another supervised machine learning algorithm that can be employed for both classification and regression problems. In terms of classification SVM is intended for binary problems, with the main purpose to find an optimal hyper-plane that separates samples of the two classes in feature space. In the example of Fig. 3.22.a several hyperplanes have been tried (dashed lines), yet only one (the thick line) is chosen as it optimally differentiates the two classes. An additional consideration when choosing the optimal hyper-plane is establishing maximal margins from the samples.

Support Vector Regression (SVR) is in generally based on the same principles with the SVM that is used for classification, yet it encompasses some differences. The regression nature of the algorithm is intended in predicting real-value numbers, as opposed to discrete categories, which has infinite possibilities. Therefore it is a more complicated problem, which targets the construction of a function that by minimizes the error between the actual and the estimated value (c.f. Fig. 3.22.b). Implementations for SVM are from MATLAB both for classification1 and regression2 approaches.

METHODOLOGY

Fusion

Many types of features were extracted, such as appearance-based features, transferlearning-based VGG features, as well as the statistical features from time-series of geometrical features. In an attempt to test the additional value of the different descriptors, fusion of the features was tested. Specifically, the feature-level fusion was used, which implements the concatenation of the different vectors, in several combinations. Featurelevel fusion was employed in experiment 3 [START_REF] Pampouchidou | Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text[END_REF] (subsection 4.4), experiment 5 [START_REF] Pampouchidou | Quantitative comparison of motion history image variants for video-based depression assessment[END_REF] (subsection 4.6), and in the main study (subsection 5.2.1).

Summary

The employed pipeline was described in this Chapter, along with the several methods used for the different steps. Specific experimental setups employing the described algorithms are presented in the next Chapter.

Chapter 4

Preliminary Experimental Evaluation

In this chapter the different preliminary experimental tests are presented, with the specific configuration of each methodology, as well as the derived results. The data collection was a long-term process, which took more than 4 years to be completed, in the meanwhile the several proposed algorithms and methodologies were tested in available benchmark datasets. The preliminary experiments took place in order to provide some insight on the proposed methodologies, until the data collection was completed. Next the two benchmark datasets employed for the preliminary tests are described, followed by the description of the respective experiments.

Employed Datasets

In spite the fact that automatic depression assessment is highly desirable, clinical data are not open to the research community, given the sensitivity of personal data involved.

However, two datasets with depression annotation, both based on self-reports and volunteer non-diagnosed participants, were made available in terms of the AVEC challenges in 2013, 2014, and 2016 respectively. Both require a signed End-User License Agreement before granting permission for download access.

Experiment 1

DAIC

During AVEC'16 [START_REF] Valstar | AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge[END_REF] a different dataset was utilized, the Distress Analysis Interview Corpus -Wizard of Oz (DAIC-WOZ) [START_REF] Gratch | The Distress Analysis Interview Corpus of Human and Computer Interviews[END_REF]. This dataset contains clinical interviews designed to support the diagnosis of psychological distress conditions such as anxiety, depression, and post-traumatic stress disorder. These interviews were conducted by a virtual character, while collecting audio and video recordings as well as questionnaire responses. However the virtual agent was not functioning independently, but was controlled by a human interviewer in another room, as the purpose at the time was not to have a fully functional intelligent agent, but rather evaluate the efficiency of the interaction of the character with humans.

DAIC-WOZ provided only the features extracted with the OpenFace software [31],

instead of the original video recordings, due to data protection issues. The shared data include 189 sessions, ranging between 7-33min (with an average of 16min). Transcript of the interaction, audio recordings, and OpenFace features are available for each recording.

Experiment 1

Results of the first complete experiment conducted in terms of the PhD work was pub- 41pseudo-images per frame), while for XZ plane the number of images was M and for YZ it was N; LBP ran for each one of them in two combinations of parameters as described above, resulting in 840 features for XY, 252 for XZ and 28 for YZ, for a total of 1120, while for the window of 180 frames the resulting feature set was of size 5320. Finally a Nearest Neighbor classifier was used to train/test the method adopted for detecting self-reported severity of depressive symptomatology according BDI-II scores.

As shown in Table 4.1, best results were obtained by integrating facial activity data over for30 frames (1 sec). According to the confusion matrix computed for the 30 frame window in Table 4.2, more than 50% of the samples were correctly classified in each class, significantly exceeding random levels (25%) in the case of four classes.

Experiment 2

Experiment 2

The work of this experiment was published in the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Pampouchidou et al. 2016) [START_REF] Pampouchidou | Video-Based Depression Detection Using Local Curvelet Binary Patterns in Pairwise Orthogonal Planes[END_REF]. Again the AVEC dataset was used, while an effort was made to find solutions in crucial issues that emerged during the first experiment.

The first issue that needed to be addressed was face detection, as previously there was a high rate of false detections (˜20%). In order for any false classification to be attributed solely to the feature extraction method 100% accurate face detection had to be established; thus a semi-automatic face detection algorithm was implemented. Face region was manually initialized, and then tracked with the Kanade-Tomasi-Lucas tracker as described in [START_REF] Tomasi | Detection and tracking of point features[END_REF], which is set to fail and to be reinitialized if the tracked points are below a threshold (20 points). Tracking fails when face goes out of the field of view, because of occlusions (e.g. hand in front of the face), or when illumination becomes too inadequate even for a human observer to distinguish facial features. Such issues are met in about 20 videos out of the total of 200, and these videos were disregarded.

The extracted facial region was resized to 256x256 pixels, followed by the Curvelet transform, with Orientation and Scale parameters were set to 1 [START_REF] Candes | Fast discrete curvelet transforms[END_REF], resulting to a 43x43

CurveFace. LBP descriptors were extracted for two sets of [Radius, Neighbourhood] Results for both Frame-based and Video-based classification algorithms, and all three cut-offs, for both cross validation methods are summarized in Table 4.3. Table 4.4

presents the confusion matrix across the four classes derived from the Frame-based algorithm (20-Fold cross-validation).

In sum, the technical advances achieved in Experiment 2 were: a) establishing an accurate face detection, b) implementing overlapping window to involve more information on the motion patterns, c) choosing the Pairwise-Orthogonal-Planes instead of the Three-Orthogonal-Planes in order to reduce the dimensionality of the features, d) testing video versus frame based representation, and e) applying additional cross-validation methods by using LOO in addition to k-fold. Given the special circumstances the LMHI methods was applied. The landmarks used for LMHI, according to the numbering of Fig. 3.2, were those corresponding to eyebrows {18-27}, eyes {37-48}, nose-tip {32-36}, and mouth {49-68}. Before computing the LMHI all landmarks were co-registered, using affine transformation, by aligning the points corresponding to the temples, chin, and inner and outer corners of the eyes {1, 9, 17, 37, 40, 43, 46}.

PRELIMINARY EXPERIMENTAL EVALUATION

The gray value was defined by a step s, which corresponded to the maximum pixel value (255) divided by the total number of frames. Thus in every frame the gray value was computed by s multiplied by the frame count (i.e. for the 4th frame the gray value was 4 x s). The morphological operation of erosion with a structural element of disk and size 2 was applied in order to remove outliers (very distant movements) and the image 4.5 Experiment 4 Gender-independent Gender-based 0.5 (0.9) 0.17 (0.83) was cropped to the non-black pixels (non-zero). The resulting LMHI was further resized to fit the average size of the LMHI. An example of the resulting LMHI is illustrated in Fig. 4.3, where the amount of movement is indexed by the multitude of bright pixels.

LBP, as well as HOG, were extracted based on LMHI. LBP was computed for two sets of parameters: radius and neighborhood for {1, 8} and {2, 16}, resulting in a total of 28 features covering the entire face. The LMHI was further partitioned using corresponding half ratios to represent three composite regions: "nose+mouth", "left eye+eyebrow", and "right eye+eyebrow". These regions were partitioned by red dashedlines in Fig. 4.3. For each subregion 28 additional features were computed. Further, HOG was computed for the entire face area with 1080 features, as well as for the remaining sub-regions of "nose+mouth", giving 360 features, "left eye+eyebrow" with 144, and "right eye+eyebrow" with 144 features as well. Additionally, the LMHI histogram bins were computed resulting in 255 additional features (black was excluded). Mean and standard deviation of the pixel values constitute the final two features, resulting in a total of 2097 LMHI features.

The performance of the proposed method was evaluated through training on the training set and subsequent testing with the development and test sets, as set by the challenge organizers. In addition, the algorithms were assessed using the LOO procedure on the joined training and development sets. Performance of each modality, in genderindependent and gender-based models, as well as the favorable comparison to the baseline performance, are reported in Tables 4.5 and 4.6.

Experiment 4

The fourth experiment was set in a different direction than the previous, as instead of appearance-based features, geometrical features with statistical descriptors were tested.

The results of this experiment were published in the 39th Annual International Confer-4.6 Experiment 5 pipeline of the methodology proposed in terms of this experiment is illustrated in Fig.

4.4.

OpenFace was employed for detection of 2D facial landmarks extracting aligned facial images of size 112×112 pixels [31]. In the present work, only successfully detected frames were retained for further processing.

Regarding specific parameters of the motion representation algorithms, the value of ξ was set to 25 for MHI, and to 8 for GMHI. These thresholds were chosen empirically, so that the static-background did not present movement in the motion image.

This effect was noted at lower ξ values, where differences in pixel intensity were attributed to illumination variations. Setting the appropriate threshold, ensures that the movement represented by the motion images is meaningful, and can be attributed solely to movements. LBP was tested with two sets of [radius, neighborhood], namely [1,8] and [2,16]. The Gaussian kernel was chosen for the SVM classifier, with the expected proportion of outliers in the training data set to 10%. The log(x/(1 -x)) transform function was applied. The number of principal components retained was also selected empirically: k=100 for the appearance-based descriptors and k=60 for VGG. LOSO was used for cross-validation, as a non-biased and person-independent method, given that the dataset contained more than one recordings of the same subject (ranging from 2 to 6).

Performance of the different configurations of the proposed algorithm is summarized in Tables 4.8 -4.12. Table 4.8 presents performance of the various appearance-based descriptors for each of the three different motion images. The descriptors were tested individually, and combined with feature level fusion (concatenated). Table 4.9 presents Additional performance metrics for the best performing model are reported in Table 4.11, whereas Table 4.12 compares the present findings to previously published results using similar data sets. It should be noted that the results presented by Senoussaoui et al. [START_REF] Senoussaoui | Model Fusion for Multimodal Depression Classification and Level Detection[END_REF] and Alghowinem et al. [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF] were obtained using different organization of the

PRELIMINARY EXPERIMENTAL EVALUATION

Main Experiment

The main experiment in terms of the PhD is consisted by two main parts: a) the data collection from the clinical study, and b) the experimental evaluation of the developed methodology on the clinically valid dataset. The respective sections follow below based on this rationale. The work described in this Chapter has been submitted for publication the the IEEE Journal of Biomedical and Health Informatics.

Data Collection

The work described in this section concerns the creation of a database, comprising recordings of human facial expressions, speech, and physiological signals, after experimental induction of discrete emotions that are considered relevant for assessing depressive symptomatology. The collected data are used in Section 5.2 for the main study of developing computational methods capable of recognizing non-verbal signs, able to differentiate mentally healthy individuals from those suffering from depression.

Although the primary aim of the dissertation did not involve construction of an emotion database, the undertaking of original data collection lies in the field of affective computing and faces similar challenges. Data collection procedures required careful design having a strong impact on the capacity to address the main research questions [START_REF] Kächele | The influence of annotation, corpus design, and evaluation on the outcome of automatic classification of human emotions[END_REF]. As already shown in Ch.2, the methods employed for collecting relevant symptomatology from individuals, along with the annotations, are the most important steps.

Table 5.1 summarizes all steps that took place in order to successfully complete the data

MAIN EXPERIMENT

collection performed in terms of the presented thesis, while the details of the different steps are described in some detail within the following sections.

Ethics and Data Protection

Obtaining the required bioethics permissions was the first and prerequisite step in order to begin the data collection. Privacy and ethics are fundamental principles when it comes to data collection involving video recordings, or other types of sensitive data [START_REF] Cowie | Emotion-Oriented Systems: The Humaine Handbook, chapter Issues in Data Collection[END_REF].

Obtaining ethical approval, informed consent, and establishing the confidentiality of the data are some key aspects. Securing approvals from the Bioethics Committee of the University Hospital of Heraklion (Decision 296, Session 7/06-04-2016) and the National Data Protection Authority (Protocol No ΓN/EΞ/392-2/21-04-2016) lasted approximately one year. In addition to providing prospective participants with an extensive description of the study details in layman terms, they were explicitly informed that they could withdraw at any point during or after completing the experiment and request deletion of all their data.

Participants

Participants were healthy volunteers aged 20-65 years without history of mental or neurological disorder and patients suffering from MDD as diagnosed by their treating psychiatrists at the Psychiatry Outpatient Clinic, University Hospital of Heraklion. Healthy volunteers were recruited through announcements on a Facebook page, flyers posted at several sites (patient waiting areas at the University Hospital, University Campus, FORTH labs and common areas), as well as through personal referrals. Patients were informed regarding the study by their physicians during regular appointments and if they consented verbally they were explained the details of the study both verbally and in writing by a research assistant (psychologist). In most cases written consent and testing were obtained at the same time to avoid bring the patient in on a separate day.

The final sample included 65 participants (control group n=45, patient group n=20).

Socio-demographic information and clinical indices can be found in Table 5.2 for each group, while individual data on the same variables can be found in Appendix A. Although an effort was made to keep demographics in similar levels, and recruit participants in the same age range the patient group was older and had completed fewer years of formal

Data Collection

Tools assessing depression-related symptoms

Two self-report questionnaires were administered to all participants to assess depression and anxiety symptomatology, namely the Greek adaptations of the BDI-II and the STAI Form Y. The BDI-II [START_REF] Giannakou | Adaptation of the beck depression inventory-II to greek population[END_REF] has been adjusted according to DSM-IV criteria for MDD, comprising 21 questions, scored on 0-3, 0-4, or 0-5 point scales. These questions assess a wide range of emotional and behavioral signs of depression such as body image, hypochondriasis, difficulty in working, sleep loss, appetite loss, thoughts of selfpunishment, suicidal ideation, and reduced libido. The higher the score, the more severe the depressive symptoms, while the standardized cutoffs for BDI-II are:

• 0-13: minimal depression The Trait-Anxiety Scale (STAI Form Y-1) [START_REF] Fountoulakis | Reliability and psychometric properties of the greek translation of the state-trait anxiety inventory form y: Preliminary data[END_REF] was designed as a self-assessment tool of persistent symptoms (feelings, somatic complaints and behaviors) that are considered as core manifestations of anxiety as a characteristic of the individual. It consists of 20 items rated on a 1-4 point scale, with higher total scores indicating higher levels of anxiety.

Emotion elicitation paradigm

Non-social context: Selection of stimuli

In the context of a pilot study involving n=10 healthy volunteers aged 26-to-39 (5 men) we evaluated thirteen video clips that were considered as suitable for eliciting each one of four of Ekman's six basic emotions [joy (4 clips), disgust (3 clips), sadness (3 clips), and fear (3 clips) [START_REF] Ekman | Are there basic emotions?[END_REF]). Clips were chosen from popular movies, TV series, or YouTube. Participants in the pilot study were asked to view each video clip (presented in a different random order across participants). Participants in the pilot study were asked to view each video clip (presented in a different random order across participants). After watching each clip they were asked to rate their emotional experience on 15 dimensions (according to Gross and Levenson [START_REF] Gross | Emotion elicitation using films[END_REF]) on a 0-8 point scale. In Fig. 5.1.a the three

MAIN EXPERIMENT

video clips that were initially selected to induce sadness are contrasted based on groupaverage ratings on Gross and Levenson's 15 emotion dimensions. Whereas sadness is the dominant emotion elicited by all three clips, the clip from the movie "Forrest Gump" was excluded as it ranked lower in the sadness dimension while also elicited higher levels in pleasure and interest. After having collected recordings from about half of the control population (n=24), the stimuli were again evaluated, to make another selection, and keep only one for the patients population. As it can be seen in Fig. 5.1.b again both clips rank high in terms of sadness, yet the clip from the movie "Terms of Endearment" was chosen because it elicited lower levels of two potentially confounding emotional states, namely arousal and tension. These states are considered as indices of psychological stress that can alter the intended facial expression profiles for sadness.

Regarding the joy clip, a scene from a popular Greek comedy series (Para pente) was chosen over a clip of a laughing baby from YouTube, a greek movie (Mpakalogatos), and another comedy series (Peninta-Peninta), using a similar rationale as for the sadness clip. Finally, 30 frames (1 sec) were added to the beginning of each clip to serve as baseline. The remaining video clips (disgust, fear) did not elicit distinct emotions (e.g., the disgust clip elicited high levels of disgust but also similar levels of repulsion, arousal, interest, tension / c.f. Fig. 5.2) Therefore these clips were not used in the main study.

Social context

In addition to the video clips, an interpersonal method for eliciting positive and negative emotions was employed as well in the form of a semi-structured interview with the research assistant. Initially (prior to viewing the Joy clip) the participant was asked to describe a positive personal experience from his/her life; if the participant hesitated he/she was encouraged by the research assistant to go into more details, and asked probing questions guiding him/her to relive this experience as vividly as possible. In a similar manner, prior to viewing the Sadness clip participants were asked to describe a negative personal experience, which involved sadness or distress. The research assistant was closely monitoring the participant's emotional state and was instructed to terminate the interview in case of extreme emotional responses. A neutral baseline for the positive/negative experience description was chosen in the form of reading aloud a 260-word narrative text describing an excursion in the country.

Data Collection

On two predetermined occasions, the participant was guided through relaxation exercises, involving controlled breathing and brief mindfulness techniques to ensure that emotional states and stress levels returned to neutral prior to the description of positive experience/joy clip and, again, prior to the description of negative experience/sadness clip.

Technical Setup

The technical setup involved a camera, lights, microphone, and biosignal recording device. As shown in Fig. 5.3, the participant was seated in front of the PC which presented the stimuli, rating scales and questionnaires and registered his/her responses. Two researchers conducted the experiment: one operated the stimulus delivery and recording devices (AP) and the second (psychologist) interacted with the participant (obtained consent, provided instructions and additional explanations if needed, performed guided relaxation, and conducted the semi-structured interviews). Additional biosignals (voice, heart rate, galvanic skin response) were also recorded but will not be considered further in this thesis. Facial video data analyzed for the purposes of this dissertation originated from conditions 7-8 and 11-13 (shown in bold in Table 5.3). The order of conditions is presented in Table 5.3 and the total duration of the experiment ranged between 60-90 min.

The camera employed for the video recordings was the Grasshopper R 3, which provides high-performance and high-quality imaging (see Table 5.4). Chosen camera was of high specification in order to investigate how the quality of the recordings affects the outcome, as comparing to the reported datasets (see Table 2.2 which are of lower standards. The camera specs support maximum specifications of 90 frames per second (fps) and 2048×2048 image resolution. Although the PC involved was high performing (32GB RAM, SSD, SATA3), still full specifications of the camera could not be supported. During 10 benchmark tests it was specified that the average speed of writing to the disk was 413.9 MB/s. Furthermore, several configurations of fps and resolution were tested, to check whether frames were being lost during acquisition due to bottle-neck problem; 80 fps and 1920×1920 was the maximum configuration without losing any frames for 10 additional tests, therefore there were the finalized settings for the camera. Indirect lights were used to establish controlled illumination; the lights did not point directly on the participants, but to the wall opposite of them, to avoid extreme brightness. 95

MAIN EXPERIMENT

Blinded Expert Annotation

Although based on the data collection protocol, 3 types of labels were already available (diagnosis, BDI-II, STAI), still it was interesting to investigate the annotation by experts.

As also supported by [START_REF] Cowie | Emotion-Oriented Systems: The Humaine Handbook, chapter Issues in Data Labelling[END_REF] "It makes more sense to construct a labeling that is rich enough to support any foreseeable use of the basic material.", who continues supporting that a parameter to be examined is redundancy. In this case however, given the nature of the rest of the labels against the annotation, it can be supported that there is no redundant information, as they entail entirely different aspects for depression manifestations. More importantly, the motivation to move forward with the annotation was to have a label directly comparable to the output of a video processing algorithm, as both diagnosis and self-reports involve much more information (verbal, full-body, repeated sessions, reporting bias, etc.) which cannot be derived solely by a single-visual video recording.

Thus, the aim would be to compare the ability of an expert to detect visual signs of depression from the entire set of facial video recordings, versus the proposed automatic methodology based on digital image processing and machine learning. It is worth noting that it is the first time that experts' annotations are used for automatic depression assessment, as to the best of the author's knowledge all relevant approaches to date employ either the diagnosis, self-reports, or HAM-D scores attributed by clinicians in the time of examination.

CARMA1 [START_REF] Girard | CARMA: Software for continuous affect rating and media annotation[END_REF], a user-friendly software, was employed for this procedure; among the annotation tools reviewed in [START_REF] Kächele | The influence of annotation, corpus design, and evaluation on the outcome of automatic classification of human emotions[END_REF] CARMA was the only one which was fully-functional and corresponded to the needs of our research. Within CARMA the video is displayed in the center (only visual information in our case), while a slider right next to the display can be adjusted real-time and in a continuous manner by the annotator, stored externally in a file of comma separated values for further statistical analysis. The rating scale was set to 0-8 points with 0 indicating complete absence of depressive signs, and 8 indicating the most severe signs of depression. Annotation was performed by two experts (psychologists with at least one year of experience with patient interviews at the University Psychiatry Clinic) who were blinded with respect to clinical diagnosis and emotion elicitation condition. In case of deviation between raters equal to or greater

Experimental Tests

than 1.5 points on the scale, a third annotator was employed. The final annotation score registered was the average of all available annotations for each elicitation condition.

Statistical Results

The dispersion of BDI-II, STAI, and expert judgment values per group is shown in Fig. the optimal cutoff reported in the Greek validation studies [START_REF] Giannakou | Adaptation of the beck depression inventory-II to greek population[END_REF] [81] for BDI-II (13/14 points) was applicable in the current data set as indicated by 70% sensitivity and 85% specificity for BDI-II. Although sensitivity associated with the standard cutoff of 39/40 points on STAI was very high (90%), specificity was quite low (55%) which is expected given that the standard cutoff was establish for identifying persons with anxiety disorders [85] [81]. The optimal cut-off value for expert judgment (i.e., the value associated with highest sensitivity (90%) and specificity (80%)) was determined at 3.4/3.5 points (on the 0-8 point scale). Bivariate scatter plots shown in Figure 5.5 further suggest that selfreported values of depressive and anxiety symptoms were only moderately correlated with expert judgments of depression, whereas the correlation between the former (STAI and BDI-II scores) was very high (r = .768).

Experimental Tests

After the data collection was completed, the question of evaluating the proposed methodology arose. Based on the preliminary evaluation of the different algorithms and experimental setups, the MHI combined with VGG features proved to be the best performing, thus there is a great motivation to proceed with this methodology for the final and main experimental test. Next, the particulars of experimental setup are being explained, followed by two set of tests, those of a) categorical assessment, and b) continuous assessment.

Experimental Setup

The video recordings were successfully preprocessed by OpenFace to derive aligned faces, which were then submitted to the MHI algorithm (c.f. subsection 3.2.

Categorical Assessment

Categorical analyses were performed for four different types of binary labels: clinical diagnosis (depression, healthy), BDI-II score, STAI score, and blinded experts judgment (using cut-offs listed in 5.1.6). Analyses were performed separately across genders as well as for each gender and task separately. Results (Cohen's Kappa values) of different approaches are presented in Fig. 5.6, whereas Table 5.5 lists the best-performing approach for each classification scheme (diagnosis, STAI, BDI-II, expert judgment) in gender-based and gender-independent modes.

Experimental Tests

Continuous Assessment

Continuous assessment (regression) was performed separately for each of the three continuous labels (i.e., BDI-II, STAI, and expert judgment score) in gender dependent and gender-independent modes. Analyses were conducted for each condition separately as well as on combined data across conditions. Participant scores were normalized on a 0-100 scale to account for the different score ranges (i.e. BDI-II: 0-63, STAI: 0-80, and annotation: 0-8). Fig. 5.7 displays average normalized RMSE values demonstrating that the proposed method succeeds best in predicting the self-reported score on the STAI.

Best-performing scheme in terms on both RMSE and MAE among those listed in Table 5.6 is the gender-independent model conducted on video recordings from the Neutral condition. Additional indices of model performance are presented in Fig. 5.8 -5.13, in the form of absolute and log-transformed differences between actual and predicted label values (BDI-II score, STAI score, or expert judgment) using the following formula:

log 2 A norm P norm (5.1)
where A norm is the normalized actual value and P norm the normalized predicted value.

Fig. 5.8 -5.13 also include Bland-Altman plots [START_REF] Bland | Statistical methods for assessing agreement between two methods of clinical measurement[END_REF] using the following formula:

M = α -β, A = α + β 2 (5.2)
where α the predicted value and β the actual value. M corresponds to the difference between actual and predicted values (plotted on the y axis) and A to their average (plotted on the x axis) .

For future clinical applications of automated assessment methods, false negative results (i.e., failure to detect significantly high scores on a psychopathological trait, are most critical. In our results, such events correspond to participants where our models significantly underestimated self-reported or expert-judgment ratings as indicated by scores >1.96 SDs from the sample mean. With respect to BDI-II scores (Fig. 5.8 -5.9), the best performance was achieved by VGG features derived from the Neutral condition in gender-independent mode. In this analysis, there were only two false negative cases (the model significantly underestimated self-reported depression severity for two participants who scored > 55 points on BDI-II). With respect to STAI scores (Fig. 5.10 -5.11),

Experimental Tests

Experimental Tests

Experimental Tests

MAIN EXPERIMENT

Algorithm Development and Performance

Categorical vs Continuous Assessment One of the main issues addressed in the present work concerned a comparison of categorical versus continuous assessment. Initially, we pursued classification problems which are computationally more straightforward. However, computational approaches developed and applied to the AVEC dataset with apparent success failed to generalize to the data obtained in the context of the main experiment. Such poor performance motivated adoption of regression approaches which, eventually, appeared to perform comparably across datasets.

Importantly, one of the depression related labels (clinical diagnosis, based on clinically acceptable BDI-II cut-off, or based on expert judgment of presence of depression based on visual cues) were predicted at an acceptable level using categorical assessment. This failure however may not be so crucial after all, as the machine learning algorithm does not need to make a diagnosis, just a recommendation for further exploration; as already made clear, the proposed methodology is supposed to serve as a decision support tool, and not as a standalone system. In addition, the relatively poor classification performance may reflect the continuous nature of depressive symptomatology. This problem is compounded by the potential moderating role of clinical factors (such as 6. DISCUSSION type of medication, type and duration of treatments) and person-specific characteristics (illness-related cognitions and personality characteristics) that may affect the intensity and quality of facial expression of depression symptoms (e.g., negative mood, apathy, helplessness). Furthermore, depression does not have clear categories, and thus cannot be easily treated as a classification problem. The apparent continuum of severity of depression symptomatology is expected to be reflected in a corresponding continuous manner in facial motion dynamic patterns across patients rendering continuous assessment more appropriate.

Low vs high level features

Although a trend toward utilizing high-level features has been observed, and promoted by AVEC'16, where features were provided after preprocessing to enable high-level feature extraction, it seems that they are not the best performing. The high-level features are more appealing though, as they enable direct interpretation and give insights more easily. In terms of the low level features, LBP did not perform well in terms of classification, for the given set of parameters, resulting in near zero recognition most of the times. A potential improvement to LBP may entail adopting larger radius and neighborhood settings, as the ones selected here may represent only micro-movement patterns, or even employing another variant of LBP, such as weighted LBP. Remarkably, LBP, although failing in classification, it was among the best performing in the main experiment for predicting individual BDI-II scores in the gender-based mode. Overall the best performing low level descriptor in many different experimental setups was HOG, which can be justified by the fact that it retains spatial information, and does not construct a unique histogram for the whole image.

Deep learning

The deep learning approach outperformed the other approaches in most settings, both in Experiment 5 (see section 4.6) and in the main study. However, given that only the generic VGG was employed in the proposed work, based on its previously reported performance, it is highly probable that after training and tuning the network, the rate of correct depression assessment could improve significantly. The highly competitive performance of deep learning-based features has been noted in several research areas. The fact that deep learning does not rely on prespecified rules, in a manner similar to human cognition, may account for its superiority. The performance of deep learning algorithms with a relatively small data set, such as the one tested in

Motion representations

Results pertaining to the visual features based on LMHI as extracted from the available data set in Experiment 3 (see section 4.4) were rather surprising given that analyses highlighted a single visual feature (LMHIFaceHOG) as the most significant. This could be explained by the fact that LMHIFaceHOG incorporates motion information, and by being registered and resized, it minimizes appearance-based variation. Furthermore, LMHI requires significantly less information (image frames vs. selected landmarks), and ensures participant anonymity by retaining only facial landmarks -an important attribute in studies with clinical samples. Among additional motion history-based algorithms evaluated (MHI and GMHI) in Experiment 5 (see section 4.6) the original MHI algorithm performed better than the proposed variants. The rest of the proposed motion representations, namely the LCBP-TOP and LCBP-POP, as well as the one based on facial geometry, did not perform as well as the motion images. Finally, in terms of the window based approaches, the duration of windows varied for the different approaches, which means it is highly depending on the specifics of the approach (e.g. method, dataset, etc).

Cross validation An important issue to consider when evaluating published reports on automatic depression assessment concerns the use of relatively small samples and suboptimal cross validation methods, which are highly susceptible to model overfitting.

Performance metrics

The experience gained through the experimental tests leads to the conclusion that performance metrics need to be considered in combination. Most of the reported approaches in the literature report only one metric (e.g. accuracy) which does not reliably show the capacity of a model, especially in cases of highly unbalanced datasets. Further metrics such as Cohen's Kappa encompasses most pertinent information and it is not surprising that the majority of complementary metrics typically follow Kappa values across studies. The F1-score, which has been very popular in previous reports, does not necessarily reflect accurate recognition across all classes (i.e., a high F1-score may be associated with good recognition in one class and poor recognition in 6. DISCUSSION another) and does not consider the level of chance. Thus it is highly relevant to report a set of metrics, rather than choosing a single one.

Data Related Issues

Sample size Many approaches, along with some of the ones proposed hereby, report very high detection accuracy rates, a fact that clearly demonstrates the clinical potential of the field, but sample sizes are often too small to enable the generalizability of these results. In order for a system to be fully evaluated and acknowledged as an assessment tool, it must be tested on considerably larger sample sizes, featuring a wider variety of demographic characteristics, clinical diagnosis methods, and ethnic-cultural backgrounds.

Comorbidity & subtypes Comorbid diagnoses, should be carefully recorded and used to evaluate potential misclassifications, given the high comorbidity rates between PTSD, anxiety and depression [START_REF] Scherer | Automatic Audiovisual Behavior Descriptors for Psychological Disorder Analysis[END_REF]. In addition the capacity to distinguish between different depression subtypes, and MDD from other mood disorders also needs to be addressed [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: the Effect of Gender[END_REF]. Individual variability due to comorbid personality disorders or characteristics, as well as the influence of ethnicity and culture requires further exploration [START_REF] Alghowinem | Multimodal Depression Detection:Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors[END_REF].

Multiple sessions Furthermore, as reported section 1.1, a one-off clinical assessment may not be sufficient neither for diagnosis or registration of facial features, as development of rapport with the participant is necessary. For this to be achieved several sessions over a fixed interval (e.g. 7 weeks as in [START_REF] Cohn | Detecting Depression from Facial Actions and Vocal Prosody[END_REF] [90] [START_REF] Girard | Social Risk and Depression: Evidence from Manual and Automatic Facial Expression Analysis[END_REF]) are advisable. Existence of baseline data would also be useful, but unfortunately this is not possible in most cases. However, given the importance of symptom/sign stability for depression diagnosis, repeated recordings over several days or weeks would render results more clinically relevant. Finally, multiple sessions can benefit the remission assessment, allowing longterm monitoring of the recovery process, as well as providing a personalized model based on the built rapport.

Data Related Issues

Real-world vs laboratory setting In the field of automatic facial expression recognition (AFER) approaches are moving toward real-world conditions [START_REF] Martinez | Advances in Face Detection and Facial Image Analysis, chapter Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition[END_REF], as exemplified by the Emotion Recognition in-the-Wild (EmotiW) challenge series [START_REF] Dhall | Emotion Recognition in the Wild Challenge[END_REF] [67] [START_REF] Dhall | Video and Image Based Emotion Recognition Challenges in the Wild: EmotiW 2015[END_REF]. The manner in which the AVEC dataset was constructed also supports this idea, as the recordings took place in independent setups and on personal computers. This choice, however, impacted performance, as shown in Table 2.5: approaches for categorical assessment of depression based on AVEC demonstrated lower than average performance, when compared to approaches based on other datasets.

Additionally, although current in-the-wild approaches may be considered as promising, they are not yet sufficiently reliable even for AFER as supported in [START_REF] Schuller | Emotions and Personality in Personalized Services: Models, Evaluation and Applications, chapter Acquisition of Affect[END_REF] [START_REF] Martinez | Advances in Face Detection and Facial Image Analysis, chapter Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition[END_REF]. Therefore, at present, such approaches do not appear to meet minimum requirements for a clinical decision support system. On the other hand, the strict requirement for standardization of data collection [START_REF] Cummins | A Review of Depression and Suicide Risk Assessment Using Speech Analysis[END_REF] may impose potentially serious limitations, such as questionable originality and genuineness of the data and lack of variance in contextual information. Although standardized medical equipment typically operates under highly controlled conditions, collection of data indicative of depressive symptomatology is highly susceptible to the dynamic nature of behavioral and underlying psychological processes of the person being evaluated.

Stimulus

In terms of the benchmark datasets employed for the preliminary experiments, the DAIC-WOZ presented a better interpersonal context for depression assessment in view of the extant literature supporting the better suitability of interviews for detecting signs of depression. However, in the tests performed in terms of the main study the processing of clips recorded during the neutral text reading (non social context) turned out to be the best performing in several setups. Some disadvantages of the neutral text reading task involved some reading problems from the participants, language issues, sight problems, etc.

Testing clips combined across tasks was never ranked among the best performing. It is obvious that comparison, either in terms of categorical assessment or continuous, must be under strictly defined protocol, and during executing the same task. Furthermore, with the exception of diagnosis, which has their best performance in both Neutral and Positive stimuli, for gender-based and gender-independent respectively, the rest labels have their best in the same stimulus without regard to the gender dependency mode.

DISCUSSION

More specifically, for BDI the best performance is with neutral stimulus in both genderbased and gender-independent modes. In the same rationale, the highest performance for STAI prediction was again with the neutral stimulus for both gender modes. In terms of the expert judgment values, the best result was for the Negative Experience Recall condition in gender-based mode.

Annotation During the preliminary experimental tests the need for clinical data, involving diagnosed patients and not just volunteers as in the benchmark datasets, is stressed. It was assumed that a score on a single self-report instrument may not be sufficient to establish reliable and valid classification of individuals according to depression status. Other factors can readily elevate self-report scores on depression scales, such as the presence of significant, long-term life stressors and anxiety. Thus, a clinical interview would be more reliable for building a robust dataset.

In the proposed work, the data were collected from both non-diagnosed individuals to form the control group, and diagnosed MDD patients. In addition, a team of psychologists rated each video and provided an overall assessment of depressive signs, to provide an alternative benchmark which would be more closely matched to the source of information employed by the proposed methodology (facial images). The annotation was another novelty by the proposed work, as it has never before been provided by other datasets, which employ just a self-report score (BDI, PHQ9, etc), or the score of a clinically-administered instrument (e.g. HAM-D). The parallel use of STAI was an additional novelty.

However, the results did not prove the hypothesis, as the training and testing based on the diagnosis did not provide good results, which can again be attributed to the fact that depressed status is not uniform. There are several and very different ways that patients appear or behave, which is highly relevant to their severity of depression.

Although all patients included in the dataset were medicated and diagnosed, they were in different stages of remission, as well as with different levels of severity, thus considering them as a uniform class is false. STAI scores were predicted much better than any other variable, which implies that anxiety can be portrayed and evaluated more accurately by the specific methodology.

Results are well above chance for all metrics involved in categorical assessment of STAI, and also in terms of continuous assessment it performs best too. Although the best

Data Related Issues

Cohen's Kappa is for diagnosis, yet it is still low (45.07%). In addition, and although if we consider each metric independently, again diagnosis has the best accuracy and precision, still it fails for F1-score and for Recall. A better insight of the results was given by checking which setup had the overall best performance, a rule of "which setup has all metrics above 70%" was set (excluding the kappa). The STAI was the only one to conform with this rule, for both gender based and gender independent. This seems quite promising, in both cases for the Sad stimulus.

Gender dependency

The gender-based model did not perform very well, while the gender-independent performed much better. This can be explained by the fact that by separating male/female subjects then the remaining training samples for each model are quite low. Therefore gender dependency was not properly evaluated, and again there is a need for a greater sample, as even the male rate is quite low. It seems that it was not really meaningful to attempt gender-based approaches in the first place, as the training sample was too low. Specifications Based on the literature review it was predicted that video acquisition conditions are of great importance, as they often affect processing tasks such as face detection, and that to this end, illumination, image resolution, frame rate, are some crucial factors to be considered. However, this prediction was not corroborated in the main study, as the categorical approach using the same methodology did not perform as well as in the AVEC dataset, while for the continuous assessment the perform in the same level. Finally, by constructing a high-specification dataset the computational cost is raised significantly, as higher resolution frames and at a higher frame rate produce a considerably higher amount of data to be processed, which directly affects the pipeline; for instance for the example of the motion history image, having a frame of 1200x1200 at a frame rate of 80 fps would be much more complex to be computed compared to an image of 480x640 at a frame rate of 30 fps.

Failed predictions

We made an effort to account for the failed predictions of the best performing method, which is the continuous prediction of the STAI self-reported score for the neutral stimulus in the gender-independent mode. The most notable failure concerns underestimation of STAI scores, given the higher associated clinical risk. Among the four 123 6. DISCUSSION patients in this category, two suffered from severe depression and were treated with high doses of combinations of antidepressants which may have affected the dynamics of facial expressions. Another patient spoke Greek as a second language and experienced some difficulty in reading the text, while the fourth patient also had difficulty reading due to reduced visual acuity. These issues can be considered as limitations of the proposed methodology, namely: individual medical treatments, language and vision problems.

Future Plans

In this section potential solutions to problems identified in the current experimental work are proposed.

Algorithms

In future work improvement of feature selection methods is probably the best avenue to enhance classification performance. Inspection of the bivariate and partial correlation matrix between individual features and using probability-based statistic indices (such as Fisher's z) to identify significant associations may help optimize feature selection.

Although performance achieved here in terms of categorical assessment for the AVEC'14 outperforms related work, there is still room for significant improvements.

Future work may attempt training and tuning the VGG, testing different versions of the network, assessing the performance of additional classifiers, as well as attempting decision fusion for the different motion representations and feature combinations. Additional improvements could involve applying windowing to the different motion image variants, using codebook approaches for dimensionality reduction (e.g. Bag of Words).

Multimodal Although this thesis is focused on image processing, work conducted independently in our lab employing multimodal approaches showed promising results [START_REF] Pampouchidou | Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text[END_REF] [163] [START_REF] Pampouchidou | Détection de la dépression par lŠanalyse de la géométrie faciale et de la parole[END_REF]. These preliminary results are in agreement with the systematic literature review presented in [START_REF] Pampouchidou | Automatic assessment of depression based on visual cues: A systematic review[END_REF]. Therefore, combining audio based features [START_REF] Simantiraki | Glottal source features for automatic speech-based depression assessment[END_REF] could potentially improve overall performance. In addition, during the data collection physiological signals were collected too, at the same time with visual and audio signals; blood volume pulse (BVP), and galvanic skin response (GSR) could also add value to the performance. or using weighted fusion methods. Stacking is another method for implementing decision fusion [START_REF] Sfakianakis | Stacking of network based classifiers with application in breast cancer classification[END_REF], and also the Posterior Probability Classification Model [START_REF] Fazlollahi | Computer-aided detection of cerebral microbleeds in susceptibility-weighted imaging[END_REF] (c.f. Fig. 6.1) which was also used in [START_REF] Pampouchidou | Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text[END_REF] during our participation the AVEC'16 challenge.

Future Plans

Data Efforts to develop methods capable of differentiating mood disorder types, and also depression from other psychiatric disorders, such as various anxiety conditions, are not many. Although some notable initial attempts in this direction can be found in [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences[END_REF] [START_REF] Stratou | Automatic Nonverbal Behavior Indicators of Depression and PTSD: the Effect of Gender[END_REF] by focusing on distinguishing PTSD from depression, and another in [START_REF] Yang | Coupled HMM-based Multimodal Fusion for Mood Disorder Detection Through Elicited Audio-Visual Signals[END_REF] attempting to distinguish cases of unipolar depression versus bipolar disorder, still they are quite limited, and require further exploration.

Additional manifestations may need to be considered, but would need also additional specifications for the setup to be adjusted. Pupil-related features, for example, would be interesting to investigate in an non-obtrusive manner, that is without using head-mounted eye-tracker devices, but still highly constrained environment would be 6. DISCUSSION required, with a very expensive camera in order to achieve resolution and illumination conditions able to monitor the pupil activity. Monitoring body gestures would also require additional cameras and a different setup; in order to be able to monitor the body, we should not have a desk or other occlusions, rendering an interview setup as the most appropriate.

As already established above, the small number of subjects in each dataset is another issue. There is a need to test the validity in terms of a clinical study, including potentially thousands of samples. This is an issue, as the recruitment of patients, and having them to commit is very hard. This is the reason that there is still a long way from standardizing such a system. An approach for cross-corpus / cross-cultural method evaluation by Alghowinem et al. [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF] highlighted this issue as well.

CONCLUSIONS

Compare interpersonal vs non-social contexts for video recording Different types of stimuli were tested, both of non-social and interpersonal contexts, involving watching video clips selected to elicit specific emotions, answering questions from interviewer, and reading out loud a neutrally charged passage. However, the results of the tests did not provide a clear answer to the question, of which context is the best to evaluate the depression state. Although for our dataset it was the neutral task which provided the best performance in most settings, for the AVEC dataset the best results came for the Freeform task, which entailed responding to various questions. Given the many differences between the two datasets, it is not so straightforward to explain why this happened.

Evaluate the proposed methodology against several independent sources of information pertaining to participant psychoemotional status: Clinical Diagnosis, Self-reported symptoms of depression and anxiety, Expert judgment of visible depression manifestations. Results indicated that the proposed method was more sensitive to facial features more closely associated with self-reported anxiety (STAI scores). This fact could imply that anxiety manifestations are better detectable using the proposed methodology than manifestations of depression. The proposed method performed well for both gender-based and gender-independent modes, with best results with video recordings obtained during a non-challenging, emotionally or cognitively, condition (reading a neutral passage).

Develop video-based methodology to extract features correlating with signs

of depression Several methodologies were developed in the context of this thesis, which were initially evaluated on benchmark datasets. The best performing methodology was also tested on our dataset, and proved to have consistent performance across two completely different datasets. The consistency in performance was also in terms of predicting BDI-II scores.

Experiment with different video acquisition parameters Different video acquisition parameters were set during the data collection intentionally, to evaluate their contribution to model performance. At this moment it is not clear whether higher specifications contribute to higher performance, as the best performing algorithm with the 7.2 Major Contributions AVEC dataset did not outperform results based on our dataset (which was obtained at higher video specifications).

Investigate categorical vs continuous depression assessment While both approaches were investigated, greater emphasis was given to the categorical assessment.

Development of the focusing on classification spanned several years, whereas continuous assessment was systematically pursued during the final stages of the thesis research. It should be noted however that continuous assessment may be more suitable to the nature of measures used to diagnose and quantify depression severity.

Major Contributions

The major contributions of this thesis can be summarized as follows:

• Systematic literature review of relevant approaches, published in IEEE Transactions on Affective Computing (I.F. 4.585 / Q1) [START_REF] Pampouchidou | Automatic assessment of depression based on visual cues: A systematic review[END_REF]. 2.455 / Q2) [START_REF] Pampouchidou | Quantitative comparison of motion history image variants for video-based depression assessment[END_REF]. All implementations will be will be soon made available in MathWorks.

• Classification using four severity classes of self-reported depressive symptomatology (minimal, mild, moderate, and severe) was performed in two of our published approaches [START_REF] Pampouchidou | Designing a Framework for Assisting Depression Severity Assessment from Facial Image Analysis[END_REF] [160], which are the only ones to the best of our knowledge.

The remaining published approaches attempted binary classification, three-class classification, or regression.

• Categorical assessment of depressive symptomatology was performed using deep learning methods, for the first time on the AVEC dataset.

• Construction of a clinically valid dataset in terms of depression and anxiety.

A. COMPLETE LIST SOCIO-DEMOGRAPHICS PARTICIPANTS PROFILES

  of my funding during the last five years. First, I would like to thank Prof. Tsiknakis and Prof. Marias, who supported me financially during the first two years of my PhD through partial funding from research and development projects executed at that time by the Computational BioMedicine Laboratory (CBML) in Foundation for Research and Technology -Hellas (FORTH).

  Thus I would like to thank D.Manousos, G.Karatzanis, O.Simantiraki, G. Zacharioudakis, and C.Chatzaki for their technical support. My sincere gratitude goes also to the psychologists that executed the protocol and interviewed the participants during the recordings, I.Apostolaki, E.Kazantzaki, E.Papastefanakis, along with the psychologists who carried the annotation of the video recordings K.Argiraki and M.Daskalaki, thank you all so much! Next I would like to thank the master students I co-advised through these years for their co-operation. C.-M.Vazakopoulou, A.Maridaki, and A.-D.

5

 5 for recruiting me as his student assistant during my undergrad studies, and xii Résumé La dépression est le trouble de l'humeur le plus répandu dans le monde avec des répercussions sur le bien-être personnel, familial et sociétal. La détection précoce et précise des signes liés à la dépression pourrait présenter de nombreux avantages pour les cliniciens et les personnes touchées. Le présent travail visait à développer et à tester cliniquement une méthodologie capable de détecter les signes visuels de la dépression afin d'aider les cliniciens dans leur décision. Plusieurs pipelines d'analyse ont été mis en oeuvre, axés sur les algorithmes de représentation du mouvement, via des changements de textures ou des évolutions de points caractéristiques du visage, avec des algorithmes basés sur les motifs binaires locaux et leurs variantes incluant ainsi la dimension temporelle (Local Curvelet Binary Patterns-Three Orthogonal Planes (LCBP-TOP), Local Curvelet Binary Patterns-Pairwise Orthogonal Planes (LCBP-POP), Landmark Motion History Images (LMHI), and Gabor Motion History Image (GMHI)). Ces méthodes de représentation ont été combinées avec différents algorithmes d'extraction de caractéristiques basés sur l'apparence, à savoir les modèles binaires locaux (LBP), l'histogramme des gradients orientés (HOG), la quantification de phase locale (LPQ) et les caractéristiques visuelles obtenues après transfert de modèle issu des apprentissage profonds (VGG). Les méthodes proposées ont été testées sur deux ensembles de données de référence, AVEC et le Wizard of Oz (DAIC-WOZ), enregistrés à partir d'individus non diagnostiqués et annotés à l'aide d'instruments d'évaluation de la dépression. Un nouvel ensemble de données a également été développé pour inclure les patients présentant un diagnostic clinique de dépression (n = 20) ainsi que les volontaires sains (n = 45). xvii Deux types différents d'évaluation de la dépression ont été testés sur les ensembles de données disponibles, catégorique (classification) et continue (régression). Le MHI avec VGG pour l'ensemble de données de référence AVEC'14 a surpassé l'état de l'art avec un F1-Score de 87,4% pour l'évaluation catégorielle binaire. Pour l'évaluation continue des symptômes de dépression "autodéclarés", LMHI combinée aux caractéristiques issues des HOG et à celles issues du modèle VGG ont conduit à des résultats comparatifs aux meilleures techniques de l'état de l'art sur le jeu de données AVEC'14 et sur notre ensemble de données, avec une erreur quadratique moyenne (RMSE) et une erreur absolue moyenne (MAE) de 10,59 / 7,46 et 10,15 / 8,48 respectivement. La meilleure performance de la méthodologie proposée a été obtenue dans la prédiction des symptômes d'anxiété auto-déclarés sur notre ensemble de données, avec une RMSE/MAE de 9,94 / 7,88. Les résultats sont discutés en relation avec les limitations cliniques et techniques et des améliorations potentielles pour des travaux futurs sont proposées. xviii DAIC-WOZ Distress Analysis Interview Corpus -Wizard of Oz DBS-SCC Deep Brain Stimulation of the Subcallosal Cingulate Cortex DCNN Deep Convolutional Neural Networks DCNN-DNN Deep Convolutional Neural Netowork -Deep Neural Network DCS Divergence-Curl-Shear DM Discriminative Mapping DNN Deep Neural Networks DoG Difference of Gaussians DSC Depression Recognition Sub-challenge DSM-Diagnostic and Statistical Manual of Mental Disorders of the (APA) fifth edition DSSS Depression and Somatic Symptoms Scale DTL Deep Transformation Learning ELM Extreme Learning Machines FC Fully Connected FDHH Feature Dynamics History Histogram fps frames per second GLM Generalized Linear Models GMHI Gabor Motion History Image GMM Gaussian Mixture Models GSR Gaussian Staircase Regression HAM-D Hamilton Depression Rating Scale HCRF Hidden Conditional Random Fields xx HDR Histogram of Displacement Range HMM Hidden Markov Model HOG Histogram of Oriented Gradients IAPS International Affective Picture System ILSVRC ImageNet competition kNN k-Nearest Neighbour LBP Local Binary Patterns LBP-TOP Local Binary Patterns in Three Orthogonal Planes LCBP-POP Local Curvelet Binary Patterns-Pairwise Orthogonal Planes LCBP-TOP Local Curvelet Binary Patterns-Three Orthogonal Planes LDA Linear Discriminant Analysis LGBP-TOP Local Gabor Binary Patterns in Three Orthogonal Planes LMHI Landmark Motion History Images LMM Landmark Motion Magnitude LogR Logistic Regression LOO Leave-One-Out LOSO Leave-One-Subject-OutLPQ Local Phase Quantization LPQ-TOP Local Phase Quantization-Three Orthogonal Planes LR Linear Regression LSTM-RNN Long Short Term Memory Recurrent Neural Networks MAE Mean Absolute Error xxi MaxEnt Maximum Entropy Model MBH Motion Boundary Histogram MDD Major Depressive Disorder MHH Motion History Histograms MHI Motion History Images MIM Mutual Information Maximization MINI Mini International Neuropsychiatric Interview MPGI Moore-Penrose Generalized Inverse mRMR minimum Redundancy Maximum Relevance NB Naïve Bayes NNet Neural Networks OLS Ordinary Least Squares ORI Oregon Research Institute PCA Principal Components Analysis PHQ Patient Health Questionnaire PLS Partial Least Square Regression PTSD Post Traumatic Stress Disorder QIDS-SR Quick Inventory of Depressive Symptomatology-Self Report ReLU Rectified Linear Unit RF Random Forest RFR Random Forest Regressor RMSE Root Mean Squared Error xxii RVM Relevance Vector Machines S.D. Standard Deviation SDA Stacked Denoising Autoencoders Self-RIs self-report scales and inventories SGD Stochastic Gradient Descent STAI State Trait Anxiety Inventory STFT short-term Fourier transform STIP Space-Time Interest Points SVM Support Vector Machines SVR Support Vector Regression VFE Variability of Facial Expression Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Depression Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.1 Clinical diagnosis & Self-Reports . . . . . . . . . . . . . . . . . . . 3 1.2.2 Nonverbal Signs of Depression . . . . . . . . . . . . . . . . . . . . 4 1.3 Research Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2 Literature Review 9 2.1 Review Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Depression Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.1 Data Collection Procedure . . . . . . . . . . . . . . . . . . . . . . . 11 2.2.2 Reported Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 State-of-the-Art Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.3.2 Feature Extraction / Manipulation . . . . . . . . . . . . . . . . . . 16 2.3.2.1 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . 16 2.3.2.2 Dimensionality Reduction . . . . . . . . . . . . . . . . . . 20 2.3.3 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3.3.1 Cross Validation Methods . . . . . . . . . . . . . . . . . . 22 xxix 2.3.3.2 Classification . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3.3 Regression . . . . . . . . . . . . . . . . . . . . . . . . . .2.4 Selected Approaches and Meta-Analysis . . . . . . . . . . . . . . . . . . . 2.4.1 Categorical Depression Assessment . . . . . . . . . . . . . . . . . . 2.4.1.1 Pittsburgh . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1.2 BlackDog . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1.3 DAIC-WOZ . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1.4 AVEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1.5 Other Datasets . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Continuous Depression Assessment . . . . . . . . . . . . . . . . . . 2.5 Scope of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4. 4

 4 Experiment 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Experiment 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Experiment 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Main Experiment 5.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Ethics and Data Protection . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.3 Psychological measurements and experimental procedures . . . . . 5.1.3.1 Tools assessing depression-related symptoms . . . . . . . 5.1.3.2 Emotion elicitation paradigm . . . . . . . . . . . . . . . . 5.1.4 Technical Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.5 Blinded Expert Annotation . . . . . . . . . . . . . . . . . . . . . .

[

  17] (c.f. Fig. 1.1). Clinically distinct depressive disorders encompass a wide range of manifestations. According to the Diagnostic and Statistical Manual of Mental Disorders of the American Psychiatric Association (APA) [28], now in its fifth edition (DSM-5), subtypes of depressive disorders include: Major Depressive Disorder (MDD), Persistent Depressive Disorder, Disruptive Mood Dysregulation Disorder, Premenstrual Dysphoric Disorder, Substance/Medication-Induced Depressive Disorder, Depressive Disorder Due to Another Medical Condition, and Other Specified Depressive Disorder or Unspecified Depressive Disorder. MDD, also commonly referred to as Clinical Depression, is considered as the most typical form of the disease [217]. According to DSM-5 MDD can be diagnosed by the presence of a) depressed mood most of the day, and/or b) markedly diminished interest or pleasure, combined with at least four of the following symptoms for a period exceeding two weeks: significant weight change of over 5% in a month, sleeping disturbances (in-1. INTRODUCTION somnia or hypersomnia), psychomotor agitation or retardation almost every day, fatigue or loss of energy almost every day, feelings of worthlessness or excessive guilt, diminished ability to concentrate or indecisiveness almost every day, recurrent thoughts of death or suicidal ideation. An additional common feature of all depressive disorders is "(...) clinically significant distress or impairment in social, occupational, or other important areas

Figure 1 . 1 :

 11 Figure 1.1: Prevalence of depressive disorders (% of population), by WHO Region (Taken from [229])

Chapter 4 .

 4 Chapter 5 presents the methods used for collecting the clinical data, in terms of technical setup, protocol, and participants, along with the main experiment conducted as part of the current study. Discussion of experimental results and future work takes place in Chapter 6. Finally, Chapter 7 presents the conclusions of this work.

  eighty studies, technical details, potential limitations of each approach and classification accuracy achieved are evaluated, focusing on image processing and machine learning algorithms applied to depression detection. State of the art methods are presented highlighting their advantages and limitations, based on a quantitative meta-analysis of their results. Datasets created to serve the various studies and the corresponding data acquisition protocols are also described and discussed. Ultimately, gaps in the literature are identified to be addressed in terms of the proposed thesis. Parts of this Chapter have been published in IEEE Transactions on Affective Computing (Pampouchidou et al. 2017) [165].

Figure 2 . 1 :

 21 Figure 2.1: Number of studies in the field of depression assessment per year of publication

Figure 2 . 2 :

 22 Figure 2.2: Typical workflow for automatic depression assessment. The output can be derived from a) single feature sets/modalities, b) feature fusion, with dimensionality reduction before (FF1) or after fusion (FF2), and c) decision fusion with any possible combination of outputs from single feature sets/modalities and feature fusion

  algorithms are also very popular for full-face based features. Among the most prevalent texture descriptors are Local Binary Patterns (LBP). Several variants of LBP have been created for automatic depression assessment, such as an extension of LBP that considers patterns on Three Orthogonal Planes (LBP-TOP). Local Gabor Binary Patterns in Three Orthogonal Planes (LGBP-TOP) extends LBP-TOP by computing patterns on the output of Gabor-filtered data, rather than on the original intensity image. Along the same lines, Local Curvelet Binary Patterns-Three Orthogonal Planes (LCBP-TOP) was introduced in some studies, which entails computing the patterns on the curvelet transform of the original image. Local Curvelet Binary Patterns-Pairwise Orthogonal Planes (LCBP-POP) is yet another variation of the algorithm operating on pairs of orthogonal planes. Additionally, the Block-Wise LBP-TOP (BW-LBP-TOP) method which computes the LBP-TOP for a specific number of nonoverlapping blocks, has also been employed. Local Phase Quantization (LPQ) is another texture descriptor with a similar extension of Local Phase Quantization-Three Orthogonal Planes (LPQ-TOP). Another popular algorithm for motion-based approaches is 2. LITERATURE REVIEW the histogram of optical flow, which estimates the motion within visual representations. Divergence-Curl-Shear (DCS) descriptors are also based on optical flow. The Motion History Histograms (MHH) algorithm, which extends Motion History Images (MHI), has also been found in the related literature. A further extension of MHH is the 1-D MHH, which is computed on the feature vector sequence, instead of the intensity image. In a similar manner, Feature Dynamics History Histogram (FDHH), captures the dynamic variation which occurs among the extracted descriptors of image sequences. Gabor Motion History Image (GMHI) is another variant, which considers the Gabor filtered image, instead of the original frames. The Difference Image is a simplified process, which considers intensity differences between the first and the last frame. Motion Boundary Histogram (MBH) is another motion based algorithm, which considers the gradient in order to suppress the constant motion. Finally, the Space-Time Interest Points (STIP) algorithm, which detects local structures characterized by significant intensity variability in both space and time.

Figure 2 . 3 :

 23 Figure 2.3: CLM landmarks fitted on a facial image (created using algorithm from [175])

117 ]-

 117 Optical flow [96][240][136] -Head aversion [25] Facial Landmarks -Displacement [147][70] [36][37][161][163][164] [214][200] -Velocity [70][161][151] [163][164][214][200] -Acceleration [70][151] [163][164][214][200] -Displacement from mid-horizontal axis [145]-Coordinates[START_REF] Kaya | Eyes Whisper Depression: A CCA based Multimodal Approach[END_REF]

Figure 2 . 4 :

 24 Figure 2.4: Taxonomy of visual features utilized in the reviewed studies for depression assessment

22 2. 3

 223 [START_REF] Ghosh | A Multimodal Context-based Approach for Distress Assessment[END_REF] [START_REF] Girard | Nonverbal Social Withdrawal in Depression: Evidence from Manual and Automatic Analyses[END_REF] [89][START_REF] Scherer | Automatic Audiovisual Behavior Descriptors for Psychological Disorder Analysis[END_REF] [START_REF] Alghowinem | Eye Movement Analysis for Depression Detection[END_REF] [23][START_REF] Joshi | Multimodal Assistive Technologies for Depression Diagnosis and Monitoring[END_REF] [70][START_REF] Pampouchidou | Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text[END_REF] and the k-fold method[START_REF] Zhou | Multimodal Prediction of Psychological Disorders: Learning Verbal and Nonverbal Commonalities in Adjacency Pairs[END_REF] [START_REF] Alghowinem | Multimodal Depression Detection:Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors[END_REF] [160][START_REF] Dibeklioglu | Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding[END_REF]. In the leave-one-out procedure, for a dataset of N samples, N training sets are created of size N-1, each time consisting of all but one sample. The algorithm is then tested N times on its capacity to classify the "left-out" cases for each 1 http://www.cs.waikato.ac.nz/ml/weka/ State-of-the-Art Methods

Figure 2 . 6 :

 26 Figure 2.6: Ranking of regression algorithms employed in relevant studies

  relevant algorithms employed during different experimental tests of this thesis included: a) illumination normalization, b) face detection, c) facial landmarks detection, and d) face alignment. The specific algorithms for each preprocessing step are explained within the following subsections.

2 Figure 3 . 1 :

 231 Figure 3.1: VJ face detection: Haar features applied on a facial image

  step in the present work. Another option, in order to obtain the face region was based on the use of facial landmarks. In this respect, two different algorithms were assessed at two different stages of the thesis. Initially, and for the works published in Pampouchidou et al. [159] and Pampouchidou et al. [160] (experiments described in subsections 4.2 and 4.3) the GNDPM 1 [205] [230] was applied on the image sequences (see Fig.3.2.a).

Figure 3 . 2 :

 32 Figure 3.2: Examples of landmarks on a facial image based a) on Tanaka face-parts detection combined with GNDPM and b) OpenFace facial landmarks

(Figure 3 . 3 :

 33 Figure 3.3: LBP from Three Orthogonal Planes (Taken from:[236])

  The coarsest level (scale=1,orientation=1) is in the center, while the number of orientations doubles for every second scale

Figure 3 . 6 :

 36 Figure 3.6: Example of curvelet pseudo-images wrapped by CurveLab toolbox, for scale=4, and orientation=4

Figure 3 . 7 :Figure 3 . 8 :

 3738 Figure 3.7: Flow Curvelet transform on facial video frames, and examples for X-Z and Y-Z planes

a) 3 . 2

 32 MHI as derived from the basic algorithm, b) LMHI which relies on facial landmarks, Motion Representation and c) GMHI. The LMHI algorithm was introduced as part of our DSC-AVEC'16 participation with Pampouchidou et al. [161], which instead of considering intensities from image sequences, considers sequences of facial landmarks. In addition a quantitative comparison of different motion history variants was published in Pampouchidou et al. [162] (GMHI). More details regarding the specific motion representation algorithms are presented below with implementation examples illustrated in Fig. 3.8. MHI is a robust, yet relatively straightforward, algorithm developed to represent the motion that occurs in the course of a complete video recording with a single image [20].

( a )Figure 3 . 10 : 3 . 3 Feature Extraction 3 . 3

 a3103333 Figure 3.10: Geometric features considered in time-series

  based attributes. The appearance-based descriptors employed in terms of the present work include the LBP, Local Phase Quantization (LPQ), HOG, and the conventional image histogram. The descriptors are explained within the next subsections and are illustrated in Fig. 3.15 for the example of MHI.

Figure 3 . 11 :Figure 3 . 12 :

 311312 Figure 3.11: Example of LBP pipeline

Figure 3 .

 3 12b first row are all uniform, each of which stands for a type of contour, such as the first one (0-valued) is a bright spot, the last one of the row (8) is a dark spot, while the intermediate (1-7) represent different types of edges. In terms of experiments 4.2, 4.3, and 4.4 University of Oulu implementation 1 was used, while for experiments 4.6 and 5.2.1 the MATLAB implementation was used 2 .

Figure 3 . 13 :

 313 Figure 3.13: Example of the LPQ pipeline

and u 4 =Figure 3 . 14 :

 4314 Figure 3.14: Example of HOG pipeline

Figure 3 . 15 :

 315 Figure 3.15: Visualization of appearance-based features

Figure 3 . 16 :

 316 Figure 3.16: Architecture of VGG16

Figure 3 . 18 :

 318 Figure 3.18: Visualization of Pool 5 activations

Figure 3 . 20 :

 320 Figure 3.20: Example of kNN. Taken from:[11]

Figure 3 . 21 :

 321 Figure 3.21: Example of RF 1

Figure 3 . 22 :

 322 Figure 3.22: Example of SVM for both classification and regression approaches

Figure 4 . 1 :

 41 Figure 4.1: Experiment 1: LCBP-TOP on eye region, example for the column passing vertically from the middle of the left eye

[ 153 ] 4 . 3

 15343 ; LBP1=[1,8] and LBP2=[2,16]. LBP1 gave a 10-bin histogram, and LBP2 an 18bin histogram; the two were concatenated to a 28 element feature vector for each frame in the Frame-based Classification. Video-based Classification was performed next by computing the XZ and YZ planes, for a window of 30 subsequent frames, with an overlap of 15 frames. Therefore, a set of 30 CurveFaces of 43x43, results in 43 XZ planes of 30x43, and 43 YZ of 43x30. LBP1 Experiment 2

Figure 4 . 3 : 4 . 4 3

 43443 Figure 4.3: Experiment 3: LMHI

Figure 4 . 4 : 4 . 6

 4446 Figure 4.4: Experiment 5: Flow of the motion history variants methodology. Dashed arrows in feature extraction indicate that features are considered individually or in combination with feature fusion. MHI: Motion History Image, LMHI: Landmark Motion History Image, GMHI: Gabor Motion History Image, HOG: Histogram of Oriented Gradients, LBP: Local Binary Patterns, LPQ: Local Phase Quantization, PCA: Principal Components Analysis, SVM: Support Vector Machine

2. Protocol design -

 design Building web-based application to administer self-report questionnaires and scales -Building the emotion perception application -Selecting stimulus video-clips o Initial selection of video-clips for pilot testing o Post-processing of the video-clips * Video-editing of clips to appropriate length * Adding Greek subtitles * Adding white screen at the beginning of each clip to serve as baseline o Pilot testing the video-clips (n=10) o Statistical Analysis of emotion self-ratings from pilot study o Final selection of video-clips based on pilot study results 3. Technical setup -Selection and purchasing of suitable, portable biosignal recording device -Setting up the experiment room (camera, lights, biosignal device, PC controlling stimuli and questionnaire/scale administration and PC performing data acquisition and storage) 4. Participant recruitment -Control group: through dedicated project webpage, email list, flyers -Patient group: through referrals by clinicians in University Hospital, Psychiatric Outpatient Clinic 5. Recordings -Technical part: handling recordings -Psychologist: guiding the participants through the protocol, and conducting interviews 6. Post-Processing -Data Compression -Selection of informative parts -Condition-and diagnosis-blind annotation of facial videos by experts (psychologists) 5.1 Data Collection

• 14 -

 14 18: mild depression • 19-28: moderate depression • 30-63: severe depression

92

 92

Figure 5 . 1 :Figure 5 . 2 :

 5152 Figure 5.1: Average participant ratings of Sad video clips on the 9-point emotion scale of Gross and Levenson [95]. Upper panel: Ratings of the three video clips tested in the pilot study; Lower panel: Corresponding ratings by control group participants in the main study (n=24)

Figure 5 . 4 :Figure 5 . 5 :

 5455 Figure 5.4: Dispersion of BDI-II, STAI, and expert judgment values per group. Median (dark horizontal line), interquartile range (boxes), and range of values on BDI-II (upper panel), STAI (middle panel), and blinded expert judgment (lower panel) per participant group. The 13/14 point cutoff on BDI-II, 39/40 point cutoff on STAI, and 3.4/3.5 point on expert judgment are indicated by a blue horizontal line

5. 4 ;

 4 it shows that the best separation between groups was achieved by expert judgment ratings of depression relying solely on facial expressions. This impression was proven by sensitivity (Receiver Operating Characteristic) analyses revealing higher Area Under the Curve for expert judgment (AUC=0.913, SE=0.035, p<0.001; Fig. 5.4 lower panel) as compared to BDI-II (AUC=0.857, SE=0.051, p< 0.001; Fig. 5.4 upper panel) and STAI (AUC=0.804, SE=0.059, p< 0.001; Fig 5.4 middle panel). These analyses confirmed that

Figure 5 . 6 :Figure 5 . 6 :

 5656 Figure 5.6: Average performance (Cohen's Kappa) of the categorical assessment schemes by number of PCA components retained during preprocessing for each experimental condition. Results for gender-dependent and gender-independent schemes are displayed in the left-and right-hand columns, respectively. Colors represent different video-based features.Continue on the next page

Figure 5 . 7 :

 57 Figure 5.7: Average normalized RMSE for the continuous assessment schemes predicting BDI-II scores (upper panel), STAI scores (middle panel), and expert judgments of depression (lower panel) by number of PCA components retained during preprocessing for each experimental condition. Results for gender-based and gender-independent schemes are displayed in the left-and right-hand columns, respectively. Colors represent different video-based features.

Figure 5 . 8 :

 58 Figure 5.8: Continuous assessment: Prediction of BDI-II scores in the main experiment (gender-based mode). Upper panel: Absolute differences between actual and predicted BDI-II scores for each participant. Predicted values were computed in the context of continuous assessment analysis using recordings from the Neutral condition (HOG features). Lower panel: (b) log transformed differences between actual and predicted values; (c) Bland-Altman plot displaying the distribution of differences (y axis) over the range of BDI-II scores (x axis).

Figure 5 . 9 :

 59 Figure 5.9: Continuous assessment: Prediction of BDI-II scores in the main experiment (gender-independent mode). Upper panel: Absolute differences between actual and predicted BDI-II scores for each participant. Predicted values were computed in the context of continuous assessment analysis using recordings from the Neutral condition (VGG features). Lower panel: (b) log transformed differences between actual and predicted values; (c) Bland-Altman plot displaying the distribution of differences (y axis) over the range of BDI-II scores (x axis).

Figure 5 . 10 :

 510 Figure 5.10: Continuous assessment: Prediction of STAI scores in the main experiment (gender-based mode). Upper panel: Absolute differences between actual and predicted STAI scores for each participant. Predicted values were computed in the context of continuous assessment analysis using recordings from the Neutral condition (HOG features). Lower panel: (b) log transformed differences between actual and predicted values; (c) Bland-Altman plot displaying the distribution of differences (y axis) over the range of STAI scores (x axis).

Figure 5 . 11 :

 511 Figure 5.11: Continuous assessment: Prediction of STAI scores in the main experiment (gender-independent mode). Upper panel: Absolute differences between actual and predicted STAI scores for each participant. Predicted values were computed in the context of continuous assessment analysis using recordings from the Neutral condition (HOG features). Lower panel: (b) log transformed differences between actual and predicted values; (c) Bland-Altman plot displaying the distribution of differences (y axis) over the range of STAI scores (x axis).

Figure 5 . 12 :

 512 Figure 5.12: Continuous assessment: Prediction of expert judgment of depression in the main experiment (gender-based mode). Upper panel: Absolute differences between actual and predicted expert judgment for each participant. Predicted values were computed in the context of continuous assessment analysis using recordings from the Negative Experience Recall Condition (HOG features). Lower panel: (b) log transformed differences between actual and predicted values; (c) Bland-Altman plot displaying the distribution of differences (y axis) over the range of expert judgment values (x axis).

Figure 5 . 13 :

 513 Figure 5.13: Continuous assessment: Prediction of expert judgment of depression in the main experiment (gender-independent mode). Upper panel: Absolute differences between actual and predicted expert judgment for each participant. Predicted values were computed in the context of continuous assessment analysis using recordings from the Joy film (VGG features). Lower panel: (b) log transformed differences between actual and predicted values; (c) Bland-Altman plot displaying the distribution of differences (y axis) over the range of expert judgment values (x axis).

Figure 5 . 14 : 6 Discussion

 5146 Figure 5.14: Comparison of our approach (Proposed) with previously reported results on the AVEC'2014 Test (upper panel) and Development (lower panel) datasets.

6. 1

 1 Algorithm Development and Performance the present work, generates great promises regarding their capacity to provide clinically meaningful results for depression assessment provided with sufficiently large training data sets.

Figure 6 . 1 :

 61 Figure 6.1: Posterior probability classification model for fusion

•

  Several methodological contributions by implementing novel motion representation algorithms, namely the Local Curvelet Binary Patterns-Three Orthogonal Planes (LCBP-TOP), Local Curvelet Binary Patterns-Pairwise Orthogonal Planes (LCBP-POP), Landmark Motion History Images (LMHI), and Gabor Motion History Image (GMHI), published in several peer-reviewed IEEE [159] [160] [163] and ACM [161] conferences, and EURASIP Image and Video Processing journal (I.F.

  .3.1.3 Histogram of Oriented Gradients . . . . . . . . . . . . . . 3.3.1.4 Image Histogram . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Transfer Learning from Pretrained Networks . . . . . . . . . . . . xxx 3.3.3 Statistical Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Dimensionality Reduction with PCA . . . . . . . . . . . . . . . . . 3.4.2 Cross Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Gender Dependency . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.4 kNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.5 Random Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.6 SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 Methodology 3.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Illumination Normalization . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Face Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Facial Landmarks Detection . . . . . . . . . . . . . . . . . . . . . . 3.2 Motion Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Local Curvelet Binary Patterns . . . . . . . . . . . . . . . . . . . . 3.2.1.1 Curvelet Transform . . . . . . . . . . . . . . . . . . . . . 3.2.1.2 Three Orthogonal Planes vs Pairwise Orthogonal Planes 3.2.2 Motion History Image . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2.1 Original MHI . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2.2 Landmark Motion History Image . . . . . . . . . . . . . . 3.2.2.3 Gabor Motion History Image . . . . . . . . . . . . . . . . 3.2.3 Time-series of Geometrical Features . . . . . . . . . . . . . . . . . 3.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Appearance-based Descriptors . . . . . . . . . . . . . . . . . . . . 3.3.1.1 Local Binary Patterns . . . . . . . . . . . . . . . . . . . . 3.3.1.2 Local Phase Quantization . . . . . . . . . . . . . . . . . . 34 Preliminary Experimental Evaluation 4.1 Employed Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 AVEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 DAIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Experiment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  5.1.6 Statistical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Experimental Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . Categorical Assessment . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Continuous Assessment . . . . . . . . . . . . . . . . . . . . . . . . Algorithm Development and Performance . . . . . . . . . . . . . . . . . . 6.2 Data Related Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Future Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	CONTENTS
	5.2.2 6 Discussion
	6.1
	xxxi

Table 1 .

 1

	Eyelid activity (openings, blinking)
	Eye gaze (limited & shorter eye contact)
	Visual fixation
	Low frequency & duration of glances
	Eyebrow activity
	"Veraguth fold" *
	Frowns
	Fewer smiles
	More frequent lip presses
	Smile intensity & duration
	Mouth corners angled down
	Mouth animation
	Listening smiles (smiling while not speaking)
	Facial expression occurrence (variability & intensity)
	Sad / negative / neutral expression occurrence
	Head pose (orientation, movement)
	Body gestures (full or upper body, or body parts)
	Slumped posture
	Limp & uniform body posture
	Reduced & slowed arm and hand movements
	Shaking and/or fidgeting behavior
	Self-adaptors
	Foot tapping
	Motor variability

1: Non-verbal manifestations of depression

Table 2 .1: Search

 2 terms and web-resources employed in the current literature review

	Keywords	Web-resources
	• Depression	• ACM Digital Library [1]
	• Facial Expression	• IEEE Xplore Digital Library [4]
	• Non-verbal communication	• Elsevier [2]
	• Image Processing	• Springer [10]
	• machine learning	• Wiley Online Library [12]
	• Biomedical Imaging	• NASA [6]
	• Face	• Oxford University Press [7]
	• Emotion	• US National Library of Medicine [8]
	• Computer Vision	• Scopus [9]
		• Google Scholar [3]
		• Medpilot [5]
	• Depression	• World Health Organization [17]
	• Definition	• Survey of Health, Ageing and Retirement in
	• Types	Europe [14]
	• Frequency or rate	• Mayo Clinic [13]
	• Diagnostic tests	• National Comorbidity Survey [16]
	• Etiology and risk factors	• World Health Organization -Regional Office for
	• Predictability	Europe [15]
	Note: The first row contains keywords and web-resources that were canvassed to identify relevant
	approaches. Elements pertaining to the clinical relevance of studies are listed in the bottom row.
	fact that there was no Depression sub-challenge in the 2015 AVEC; similarly, the sharp
	rise of interest in 2013, 2014, 2016, and 2017 can be attributed to the respective AVEC
	challenges.

Table 2

 2

	.2. Partici-

Table 2 .2: Datasets employed by the reviewed studies for depression assessment Corpus Population / Total (Control / Study) Symptomatology Collection Methods Ground Truth Selection Criteria Research Question Image Resolution / Frame Rate Availability to Third Parties

 2

	Pittsburgh	Adults / 49 (-/-)	Inter	Clinical Assess-ment	DSM-IV, HAM-D>15	Detection	640x480 / 29.97	Visual & Audio Features (Expected)
	BlackDog	130 (70/60) Adults /	Comb	Assess-ment Clinical	history of mental illness, Study: DSM-IV, Control: No	Detection	24.94 800x600 /	-
					HAM-D>15
	DAIC-WOZ	Adults / 189 (-/-)	Comb	Self-report	Age, language, eye-sight	Detection, Severity	-	Visual & Audio Features, Audio Recordings, Transcripts
	AVEC	Adults 58 (-/-)	Non-social	Self-report	-	Severity	-	Full Video Recordings, Visual & Audio
								Features
	ORI	/ 8 (4/4) Adolescents	Inter	-	-	Detection	-	-
	ORYGEN	/ 30 (15/15) Adolescents	Inter	Assess-ment Clinical	depression, age 9-12 years Stage2: Depression, 2 Stage1: No	Prediction	-	-
					years after
	CHI-MEI	26 (13/13) Adults /	Comb	Assess-ment Clinical	DSSS, HAM-D	Depres-sion / Bipolar Disorder Unipolar	30 640x480 /	-
	EMORY	(-/7) Adults / 7	Inter	Assess-ment Clinical	Treatment DBS-SCC	Recovery	-/30	-
			Inter: Interpersonal, Comb: Combination, Non: Non-social

2.3 State-of-the-Art Methods

Table 1

 1

	.1, reveals that the majority of signs involve muscle activity, which accounts
	for the temporal nature of the features. Features can be further categorized as high
	or low level; high level features directly translate to human common sense, while low
	level features are based on "traditional" image processing descriptors. Depending on the
	approach, the software packages mentioned in Preprocessing could also serve as feature
	extraction methods (e.g., OpenFace, CERT, etc). In the present work feature extraction
	algorithms are grouped into those focusing on the face region and those relying on the

.3 State-of-the-Art Methods

	body region. A pictorial taxonomy of the various algorithms, including the region of
	interest on which they are applied, the features computed, and references to respective
	studies, is presented in Fig. 2.4. The various features, retrieved from relevant studies,
	are described below in detail.
	Face
	Features related to the face are classified here into features from full face, AUs, facial
	landmarks, and mouth/eyes.
	Full Face As it becomes apparent from Fig. 2.4, approaches employing feature
	extraction from the entire face region comprise the most popular category. Certain
	high level features extracted from the face as a whole concern basic emotional expres-
	sions displayed, given that depression is associated with reduced variability of emotional
	expression and greater persistence of a neutral expression. As expected, geometrical
	features, such as edges, corners, coordinates, and orientation, are often used to repre-
	sent facial expressions. Functionals derived from the time series of geometric features
	are quite popular. Some examples are average, minimum, and maximum values of dis-
	placements, velocities, or accelerations of the coordinates that define the face region as

Table 2 . 3 :

 23 Classification algorithms employed in relevant studies SVM [237] [53] [147] [89] [176] [239] [22] [24] [23] [114] [118] [115] [116] [25] [117] [184] [107] [162] [141] [174] [133] [119]

	k-Nearest Neighbour (kNN) [159] [157] [156] [160] [163] [133]

  [START_REF] Gong | Topic modeling based multi-modal depression detection[END_REF]

	0 1 2 3 4 5 6 7 8 9 10 11 12	SVR LR PLS GSR CCA RVM RFR DCNN DCNN-DNN DTL CRF OLS DM MPGI ELM LSTM-RNN SGD

Table 2 . 5 :

 25 Comparison of approaches for categorical assessment of depression grouped according to the dataset used, ranked within group based on Kappa Three classes were considered, corresponding to the annotation in the table as follows:

	Paper	Population	Features	Classification	Reported	Kappa
		(Study/-		Algorithm	Accuracy
		Control) /			(or as
		Male rate			otherwise
					noted)
			Pittsburgh
	Joshi (2013)	36 (18/18) /	Body, Full	SVM	97.2%	0.94
	[116] [114]	36.1%	Face
	Alghowinem et	38 (19/19) /	Eyes, Full	SVM	mean recall	0.89
	al. (2015) [24]	36.8%	Face		=94.7%
	Dibeklioglu et	95 (58/37) *1	Facial	LR	91.38%	0.78
	al., (2015) [70]	/40.4%	Landmarks,
			Full Face,
			Audio
	Dibeklioglu et	130	Facial	SDA	78.67%	0.73
	al., (2017) [71]	([58/35]/37) *1*2	Landmarks,
		/40.4%	Full Face,
			Audio
	Kacem et al.,	126	Full Face	SVM	70.8%	0.65
	(2018) [119]	([56/35]/35) *1*2
	Dibeklioglu et	130	Facial	Logistic	84.49%	0.63
	al., (2015) [70]	([58/35]/37) *1*2	Landmarks	Regression
		/40.4%
	Dibeklioglu et	130	Facial	SDA	72.59%	0.62
	al., (2017) [71]	([58/35]/37) *1*2	Landmarks
		/40.4%
	Dibeklioglu et	-//-	Full Face	LR	86.21%	0.60
	al., (2015) [70]
	([Moderate to Severe/Mild]/Remission)
	*3 The confusion matrix was computed based on performance metrics provided by the authors
	of the original report

*1 Reported sample size corresponds to number of sessions

*2

Continue on the next page 2.4 Selected Approaches and Meta-AnalysisTable 2 . 5 :

 25 Comparison of approaches for categorical assessment of depression (cont.) The confusion matrix was computed based on performance metrics provided by the authors of the original report

	Paper	Population	Features	Classification	Reported	Kappa
		(Study/-		Algorithm	Accuracy
		Control) /			(or as
		Male rate			otherwise
					noted)
	Cohn et al.	107 (66/41) *1	Facial	SVM	79%	0.53
	(2009) [53]	/35%	Landmarks
			BlackDog
	Joshi	60 (30/30) /	Upper Body,	SVM	91.7%	0.83
	(2013)[114] [117]	50%	Full Face,
			Audio
	Alghowinem et	-//-	Full Face,	SVM	mean recall	0.77
	al. (2016) [25]		Audio		=88.3%
	Alghowinem et	-//-	Eyes, Full	SVM	mean recall	0.57
	al. (2016) [25]		Face		=78.3%
	Alghowinem et	-//-	Full Face	SVM	mean recall	0.53
	al. (2013) [23]				=76.8%
	Joshi 2013, Joshi	-//-	Upper Body	SVM	76.7%	0.53
	et al. (2013)
	[114] [118]
	Alghowinem et	-//-	Eyes, Full	SVM	mean recall	0.53
	al. (2015) [24]		Face		=76.7%
	Alghowinem et	-//-	Eyes	SVM	mean recall	0.50
	al. (2013) [22]				=75%
			DAIC-WOZ
	Yang et al.	35 (7/28) /	Facial	Random Forest	F1 = 0.86	0.82
	(2016) [231]	45.7%	Landmarks,
			AU, Full
			Face, Audio,
			Text
	Scherer et al.	39 (14/25) /	Full Face,	SVM	89.74%	0.76 *3
	(2013) [176]	62%	Audio
	Yu et al. (2013)	130 (30/100)	Full Face,	HCRF	F1=0.644	0.58 *3
	[239]	/ 53%	Audio

*1 Reported sample size corresponds to number of sessions *2 Three classes were considered, corresponding to the annotation in the table as follows:

([Moderate to Severe/Mild]/Remission)

*3

Continue on the next page 2. LITERATURE REVIEWTable 2 . 5 :

 25 Comparison of approaches for categorical assessment of depression (cont.) The confusion matrix was computed based on performance metrics provided by the authors of the original report

	Paper	Population	Features	Classification	Reported	Kappa
		(Study/-		Algorithm	Accuracy
		Control) /			(or as
		Male rate			otherwise
					noted)
	Nasir et al.	35 (7/28) /	Facial	SVM	F1=0.63	0.55 *3
	(2016) [151]	45.7%	Landmarks
	Pampouchidou	-//-	Facial	RF	F1=0.62	0.53
	et al. (2016)		Landmarks,
	[161]		Audio
	Valstar et al.	-//-	Facial	SVM	F1=0.5	0.45 *3
	(2016) [212]		Landmarks,
			AU, Audio
	Valstar et al.	-//-	Facial	SVM	F1=0.5	0.45 *3
	(2016) [212]		Landmarks,
			AU
	Pampouchidou	-//-	Facial	RF	F1=0.5	0.39
	et al. (2016)		Landmarks
	[161]
	Scherer et al.	39 (14/25) /	Full Face	SVM	64.1%	0.20 *3
	(2013) [176]	62%
			AVEC
	Pampouchidou	200 (34/166)	Full Face	SVM	89%	0.78
	et al. (2017)
	[162]
	Senoussaoui et	50 (25/25)	Full Face	SVM	82%	0.64
	al. (2014) [182]
	Alghowinem et	32 (16/16)/	Eye, Full	SVM	mean recall	0.38
	al. (2015) [24]	28.1%	Face		=68.8%
	Pampouchidou	200 (34/166) *1	Full Face	kNN	74.5%	0.13
	et al. (2016)	/ 33%
	[160]

*1 Reported sample size corresponds to number of sessions *2 Three classes were considered, corresponding to the annotation in the table as follows:

([Moderate to Severe/Mild]/Remission)

*3

Table 2 . 6 :

 26 Approaches for categorical assessment of depression employing datasets, or combination of datasets, which have not been reported elsewhere

	Regarding the categorical approaches tested on the AVEC and DAIC-WOZ datasets,
	there are additional reasons which could have affected performance, as revealed through
	comparison of κ values cross studies. Both datasets were collected in a way that limited

3, as well as key issues that emerged from reviewed literature. The most important issue concerning the data is availability. As mentioned before, obscuring participant identity is practically impossible in raw video data compared to other modalities (e.g. speech or physiological signals). Consequently, open access is strictly prohibited, while licensing to a third party is seriously restricted. * Confusion matrix was computed, and not provided by the authors of the original research report

Table 2 . 7 :

 27 Summary of methods and results of studies employing continuous depression assessment based on the dataset provided by AVEC'13, AVEC'14 and AVEC'16. Performance metrics are given in the form of MAE / RMSE and approaches have been ranked according to reported performance primarily on Test-RMSE, and secondly on Development-RMSE

	Paper	Regression	Features	Development	Test
		AVEC 2013 -BDI prediction -Complete recordings
	Zhou et al. (2018)	CNN	Full Face, Eyes,	-	6.20 / 8.28
	[238]		Mouth
				Continue on the next page
			35

Table 2 . 7 :

 27 Summary of studies employing continuous depression assessment (cont.)

	Paper	Regression	Features	Development	Test
	Ma et al. (2017)	LDA+OLS	Full Face	-	7.26 / 8.91
	[136]
	Zhu et al. (2017)	DCNN	-	-	7.58 / 9.82
	[240]
	Kaya et al. (2014)	MPGI		-	8.254 / 10.315
	[126]
	Cummins et al.	SVR		-/ 12.08	-/ 10.45
	(2013) [59]
	Meng et al. (2013)	PLS		7.09 / 8.82	9.14 / 11.19
	[148]
	Valstar et al.	SVR		8.74 / 10.72	10.88 / 13.61
	(2013) [209]
	AVEC 2014 -BDI prediction -Northwind / Freeform tasks
	Jan et al. (2017)	PLS	Full Face	7.25 / 9.52	6.68 / 8.01
	[109]
	Zhou et al. (2018)	CNN	Full Face, Eyes,	-	6.21 / 8.39
	[238]		Mouth
	Kang et al. (2017)	DTL	Full Face	-	7.74 / 9.43
	[123]
	Zhu et al. (2017)	DCNN		-	7.47 / 9.55
	[240]
	Kaya & Salah	CCA		-	7.86 / 9.72
	(2014) [125]
	Jain et al. (2014)	SVR		6.969 / 8.167	8.399 / 10.249
	[107]
	Jan et al. (2014)	PLS+acslr		7.36 / 9.49	8.44 / 10.50
	[108]
	Kächele et al.	SVR		7.03 / 8.82	8.97 / 10.82
	(2014) [120]
	Valstar et al.	SVR		7.577 / 9.314	8.857 / 10.859
	(2014) [210]
	Senoussaoui et al.	GLM+acsrvm + SVR		6.95 / 8.52	-
	(2014) [182]
	Smailis et al.	SVR		-/ 9.07	-
	(2016) [189]
	He et al. (2015)	SVR		7.99 / 9.63	-
	[100]

Continue on the next page 2.5 Scope of the ThesisTable 2 . 7 :

 27 Summary of studies employing continuous depression assessment (cont.)

	Paper	Regression	Features	Development	Test
	Sidorov & Minker	SVR		14.843 / 17.667	-
	(2014) [184]
	Chapter 3	AVEC 2016 -PHQ-8 prediction -DAIC-WOZ
	Sun et al. (2017)	CRF	Facial	4.6 / 5.9	4.89 / 6.23
	[198]		Landmarks,
			Eyes, AU
	Stepanov et al.	LSTM-RNN	Facial	4.66 / 6.09	5.36 / 6.72
	(2017) [192]		Landmarks
	Valstar et al.	RFR		5.88 / 7.13	6.12 / 6.97
	(2016) [212] [172]
	Song et al. (2018)	CNN	Full Face, AU,	4.37 / 5.84	-
	[191]		Eyes
	Williamson et al.	GSR		5.33 / 6.45	-
	(2016) [226]
	Dham et al.	SVR	Facial	4.91 / 6.46	-
	(2017) [69]		Landmarks
	Dang et al. (2017)	GSR	AU	5.34 / 6.67	-
	[63]

  The OpenFace 1 by Baltrušaitis et al.[31] (see Fig. 3.2.b) served in the remaining published approaches developed during this thesis [161] [163] [162] [164] (described in sections 4.4, 4.5, 4.6, and 5.2.1). OpenFace, which was applied directly on video recordings, is an open source tool that provides a variety of options apart from detecting landmarks; it also detects head pose, eye gaze, HOG features, AU, and basic emotions. In some experiments (e.g. experiment 4.6) features provided by OpenFace were tested for classification in addition to the ones proposed by the thesis work. OpenFace is based on Conditional Local Neural Fields, which is an instance of CLM, advanced in the use of patch experts and optimization function. The Point Distribution Model implemented

within OpenFace captures landmark shape variations, while the patch experts capture local appearance variations of each landmark.

Table 4 . 1 :

 41 Results of Experiment 1 for four levels of depression

	Window Size (frames#) Classification accuracy
	5	41.36
	15	48.52
	30	55.42
	60	47.83
	90	49.31
	120	48.66
	150	46.32
	180	50.33

Table 4 . 2 :

 42 Experiment 1: Confusion matrix for best result (4 classes)

		Minimal Mild Moderate Severe
	Minimal	51.0%	19.6%	20.2%	9.2%
	Mild	10.7%	63.4%	16.2%	9.7%
	Moderate	11.2%	19.3%	55.0%	14.6%
	Severe	11.9%	22.1%	10.5%	55.5%

Table 4 . 3 :

 43 Experiment 2 results (%)

		Leave-One-Out	20-Fold
		Frame Video Frame Video
	Minimal/Mild	60.5	74.5	97.6	83.8
	Mild/Moderate	59.0	63.5	96.9	85.4
	Moderate/Severe	72.5	74.5	95.8	81.3

Table 4 . 4 :

 44 Experiment 2: Multi-class confusion matrix based on the Frame-based algorithm and 20-Fold cross validation (%)

		Minimal Mild Moderate Severe
	Minimal	89.3	3.6	3.7	3.4
	Mild	1.9	95.3	1.4	1.4
	Moderate	1.3	1.2	96.1	1.5
	Severe	0.8	1	0.8	97.4
	can be observed in both X and Y axes respectively.

Table 4 . 5 :

 45 Experiment 3 results: F1 score reflecting binary classification (high vs. low self-reported depressive symptomatology) in gender-independent and gender-based modes

Table 4 .

 4 Dash represents unavailable F1-score due to zero depressed individuals classified correctly.

	8: Experiment 5: Experimental results employing appearance-based descriptors
	(F1-score %)
	LBP{1,8} # LBP{2,16} # HOG	LPQ	Hist + Mean	Feature
						+ Std	Fusion
	MHI	*	*	81.9	59.3	81.9	36.6
	LMHI	*	66.4	64.9	45.8	64.8	72.7
	GMHI	*	*	80.0	69.8	80	74.0

# LBP parameters in brackets correspond to {radius, neighborhood} respectively. *

Table 4 . 9 :

 49 Experiment 5: Experimental results employing VGG features, before and after fully connected layer (F1-score %)

		Before		After		Feature
		Max	Mean	FC6	FC7	FC8	Fusion
	MHI	87.1	63.0	51.5	84.8	70.1	87.4
	LMHI	64.0	67.7	56.6	18.2	66.4	64.0
	GMHI	85.7	62.7	55.6	76.0	65.5	84.3
	Combined	64.6	52.1	51.8	74.7	46.3	65.1

Table 4 .

 4 10: Experiment 5: Confusion matrix for the best F1-Score of the proposed approach (VGG-MHI feature fusion/2 classes)

	Self-Reported	Predicted Non-depressed Depressed
	Non-depressed	102	2
	Depressed	20	76

the performance of VGG for the different configurations. The confusion matrix corresponding to the best-performing model (VGG feature fusion) is presented in Table

4

.10.

Table 4 .11: Experiment

 4

	5: Additional performance metrics for the best-performing ap-
	proach (VGG-MHI feature fusion / %)
	F1	87.4
	Accuracy	89.0
	Sensitivity	79.1
	Specificity	98.1
	Precision	97.4
	Cohen's Kappa	77.8
	.95 Confidence Interval	± 8.6

Table 4 . 13 :

 413 Summary of achievements during preliminary experiments

	Publication	Type	Name	Dataset	No	Acc	Kappa
					Classes
	Pampouchidou et	Intern.	ACM	DAIC-	2	F1=0.5 39.00%
	al. [161] (2016)	Conf.	AVEC	WOZ
	Pampouchidou et	Intern.	IEEE	AVEC	4	55.42%	41.61%
	al. [159] (2015)	Conf.	ICSIPA
	Pampouchidou et	Intern.	IEEE	AVEC	2	74.50%	13.00%
	al. [160] (2016)	Conf.	EMBC
	Pampouchidou et	Intern.	IEEE	AVEC	2	F1=58.6 20.84%
	al. [163] (2017)	Conf.	EMBC
	Pampouchidou et	Article	Springer	AVEC	2	89%	78.00%
	al. [162] (2017)		EURASIP

Table 5 .

 5

	1: Summary of work toward data collection
	1. Obtaining necessary research permissions from the
	-Bioethics Committee, University of Crete Hospital
	-National Data Protection Authority

Table 5 . 3 :

 53 Data collection protocol in the main study

	# Task	Method
	1 Acquaintance	Orally
	2 Read and sign information sheet and	In writing
	consent form
	3 Demographics & Clinical History	Webpage
	4 Relaxation (breathing exercise)	BioTrace
	5 Prolonged /a/ utterance	Orally
	6 Positioning in front of the camera	-
	(aligned with a spot on the wall)
	7 Description of positive experience	Orally
	8 Joy clip	Webpage
	9 Emotion ratings for Joy clip	Webpage
	10 Relaxation (breathing exercise)	BioTrace
	11 Reading aloud a neutral text	Webpage /
		Orally
	12 Description of negative experience	Orally
	13 Sadness clip	Webpage
	14 Emotion ratings for Sad-clip	Webpage
	15 Complete STAI	Webpage
	16 Complete BDI-II	Webpage
	17 Prolonged /a/ utterance	Orally

Table 5 .4: Camera Specifications

 5

	Resolution	2048×2048	Frame Rate	90 FPS
	Megapixels	4.1 MP	Chroma	Color
	Sensor Name	CMOSIS CMV4000-3E5 Sensor Type	CMOS
	Readout Method	Global shutter	Sensor Format	1$$
	Pixel Size	5.5 µm	Lens Mount	C-mount
		96

Table 5 . 5 :

 55 Best-performing classification schemes in gender dependent and genderindependent modes.

	Label Gender	Cond Feature PCA Kappa	F1	Acc	Prec	Rec
		mode
	Diag	Based	Neu	HOG	10	33.4% 41.7% 78.1% 100.0% 26.3%
	Diag	Indep	Pos	V19FC8	40	45.1% 61.5% 76.9% 63.2% 60.0%
	BDI	Based	Neu	FF	5	33.7% 44.4% 76.6% 85.7% 30.0%
	BDI	Indep	Neu	V16FC7	20	37.3% 56.4% 73.4% 57.9% 55.0%
	STAI	Based	Sad	V16FC7	10	36.2% 81.7% 73.8% 77.6% 86.4%
	STAI	Indep	Sad	V19FC7	10	34.5% 82.1% 73.8% 76.5% 88.6%
	Exp	Based	Joy	V19FC7	20	36.8% 57.1% 72.3% 60.0% 54.5%
	Exp	Indep	Joy	FF	5	41.1% 57.9% 75.4% 68.8% 50.0%
	Cond:Condition, Acc: Accuracy, Prec: Precision, Rec: Recall, Diag:Diagnosis, Exp: Expert Judgment, Indep:
	Independent, Neu: Neutral, Pos: Positive, FF: Feature Fusion. PCA: Number of extracted components.
						104

Table 5 . 7 :

 57 Best-performing continuous assessment schemes conducted on various partitions of the AVEC'14 dataset in gender-based and gender-independent modes. LOSO cross validation on the entire dataset protocols, and of different cultures. It is remarkable that across different setups for the AVEC dataset HOG is always the best performing feature set.

	Test Partition Task	Gender PCA Feature RMSE MAE
			mode
	Test	Freeform	Based	10	HOG	10.63	8.58
	Test	Northwind Based	10	HOG	11.45	9.92
	Test	Freeform	Indep	20	HOG	10.15	8.48
	Test	Northwind Indep	40	HOG	10.95	9.22
	Development	Freeform	Based	5	HOG	9.20	7.81
	Development	Northwind Based	40	HOG	10.74	8.91
	Development	Freeform	Indep	45	HOG	9.15	7.83
	Development	Northwind Indep	40	HOG	10.97	9.33
	LOSO	Both	Based	90	HOG	10.96	8.89
	LOSO	Both	Indep	100	HOG	10.89	8.87
	LOSO:

Table A . 1 :

 A1 Complete socio-demographics participants profiles (cont.)

	#	Diagnosis Gender Age	Years of	BDI STAI Annotation
					Education
	5	0	1	30	20	21	0.5
	6	0	0	41	16	0	2
	7	0	1	35	22	16	3
	8	0	1	43	16	1	4
	9	0	0	32	19	13	2.5
	10	0	1	53	17	11	5.33
	11	0	0	37	15	4	2.5
	12	0	1	49	17	1	2.5
	13	0	0	37	16	0	1.33
	14	0	1	54	7	15	2.5
	15	0	1	50	12	14	4.75
	16	0	1	51	18	1	3.25
	17	0	0	36	23	2	5
	18	0	0	34	25	0	1
	19	0	1	34	14	2	1
	20	0	1	56	14	10	3.25
	21	0	1	26	17	8	1.5
	22	0	0	37	16	3	3.5
	23	0	1	51	12	11	6
	24	0	0	39	17	11	2.25
	25	0	1	35	16	0	1
	26	0	1	38	16	11	1.5
	27	0	1	37	9	12	4
	28	0	1	32	16	2	1
	29	0	0	41	17	0	1.5
	30	0	0	42	12	4	1.5
	31	0	0	39	17	7	2
	32	0	0	46	17	0	3.5
	33	0	0	43	24	6	2.5
	34	0	1	40	18	4	1.5
	35	0	1	43	33	15	1
	36	0	1	24	14	8	2
	37	0	0	49	20	2	1.5
	38	0	1	43	18	5	3.5
	39	0	1	34	19	14	3.17
	40	0	0	51	12	1	1.5
						Continue on the next page

http://avec2013-db.sspnet.eu/

http://dcapswoz.ict.usc.edu/

https://www.cl.cam.ac.uk/ tb346/res/openface.html

https://www.mathworks.com/matlabcentral/fileexchange/26523-the-inface-toolbox-v2-0-forillumination-invariant-face-recognition

https://ibug.doc.ic.ac.uk/resources/gauss-newton-deformable-part-models-face-alignment/

https://github.com/TadasBaltrusaitis/OpenFace/

http://www.curvelet.org/software.html

http://www.cse.oulu.fi/wsgi/MVG/Downloads/LBPMatlab

https://www.mathworks.com/help/vision/ref/extractlbpfeatures.html

http://www.cse.oulu.fi/wsgi/MVG/Downloads/LPQMatlab

https://www.mathworks.com/help/vision/ref/extracthogfeatures.html

https://www.mathworks.com/help/images/ref/imhist.html

https://www.mathworks.com/help/nnet/ug/pretrained-convolutional-neural-networks.html

https://www.mathworks.com/help/stats/fitcknn.html 2 https://machinelearning-blog.com/2018/02/06/the-random-forest-algorithm/

https://www.mathworks.com/help/stats/fitctree.html

https://www.mathworks.com/help/stats/fitcsvm.html

https://www.mathworks.com/help/stats/fitrsvm.html

https://github.com/jmgirard/CARMA/

Acknowledgements

This has been a long project, with many challenges, but with much of support too. Beginning the acknowledgments I would like to thank the sources

Motion Representation

Feature Extraction

were utilized; this technique is known as transfer learning. In particular the MATLAB model used hereby is trained on a subset of the ImageNet database (from ILSVRC).

Both VGG16 and VGG19 are trained on more than a million images, for classifying 1000 object categories (e.g. keyboard, pencil, animals, etc), enriched by a wide range of feature representations. Figures 3.17

Statistical Descriptors

The time-series derived based on the motion representation method described in subsection 3.2.3 (e.g. Fig. 3.19) were further processed as signals to extract several statistical features. The measures selected included mean, median, mode, range, standard deviation, variance, skewness, kurtosis, energy, entropy, correlation, and interquartile range [START_REF] Acharjee | Activity Recognition System Using Inbuilt Sensors of Smart Mobile Phone and Minimizing Feature Vectors[END_REF]. Additional statistics included max, min, mad, mean frequency, and band power [START_REF] Pediaditis | Computerised Analysis of the Clinical Image of Absence Seizures[END_REF]. MATLAB implementations were employed for extracting all the statistical descriptors.

PRELIMINARY EXPERIMENTAL EVALUATION

AVEC

This dataset was introduced in the 3rd and 4th AVEC [START_REF] Valstar | AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge[END_REF], at the time being the only freely available dataset, annotated for depression, which included video recordings of participants. The dataset consisted of video feeds of undiagnosed volunteers performing the following tasks in the German language: vowel pronunciation, solving a task out loud, counting from 1 to 10, reading novel excerpts, singing, and describing a scene displayed in pictorial form. During AVEC'13 the complete recording was used for testing the performance of the different approaches, while in AVEC'14 two tasks were selected: the Northwind (reading a novel passage) and Freeform (answering a series of questions-both neutral and potentially emotionally challenging) [START_REF] Valstar | AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge[END_REF]. In the present work the subset of the two tasks, as employed in AVEC'2014, was used. In an effort to make the problem binary, all three possible pairs of depression severity subgroups were initially considered: {minimal/mild}, {mild/moderate}, {moderate/severe}. However, the 4 subsets were highly unbalanced, with the 'minimal' class having as many recordings as all the rest together. Consequently random data-sampling was used in order to keep equal number of samples from each class.

PRELIMINARY EXPERIMENTAL EVALUATION

Two main sets of tests were conducted, with either 20-Fold or LOO cross-validation methods. The 20-Fold classification was applied to individual samples, with the sets being partitioned 20 times. In this manner, 20 different randomly selected train/test sets were used in cross-validation. In the LOO method, all recordings belonging to the same participant were excluded from the training process, and were only used for testing. AVEC datasets, thus a direct comparison with the present results is not possible. In the cross-corpus approach of Alghowinem et al. [START_REF] Alghowinem | Cross-Cultural Detection of Depression from Nonverbal Behaviour[END_REF] the used set was carefully selected from the original dataset (AVEC'13) in terms of the total number and duration of recordings per participant, in order to match the other two datasets. On the other hand, in [START_REF] Senoussaoui | Model Fusion for Multimodal Depression Classification and Level Detection[END_REF] the algorithm was applied to the training data set provided by the challenge organizers (AVEC'14) and tested on the development dataset. Although the AVEC dataset has been widely employed in approaches for continuous depression assessment, the aforementioned approaches, to the best of the authors' knowledge, are the only ones attempting categorical depression assessment on the specific dataset. Results are reported for F1-score, unless indicated otherwise, which is given by equations 2.5, 2.3, and 2.4

Summary

Classification results using previously available data sets are listed in Table 4.13 for each of the five preliminary experiments. In total 4 conference papers were produced, three published by IEEE, while the one by ACM was in terms of the AVEC'16 Depression Sub-Challenge, with the participation being included in the finalists of the challenge. Finally, one article was produced as well, in Springer EURASIP Image and Video Processing. 1 Conducted by psychologists based on participants' face videos alone education than the control group. The two groups did not differ significantly on the percentage of women which was higher than the percentage of men in both groups and especially among patients (85 vs. 15%) in accordance with the literature [START_REF] Bogner | Are higher rates of depression inwomen accounted for by differential symptomreporting?[END_REF]. As expected, self-reported depression and anxiety were higher for the patients (BDI-II, STAI, and annotation). Three participants did not complete one task (each for different reasons and a different task), bringing the total number of recordings to 322 (out of a possible of 325: 65 participants × 5 tasks).

MAIN EXPERIMENT

Psychological measurements and experimental procedures

In designing the study special care was given to two key aspects:

(a) Tools employed to assess depression-related symptoms in everyday life of participants (including those not meeting formal criteria for MDD or other mood disorder), and (b) Defining the experimental conditions to elicit specific emotions in the laboratory under the assumption that the quality and intensity of such emotions and facial expressions of emotions will be altered in the presence of significant depressive symptomatology [START_REF] Davies | Facial expression to emotional stimuli in non-psychotic disorders: A systematic review and meta-analysis[END_REF]. We adopted techniques that involved "human-human interaction" as well as "human-computer interaction" [START_REF] Kächele | The influence of annotation, corpus design, and evaluation on the outcome of automatic classification of human emotions[END_REF] in both "social" and "nonsocial" context as described in Subsections of 5.1.3.2 and summarized in Table 5.3. both best-performing models underestimated self-reported anxiety severity in a two participants scoring >50 points in the scale. Both models relied on HOG features derived from the Neutral condition. With respect to expert judgment values (Fig. 5.12 -5.13) was achieved marginally by HOG features derived from the Negative Experience Recall condition in gender-based mode. In this analysis, the model significantly underestimated expert ratings of depression severity in two participants.

MAIN EXPERIMENT

The generalizability of the proposed method was tested on the AVEC'14 dataset comprised of 300 video recordings from 83 participants using continuous assessment against the only available continuous variable (i.e., individual BDI-II total scores). Comparison of results presented in Tables 5.6 and 5.7 show that our method relying on HOG features in gender-based mode produces similar results across datasets: RMSE=10.59 and MAE=7.46 in our dataset (neutral condition); RMSE=11.45/10.74 and MAE=7.81/9.92 (test/development data sets) in the comparable condition of the AVEC dataset (Northwind/passage reading).

Results using the AVEC data set permit comparison of our method with previously reported approaches listed in Fig. 5.14. This comparison shows that although the proposed methodology does not outperform the other approaches, still succeeds in performing at the same levels as state-of-the-art methods, and on top of that it sustains the same performance over two different datasets, with completely different specifications,

MAIN EXPERIMENT

Research Questions Addressed

Several research questions that were set at the beginning of this research were addressed in this thesis as described in some detail below.

Construct a clinically valid dataset, comprising diagnosed patients as well as healthy control individuals

This question was successfully addressed, overcoming several, mostly expected, obstacles in this line of work. In the end data were collected from 65 individuals (45 controls and 20 patients), providing 322 video recordings across 5 experimental conditions. Facial image, speech, and BVP and GSR were recorded through video, audio, and physiological signals respectively.

CONCLUSIONS

• The proposed methodology, as presented in the main study was validated across datasets (AVEC and our dataset) presenting a competitive performance for predicting individual BDI-II scores, submitted for publication to the IEEE Journal of Biomedical and Health Informatics (I.F. 3.85 / Q1).

Appendix A

Complete List socio-demographics participants profiles

In the following table the detailed list of socio-demographics participants profiles is presented. The first column correspond to the serial number of the participant. The column 'Diagnosis' takes binary values, and corresponds to the participant being a diagnosed for depression (1) or not (0). Gender takes the value 0 for male participants and 1 for female, while the column age is for how many years old they are.