Thomas Treal

Philip L Jackson

Nicolas Vignais

Jean Jeuvrey

Aurore Meugnot

Natural human

Keywords: Idle motion, Biological motion, Empathic response, Pain communication, Virtual character, Behavioural realism

come

Introduction

Do you remember how you reacted when you saw a street artist mimicking a stone statue for the first time? Perhaps you tried to detect blinking or small variations in their head or body posture that would help you differentiate between a human or a stone statue? These subtle movements are usually ignored in social interactions, except in this type of situation where the humanness of the person is ambiguous. In particular, this anecdote illustrates how much these non-communicative movements-coined idle motion by Egges and collaborators [START_REF] Egges | Personalised real-time idle motion synthesis[END_REF]-may be a core ingredient when animating a virtual character that is talking or displaying a dynamic facial expression, and that is being used as a research tool to study emotion communication in humans (i.e., emotion recognition, empathy, emotional contagion, etc.). The ability of virtual characters to create an ecological interaction while maintaining high control of the interaction is well accepted [START_REF] De Gelder | Virtual reality and the new psychophysics[END_REF][START_REF] Egges | Personalised real-time idle motion synthesis[END_REF][START_REF] Pan | Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape[END_REF]Wykowska et al., 2016). The current challenge is not necessarily to create a fully interactive virtual character, but one that is realistic enough for participants to accept that it might behave like a real person, such as feeling and expressing emotions. Indeed, its emotion must be believable and provoke a spontaneous and socially expected reaction from participants in order to capture the unintentional and automatic nature of emotional processes [START_REF] De Gelder | Virtual reality and the new psychophysics[END_REF].

However, a virtual character with a fixed body may be perceived spontaneously as a fictional character rather than a virtual social agent. Inconsistency between a fixed body and a dynamic face could trigger a sense of eeriness towards the virtual agent and hinder interaction with it. Thus, idle motion is generally implemented by default on virtual characters to increase their behavioural realism [START_REF] Groom | Evaluating the effects of behavioral realism in embodied agents[END_REF] or to make the character seem 'alive' in a neutral (control) condition without introducing confusing factors [START_REF] Buisine | The Role of Body Postures in the Recognition of Emotions in Contextually Rich Scenarios[END_REF][START_REF] De Melo | Influence of autonomic signals on perception of emotions in embodied agents[END_REF][START_REF] Krämer | Social snacking with a virtual agent -On the interrelation of need to belong and effects of social responsiveness when interacting with artificial entities[END_REF]. Idle motion includes a "varied range of features such as subtle variations or shift in body posture caused by small muscle contraction, eye movements or blinking, breathing" [START_REF] Egges | Personalised real-time idle motion synthesis[END_REF][START_REF] Kocoń | Idle Motion Synthesis of Human Head and Face in Virtual Reality Environment[END_REF]. For instance, in Groom and colleagues (2009), the conversational agent "blinked and moved her eyes in a natural fashion". In another study [START_REF] De Melo | Influence of autonomic signals on perception of emotions in embodied agents[END_REF], the agent in the control condition of Experiment 2 "displays a neutral respiration". In [START_REF] Krämer | Social snacking with a virtual agent -On the interrelation of need to belong and effects of social responsiveness when interacting with artificial entities[END_REF], "the virtual human did not show any kind of feedback but merely slight and randomised posture shifts and eye blinking in order to appear alive (idle behavior)". Surprisingly, except for one study [START_REF] Treal | Combining trunk movement and facial expression enhances the perceived intensity and believability of an avatar's pain expression[END_REF], no empirical evidence supports that these subtle movements influence the perception of emotions expressed by a virtual character. Treal and collaborators (2020) demonstrated that the facial pain expression of a virtual character was perceived as more intense and more believable when a subtle to-and-fro movement mimicking human dynamic equilibrium (postural oscillations) was added to its trunk, compared with a static condition.

Although the method was artificial (i.e., the movement was mechanical and manually implemented in the animation software), these results prompted further exploration into how much idle motion might contribute to make the facial expressions of a virtual person more believable.

Such work is of great interest given the growing number of platforms using virtual humans as a therapeutic tool for clinical populations with social cognition deficits [START_REF] Brunet-Gouet | Editorial: Advances in Virtual Agents and Affective Computing for the Understanding and Remediation of Social Cognitive Disorders[END_REF], or in serious games to improve clinical reasoning skills in healthcare professionals [START_REF] Blanié | Comparative value of a simulation by gaming and a traditional teaching method to improve clinical reasoning skills necessary to detect patient deterioration: A randomized study in nursing students[END_REF][START_REF] Dukes | A multitasking healthcare simulation for training nurses to recognize patient deterioration[END_REF][START_REF] Wang | A Systematic Review of Serious Games in Training Health Care Professionals: Simulation in Healthcare[END_REF]Wu et al., 2014). For instance, by improving a virtual character's anthropomorphism, a higher affective reaction may be elicited towards its pain, which may be desirable in order to train healthcare professionals in emotional regulation. This is one of the three components of empathy and a critical skill in medical care. Empathy, which has been defined as the ability to share and understand others' emotion or thinking, and feel motivated to help [START_REF] Decety | Dissecting the Neural Mechanisms Mediating Empathy[END_REF], is underlaid by two other main processes [START_REF] Cuff | Empathy : A Review of the Concept[END_REF][START_REF] Decety | The Functional Architecture of Human Empathy[END_REF][START_REF] Shamay-Tsoory | The Neural Bases for Empathy[END_REF]Zaki & Oschner, 2012;[START_REF] Tremblay | I Can But I Shall Not Always Be Empathic[END_REF]: (1) perspective-taking, which refers to explicit reasoning about others' mental states; and (2) an automatic and emotional component called affective resonance, resulting in the sense of mixed emotions, e.g. distress, sympathy or compassion when seeing pain in others [START_REF] Decety | Dissecting the Neural Mechanisms Mediating Empathy[END_REF][START_REF] Decety | Why empathy has a beneficial impact on others in medicine: Unifying theories[END_REF][START_REF] Goubert | Facing others in pain: The effects of empathy[END_REF]. The above-mentioned mechanism of emotional regulation is then paramount to contain one's own emotional reaction to avoid being overwhelmed by others' emotions [START_REF] Decety | The Functional Architecture of Human Empathy[END_REF]Jackson et al., 2015a). However, it can be challenging for health professionals to both maintain compassionate care and cope with self-oriented unpleasant emotions provoked by emotionally distressing situations [START_REF] Gleichgerrcht | Empathy in Clinical Practice: How Individual Dispositions, Gender, and Experience Moderate Empathic Concern, Burnout, and Emotional Distress in Physicians[END_REF], 2014;[START_REF] Shapiro | Walking a mile in their patients' shoes: Empathy and othering in medical students' education[END_REF]. Thus, platforms using virtual patients may be an innovative tool to help caregivers better recognise their own emotional reactions and develop strategies enabling them to reduce emotional contamination.

The present study aimed to examine the impact of idle motion in increasing the humanness of a virtual agent by using a new and ecological paradigm based on human motion capture to reproduce natural postural oscillations in a virtual character. This method captured the complex pattern of body movements reflecting postural oscillations, and in particular, included respiratory movements that are closely linked to postural control [START_REF] Hunter | Respiratory components of human postural sway[END_REF]. Moreover, a recent study showed that visual attention was more sensitive to biological rather than mechanical motion, whether toward a human or a virtual body (Williams et al., 2019). In line with this work, biological movement seems critical to elicit acceptance that a virtual character might appear to be real at first glance. While in past studies, idle motion has usually been generated using algorithms [START_REF] Egges | Personalised real-time idle motion synthesis[END_REF][START_REF] Kocoń | Idle Motion Synthesis of Human Head and Face in Virtual Reality Environment[END_REF] or a hybrid system combining motion capture data with procedural motion generation [START_REF] Luo | Augmenting Gesture Animation with Motion Capture Data to Provide Full-Body Engagement[END_REF], to our knowledge, applying unconstrained, natural human postural oscillations to a virtual character paired with dynamic facial expression has not been previously examined. It was hypothesised that these subtle movements (i.e., natural postural oscillations) would enhance the humanness of the virtual character, and thus significantly influence participants' empathic response towards its facial pain expression. In this paper, the term empathic response is used to describe the behavioural manifestation of empathy resulting from a specific combination of its inner emotional and cognitive components. As proposed by some authors, an empathic response may be characterised according to two facets: emotion identification and affect sharing [START_REF] Coll | Are we really measuring empathy? Proposal for a new measurement framework[END_REF]; see also [START_REF] Bird | The self to other model of empathy: Providing a new framework for understanding empathy impairments in psychopathy, autism, and alexithymia[END_REF]. Namely, a person would be considered empathic if he/she correctly identifies and shares another person's emotion. These two distinct yet interacting facets of empathic response were assessed through two experiments: in Experiment 1, the participants rated the intensity of a dynamic facial pain expression in a virtual character whose body was or was not animated with natural postural oscillations. It was expected that they would perceive the virtual character's pain as more intense in the idle condition compared with the still condition. Using the same materials, Experiment 2 aimed to determine whether participants would report greater affective sharing when seeing a virtual character's pain expression in the presence of idle motion, compared with a still condition. It was predicted that participants would report being touched more by the character's pain expression when its body was animated with natural postural oscillations (idle condition) than when it was in the still condition.

Experiment 1 2.1 Method

Participants

Thirty-two young adults from the Faculty of Sport Science of Paris-Saclay University took part in the experiment (mean age = 18.31 years, SD = 0.53, range = 18-19 years, 16 women). Each participant had normal or corrected-to-normal vision. The study was approved by the Ethics Committee of Paris-Saclay University (#CER-Paris-Saclay-2020-147) and the participants gave written informed consent to take part in this study.

Material and Tasks

The stimuli were composed of ten video clips (duration = 4 seconds) showing the upper body of a male virtual character (Figure 1) from the Empathy-Enhancing Virtual Evolving Environment (EEVEE) (Jackson et al., 2015b). The video clips were created using Blender® (Blender Foundation, Amsterdam, Netherlands). The virtual character was animated with a dynamic facial pain expression at five different intensities (20%, 40%, 60%, 80% and 100%), either with body animation (idle condition, see the Body Animation section below for a detailed description of the method) or without body animation (still condition).

The video clips are available online via a link in the Supplementary Material section.

Facial pain expression animation. The character's facial pain expression was created using Action Units (AUs) according to the Facial Action Coding System (FACS) [START_REF] Ekman | Manual for the facial action coding system[END_REF]. The characteristic AUs from the pain expression (AU4 (brow lowerer), AU6 (cheek raiser), AU7 (lid tightener), AU9 (nose wrinkler), AU10 (upper lip raiser) and AU43 (eyes closed)); and five occasional AUs (AU12 (lip corner puller), AU20 (lip stretcher), AU25 (lips part), AU26 (jaw drop) and AU27 (mouth stretch)) [START_REF] Prkachin | The consistency of facial expressions of pain : A comparison across modalities[END_REF] were manipulated using Blender to create five intensities of facial expression (20%, 40%, 60%, 80% and 100% of their maximum contraction). Each video began with the character showing a neutral expression (AUs set at 0). The level of the AUs were increased linearly to reach the set intensity over one second. The pain expression was then maintained for three seconds until the end of the clip (Figure 1). Body Animation. For the idle motion condition, the character's body was animated with pre-recorded human postural oscillations, i.e., subtle motion specific to dynamic human equilibrium and breathing. For the still condition, the character's body was fixed at the initial posture of the person in the pre-recorded motion capture sequence.

Motion capture data: kinematic data were collected on a young man (age = 20, with anthropometric characteristics similar to the virtual character). He was instructed to stand naturally with his arms alongside his body without moving his feet and looking straight ahead. Data were collected using a Qualisys optoelectronic motion capture system (Göteborg, Sweden) composed of eight infrared cameras sampled at 50 Hz. Markers were placed on the following anatomical landmarks: head of the fifth and first metatarsals, medial and lateral malleolus, medial and lateral femoral condyles, anterior superior iliac spine, zyphoid process at the lower part of the sternum, L5, C7, nasion, acromion process, olecranon, processus styloideus of the ulna and radius (Le [START_REF] Goic | An Initial Passive Phase That Limits the Time to Recover and Emphasizes the Role of Proprioceptive Information[END_REF]. Three more markers were added to record respiratory movements: two on the sternal extremity of the clavicle and one on the navel. The markers' positions were imported and applied on a skeleton to animate the virtual character using Blender® (Blender Foundation, Amsterdam, Netherlands). The corporeal envelope of the character was mapped on the skeleton in Blender and the link between the envelope and the skeleton was set using the "automatic weighted" option available in Blender.

Judgement of Pain Intensity. The task was programmed using Psychopy [START_REF] Peirce | PsychoPy2: Experiments in behavior made easy[END_REF] and displayed on a computer screen (resolution = 1920 x 1080) at a viewing distance of 70 cm from the participants. Each trial began with the presentation of a video clip (duration = 4 seconds). After the video clip, the labels "Mild pain" or "Very intense pain" [In French "Douleur peu intense" and "Douleur très intense"] were displayed on each side of the screen.

The participants were asked to report the intensity of the virtual character's pain expression by clicking one of the two labels. Each stimulus was displayed ten times for a total of 100 trials for the task (ten trials * five facial expressions * two body conditions). The order of the stimuli was randomised across the task. After every 20 trials, a pause screen was displayed and the participants clicked on a rectangle in the centre of the screen when they were ready to resume the task. Before the experimental phase, the participants performed a familiarisation phase composed of ten trials presenting each stimulus randomly.

Data analysis

The Very Intense Pain rating, corresponding to the percentage of "Very Intense Pain" responses, was calculated for each degree of facial pain expression (20%, 40%, 60%, 80% and 100%) in the two body conditions (still and idle). All statistical analyses were performed using Python libraries (LMFIT [START_REF] Newville | LMFIT: Non-Linear Least-Square Minimization and Curve-Fitting for Python[END_REF] and SciPy [START_REF] Virtanen | SciPy 1.0: Fundamental algorithms for scientific computing in Python[END_REF].

A logistic function was fitted on each participant's Very Intense Pain rating using the formula: y = 1/(1+exp(-(x-PSE)/JND)) in order to collect the Point of Subjective Equality (PSE) and the Just Noticeable Difference (JND) (see Figure 2). The PSE corresponded to the theoretical facial expression intensity at which a participant switched from "Mild Pain" to "Very Intense Pain" (i.e., the intensity of facial expression for which a participant responded "Very Intense Pain" 50% of the time). The JND determined the precision of the categorisation and corresponded to the minimum AU contractions added to the PSE to trigger a quasi-systematic "Very Intense Pain" response. Two participants' Very Intense Pain ratings were excluded from analysis because they did not fit the logistic analysis. Thus, the final sample for analysis of the psychophysical variables (PSE and JND) included 30 participants. Two-tailed paired t-tests were performed on the PSE and JND to compare the perception of facial pain expression in the idle vs. still conditions.

Results

The two-tailed paired t-test on PSE revealed a significant difference between the idle and still conditions [t(29) = -2.22, p = .035, d = -.41]. This result indicated that a lower intensity of facial pain expression was required to be categorised frequently as "Very Intense Pain" in the idle condition (PSE mean = 56.19%,95% CI [52.59,59.78]) compared with the still condition (PSE mean = 58.14%, 95% CI [54.31,61.97]). The two-tailed paired t-test on the JND also revealed a significant difference between idle and still conditions [t(29) = -2.24, p = .033, d = -.42], indicating that the participants were more precise in categorising intensity in the idle condition (mean JND = 2.78%, 95% CI [1.57,3.98]) than in the still condition (mean JND = 4.08%, 95% CI [2.46, 5.69]) (see Figures 2 and3).

Summary

The results of this first experiment showed that the presence of idle motion influenced the perceived intensity of the virtual character's facial pain expression. Indeed, the significantly lower PSE in the idle condition compared with the still condition indicated that a lower intensity of facial pain expression was necessary to be frequently categorised as "Very Intense Pain" in the presence of postural oscillations compared with the still condition. The significantly lower JND in the idle condition compared with the still condition suggested that a lower intensity of facial expression was needed to reach a quasi-systematic "Very Intense Pain" response threshold in the presence of postural oscillations.

These results showed that identification of the virtual character's pain was modulated by the presence of postural oscillations, compared with a still condition. Since these movements are non-communicative, it is likely that natural postural oscillations increased behavioural realism by creating the illusion that the virtual character is alive, which may increase its anthropomorphism and favour perception of its pain expression.

Experiment 2

Experiment 2 aimed to determine whether adding natural postural oscillations to the virtual character might enhance affect sharing when considering its pain expression. In contrast with [START_REF] Coll | Are we really measuring empathy? Proposal for a new measurement framework[END_REF], this construct is not restricted to emotional contagion, yet it reflects affective arousal in response to the valence and intensity of another's emotion.

Consequently, affect sharing would elicit a varied range of feelings, including self-oriented (e.g., distress) or other-oriented affective responses (e.g., compassion, sympathy) [START_REF] Goubert | Facing others in pain: The effects of empathy[END_REF], which depend on contextual and individual factors (e.g., individual capacity of emotion regulation). Concretely, affect sharing was assessed in Experiment 2 by asking the participants to report whether they felt affectively aroused (i.e., "Touched" [In French "Touché(e)"]) when viewing pain in the virtual character. It was assumed that the participants would have a greater subjective sense of being touched by the virtual character's pain in the idle condition compared with the still condition.

Method

Participants

A different group of 34 young adults from the Faculty of Sport Science of Paris-Saclay University took part in the experiment (mean age = 18.56 years, SD = 1.35, range = 18-25 years, 17 women). Every participant had normal or corrected-to-normal vision. The study was approved by the Ethics Committee of Paris-Saclay University (#CER-Paris-Saclay-2020-147) and the participants gave written informed consent to take part in this study.

Material and Tasks

Self-report of affect sharing. The task design was the same as in Experiment 1 using the same video clips. Following the video clip, the labels "Not Touched" or "Touched" [In French "Pas Touché(e)" and "Touché(e)"] were presented on either side of the screen. The participants were asked to report whether they were touched by the virtual character's pain by clicking on one of the two labels. Each stimulus was displayed ten times for a total of 100 trials for the task (ten trials * five facial expressions * two body conditions). The order of stimuli was randomised across the task. Every 20 trials, a pause screen was displayed and the participants clicked on a rectangle in the centre of the screen to resume the task. Before the experimental phase, the participants performed a familiarisation phase composed of ten trials presenting each stimulus in random order.

Data Analysis

The Sense of Being Touched rating, corresponding to the percentage of "Touched" responses, was calculated for each degree of facial pain expression (20%, 40%, 60%, 80% and 100%) in the two body conditions (still and idle). All statistical analyses were performed using Python libraries (LMFIT [START_REF] Newville | LMFIT: Non-Linear Least-Square Minimization and Curve-Fitting for Python[END_REF] and SciPy [START_REF] Virtanen | SciPy 1.0: Fundamental algorithms for scientific computing in Python[END_REF].

A logistic function was fitted on the Sense of Being Touched rating for each participant using the same formula as in Experiment 1: y = 1/(1+exp(-(x-PSE)/JND)) in order to collect the Point of Subjective Equality (PSE) and the Just Noticeable Difference (JND) (see Figure 2 and Experiment 1 for details about this indices). Four participants were excluded from analysis because their Sense of Being Touched rating did not fit the logistic analysis. Thus, the final sample for the analysis on the psychophysical variables (PSE and JND) included 30 participants. Two-tailed paired t-tests were performed on PSE and JND to compare the perception of facial pain expression in idle vs. still conditions.

Results

The two-tailed paired t-test on PSE revealed a significant difference between the idle and still conditions [t(29) = -2.15 ; p = .040, d = -.40], which indicated that the participants reported being "Touched" by the pain expression at a lower intensity of facial expression in the idle condition (PSE mean = 53.9 %, 95% CI [48.9%, 58.8%]) than in the still condition (PSE mean = 56.9%, 95% CI [51.0%, 62.8%]). The two-tailed paired t-test on JND did not reveal a significant difference between idle (mean JND = 4.1%, 95% CI [2.4%, 5.8%]) and still (mean JND = 3.6%, 95% CI [2.1%, 5.2%]) conditions [t(29) = 0.55 ; p = .59, d = .10] (see Figures 4 and5).

Summary

The results showed that the participants were aroused more by the virtual character's pain expression in the presence of idle motion (natural postural oscillations) than in the still condition. Indeed, the PSE was significantly lower in the idle condition than in the still condition, indicating that a lower intensity of facial pain expression was necessary to touch participants in the idle condition than in the still condition. By contrast, the JND did not differ between the idle and still conditions.

In agreement with Experiment 1, it is likely that the presence of postural oscillations made the avatar seem more human and its emotion more genuine. Thus, the participants would be more sensitive to its pain expression, as reflected by their enhanced empathic response. They would identify its pain as being more intense and would share more in its distress.

General discussion

The present study revealed the potential of idle motion, and natural postural oscillations in particular, to increase behavioural realism in a virtual character, resulting in an increased empathic response towards its facial pain expression. Indeed, the participants perceived its pain as more intense (Experiment 1) and reported being more touched (Experiment 2) when seeing its facial pain expression in the presence of natural postural oscillations compared with the still condition. These results confirm previous outcomes obtained by Treal and collaborators (2020), who showed that participants perceived the virtual character's facial pain expression as more intense and more believable when its trunk was animated with a subtle mechanical oscillatory movement. In the same vein, some studies on robots revealed that robots' gestures during idle moments (i.e., moments when the robot is not performing a task) may enhance its anthropomorphism [START_REF] Cuijpers | Motions of Robots Matter! The Social Effects of Idle and Meaningful Motions[END_REF] or 'friendliness' [START_REF] Asselborn | Keep on moving! Exploring anthropomorphic effects of motion during idle moments[END_REF].

The originality of the present work was the use of human motion-captured postural oscillations, based on the bulk of evidence showing that the human visual system is particularly sensitive to biological motion, that would be detected in an early and preattentive stage [START_REF] Hirai | The two-process theory of biological motion processing[END_REF] for a review and a recent theoretical model). The social function of biological motion in the context of voluntary action is well documented (e.g., [START_REF] Alaerts | Action and Emotion Recognition from Point Light Displays: An Investigation of Gender Differences[END_REF][START_REF] Dayan | Motion cues modulate responses to emotion in movies[END_REF][START_REF] Dittrich | Perception of Emotion from Dynamic Point-Light Displays Represented in Dance: Perception[END_REF][START_REF] Pavlova | Biological Motion Processing as a Hallmark of Social Cognition[END_REF][START_REF] Troje | What is biological motion? Definition, stimuli, and paradigms[END_REF]. To our knowledge, this is the first study to demonstrate the contribution of postural oscillations in emotion perception research when kinematics contain no socially relevant information (i.e., automatic movements like postural oscillations). It is proposed that idle motion implemented naturally rather than mechanically would be critical to elicit a 'suspension of disbelief', a term coined by [START_REF] De Gelder | Virtual reality and the new psychophysics[END_REF] to describe acceptance that a virtual character might feel emotion despite knowing it is virtual. This hypothesis is supported by recent studies [START_REF] Piwek | Empirical evaluation of the uncanny valley hypothesis fails to confirm the predicted effect of motion[END_REF]Williams et al., 2019) showing differences in emotional responses from participants when observing an agent moving naturally or mechanically. Piwek and colleagues (2014) asked participants to judge the acceptability of different types of virtual characters animated with biological motion or with the same motion distorted at different levels (mechanical motion). They showed that the implementation of biological motion enhanced the characters' acceptability compared with mechanical motion. Williams et al. (2019) showed that a human body moving naturally elicited both greater visual attentional engagement and autonomic arousal (assessed by pupil size changes) compared with a human body moving mechanically.

Thus, the presence of biological motion in virtual characters might provoke an early unconscious identification of the virtual agent as a natural object (i.e., a human) rather than an artificial one (i.e., a cartoon). This early, pre-attentive identification would significantly influence the latter stages of facial pain expression processing, and ultimately the believability of its pain experience, suggesting a higher empathic response to its pain. This view is in agreement with a recent hierarchical model of social perception [START_REF] Teufel | A hierarchical model of social perception: Psychophysical evidence suggests late rather than early integration of visual information from facial expression and body posture[END_REF] supporting the idea that different sources of visual information would be initially processed and then integrated in later stages of the visual system to shape a whole-person percept that would be consciously experienced. In other words, natural postural oscillations might be a critical component to trigger a more spontaneous emotional reaction, which in turn will favour a greater empathic response towards the pain expression of a virtual character. Based on the theoretical models of empathy establishing a clear difference in timing and networks between its components [START_REF] Coll | Meta-analysis of ERP investigations of pain empathy underlines methodological issues in ERP research[END_REF][START_REF] Shamay-Tsoory | The Neural Bases for Empathy[END_REF], it is proposed that the increased empathic response when the virtual human is animated with natural postural oscillations might be due to a greater affective resonance provoked by its facial pain expression, rather than stemming from a modulation of the cognitive component (i.e., perspective taking) of empathy. Indeed, in a seminal electroencephalography study, [START_REF] Fan | Temporal dynamic of neural mechanisms involved in empathy for pain : An event-related brain potential study[END_REF] investigated the temporal dynamics of neural activity underlying empathy processes using the event-related potential (ERP) technique. Participants were shown real pictures or cartoons of hands in painful situations. The results revealed that only the early brain response reflecting affective sharing was modulated by the type of painful hand stimuli, as indicated by a later early affective response evoked by painful cartoons compared with human pictures of painful hands. In the same vein, a recent study using fMRI showed that dynamic fear-expressing humans activated stronger brain regions associated with affective resonance (i.e., ventral anterior and posterior cingulate gyrus, the anterior insula and the inferior frontal gyrus) compared with similar face expressions in virtual characters (avatars) [START_REF] Kegel | Dynamic human and avatar facial expressions elicit differential brain responses[END_REF]. These works support the idea that the humanness of a virtual character would affect the mechanism of affective resonance more specifically. Nevertheless, future investigations using neuroimaging methods are needed to explore these theoretical proposals, i.e., the idea of an unconscious primary categorisation of an animated agent as human (i.e., natural object) resulting in greater affective resonance when empathising with its pain, due to the presence of natural postural oscillations.

The positive influence of naturalistic (i.e., based on human motion capture) idle motion on the perception of a virtual character's facial expression needs to be validated with other emotions (e.g., joy, disgust, fear, surprise, etc.) and demonstrated in more complex tasks including contextual and situational information (e.g., a narrative item before stimuli presentation, a contextual background or even more complex scenarios). As stated by scholars in affective neuroscience, the effect of context on the perception of others' emotion is piecemeal and not sufficiently accounted for in the scientific literature [START_REF] Barrett | Emotional Expressions Reconsidered : Challenges to Inferring Emotion From Human Facial Movements: Psychological Science in the Public Interest[END_REF]. In the same vein, some authors in affective computing are using virtual settings (i.e., video games) in fundamental research, showing that participants rely on contextual information even more than facial expression to judge a virtual character's emotion [START_REF] Lankes | An experimental setting to measure contextual perception of embodied conversational agents[END_REF][START_REF] Lankes | Using embodied conversational agents in video games to investigate emotional facial expressions[END_REF]. Moreover, a recent study using a medical virtual system for nursing education questioned the impact of three levels of behavioural realism on nurse trainees' visual attention (assessed by gaze measures) when interacting with a virtual patient to medically assess it as its health condition deteriorated [START_REF] Volonte | Empirical Evaluation of Virtual Human Conversational and Affective Animations on Visual Attention in Inter-Personal Simulations[END_REF]. The conditions of behavioural animation were: (A) no animation, i.e., the virtual patient was static; (B) passive animation with idle motions involving interactions with the environment (e.g., hand and head scratching, looking at TV); (C) communicative (or conversational) animation in which the virtual patient also gazed at participants, displayed communicative lip-synced speech, and facial expressions and gestures, in addition to passive animation). The authors observed that the type of animations did not significantly change visual attention.

However, in this study, idle motion was artificially implemented using morphing (see Wu et al., 2014, for a description of the animation method). Thus, the debate remains open on the influence of biological postural oscillations on the way one would empathise with a virtual person in ecological situations, and its use to enhance the quality of human-machine interactions in an applied domain. Finally, the present study used only a male character, while it is evidenced that the perception of pain in others can be modulated by the gender of the person in pain (e.g., [START_REF] Coll | The Role of Gender in the Interaction Between Self-Pain and the Perception of Pain in Others[END_REF]. So, it would be of interest to examine whether the potential of natural postural oscillations to enhance the genuineness of a virtual human's facial pain expression would be modulated by the gender of a virtual character relative to the gender of the participant.

Conclusion

The use of highly realistic virtual characters to study emotion perception and social cognition offers the possibility to create an interactive, yet well-controlled interaction. Idle motion appears to be a key component of behavioural realism. Surprisingly, there is no experimental data supporting this consensual stance, except for one recent study [START_REF] Treal | Combining trunk movement and facial expression enhances the perceived intensity and believability of an avatar's pain expression[END_REF] and the present work that specifically demonstrated the potential of natural postural oscillations to enhance the empathic response towards pain expressed by a virtual character.

It was argued that human motion capture would be particularly relevant for idle movement animation, which is by essence a 'by default component', not designed to be interactive and flexible according to the user's responses. Overall, further research using contextualised settings (i.e., rich and lifelike situations) will help elucidate the contribution of natural idle motion when perceiving emotion in a virtual agent, and more generally when interacting with it.

Figure 1 -

 1 Figure 1 -Left side: virtual character at the beginning of the clip (neutral expression). Right side: virtual character expressing pain (AUs at 60% of maximum contraction)

Figure 2 -

 2 Figure 2 -Logistic modelling of "Very Intense Pain" responses according to facial pain expression intensity in the still and idle conditions. PSE and JND correspond to the mean of PSE and JND obtained from the modelling of each participant.

Figure 3 -

 3 Figure 3 -Mean of PSE and JND obtained from the logistic modelling of each participant's "Very Intense Pain" responses in the still and idle conditions. Error bars correspond to confidence intervals at 95%.

Figure 4 -

 4 Figure 4 -Logistic modelling of "Touched" responses according to facial pain expression intensity in the still and idle conditions. PSE and JND correspond to the mean of PSE and JND obtained from the modelling of each participant.

Figure 5 -

 5 Figure 5 -Mean of PSE and JND obtained from the logistic modelling of "Touched" responses of each participant in the still and idle conditions. Error bars correspond to the confidence intervals at 95%.

Acknowledgment

TT received scholarships from the Paris-South University, France.

Declaration of competing interest

None.

CRediT authorship contribution statement

Thomas Treal: Methodology, Software, Formal analysis, Investigation, Data


DEIB Jorge F Lazo

Aldo Marzullo
email: marzullo@mat.unical.it

Sara Moccia

Michele Catellani
email: michele.catellani@ieo.it

Benoit Rosa
email: b.rosa@unistra.fr

Francesco Calimeri
email: calimeri@mat.unical.it

A Lumen Segmentation Method in Ureteroscopy Images based on a Deep Residual U-Net architecture

Keywords: learning, ureteroscopy, convolutional neural networks, lumen segmentation

th Elena De Momi

DEIB Politecnico di Milano

Milan, Italy elena.demomi@polimi.it Abstract-Ureteroscopy is becoming the first surgical treatment option for the majority of urinary affections. This procedure is performed using an endoscope which provides the surgeon with the visual information necessary to navigate inside the urinary tract. Having in mind the development of surgical assistance systems, that could enhance the performance of surgeon, the task of lumen segmentation is a fundamental part since this is the visual reference which marks the path that the endoscope should follow. This is something that has not been analyzed in ureteroscopy data before. However, this task presents several challenges given the image quality and the conditions itself of ureteroscopy procedures. In this paper, we study the implementation of a Deep Neural Network which exploits the advantage of residual units in an architecture based on U-Net. For the training of these networks, we analyze the use of two different color spaces: gray-scale and RGB data images. We found that training on gray-scale images gives the best results obtaining mean values of Dice Score, Precision, and Recall of 0.73, 0.58, and 0.92 respectively. The results obtained shows that the use of residual U-Net could be a suitable model for further development for a computer-aided system for navigation and guidance through the urinary system.

I. INTRODUCTION

Ureteroscopy is a procedure dedicated to the exploration of the upper urinary tract to perform diagnosis and treatment of different conditions, such as kidney stones or carcinoma of the upper urinary tract. The procedure involves the passage of a flexible or semi-rigid ureteroscope through the urethra and bladder, and up to the ureter to the point where the stones or tumor is located [START_REF] Monga | Ureteroscopy: indications, instrumentation & technique[END_REF]. Urinary calculi may be located anywhere in the urinary tract, but they are more common to appear in the kidneys. In the case of cancer, tumors can be found at any point in the renal pelvis, renal calyces, and ureters, but the most common location is the renal pelvis and renal calyces, being present in about 4-15% of the cases [START_REF] Latchamsetty | Treatment of upper tract urothelial carcinoma: a review of surgical and adjuvant therapy[END_REF]. The total traveling distance from the ureter orifice to the kidneys ranges from 28cm to 34cm and the diameter of the ureter is 5mm in the pelvis [START_REF] Giusti | Current standard technique for modern flexible ureteroscopy: Tips and tricks[END_REF].

A review of the intraoperative complications in ureteroscopy shows that avulsion, major and minor perforation, mucosal abrasion and stricture are considered the most common ones [START_REF] De La Rosette | Handling and prevention of complications in stone basketing[END_REF]. Minor perforation is the one with a higher frequency, appearing with an average of 1.99% of the analyzed procedures, followed by stricture (0.58%), and major perforation is the one with lower probability (0.06% of the cases). Navigation and diagnosis inside the urinary tract are highly dependent upon the operators expertise, and image-related conditions such as the presence of image artifacts, floating debris, the low quality of the video, occlusions in the video, or image noise [START_REF] Ali | Endoscopy artifact detection (ead 2019) challenge dataset[END_REF], could add additional challenges for non-experienced operators. Furthermore, the heterogeneity in the anatomical structures, and the different shapes in which the lumen is deformed at different points in the urinary tract adds additional challenge to the task of segmenting the lumen. Some examples which highlight the variability in the deformation, colors and shape of the lumen are shown in Fig. 1.

The task of lumen segmentation is a fundamental part in the development of computer vision methods for surgical assistance, since this is the reference which marks the path that the endoscope should follow. These assistance systems intend to deal with the mentioned challenges by enhancing relevant information to potentially increase the performance of the surgeon and minimize the probability of complications. Besides, the combination of this visual information with the information from some other sensors such as EM sensors or Fiber Bragg Grating sensors, that can provide positional information, is a primary step in the development of computer vision systems suitable for more advanced robot-assisted ureteroscopy [START_REF] Chew | Robotic ureteroscopy: The future of stone management?[END_REF].

Recent advancements in the field of computer vision applied to biomedical image analysis and particularly, the use of Deep Learning (DL) methods such as Convolutional Neural Networks (CNNs) have shown remarkable results in biomedical image segmentation challenges such as EM ISIBI 2012 [START_REF] Arganda-Carreras | Crowdsourcing the creation of image segmentation algorithms for connectomics[END_REF], the Longitudinal MS Lesion Segmentation Challenge [START_REF] Van Ginneken | 3D segmentation in the clinic: A grand challenge[END_REF] or the Endoscopic Artefact Detection Challenge EAD2019 [START_REF] Ali | Endoscopy artifact detection (ead 2019) challenge dataset[END_REF].

In this paper, we propose the application of a variation of the well-known CNN architecture U-Net by using residual blocks [START_REF] Szegedy | Going deeper with convolutions[END_REF]. This have shown to have beneficial effects dealing with the vanishing gradient problem [START_REF] Drozdzal | The importance of skip connections in biomedical image segmentation[END_REF]. We compare the results of training the network in the same datasets, but in different image modalities, one composed only of gray-scale images and another of RGB images to determine whether the information contained in a color-space of one single channel is enough to perform the task, or a bigger color-space is needed.

The rest of the document is organized as follows: in Sec. II we provide a survey of lumen segmentation methods used in data from other surgical procedures as well as image segmentation methods used in the analysis of ureteroscopy Fig. 1: Sample of images in the dataset collected showing the variability of the hue as well as the shape and location of the tissue which sometimes merges with the tool and the guide-wire. images, Sec. III presents the details of the implementation used in this study, in Sec. IV the results obtained are given and discussed. Finally Sec. VI concludes the paper.

II. RELATED WORK

Lumen segmentation has been explored in different surgical procedures such as colonoscopy procedures and also in different image modalities such as Intravascular Ultrasound images. We limit the literature research to colon procedures, which not only has a more similar anatomical structure, but the procedure itself is more similar to ureteroscopy regarding the kind of image data and instruments used during the intervention.

The study in [START_REF] Gallo | Lumen detection in endoscopic images: a boosting classification approach[END_REF] implements a intestinal detection method based on the application of a Haar-like features-based algorithm to recognize the lumen. A combination of the algorithm with AdaBoost to select discriminative features is later incorporated in a cascade classifier. The method is sensitive to light conditions, and only works for lumen detection. The method proposed in [START_REF] Zabulis | Lumen detection for capsule endoscopy[END_REF] makes use of a coarse-to-fine version of the mean-shift algorithm to detect and track the lumen. The algorithm runs for several times with different seed points and the detected extreme points are spatially clustered. In [START_REF] Wang | A lumen detectionbased intestinal direction vector acquisition method for wireless endoscopy systems[END_REF], two methods are presented, one based on adaptative thresholding for the detection of lumen when is not completely contracted and an alternate method for the case in which lumen is contracted. The second method is based on texture detection where, first adaptative thresholding is used to locate the minimum gray pixel and then the strong edges (wrinkles) of the folded intestinal wall are detected. Even if this method is capable of detecting the lumen shape no results in term of pixel accuracy are given and it is not optimal when there are considerably amounts of turbid liquid. The main limitation in the implementation of these methods is the fact that they are parameter-sensitive, which means that they cannot generalize very well to unseen data and in general to the inter-patient anatomy variability and imaging conditions. To deal with this more recent approaches have started to make use of Neural Networks-based methods.

A benchmark for segmentation in colonoscopy images is presented in [START_REF] Vázquez | A benchmark for endoluminal scene segmentation of colonoscopy images[END_REF]. The authors use a multi-class image segmentation method based on the implementation of a 8layer Fully Convolutional Network (FCN) and the classes they segment are: background, lumen, polyp and image specularities. Considering the specific case of ureteroscopy images, no study related with the segmentation of lumen has been found. We relate this to the lack of public available datasets. A possible reason for such fact could be difficulty of obtaining this image data and the lack of an automated or semi automated device to perform this procedure.

The review of image segmentation methods in ureteroscopy images, even if it refers to the segmentation of different structures, is relevant for our work given the specific challenges in the analysis of uretersocopy images. Some of these challenges include a low image quality because of breathing motion [START_REF] De Koste | Renal mobility during uncoached quiet respiration: An analysis of 4D CT scans[END_REF], presence of floating debris from the dusted tissue and the irrigation fluid used to remove it [START_REF] Monga | Ureteroscopy: indications, instrumentation & technique[END_REF].

A kidney stone detection method using ureteroscopy images is proposed in [START_REF] Rosa | An algorithm for calculi segmentation on ureteroscopic images[END_REF] and it is based on a region growing algorithm, this algorithm requires of a seed, a similarity criterion and a stopping criterion. However, the definition of the criterion is not optimal since the position of the seed must be defined by the user (with the requirement that it should be placed in the center of the calculus), and the stopping criterion, which need to be optimized is empirically determined. A method based on Single Shot MultiBox Detector for Ureteral Orifice detection and segmentation was proposed in [START_REF] Peng | Real-time detection of ureteral orifice in urinary endoscopy videos based on deep learning[END_REF]. The authors use images from resctoscopy, which have a larger field of view, to train their model and images from ureteroscope to validate it. Recently in [START_REF] Gupta | Mi-unet: Improved segmentation in ureteroscopy[END_REF] an implementation for kidney stones segmentation based on the combination of U-Net and a Deformation Vector Field which makes use of consecutive frames to determine the deformation before being fed the regular U-Net architecture has been implemented. Nevertheless, these approaches have the peculiarity that they focus only in a specific and small part of the urinal system and they do not deal with most of the irregularities that appears in uretersocopy images. In our work, we deal not only with the variability of the image quality dependant on the device used in the procedure, the inter-patient anatomical variability, but also the innervariability of the shape, texture and deformation of the ureter through the urinary tract. Furthermore, we show that the method we propose can be tested on data with the presence of elements that were not present in the data used for training, such as lasers and tumors, and still detect the lumen to some extent.

To deal with this high-variability images we propose the use of Deep CNNs. To the best of our knowledge, CNNs have not been applied to image segmentation of the lumen in ureteroscopy images. We study the implementation of a model based in deep residual networks and compare it with other 2 models in two datasets coming from the same sources, but in different color-space. Furthermore, we implement different data augmentation techniques to improve the results obtained.

III. PROPOSED METHOD

Inspired by the paradigm set by residual networks [START_REF] He | Deep residual learning for image recognition[END_REF] which have shown to achieve state of the art performances in segmentation tasks such as road segmentation from aerial images [START_REF] Zhang | Road extraction by deep residual u-net[END_REF] or nuclei segmentation [START_REF] Zeng | Ric-unet: An improved neural network based on unet for nuclei segmentation in histology images[END_REF]. We propose the implementation of a segmentation network based on U-Net architecture which makes use of residual blocks. Among the advantages these kind of networks presents there is the reduction in the training time, as result of the residual blocks the convergence of the learning process is faster. Thanks to the short-skip connections, present in this network, the spatial information propagates without degradation and in general, it has been reported that the segmentation with this architectures results in a better delineation of the borders of the figures [START_REF] Drozdzal | The importance of skip connections in biomedical image segmentation[END_REF]. We compare the results obtained with the aforementioned network with two other architectures, in this case, a model similar to the standard version of U-Net, which only differs by the addition of Batch Normalization, and FCN-8 network which has been used previously for lumen segmentation.

A. FCN-8

Fully Convolutional Neural Networks (FCNs) are an extension of CNNs to deal with the task of pixel prediction. These networks make use of long skip connections in order to recover the fine-grained spatial information from the downsampling path by merging it in the up-sampling layers. In this work we implemented a FCN8 architecture as proposed in [START_REF] Vázquez | A benchmark for endoluminal scene segmentation of colonoscopy images[END_REF] for endoluminal scene segmentation in endoscopic images.

B. U-Net

U-Net is an architecture which originally was designed for biomedical image segmentation applications [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. It is composed of a contracting path and an up-sample path which are symmetrical. The convolutional blocks of U-Net uses Convolutional Layers of kernel size 3x3 followed by 2x2 max pooling layers. The number of kernels used in each layer doubles at every block and the output of the contracting stage is connected to the feature maps in the corresponding up-sampling layer. This can ensure that the features learned in the contracting path are used in the up-sampling path for the reconstruction of the segmented images. In the implementation used in this work we added Batch Normalization layers to the output of every convolutional layer. This helps to reduce the amount of covariance shift off the hidden layers and it has the advantage that it has regularization effects, reducing overfitting and allowing the use of higher learning rates [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF].

The loss function used in this implementation was based on the Dice similarity coefficient (L DSC ) defined as:

L DSC = 1 - 2T P 2T P + F N + F P (1)
where T P is the number of pixels that belong to the lumen, which are correctly segmented, F P is the number of pixels miss-classified as lumen, and F N is the number of pixels which are classified as part of lumen but actually they are not.

C. U-Net based on residual blocks

Residual Blocks were first introduced in the ResNet architecture for image classification task. Their application was later expanded to image segmentation tasks and inspired the construction of networks which take the advantages of this type of convolutional blocks. These kind of networks have proven to achieve performances above the state of the art in both, classification and segmentation tasks. The residual units intend to address the degradation problem by adding a skip connection between the input and the output of each convolutional block, which can be defined as:

y i = h(x i ) + F (x i , W i ) (2)
x i+1 = f (y i ) (3)
where x i and x i+1 are the input and the output respectively of the i th residual block, F is the residual function, f (y i ) is the activation function and h(x i ) is the identity mapping function.

The implementation used in this study consisted of using a basic residual block as defined in [START_REF] Drozdzal | The importance of skip connections in biomedical image segmentation[END_REF]. This is composed of a convolution layer followed by batch normalization and a ReLU activation function which later was adhered to the output of the conventional convolutional block as depicted in Fig. 2. The loss function used in the network was the same as the used in the previous network and defined by Eq. 1.

D. Performance Metrics

The performance metrics chosen were the DSC, the Precision (P rec) and Recall (Rec) which are defined as

DSC = 1 -L DSC (4)
P rec = T P T P + F P (5)

Rec = T P T P + F N (6)

E. Dataset

For this study 7 videos were collected. The videos were acquired from the European Institute of Oncology (IEO) at Milan, Italy.All patients gave their informed consent for data collection and use of it for research. The data collection followed the ethical protocol approved by the IEO and in accordance with the Helsinky Declaration. The videos collected are from ureteroscopy procedures targeting upper tract tumor ablation and kidney stone removal. From these videos a different number of frames were extracted as described in Table I and manually annotated. Some samples of the images from the dataset which depict the variability in the images is shown in Fig. 1 A prepossessing stage included cropping of the frames to cut out the black region surrounding the field of view of the endoscopic images, and the conversion to gray-scale for the training and testing in the gray-scale dataset. The former one, was done by applying a canny filter for edge detection and subsequently an elliptical Hough transform algorithm based on [START_REF] Xie | A new efficient ellipse detection method[END_REF]. The final shape of the images was set to 256x256 to be consistent with the input layer of the networks used.

F. Training Setting

To handle memory limitations the networks were trained with a mini-batch size strategy. The networks were trained using data augmentation, the operations used for this purpose were rotation in intervals of 90 • , 180 • , and 270 • ; horizontal and vertical flips and zooming in and out in a range of ± 0.02. The learning rate, and mini batch size for each of the models was chosen by trying the different combinations between several possible values of the hyper-parameters and using a 5-fold cross validation strategy with the data from patients 2, 3 and 4. The 5-fold process was repeated independently with both the RGB dataset and the gray-scale dataset. The DSC was set as the evaluation metric to chose the best model with respect to the hyper-parameters. Adam optimization was used during all the trainings. Once the hyper-parameter values were chosen, an additional training process was carried out using these values. This training was done by using all the annotated frames obtained from patients 2, 3 and 4 for training and validation, while the data from patient 1 was used for testing. In total 798 frames were used for training and 417 for validation in this stage. The Kruskal-Wallis test on the DSC was used to determine statistical significance between the different models trained.

The Networks were implemented using Tensorflow and Keras frameworks in Python trained on a NVIDIA Quadro M5000 GPU.

IV. RESULTS

The box plots of the DSC, P rec and Rec are depicted in Fig. 4 for each of the networks tested. From these results is possible to see that Residual U-Net has the best performance overall in DSC, P rec and Rec with average values of 0.73, 0.58, and 0.92 respectively. In comparison with U-Net in the gray-scale dataset Residual U-Net achieves a DSC 8% better than the standard U-Net (p<0.05) and in the case of the RGB dataset achieves an average value 10% (p<0.01) better than U-Net. The difference between the FCN8 model and the residual U-Net results higher with a difference of 39% (p<0.001) for the case of the gray-scale image dataset and 33% (p<0.001) for the RGB one. In general for all the 3 models it was seen that the training in gray-scale images achieve better results among the same models. In the case of the simple U-Net a difference of 14% was observed (p<0.001) while in the case of Residual U-Net the difference was of 12% (p<0.05). Even there is a difference of 6% between the use of gray-scale images and RGB ones, no statistical difference was found among them.

V. DISCUSSION

The collective difference between the results obtained using gray-scale images and the RGB ones might be related Fig. 3: Sample of results obtained using U-Net with skip connections with its respective dice value. The colors in the Overlay images are as follows. TP: Green, FP: Pink, FN: Blue, TN: Black.

with the nature itself of the dataset as well as the task. Given that the gray-scale images might have enough information to perform the binary classification of pixels into lumen and no-lumen, the information in the RGB images provide two extra channels that does not provide any substantial new information. This implies the need of more parameters for the extra channels, and the addition of parameters will require a larger amount of data. Considering that the dataset used is not precisely large the networks might tend to overfit even if a comprehensive exploration on the combination of hyperparameters was done, and therefore regularization techniques might be needed to overcome this problem.

It is also worth to mention that during the training it is observed that the DSC and loss curves were in general smoother for the cases in which the Residual U-Net was trained than when only the standard version of U-Net was used, and the convergence time was shorter when using the former one, just as reported in [START_REF] Drozdzal | The importance of skip connections in biomedical image segmentation[END_REF].

Some sample of the results obtained with U-Net based on residual blocks in the gray-scale dataset are shown in Fig. 3. In this case it is possible to see that at least in the majority of the cases some part of the lumen is correctly segmented and that the issues come from the border areas of the lumen where it is hard to define where it ends. This is in general a hard task for the networks given the fact that the lumen, which is the region of interest, has similar hue values as the region surrounding the field of view of the camera.

Nevertheless, by using the residual-based U-net architecture it was possible to achieve a reasonable level of generalization. Given the fact that the video-frames used as test dataset come from a surgical-procedure of upper-tract carcinoma removal and in these, there are some elements which do not exist in the rest of the dataset such as tumors and laser beams. In fact, the cases in which a DSC lower than 0.5 was obtained, correspond to frames in which these elements appear, or which correspond to anatomical structures which are also not present in the training dataset such as the ureteral orifice. This leads to encouraging results, but further explorations regarding the way in how to augment the training data to cover the same conditions present in the test dataset are needed, in case that the collection of raw data with the same conditions and elements mentioned is not feasible.

Furthermore, the high values of recall obtained indicate that there are very low values of false negatives. This is important recalling the aim for which a lumen-segmentation system is intended, which is to aid the surgeon in the navigation through the urinary tract. Having high values of false negatives indicate that there are low possibilities that the system could lead the tip of the catheter to a position in which it could cause harm to the patient. Moreover, the cases in which the values of precision are not exactly high correspond mainly to the fact that the current method is missing some of the pixels in the outskirts, but the center of the lumen is always detected, which in any case, is the most important region.

Further development to properly segment the complete area of the lumen is needed and this could be achieved by implementing more complex models which can deal with shape recognition of structures in the frames such as the one proposed in [START_REF] Casella | Inter-foetus membrane segmentation for TTTS using adversarial networks[END_REF] or the use of models which can exploit temporal features [START_REF] Colleoni | Deep learning based robotic tool detection and articulation estimation with spatio-temporal layers[END_REF].

VI. CONCLUSION

In this paper, we addressed the challenging task of lumen segmentation in the ureter, for this purpose we proposed the implementation of a Deep CNN which makes use of residual units as building blocks of a U-Net-like architecture. The proposed network achieves in unseen data values of DSC, P rec and Rec of 0.73, 0.58 and 0.92 respectively. Furthermore, the model is able, to some extent, to detect part of the lumen in images with elements, such as lasers beams and anatomical structures, such as tumors that was not present during the training, but more investigation needs to be done in order to properly generalize in such type of data. In conclusion the method has the potential to be further developed in order to be integrated systems which could aid surgeons in the navigation through the urinary system.

Fig. 2 :

 2 Fig. 2: Comparison of the different convolutional blocks used to build the models. a) General sketch of U-Net architecture. b) Diagram of the convolutional block adding batch normalization c) residual block with short skip-connections.

Fig. 4 :

 4 Fig. 4: Box plots obtained with the different models tested. a) DSC, b) Precision, c) Recall. the asterisks represent the significant difference between the different architectures in terms of the median DSC Kruskal-Wallis signrank test (* p < 0.05, ** p < 0.001, *** p < 0.001). The red line in the box indicates the median value.

TABLE I :

 I Detailed information about the dataset collected. The videos marked with * indicate the videos that were set apart to be used only during testing.

		No. of	Image Size	Patient No.
		annotated	(pixels)
		frames
	Video 1*	7	356x256	1
	Video 2*	80	256x266	1
	Video 3	462	296x277	2
	Video 4	245	256x257	3
	Video 5	148	256x257	3
	Video 6	168	256x257	3
	Video 7	235	256x262	4
	Total	1,445	-	-

TABLE II :

 II Resume of the average with its respective standard deviation of the best results obtained with the 5-fold cross validation for each of the models tested in each of the datasets: (RGB dataset white rows, grayscale-dataset gray rows).

	Model	DSC	P rec	Rec	Acc
	U-Net	0.67 ± 0.01	0.57 ± 0.01	0.69 ± 0.03	0.94 ± 0.02
	U-Net	0.55 ± 0.09	0.43 ± 0.11	0.57 ± 0.07	0.87 ± 0.17
	Residual	0.68 ± 0.05	0.58 ± 0.01	0.78 ± 0.03	0.92 ± 0.10
	U-Net
	Residual	0.59 ± 0.15	0.49 ± 0.03	0.63 ± 0.13	0.94 ± 0.01
	U-Net
	FCN-8	0.36 ± 0.08	0.25 ± 0.09	0.64 ± 0.01	0.78 ± 0.07
	FCN-8	0.23 ± 0.04	0.19 ± 0.03	0.52 ± 0.10	0.64 ± 0.15

This work was supported by the ATLAS project. ATLAS has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 813782.