Keywords: Analyse d'image médicale, apprentissage profond, segmentation, fonction de perte, CNN régression, biomarqueurs, interprétabilité, circonférence de la tête du foetus, volume des structures cardiaques Medical imaging analysis, Deep learning, Segmentation, Loss function, Regression CNN, Biomarkers, Interpretability, Fetus head circumference, Cardiac structure volume vi

Segmentation is one of most prominent task in medical image processing and analysis.

For a few years now, convolutional neural networks (CNN) have been the state-of-the-art in this domain. We will focus on CNN for medical image segmentation and analysis from the following standpoints. First, the loss function is an important component that drives the CNN training and decides on the relation between target labels and the predictions.

As such, a lot of research is made on loss design, especially since the standard losses, such as the Dice loss, have shown their limitations. Second, segmentation is often the first step to subsequently estimate parameters (also called biomarkers) from the image. Medical experts use biomarkers to diagnose patients' health status and monitor treatment. Thus accurate biomarkers estimation is of paramount importance. However, errors are prone to occur in the intermediate segmentation step. Very recently, deep learning techniques have open the way to directly estimate biomarkers from images, without segmenting them.

Research on this topic is still as its early stage.

To address the above issues, this thesis proposes the following contributions, summarized in three points : first, we propose a new loss function, that is based on the Kappa coefficient, that has the ability to take into account all the pixels in the image, including the true negative, contrary to the standard Dice loss. We illustrate its added value on a public set of skin lesion images. Second, we contribute to segmentation-free direct biomarker prediction, from a methodological perspective, so as to provide a reasonable and effective analysis solution for clinical applications. We propose and study several regression CNN architectures, that learn directly to estimate the parameters of interest without resorting to segmentation. One application case is the prediction of fetus head circumference (HC) from ultrasound images: we comprehensively compare segmentation-based method and regression (i.e. segmentation-free) method under a fair experimental protocol and are able to show promising results, even though room for improvement is left. Another case is prediction of cardiac structure volumes from 3-dimensional (3D) magnetic resonance images, in which a multi-objective prediction method is achieved. Third, we investigate the interpretability of the deep regression models, by extending standard saliency maps techniques to regression CNN. We explained the inner world of the regression CNN models both qualitatively and quantitatively and are able to show that indeed the regression CNN is learning to identify the target area.

First of all, I would like to sincerely thank my two lovely supervisors, professor Samia Ainouz and professor Caroline Petitjean who are from LITIS lab. They are great female scientists with the qualities of care, patience, and wisdom. They have given me very constructive guidance and encouragement in my work, which has made me have further insight and confidence in doing scientific research. In addition, their rigorous style in work and humor in life have deeply infected me. I am fortunate to be their student and hope I can learn from their good qualities.

During my first year as a PhD student, as I was new to France and not familiar with the language and environment, my supervisor Samia helped me find accommodation in advance with a French-speaking roommate and Caroline put me in an office with a Chinese colleague. My roommate Grace and my colleague Leo (Hongliu Cao) helped me through my first year of life and research respectively. I would also like to thank the leaders, teachers and other colleagues of the LITIS laboratory for their enthusiasm and commitment to my work, such as the secretaries Fabienne, Mathieu, Brigitte, technical staff Fabrice, colleagues Rosana, Wassim, Guillaume, Cyprien, etc (I wish I could remember everyone's name !).

As the saying goes, you can rely on your parents at home and your friends when you go out. I would like to thank my dear friends for their help and the happy times I had with them (Travelling, get-together, playing, communications). They are Yang Ruiping, Wang Yanjun, Liu Xuelian, Lou Yuzhen, Jia Linlin, Wang Jundong, Xu Jie, Ren Chao, Wang Xuefei, Huang Ling, Zhang Haodi, Zhang Ce, Yuan Wenbin, and due to space limitation, forgive me for not being able to list all the names of my friends.

iii As the Covid-19 pandemic began to spread across Europe in early 2020, countries began to close their cities, schools began to telecommute, and we were all suffering from varying degrees of physical and psychological discomfort. No one was more worried about me than my family, my parents and my brother and sisterin-law, who were thousands of kilometers away in China. From time to time, they were aware of the epidemic situation on my side and concerned about my health in body and mental, and it was a difficult time. My two adorable little nieces often tell me interesting stories about them, and I envy their childishness and carefree nature. During this time, the Chinese Embassy in France also cared about the students in France, they gave us "health kits" (containing masks, wet wipes, and some cold medicine) and the LITIS lab gave us a box of masks and hand sanitizer. In 2021, I got two doses of the Covid-19 vaccine for free, hoping that the epidemic would end sooner and everyone would return to their normal work and life as before. I'm so grateful and thankful for their support and care.

Résumé

La segmentation est l'une des tâches les plus importantes dans l'analyse des images médicales. Depuis quelques années, les réseaux de neurones convolutifs (CNN) en constituent l'état de l'art. Dans ce contexte, nous allons nous focaliser sur les problématiques suivantes. Premièrement, la fonction de perte (loss) est une composante importante qui dirige l'apprentissage des CNN et décide de la relation entre les étiquettes cibles et les prédictions. Les fonctions de loss standard en particulier, telle que la loss de Dice, ont montré leurs limites. Deuxièmement, la segmentation est souvent la première étape pour ensuite estimer les paramètres (également appelés biomarqueurs) de l'image. Ces biomarqueurs sont utilisés pour établir un diagnostic et un suivi des patients. Une estimation précise des biomarqueurs est donc capital. Cependant, des erreurs sont susceptibles de se produire lors de l'étape intermédiaire de segmentation. Récemment, les techniques d'apprentissage profond ont ouvert la voie à l'estimation directe des biomarqueurs à partir des images, sans segmentation ou extraction de caractéristique adhoc. La recherche sur ce sujet en est encore à ses débuts.

Pour répondre à ces questions, cette thèse propose les contributions suivantes, résumées en trois points : tout d'abord, nous proposons une nouvelle fonction de perte, basée sur le coefficient Kappa, qui a la capacité de prendre en compte tous les pixels de l'image, y compris le vrai négatif, contrairement à la perte standard de Dice. Nous illustrons sa valeur ajoutée sur un jeu de données public d'images de lésions cutanées. Deuxièmement, nous contribuons à la prédiction directe de biomarqueurs sans segmentation afin de fournir une solution d'analyse raisonnable et efficace pour les applications cliniques. Nous proposons plusieurs architectures de CNN de régression, qui apprennent directement à estimer les paramètres d'intérêt sans recourir à la segmentation. Un cas d'application est la prédiction de la circonférence de la tête du foetus à partir d'images échographiques : nous comparons segmentation et régression avec un protocole expérimental judicieux. De ce fait, nous avons pu montrer des résultats prometteurs pour la régression, même si des améliorations restent possibles. Un autre cas est la prédiction des volumes de la structure cardiaque à partir d'images de résonance magnétique tridimensionnelles, dans lequel une méthode de prédiction multi-objectifs est réalisée. Troisièmement, nous étudions l'interprétabilité des modèles de régression, en étendant les techniques standard de cartes de saillance aux CNN de régression, qualitativement et quantitativement. Nous avons pu montrer que, sur la plupart des images, le CNN de régression apprend réellement à identifier la zone cible.

Background

Computer science and technology has helped mankind increase productivity in every aspect since its inception. As a branch of it, computer vision (CV) tries to imitate the human eye to acquire, process, and analyze everything related to media, such as images, videos. Artificial intelligence (AI) emerged almost simultaneously with computer science. In the 21st century, techniques based on artificial intelligence have progressed considerably, and one specific typical approach that has been successful is deep learning (DL) techniques. Undoubtedly, computer vision based on deep learning techniques has contributed greatly in many fields [START_REF] Chai | Deep learning in computer vision: A critical review of emerging techniques and application scenarios[END_REF]; for instance, medical image analysis or computing (MIC), which is the central theme to be highlighted in this thesis.

Medical image analysis is an interdisciplinary discipline that combines medical imaging and computer science. The most commonly used imaging modalities in clinical medicine include radiography (e.g. X-ray), computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound (US), among others. X-Rays (radiography) is first discovered in 1895. The principle is emitting electromagnetic waves to inside the human body, and the projection image is formed. Computed tomography (CT scan) developed in 1970s, uses multiple X-Ray machines in different angles to detect various parts (the soft tissues, blood vessels and bones etc.) of the body, then, reconstruct these images through computers to create cross-sectional images of the body. These images provide more detailed information than a normal X-ray image. Ultrasound (US) was first used for clinical purposes in 1956. Ultrasound is an imaging modality that uses high-frequency sound waves rather than radiation. The advantages of US images are real-time, fast, low cost, and not harmful to human. However, the quality of US images is bad, and noise is included sometimes. Magnetic Resonance Imaging (MRI) is a non-invasive imaging technology that produces three dimensional detailed anatomical images. The first MRI scan of the human body was performed in 1977. Based on the principles of nuclear magnetic resonance (NMR), MRI techniques use a strong magnetic field to force the protons inside a substance to align with that field. Based on the electromagnetic waves emitted by the decaying energy of the nucleus, the location and type of that nucleus can be known and an image is formed. One special type of MRI is functional MRI (fMRI), which is used to observe brain structure and determine which areas of the brain are "activated" when performing certain cognitive tasks. Thus, the brain US of a cardiac [START_REF] Leclerc | Deep learning for segmentation using an open large-scale dataset in 2d echocardiography[END_REF] (c) X-Ray of a hand1 (d) CT of a chest with COVID-19 [START_REF] Kwee | Chest ct in covid-19: what the radiologist needs to know[END_REF] (e) MRI of a cardiac [START_REF] Bernard | Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved[END_REF] (f ) PET of a body 2 1 The image is from https://www.imaginghealthcare.com/diagnostic-imaging/digital-x-ray/ 2 The image is from https://www.itnonline.com/article/what-pet-imaging Figure 1.1 -Medical image types organization can be understood through this way. Positron emission tomography (PET) is a nuclear imaging technology. The principal of PET is that the tracer is injected into a vein first, then PET systems detect and reconstruct the radiations from inside the body. Similar technology is Single Photon Emission Computerized Tomography (SPECT). And hybrid PET imaging systems (with CT or MRI) are practical in recent decades [Lee, 2010]. These five common modalities of medical imaging and their usages are summarized in Table 1.1 1 . Besides the image types mentioned above, there are other types of images, such as skin lesion images, fundus images, histopathology images, etc. Some examples are given in Figure 1.1. Imaging data accounts for approximately 90% of all medical data and is therefore one of the most important sources of evidence for clinical analysis and medical intervention [START_REF] Zhou | A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises[END_REF]. The goal of medical image analysis is developing computational and mathematical methods to solve problems related to medical images and use them in biomedical research and clinical care [Wikipedia, 2021]. This field involves several broad tasks: image segmentation, image registration, image classification, etc.

CHAPTER 1. INTRODUCTION

Nowadays, deep learning techniques have been successfully applied in different medical imaging analysis tasks, such as image classification, image segmentation, image registration, image reconstruction, object detection, etc. Medical image analysis based on AI can extract useful information from images, which can help doctors or experts to diagnose or make decisions about patients. If medical image analysis is aided by AI, it can greatly reduce the amount of effort doctors spend on a patient, especially in less developed areas where medical resources are not sufficient [START_REF] Vuong | Artificial intelligence vs. natural stupidity: Evaluating ai readiness for the vietnamese medical information system[END_REF]. Therefore, it is a very meaningful thing in terms of research and clinical applications. At the same time, one should also be wary of whether current AI technology (represented by DL) is safe and reliable in facing sensitive subjects with the AI techniques prospering, and why it makes this or that decision. In other words, these deep learning models should be developed with great reliability and transparency in sensitive areas such as medicine or autonomous driving. Consequently, there is a branch of AI called explainable AI (XAI) [START_REF] Samek | Explaining deep neural networks and beyond: A review of methods and applications[END_REF], which is aiming to make the AI reliable and trustworthy.

Motivation

In this thesis, we focus on medical image segmentation. Image segmentation is the process of partitioning the image into meaningful regions. In medical imaging, segmentation is often the first step required to estimate parameters (also called biomarkers) from the image, such as the volume of the segmented region, and is one of the major task in medical image analysis, useful for computer-aided patient diagnostic, pronostic and follow-up. More specifically, we address 3 specific issues.

The first issue is the class imbalance problem in supervised learning that occurs in medical image segmentation; the second issue is biomarker estimation from medical imaging based on deep learning; the third issue is the explainability of some deep learning model that is applied in medical imaging analysis.

CHAPTER 1. INTRODUCTION

First of all, the deep learning techniques are widely used in various fields. It is a data-driven, automated predictive machine. The architectures of DL are evolving rapidly with supervised learning, unsupervised/semi-supervised learning, transfer learning, federated learning, etc. Specifically, Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), attention mechanism and other models are widely used according to different needs. Recently, the Transformer networks [START_REF] Vaswani | Attention is all you need[END_REF] are quite popular in Natural Language Processing (NLP) and computer vision. No matter in which method above, the loss function is an important and integral part of neural networks. For instance, in supervised learning, the loss function is used in the neural networks to update the weight parameter of each neuron in back propagation stage, thus closing the gap between predicted and target values. The better the loss function is, the successful the performance of the model is usually, making other variables more consistent. In general, there are several types of loss functions in image segmentation, including Cross Entropy loss series and Dice losses which are derived from evaluation metrics [START_REF] Ma | Loss odyssey in medical image segmentation[END_REF].

In medical image segmentation, one prominent issue is the class imbalance problem, which refers to the ratio of foreground (segmentation target) and background in an image is severely unbalanced. For example, when segmenting a tumor from organ image or a lesion from skin image, in which the tumor or lesion is far smaller than the background (See Figure 1.2). So in this case, even though the segmentation results is not well matched the ground truth, the accuracy can still be high, because the model incorrectly takes into account the correct prediction of the background to count as the accuracy. Therefore, to this end, this thesis tries to find an optimization scheme i.e. loss function that can avoid the class imbalance problem and thus can really improve the image segmentation accuracy.

Secondly, the biomarker is a vital concept in clinical examination and diagnosis.

Broadly speaking, the definition of a biomarker [Califf, 2018] is deceptively simple:

"A defined characteristic that is measured as an indicator of normal biological processes, pathogenic processes or responses to an exposure or intervention." Specifically, there are two categories of biomarkers: imaging biomarkers and molecular biomarkers. Obtaining a biomarker from a medical image is relatively straightforward and easy, whereas obtaining the biomarker at the molecular level requires rigorous biochemistry-based experiments. In general, some known biomarkers are mainly achieved by two steps, which are segmentation step and geometry computation based on segmentation results. That is to say, medical image segmenta- [START_REF] Codella | Skin lesion analysis toward melanoma detection: A challenge at 2017 isbi, hosted by the international skin imaging collaboration (isic)[END_REF].

tion is only an intermediate step, and its further goal is to compute some kind of biomarker [Califf, 2018] or to serve image classification for determining which disease is present. Moreover, the problem associated with segmentation methods is that they are prone to errors and take an extra post processing steps and biomarker computation. This thesis is thus dedicated to exploring the feasibility to implement a direct prediction biomarker method so that it can bypass the segmentationbased approaches, which the regression-based methods just fit this scenario. Regression CNNs were first implemented for head pose estimation and facial landmark detection [START_REF] Riegler | Hough networks for head pose estimation and facial feature localization[END_REF], Ahn et al., 2014]. Afterwards, this idea has been applied in medical imaging analysis in order to solve different kinds of medical data and improve the performance as well as possible; for example, for left ventricular volumes prediction [START_REF] Luo | A novel left ventricular volumes prediction method based on deep learning network in cardiac mri[END_REF], Degrave et al., 2016[START_REF] Ge | Echoquan-net: Direct quantification of echo sequence for left ventricle multidimensional indices via global-local learning, geometric adjustment and multi-target relation learning[END_REF], mitosis counting for breast cancer diagnosis [START_REF] Chen | Automated mitosis detection with deep regression networks[END_REF], aortic diameters estimation [Fernández, 2021], carotid artery indices estimation [START_REF] Zhao | Of-umrn: Uncertaintyguided multitask regression network aided by optical flow for fully automated comprehensive analysis of carotid artery[END_REF].

The specific application that we will target in this thesis is examination of fetus growth and development during pregnancy. Head circumference (HC) is one of the key indexes to check a fetus growing state in clinical diagnose. Therefore, this thesis is aiming to use a segmentation-free method to directly predict HC. Healthy people have ejection fractions between 50% and 65% [START_REF] Kumar | Robbins and Cotran pathologic basis of disease[END_REF].

If one's EF is lower than normal index, which means that ejection volume is low, then reference vessel diameter, RVD) and lengths (lesion length, LL) of these vessels [START_REF] Zhang | Direct quantification for coronary artery stenosis using multiview learning[END_REF]. Changes in kidney volume may reflect whether it is functioning properly. Based on this criterion, clinical medicine has defined a kidney disease called renal artery atherosclerosis (RAS) [START_REF] Hussain | Segmentation-free estimation of kidney volumes in ct with dual regression forests[END_REF]. Therefore, it is necessary to estimate the volume of the kidney from the scanned images with the help of segmentation or segmentation-free methods. The problem of adolescent scoliosis has also attracted a lot of attention in recent years. The scoliosis diagnosis is generally based on the idea of Cobb angle, which is defined as the largest angle at a particular region of the vertebral column [START_REF] Sun | Direct estimation of spinal cobb angles by structured multi-output regression[END_REF].

Hence estimation of biological indicators has a great demand in clinical medicine. On the top of that, direct prediction-based methods are beginning to emerge with an accuracy that remains to be fairly compared to the accuracy of segmentation-based methods. Therefore, it's necessary to evaluate and compare these two kinds of methods from methodological and practical perspectives.

CHAPTER 1. INTRODUCTION

Finally, deep learning models have long been known for their groundbreaking performance. However, DL models are used like a black box; little is known about the decision process inside the DL model. In other words, the DL models should become more explainable or interpretable when making decisions on specific tasks [Rudin, 2019]. If this technology is to be implemented into practical applications, such as smart healthcare, autonomous driving and other cutting-edge areas, then it must be understandable and trustworthy, otherwise it could lead to fatal accidents. In particular, in the segmentation-free approach, we cannot visualize the prediction results like in the segmentation-based approach. This requires that the segmentation-free model is evidence-based when making decisions. Therefore, this study attempts to make an interpretation of the deep learning model according to the specific medical imaging problem.

Contributions of the research A new metric-based loss function

We proposed a new metric-based loss function, called Kappa loss, which considers all the pixels including background information that Dice loss ignores, the proposed loss function is proved to be reasonable and superior to Dice loss both in theoretical and experimental (on several skin lesion datasets) aspects.

Direct biomarker prediction using regression CNNs

We proposed a direct fetus head circumference prediction method (regression CNNs) from ultrasound images that bypasses the segmentation based approaches on the public HC18 dataset [START_REF] Van Den | Automated measurement of fetal head circumference using 2d ultrasound images[END_REF]. We compared the segmentation-free methods with the segmentation-based methods in a fair experimental environment.

We utilized regression CNN model to directly predict the three volumes of cardiac structures simultaneously from 3D magnetic resonance images on the public ACDC dataset [START_REF] Bernard | Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved[END_REF], multiple-channel based transfer learning was achieved on 3D medical images. To address the problem of insufficient data, data augmentation based on grid search is applied. Moreover, we performed cardiac data preprocessing including data cropping and slice number unifying and statistically analyzed and discussed the prediction results.

The explainability of regression CNN

We explained the black box of regression CNNs by several explaining methods in the forms of saliency maps and quantitative results. Besides, we achieved a customized evaluation metrics based on perturbation to quantitatively criticise different explaining methods on regression CNNs. The contributions of this thesis is concluded in Figure 1.5.

Skin lesion photo

Fetus head ultrasound

Structure of the thesis

The structure of the thesis is organized as follows:

Chapter 2 introduces the state of the art of the medical image segmentation methods, direct biomarker estimation methods, and explainable AI.

Chapter 3 describes the proposed Kappa loss function.

Chapter 4 presents the work of fetus head circumference prediction.

Chapter 5 focuses on the multi-structure of cardiac volume prediction.

Chapter 6 concludes the thesis and provides perspectives on future work.

The organization of this thesis is shown in Figure 1. we also provide the evaluation metrics and tools used to assess the methods.

Medical image segmentation methods

Traditional medical image segmentation methods

The definition of segmentation is subdividing an image into its constituent parts that are homogeneous in certain feature [START_REF] Ramesh | A review of medical image segmentation algorithms[END_REF]. Traditional segmentation methods can be divided into the following categories:

Threshold segmentation As the name implies, is an algorithm that divides the image into two parts (background and foreground) based on a pixel threshold given in advance. Otsu's method [Otsu, 1979] is the representation of this idea.

Region-based methods Three methodologies are included in this scope. One is region growing algorithm [START_REF] Adams | Seeded region growing[END_REF]. A seed point and similarity criteria decide the segmentation result. The other one is region split and merge algorithm [START_REF] Chen | Segmentation by texture using a co-occurrence matrix and a split-and-merge algorithm[END_REF]. The image is divided into 4 pieces, and if one of these pieces meets the splitting conditions, then this piece is split into 4 pieces again, and so on. When the number of splits reaches a certain level, the adjacent blocks are merged if they meet certain conditions. The third one is watershed approach [Serge and Lantuéj, 1979]. The idea is that low-intensity pixels are regarded as valleys of the surface, high-intensity pixels are peaks. When the level rises to a certain height, water overflows the current valley. This can be achieved by building dams on the watershed, thus avoiding the pooling of water from both valleys, so that the image is divided into 2 sets of pixels, one for the valley flooded by water and one for the watershed line pixels. Eventually the lines formed by these dams then partition the whole image and achieve segmentation of the image.

Clustering methods

Clustering is the partitioning of a data set into different classes or clusters according to a specific criterion (e.g. distance), so that the similarity of data objects within the same cluster is as large as possible, while the difference of data objects not in the same cluster is also as large as possible. The classical clustering algorithm is the K-Means algorithm [START_REF] Hartigan | Algorithm as 136: A k-means clustering algorithm[END_REF].

Edge detection

It is a fundamental problem in image processing and computer vi-sion. The purpose of edge detection is to identify points in a image that have significant changes in intensity. The edge is formed by separating two areas according to distinct intensity. Mathematically, edge detection is roughly the calculation of the derivative of brightness change. Once we have calculated the derivatives, the next step is to give a threshold to determine where the edges are located. Commonly used algorithm is Canny algorithm [Canny, 1986].

Graph theory based segmentation

The idea of this type of methods is to transform the pixel points of an image and their neighbors into vertices and edges and weights on edges in graph theory. Graph cuts and Grab cuts [START_REF] Rother | grabcut" interactive foreground extraction using iterated graph cuts[END_REF] are two examples. They utilize min cut algorithm to cut the edges connected between foreground and background.

Energy optimization algorithm

The basic idea is to use a continuous curve to express the target edge and define a generalized energy function so that the independent variable includes the edge curve, so the segmentation process is transformed into the process of solving the minimum value of the generalized energy function, which can be generally achieved by solving the Euler equation corresponding to the function (Euler Lagrange) equation, the position of the curve where the energy is minimized is where the target profile is located. According to the different forms of curve expression in the model, the active contour models can be divided into two categories: parametric active contour model (Snake model [START_REF] Kass | Snakes: Active contour models[END_REF]) and geometric active contour model (Level set method [START_REF] Malladi | Shape modeling with front propagation: A level set approach[END_REF]).

Deep learning based image segmentation

In recent decade, the deep learning techniques have been a great success due to the excellent performance than the traditional approaches in computer vision, natural language processing, etc is from ground truth value, for example Cross Entropy loss is used in U-Net. Besides, this kind of model is actually data-driven, that is to say, a model will have a robustness and generalizability when training with a great deal of data. Thus, data augmentation is usually needed to increase the quantity of the data. The decoder of segmentation models is used to restore the segmentation map for final output, it can be deconvolution or upsampling layer, the difference is with trainable parameters or not. As for the output segmentation map, it is actually the pixel-wise as input. First, this network can localize. Secondly, the training data in terms of patches is much larger than the number of training images. The resulting network won the EM segmentation challenge at ISBI 2012 by a large margin.

Obviously, the strategy in Ciresan et al.

[1] has two drawbacks. First, it is quite slow because the network must be run separately for each patch, and there is a lot of redundancy due to overlapping patches. Secondly, there is a trade-off between localization accuracy and the use of context. Larger patches require more max-pooling layers that reduce the localization accuracy, while small patches allow the network to see only little context. More recent approaches [11,4] proposed a classifier output that takes into account the features from multiple layers. Good localization and the use of context are possible at the same time.

In this paper, we build upon a more elegant architecture, the so-called "fully convolutional network" [9]. We modify and extend this architecture such that it works with very few training images and yields more precise segmentations; see Figure 1. The main idea in [9] is to supplement a usual contracting network by successive layers, where pooling operators are replaced by upsampling operators. Hence, these layers increase the resolution of the output. In order to localize, high resolution features from the contracting path are combined with the upsampled x j ), where x is element of input vector, K is the number of classes in multi-class classifier.

Then, 3D U-Net [Çiçek et al., 2016] and V-Net [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF] were proposed to deal with 3D medical data. Both these two models have skip connections among encoder and decoder. Similar encoder-decoder models are SegNet [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF], LinkNet [START_REF] Chaurasia | Linknet: Exploiting encoder representations for efficient semantic segmentation[END_REF], Deeplab [START_REF] Chen | Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[END_REF], PSPNet [START_REF] Zhao | Pyramid scene parsing network[END_REF], FPN [Lin et al., 2017a], U-Net++ [Zhou et al., 2018], Double U-Net [START_REF] Jha | Doubleu-net: A deep convolutional neural network for medical image segmentation[END_REF], etc.

Attention-based models

In psychology, attention is the cognitive process of selectively focusing on one or several things at the expense of others. The attention mechanism was first used in natural language processing. Then the Transformer [START_REF] Vaswani | Attention is all you need[END_REF] with only attention mechanism makes further progress. In computer vision including image segmentation, attention mechanisms are also starting to come to the fore, such as Squeeze-and-excitation networks [START_REF] Hu | Squeeze-and-excitation networks[END_REF], Attention U-Net [START_REF] Oktay | Attention u-net: Learning where to look for the pancreas[END_REF], Vision Transformer [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF], Swin Transformer [Liu et al., 2021a], and the very recent CHAPTER 2. STATE OF THE ART ConvNeXT [Liu et al., 2022].

Models for image sequence

In medical image segmentation, some data are based on time series, for instance, the state of the heart/cardiac is different in each frame, and if we want to know the two states of the heart in systole and diastole, we have to find out these two states from different frames. To this end, the Recurrent Neural Networks (RNN) [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF] and long short-term memory (LSTM) neural networks [START_REF] Hochreiter | Long short-term memory[END_REF]] can be applied for extraction of spatial and temporal information from specific medical data and tasks.

Multi-task models

Multi-task learning [Caruana, 1997] is a machine learning method based on shared representation, where multiple related tasks are put together to learn. And the purpose of the shared representation among different tasks is to improve generalization. The concept of multi-task learning exists because previous models have been single-task learning. In medical image analysis, tasks such as object (organs) detection, segmentation (lesion), regression, classification (disease) have been achieved by multi-task models [START_REF] Zhang | Multimodal multi-task learning for joint prediction of multiple regression and classification variables in alzheimer's disease[END_REF], He et al., 2019, Si and Roberts, 2019, Lian et al., 2021, Jia et al., 2021].

Weakly supervised models

Due to the annotation of medical images is time-consuming and laborious in reality. Therefore, the weakly/semi-supervised or unsupervised learning model [START_REF] Chapelle | Semisupervised learning[END_REF] is an expedient way to compensate for the situation where there is only a small amount of labeled data and a large amount of unlabeled data, hoping to achieve the same or similar learning results as supervised learning with a fully labeled dataset. The overall idea of semi-supervised deep learning covers three types of architectures [Chaudhary, 2020]:

• Self training.
-Pseudo-label [Lee et al., 2013] -Noisy Student [START_REF] Xie | Selftraining with noisy student improves imagenet classification[END_REF] • Consistency Regularization.

CHAPTER 2. STATE OF THE ART

π-model [START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF] -Temporal Ensembling [START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF] -Mean Teacher [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semisupervised deep learning results[END_REF] -Virtual Adversarial Training [START_REF] Miyato | Virtual adversarial training: a regularization method for supervised and semi-supervised learning[END_REF] -Unsupervised Data Augmentation [START_REF] Xie | Unsupervised data augmentation for consistency training[END_REF] • Hybrid Method (Combining self training and consistency regularization).

-MixMatch [START_REF] Berthelot | Mixmatch: A holistic approach to semi-supervised learning[END_REF] -FixMatch [START_REF] Sohn | Fixmatch: Simplifying BIBLIOGRAPHY semi-supervised learning with consistency and confidence[END_REF]

Loss functions

The loss function (also called cost or objective function) is one of key components in deep learning models that drives the optimization of the neural networks. Because it dictates how the error between the predicted value and the ground truth is computed and backpropagated throughout the networks. In this section, we will tions are usually agnostic to network architectures, and can be used for any segmentation tasks in a plug-and-play way.

We have witnessed the popularity of medical image segmentation challenges during the past years. These challenges serve as public benchmarks to evaluate and compare different segmentation methods proposed by researchers around the world. However, to the best of our knowledge, there is no comprehensive comparison and evaluation of these loss functions. Most existing loss functions are proposed and evaluated with different network structures as well as on different datasets. Moreover, all the studies only com-

Loss function taxonomy

We classify loss functions into four categories based on how they are derived, namely, the mismatch in distribution, region, boundary or some combination of these. Moreover, we explore the relationships between these loss functions. Fig. 1 shows the four categories and the connections between loss functions.

Let I be an image on a domain ⊂ R 2 or R 3 , and S, G denote the corresponding segmentation result and ground truth, respectively. s , g denote the predicted segmentation and ground truth

(p i ) of Sigmoid (p i = 1 1+e -x i ) or Softmax (p i = e x i C j =1 e
x j ) in multi-class (C), x is weight value of each pixel/neuron (i ). The other one is the corresponding class ground truth (g i ). Thus, the average of Binary Cross Entropy loss (BCE) over N pixels in an predicted image is composed by foreground and background two parts:

BCE = - 1 N N i =1 [g i • l og (p i ) + (1 -g i ) • l og (1 -p i )] (2.1)
If the background is much larger than the foreground, the loss is still small even though the segmentation result is inaccurate. This is the so-called class imbalance problem. To solve this class problem, then different weights need to be set on different terms.

Weighted Cross Entropy

The wighted cross-entropy (WCE) has been used in [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. The two-class form of WCE can be expressed as

WCE = - 1 N N i =1 ωg i log(p i ) + (1 -g i ) log(1 -p i ), (2.2)
where ω = (N -N i =1 p i )/ N i =1 p i , which is the weight of foreground class. ω is inversely proportional to the class frequency in order to penalize the major class (in this case is the background).

α = 0.75. FL = - 1 N N i =1 [α • g γ i • log(p i ) + (1 -α) • (1 -g i ) γ • log(1 -p i )] (2.3)
Distance map penalized cross entropy loss (DPCE), it [START_REF] Caliva | Distance map loss penalty term for semantic segmentation[END_REF] is also a variant of Cross Entropy loss. In Equation 2.4, D is the distance penalty term of CHAPTER 2. STATE OF THE ART foreground class, specifically, D is euclidean distance matrix1 of the ground truth.

And is the Hadamard product2 . In this way, pixels on the boundary can be given greater weights.

DPCE = - 1 N N i =1 (1 + Di st (g i )) g i log(p i ) + (1 -g i ) log(1 -p i ) (2.4)

Region based loss

Sensitivity-Specificity error (SSE) This loss function [START_REF] Brosch | Deep convolutional encoder networks for multiple sclerosis lesion segmentation[END_REF] combines mean squared difference between lesion region (sensitivity) and non-lesion region (specificity), regularized by a parameter r to control the ratio between this two parts.

The benefit of mean squared errors is generating smooth gradients, so that making robust optimization results.

SSE = r N i =1 (p i -g i ) 2 p i N i =1 p i + (1 -r ) N i =1 (p i -g i ) 2 (1 -p i ) N i =1 (1 -p i ) (2.5)
Dice loss It originates from the Dice coefficient [Dice, 1945] which calculates the overlap between ground truth and the segmented image. If the Dice score is 1, which indicates that the predicted image matches perfectly with ground truth data.

Here, in order to make loss converge, let the Dice be negative and plus 1. It was first used in V-Net [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF], now it has been widely used in medical image segmentation tasks.

DICE = 1 - 2 N i =1 p i g i N i =1 (p i + g i ) (2.6)
IoU loss Intersection over Union (IoU) loss [START_REF] Rahman | Optimizing intersection-over-union in deep neural networks for image segmentation[END_REF] is similar to the Dice loss, also called Jaccard loss, which is defined as:

IoULoss = 1 - N i =1 p i g i N i =1 (p i + g i -p i g i ) (2.7)

Generalized Dice loss

The authors [START_REF] Sudre | Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations[END_REF] add weights on the Dice loss for multi-class segmentation problem. The weight (w

= 1/( N i =1 g i ) 2 ) is inversely CHAPTER 2.

STATE OF THE ART

proportional to the ratio of that class.

GDL = 1 - 2 C c w c N i =1 p i g i C c w c N i =1 (g i + p i ) (2.8)

Lovász loss

The idea of Lovász loss [START_REF] Berman | The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks[END_REF]

m i =    1 -p i , if g i = 1 p i , otherwise (2.9)
Second, because the Jaccard loss ∆ J is submodular, then the Lovász extension can be used to compute the loss.

∆ J = 1 -IoU = N i =1 m i N i =1 g i ∪ N i =1 m i (2.10) Lovász l oss : ∆ J = N i =1 m i d el i (Sor t ed (m i )) (2.11) with d el i (•) = ∆(•) -∆(•), Sor t ed (m i )
, being a decreasing ordering the m i .

The author mentions in the paper and in the code that it is best to use it in combination with Cross Entropy loss, or to train the network with Cross Entropy first and then use the Lovász loss to finetune.

Tversky loss It [START_REF] Salehi | Tversky loss function for image segmentation using 3d fully convolutional deep networks[END_REF] adapts the Dice loss (Equation 2.6) in order to achieve a trade off between Precision (Equation 2.20) and Recall (Equation 2.21).

Note that when α = β = 0.5, the Tversky loss becomes Dice loss.

TL = 1 - C c N i =1 p c i g c i C c N i =1 p c i g c i + α C c N i =1 p c i (1 -g c i ) + β C c N i =1 (1 -p c i )g c i (2.12)
Focal Tversky loss (FTL) The Focal Tversky loss [START_REF] Abraham | A novel focal tversky loss function with improved attention u-net for lesion segmentation[END_REF] is proposed to improve Precision and Recall balance. The definition is as below, the γ in the paper is in 1,2,3. Note that when γ = 1, it becomes Tversky loss.

FTL = (TL) 1 γ (2.13) CHAPTER 2.

STATE OF THE ART

Asymmetric similarity loss (ASL)

The motivation for ASL loss [START_REF] Hashemi | Asymmetric loss functions and deep densely-connected networks for highly-imbalanced medical image segmentation: Application to multiple sclerosis lesion detection[END_REF] is also to better adjust the weights of FP and FN (and to achieve a better balance between Precision and Recall), for which a weighting parameter β is introduced, defined as follows:

ASL = 1 - C c N i =1 p c i g c i C c N i =1 p c i g c i + β 2 1+β 2 C c N i =1 p c i (1 -g c i ) + 1 1+β 2 C c N i =1 (1 -p c i )g c i (2.14)
Note that when α + β = 1, the ASL becomes Tversky loss.

Distance based loss

This type of loss functions is aiming to minimize the distance between predicted results and the ground truth.

Boundary loss (BL)

This loss [START_REF] Kervadec | Boundary loss for highly unbalanced segmentation[END_REF]

N i =1 [Di st (1 -g i )(1 -g i ) -(Di st (g i ) -1)g i ]p i (2.15)
Hausdorff Distance loss (HDL) It comes from the HD evaluation metric (See Equation 2.26). Because the HD metrics can't be used as loss functions directly, so the authors [START_REF] Karimi | Reducing the hausdorff distance in medical image segmentation with convolutional neural networks[END_REF] utilize the distance map to approximate the distance.

HDL = 1 N N i =1 [(p i -g i ) (Di st (g i ) 2 + Di st (p i ) 2 )] (2.16)
One should note that both of these two distance-based loss functions are combined with region-based loss in order to keep stability as mentioned in their experiments. [START_REF] Ribera | Locating objects without bounding boxes[END_REF] have the similar idea based on Hausdorff distance who proposed a loss function called "weighted Hausdorff distance" loss for object localization. Another loss function called "contour loss" that takes into account distance information via the distance map of the ground truth, has shown interesting smoothing effect in a 3D segmentation setting [START_REF] Jia | Automatically segmenting the left atrium from cardiac images using successive 3d u-nets and a contour loss[END_REF].

CHAPTER 2. STATE OF THE ART

Hybrid loss

Another type of loss function is to combine one loss function with another loss function with a weighted value in order to dealing with the same issue, which is class unbalanced problem. In the work of [START_REF] Trullo | Segmentation of organs at risk in thoracic ct images using a sharpmask architecture and conditional random fields[END_REF], they combined Cross Entropy loss and Weighted Cross Entropy loss. Three hybrid loss functions are listed below:

Dice+Cross Entropy [START_REF] Taghanaki | Combo loss: Handling input and output imbalance in multi-organ segmentation[END_REF]

Evaluation metrics in segmentation

Evaluating the segmentation results can reflect the strengths and weaknesses of a segmentation method. The following evaluation metrics are coefficients commonly used in medicine, and some are also statistical concepts often used in industrial production. Meanwhile, some loss functions are also evolved based on the evaluation metrics described in Section 2.1.3. Generally, there are two categories of evaluation metrics in segmentation results, one is region based metrics, the other on is distance based metrics. Here, we use Seg as segmentation results and GT as ground truth in the following mathematical expressions.

Region based metrics

Precision Precision (also called positive predictive value) is the proportion of true positives (TP = Seg • GT) out of all detected positive instances including false posi-

Dice coefficient

The definition of Dice coefficient [Dice, 1945] is the proportion of overlap region over segmentation and ground truth in foreground part.

Di ce = 2Seg • GT Seg + GT (2.23)

Jaccard coefficient

The idea of Jaccard coefficient is similar with Dice coefficient, but a little different in mathematical formula.

Jacc

ar d = Seg • GT Seg + GT -Seg • GT (2.24) CHAPTER 2.

STATE OF THE ART

The relationship of Jaccard coefficient and Dice is as below [START_REF] Taghanaki | Deep semantic segmentation of natural and medical images: a review[END_REF]:

Jacc ar d = Di ce 2 -Di ce (2.

Direct biomarker estimation methods

In the last section (Section 2.1), we introduce image segmentation methods based on traditional algorithms and deep learning models. In fact, in medical image segmentation, in most cases, the segmentation result is only an intermediate step.

This is because segmented areas are designed to quantify geometric factors such as perimeter, area or volume, which are then further translated into some sort of biomarker in clinical medicine. Therefore, there have been researches that try to skip segmentation, and focus on direct estimation the biomarker. In the following sections, we will introduce the direct prediction objects as well as methods. [START_REF] Afshin | Global assessment of cardiac function using image statistics in mri[END_REF][START_REF] Zhen | Direct volume estimation without segmentation[END_REF], histogram of oriented gradients (HoG) [START_REF] Zhen | Direct estimation of cardiac bi-ventricular volumes with regression forests[END_REF], supervised descriptor learning (SDL) [Zhen et al., 2015a] etc at first. Then these features are sent into different models such as artificial neural networks (ANN) [START_REF] Afshin | Global assessment of cardiac function using image statistics in mri[END_REF], support vector machine (SVM) [START_REF] Afshin | Regional assessment of cardiac left ventricular myocardial function via mri statistical features[END_REF], Sun et al., 2017], Bayesian model [START_REF] Wang | Direct estimation of cardiac biventricular volumes with an adapted bayesian formulation[END_REF][START_REF] Zhen | Direct volume estimation without segmentation[END_REF]] and random forest [START_REF] Zhen | Direct estimation of cardiac bi-ventricular volumes with regression forests[END_REF][START_REF] Zhen | Direct volume estimation without segmentation[END_REF], Zhen et al., 2015a, Zhen et al., 2016a, Li et al., 2017] to regress the estimated results by regressional objective function like mean absolute error or mean square error. Clustering method can also be used in biomarker estimation, in [START_REF] Ivanov | Automatic calculation of left ventricular volume in magnetic resonance imaging using an image-based clustering approach[END_REF], the authors address with

Traditional machine learning methods on direct estimation

LV volume estimation problem in 3 steps: 1.Locate LV; 2. Identify ED, ES, calculate area of LV by performing clustering algorithm so that the largest cluster of the image is considered to be the left ventricle; 3. Compute volume of LV. There are other machine learning algorithms such as manifold learning [START_REF] Wang | Prediction of clinical information from cardiac mri using manifold learning[END_REF], Sun et al., 2017, Tan et al., 2020], multi-output and multi-target regression [START_REF] Zhen | Descriptor learning via supervised manifold regularization for multioutput regression[END_REF], Zhen et al., 2017a[START_REF] Zhen | Direct and simultaneous estimation of cardiac four chamber volumes by multioutput sparse regression[END_REF] and regularization method [START_REF] Gu | Sparse regression with output correlation for cardiac ejection fraction estimation[END_REF].

Table 2.1 -Traditional machine learning methods for biomarker estimation from cardiac images.

Reference Estimation object

Method Data [START_REF] Afshin | Global assessment of cardiac function using image statistics in mri[END_REF]] EF Manual feature extraction+ANN 2D Cardiac MRI data [START_REF] Afshin | Regional assessment of cardiac left ventricular myocardial function via mri statistical features[END_REF] volume of LV Manual feature extraction+SVM 2D Cardiac MRI data [START_REF] Wang | Direct estimation of cardiac biventricular volumes with an adapted bayesian formulation[END_REF] volume of bi-LV Manual feature extraction+Bayesian model 2D Cardiac MRI data [START_REF] Zhen | Direct estimation of cardiac bi-ventricular volumes with regression forests[END_REF] volume of bi-LV Manual feature extraction+Random forest 2D Cardiac MRI data [START_REF] Wang | Prediction of clinical information from cardiac mri using manifold learning[END_REF] clinical variables Manifold learning 2D Cardiac MRI data [START_REF] Zhen | Direct and simultaneous estimation of cardiac four chamber volumes by multioutput sparse regression[END_REF]

volume of four chamber

Multi-ouput and multi-target regression 2D Cardiac MRI CT data [START_REF] Ivanov | Automatic calculation of left ventricular volume in magnetic resonance imaging using an image-based clustering approach[END_REF] volume of LV Clustering method 2D Cardiac MRI data

Deep learning methods on direct estimation

Compared to early machine learning methods, which are featured with multi-stage learning. The deep learning methods often come with end-to-end learning, more importantly, they can automatically learn features from images by various CNN architectures. Therefore, researchers are dedicated to designing high-efficiency networks to specific applications. We summarize the deep learning methods on direct quantification of different applications into Table 2.2 from its origins to recent re-

CHAPTER 2. STATE OF THE ART

search findings, which is mainly in the area of medical imaging.

Table 2.2 -Deep learning methods on direct quantification of different applications.

Reference Application Method

CNN+regression (Reg) model [START_REF] Riegler | Hough networks for head pose estimation and facial feature localization[END_REF] Head pose estimation Hough Forests with CNNs+Reg [START_REF] Luo | A novel left ventricular volumes prediction method based on deep learning network in cardiac mri[END_REF] LV volume estimation 8-layer CNN models with MSE loss [START_REF] Chen | Automated mitosis detection with deep regression networks[END_REF] Mitosis counting for breast cancer CNN+Reg layer [START_REF] Zhang | Direct estimation of fetal head circumference from ultrasound images based on regression cnn[END_REF] Head Circumference prediction Pretrainded CNN+Reg

Multi-scale [Zhen et al., 2016a] Bi-ventricle volume estimation Multi-scale kernels [START_REF] Luo | Multi-views fusion cnn for left ventricular volumes estimation on cardiac mr images[END_REF] LV volume estimation Multi-view input(2CH,top+mid) [Zhang et al., 2020a] Quantify Coronary Artery Stenosis Multi-view parallel feature fusion [START_REF] Li | Direct estimation of left ventricular ejection fraction via a cardiac cycle feature learning architecture[END_REF] LV volume estimation Cascaded feature fusion [START_REF] Luo | Dynamically constructed network with error correction for accurate ventricle volume estimation[END_REF] Bi-ventricle volume estimation Multi-view input and feature fusion Multi-task [START_REF] Xue | Full left ventricle quantification via deep multitask relationships learning[END_REF] Quantify all LV indices(11) CNN+RNN [START_REF] Dangi | Left ventricle segmentation and quantification from cardiac cine mr images via multi-task learning[END_REF] LV Seg, cardiac indices estimation U-Net+Reg [START_REF] Xu | Mutgan: Simultaneous segmentation and quantification of myocardial infarction without contrast agents via joint adversarial learning[END_REF] MI Seg and quantification Multi-task GAN, Generator: Reg+Seg; Discriminator:Bi-LSTM networks [Luo et al., 2020a] Bi-ventricle volume estimation Seg and Reg module and mutual authentication module between them [START_REF] Liu | Multislice left ventricular ejection fraction prediction from cardiac mris without segmentation using shared sptdennet[END_REF] EF estimation Classification+Regression [START_REF] Vesal | Spatiotemporal multi-task learning for cardiac mri left ventricle quantification[END_REF] LV indices quantification Classification+Segmentation+Regression [START_REF] Yu | Multitask learning for estimating multitype cardiac indices in mri and ct based on adversarial reverse mapping[END_REF] LV indices quantification Shared parameters between MRI and CT. [START_REF] Zhao | Of-umrn: Uncertaintyguided multitask regression network aided by optical flow for fully automated comprehensive analysis of carotid artery[END_REF] Carotid artery indices estimation Cell detection, segmentation, classification

Attention mechanism [START_REF] Pang | Direct automated quantitative measurement of spine by cascade amplifier regression network with manifold regularization[END_REF] Multiple indices of spine estimation Cascade feature amplifier network [Ge et al., 2019a] LV indices quantification Attention junction from Seg to Quantify [START_REF] Huang | Direct full quantification of the left ventricle via multitask regression and classification[END_REF] LV indices quantification Attention integrated into decoder Segmentation (Seg)/reconstruction based regression [START_REF] Du | Direct segmentation-based full quantification for left ventricle via deep multi-task regression learning network[END_REF] estimate the EF Seg results as regression CNN input [Liu et al., 2018] LV volume estimation Seg module(U-Net) and Regression CNN [START_REF] Wang | Quantification of full left ventricular metrics via deep regression learning with contour-guidance[END_REF] LV indices quantification Seg module and Regression module [START_REF] Pereira | Fully automated quantification of cardiac indices from cine mri using a combination of convolution neural networks[END_REF] LV indices quantification Seg results and original images as regression model input [START_REF] Gessert | Left ventricle quantification using direct regression with segmentation regularization and ensembles of pretrained 2d and 3d cnns[END_REF]

Multi-model fusion

Another idea [START_REF] Zhang | Direct quantification for coronary artery stenosis using multiview learning[END_REF], Zhang et al., 2020a] utilise isolated 3D convolution networks in each view then fuse each corresponding regression models for extracting multi-view features of coronary artery. [START_REF] Li | Direct estimation of left ventricular ejection fraction via a cardiac cycle feature learning architecture[END_REF] proves that feature fusion (through 3 cascaded modules,the cardiac cycle extraction module, the motion feature extraction module, and the fully connected regression module) has a positive effect on the direct estimation of the LV. [START_REF] Luo | Dynamically constructed network with error correction for accurate ventricle volume estimation[END_REF] not only fuses different views of input medical image slices, but also fuses different models to dynamically rectify the prediction results. this work is real-valued. The hidden layer is composed of K groups each of which is a binary array of N H × N H . Each of the K group in the hidden layer is associated with an N W × N W filter.

The energy function for a CRBM is defined as

ll E(v , h ) = - K k =1 N H i, j=1 N W r,s =1 h k i j W k rs v i + r-1 , j+ s -1 (1) - K k =1 b k N H i, j=1 h k i j -c N V i, j=1 v i j , ( 2
)
convolution as .

E(v , h ) = K k =1 h k • ( ˜ W * v ) - K k =1 b k i, j h k i, j -c i, j v i j

Cardiac image representation by deep CNN

To obtain expressive representations of cardiac images, we den for cardiac images a new CNN which is adequate to capture variations of cardiac structures and appearances with a small e of network parameters ( ∼ 1M). Given the fact that all cardiac ages share approximately the same spatial layout, which is of s variation than natural images, we configure a relatively low mber of filters for each convolution layer to avoid model reduncy. As for the kernel size of convolution and pooling, 5 × 5, inad of the frequently used 3 × 3, is deployed to introduce more ft invariance. Dropout and batch normalization are adopted to viate the training procedure. As can be seen in our experints, our CNN is very effective for cardiac images even without nsfer learning. As a representation learning network, our CNN ps each cardiac image X s, f into a fixed-length low dimensional tor:

f = f cnn (X s, f | w cnn ) , (1)
ere e s, f ∈ R 100 , and w cnn is the set of parameters in our CNN. related features in each local area from the structures of all feature maps during the training procedure.

Dynamic modeling of cardiac sequences by RNN

Accurate modeling of cardiac temporal dynamics improve quantification accuracy of current frame with information neighboring frames. The four tasks in this work can be divide two types: (1) the three types of LV indices, which are main lated to the spatial structure of cardiac LV in each frame, an cardiac phase, which is mainly related to the structure diffe between successive frames. Therefore, we design two RNN ules, as shown in Fig. 5 , with each of them extracting the re features and modeling the corresponding temporal dynamics two RNN modules have the same architecture and share the CNN embeddings as input. They differ in the parameters tha trained with different supervised information. RNN-1 modul be trained to predict the LV indices, while RNN-2 modules w trained to identify cardiac phase. The outputs of RNN module Multi-task learning, which literally means different tasks (models) are integrated in one networks and they are learning simultaneously. The benefit of multi-task learning is that the weights are shared in different tasks, so that each task can learn sufficient and complementary features from input medical data and corresponding loss functions. In [START_REF] Xue | Full quantification of left ventricle via deep multitask learning network respecting intra-and inter-task relatedness[END_REF], Xue et al., 2018, Du et al., 2018], the authors combine the CNN and RNN architectures to estimate the indices of cardiac and two phases ED and ES (see Figure 2.7). In [START_REF] Dangi | Left ventricle segmentation and quantification from cardiac cine mr images via multi-task learning[END_REF], Chen et al., 2020b], the authors combine the U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] and regression layer as multiple output layers, so that the regression layer can not only learn the input image but also learn from segmented images, and this two parts of weights are shared. More advanced, [Luo et al., 2020a] combines the segmentation model (FCN [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]) and regression model, they also build a mutual authentication bridge between this two model through a loss function to minimize the difference between two output modules. In [START_REF] Xu | Mutgan: Simultaneous segmentation and quantification of myocardial infarction without contrast agents via joint adversarial learning[END_REF], the authors combine the generator and discriminator network with regression layer to achieve segmentation and direct estimation of multiple cardiac indices (Myocardial Infarction, MI).

{ h s, 1 m , . . . h s,n F m } = f rnn ([ e s, 1 , . . . e s,n F ] | w m ) ,
In [Ge et al., 2019a], the authors combine segmentation models with RNN models as well as regression models. In [START_REF] Huang | Direct full quantification of the left ventricle via multitask regression and classification[END_REF], Liu et al., 2020], there is one model with two output branches, they are regression layer for direct indices prediction and classification layer, respectively. In [START_REF] Vesal | Spatiotemporal multi-task learning for cardiac mri left ventricle quantification[END_REF], their model perform 3 tasks simultaneously, respectively are segmentation, regression and classification.

The multi task learning can also happen in different image modalities. That is to say, the parameters from the two networks learned from the source modality (MRI) are shared with the target modality (CT) [START_REF] Yu | Multitask learning for estimating multitype cardiac indices in mri and ct based on adversarial reverse mapping[END_REF].

Segmentation and reconstruction based regression

Due to the original images have unclear features and noise, it's natural to think up of using segmentation or reconstruction results to estimate the indices from medical images. There are several indices estimation methods are based on the segmentation results [Liao et al., 2017, Du et al., 2018, Wang et al., 2019, Liu et al., 2018, Tao et al., 2019, Pereira et al., 2020], they first utilise segmentation neural networks (e.g. U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]) to obtain binary or multi-class segmentation results, then it will be is easier to estimate indices through regression CNN.

The same idea was used in [START_REF] Gessert | Left ventricle quantification using direct regression with segmentation regularization and ensembles of pretrained 2d and 3d cnns[END_REF], but the models (2D and 3D CNN) are initialized with pretrained weights from ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF].

This operation require two kinds of labels, one is ground truth contour of to-besegmented images, the other is ground truth of indices. In [Xue et al., 2017a], they is effective enough for the regression task, since it can easily learn more task-relevant representations from the segmented images.

C. TRAINING STRATEGY 1) PRE-TRAINING THE DENSENET

We first pre-train the DenseNet to provide good initial parameters for the following end-to-end training. Objective function of the DenseNet can be expressed as follows:

ŷc s,f ,(m,n) = f Dense (x s,f |w Dense ), (2)
L log = - s,f ,(m,n) c y c s,f ,(m,n) logŷ c s,f ,(m,n) S × F × M × N , (3)
where y c s,f ,(m,n) is the annotated category for each pixel.

Hybrid statistical learning with deep learning

Although deep learning-based methods have excellent performance, the previous statistical or machine learning methods also performed well. Thus, some researchers combine machine learning and deep learning methods to get a better results. In [Zhen et al., 2016a], they combine CNNs with random forest to estimate bi- this work is real-valued. The hidden layer is composed of K groups each of which is a binary array of N H × N H . Each of the K group in the hidden layer is associated with an N W × N W filter.

The energy function for a CRBM is defined as

ll E(v , h ) = - K k =1 N H i, j=1 N W r,s =1 h k i j W k rs v i + r-1 , j+ s -1 (1) - K k =1 b k N H i, j=1 h k i j -c N V i, j=1 v i j , ( 2
)
where b k is the bias for each group and c is the bias shared by all visible nodes. The energy function can be represented in terms of convolution as

E(v , h ) = K k =1 h k • ( ˜ W * v ) - K k =1 b k i, j h k i, j -c i, j v i j

Temporal and spatial networks

In cardiovascular disease diagnose, Ejection Fraction (EF, see Equation 1.1) is a common metric, of which doctors need to know the volume in ED and ES two phases, one way for identifying ED or ES is recognizing them by experienced doctors' eyes. For instance, in the ACDC dataset [START_REF] Bernard | Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved[END_REF], the ED and ES are already labeled by experts. However, with the number of images increasing, manual recognizing two phases is labouring. Thus, researchers come up with automatic identifying ED and ES by using temporal and spatial networks [START_REF] Xue | Direct estimation of regional wall thicknesses via residual recurrent neural network[END_REF], Luo et al., 2019[START_REF] Ge | Pv-lvnet: Direct left ventricle multitype indices estimation from 2d echocardiograms of paired apical views with deep neural networks[END_REF]. Specifically, the networks [START_REF] Xue | Direct estimation of regional wall thicknesses via residual recurrent neural network[END_REF] are composed of Recurrent Neural Networks (RNN) or Long Short-term Memory (LSTM) and CNNs. Fig. 2.10 is one temporal regression CNN networks. The input data usually is 4D MRI or CT data, that is 3D image as well as different frames of the process of a heart beat. The networks [START_REF] Luo | A novel spatio-temporal self-supervised framework to improve the generalization ability for left ventricle volume quantification based on cmr data[END_REF] consists of a shared weights framework but has two outputs, one is the temporal ordered value (judging ED or ES), the other one is estimated volume of LV, each task has a corresponding loss function, this is also counted as multitask learning. A more flexible networks [START_REF] Huang | Direct full quantification of the left ventricle via multitask regression and classification[END_REF]] that can support any number of frame input so that it predicts the left ventricle indices frame-by-frame through encoder (gated recurrent unit, GRU [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF]) and decoder (GRU with attention mechanism). poral patterns of the cardiac sequences, and give predictions for the ED and ES frames during testing. The contribution of our work is twofold: 1) A deep temporal regression network is designed to recognize the ED and ES frames; and 2) A temporal structured loss is proposed to improve the accuracy of the network. Although deep learning has been widely used for medical image analysis [2,7,12,16], our network architecture is novel and carefully designed for this use case. This approach has several advantages compared to the previous methods: 1) No prior information or interaction is needed in the detection framework, since our system automatically learns everything from the patterns of the data.2) Since RNN is able to learn long-term patterns, our framework can detect the complex and long temporal dynamics in the cardiac sequence.

Methodology

In this section, we provide an overview of our TempReg-Net framework. Then, we show that our framework can be trained end-to-end by jointly optimizing the regression and temporal structured constraints. As illustrated, the channel sub-module utilizes both max-pooling outputs and average-pooling outputs with a shared network; the spatial sub-module utilizes similar two outputs that are pooled along the channel axis and forward them to a convolution layer.

TempReg-Net Architectures

We first aggregate spatial information of a feature map by using both averagepooling and max-pooling operations, generating two different spatial context descriptors: F c avg and F c max , which denote average-pooled features and max-pooled features respectively. Both descriptors are then forwarded to a shared network to produce our channel attention map M c ∈ R C×1×1 . The shared network is composed of multi-layer perceptron (MLP) with one hidden layer. To reduce parameter overhead, the hidden activation size is set to R C/r×1×1 , where r is the reduction ratio. After the shared network is applied to each descriptor, we merge the output feature vectors using element-wise summation. In short, the channel attention is computed as: The attention mechanism is an additional option to amplify the learned features so that the model can well predict the values. It origins from natural language processing, then popular in computer vision. There are several kinds of attention methods: content-base attention [START_REF] Graves | Neural turing machines[END_REF], additive at-CHAPTER 2. STATE OF THE ART tention [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], [START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF] proposed location base attention, general attention and dot-product attention, scaled dot-product attention [START_REF] Vaswani | Attention is all you need[END_REF]. The attention mechanism in computer vision is usually divided into three major attention domains. The main ones are: spatial domain, channel domain, and mixed domain [START_REF] Woo | Cbam: Convolutional block attention module[END_REF], see Figure 2.11. Now quite a few literature start to utilize attention mechanism in different biomarker prediction tasks [START_REF] Pang | Direct automated quantitative measurement of spine via cascade amplifier regression network[END_REF], Pang et al., 2019, Ge et al., 2019a[START_REF] Huang | Direct full quantification of the left ventricle via multitask regression and classification[END_REF].

M c (F) = σ(M LP (AvgP ool(F)) + M LP (M axP ool(F))) = σ(W 1 (W 0 (F c avg )) + W 1 (W 0 (F c max ))), (2)

Medical image datasets used for direct estimation

We list the public medical image datasets that have been used for direct estimation, see

Evaluation metrics in regression

In general, the evaluation criteria in regression mainly compares the error between prediction results and ground truth value from medical images in different statistical methods. Some of these evaluation metrics can be also used as regression loss functions.

Mean absolute error (MAE) measures the error from predicted value and ground true value, see Formula 2.28, where N is the number of total samples, x i and y i are estimated and ground truth values.

MAE = 1 N N i =1 |x i -y i | (2.28)
Percentage MAE Usually only the MAE is not enough, one should also know the ratio of the prediction error compared to the true result. Thus the percentage MAE (PMAE) is defined by Formula 2.29, which is also called error rate.

Error rate

= N i =1 |x i -y i | y i * N (2.29)
Correlation coefficient Another common used metrics is correlation coefficient r , see Formula 2.30, x and y are the mean values estimated and ground truth values.

r = N i =1 (x i -x)(y i -y) N i =1 (x i -x) 2 (y i -y) 2
(2.30)

Root mean square error (RMSE) is also used in comparing the estimated value and ground truth value.

RMSE = N i =1 (x i -y i ) 2 N (2.31)
Statistical significance tests. These tests can be performed when the scores obtained by two methods are close, in order to decide the wheter the difference between them is significant or not. The t-test helps to determine whether the difference between two set of values, based on the average, is significant. The paired Wilcoxon test is a non-parametric alternative to paired t-test used to compare paired data. It's used when the data are not normally distributed.

Perspectives

Through this survey one can find that the direct quantification methods have promising prospects on medical images, such as in the fields of head circumference

CHAPTER 2. STATE OF THE ART

In the future, in addition to the existed research objects, the author believes that there will be more and more clinical medical indices that can be directly estimated by various efficient deep learning methods with less errors. So that they can assist doctors in their judgment and decision-making. For example, in oncology, the anthropometric parameters like muscle body mass (MBM), fat body mass (FBM), lean body mass (LBM), visceral adipose tissue (VAT), and subcutaneous adipose tissue (SAT) etc [START_REF] Decazes | Anthropometer3d: automatic multi-slice segmentation software for the measure-BIBLIOGRAPHY ment of anthropometric parameters from ct of pet/ct[END_REF] are important indices to evaluate a human's health state, which are potential possible applications.

Explainable Artificial Intelligence

In this section, we will present the explanation methods and tools that have been developed for deep learning models for vision applications, as well as how to evaluate these methods. A saliency map is supposed to highlight the pixels that most contributed to the network's decision, with regards to one specific image; it is also called pixel attri-CHAPTER 2. STATE OF THE ART bution [Molnar, 2019]. Figure 2.12 is an example of saliency map based on a image classification task from CIFAR-10 (Canadian Institute For Advanced Research)

Explanation methods

Saliency

dataset [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF]. The saliency map is a main carrier to visually explain the deep learning models. In the following contents, we will introduce several types of explanation methods that can generate saliency maps.

One can also extract the feature maps in the intermediate of the model in order

to observe what each layer has learned.

(( x i ) i )
, that is close to the input, but which's output value is 0, and uses this difference to estimate the attribution of each neuron recursively. Decomposition is continuous everywhere in the input domain: Two nearby points in the input space always have a similar explanation (provided that the function is continuous). Furthermore, the magnitude of the decomposition (size of the arrow) is proportional to the function value at a given point in space. Whereas sensitivity analysis measures a local effect.

[

x f ] j = i c j ∂x j ∂x i (x i ) i =( x i ) i • (x i -x i ) (2.34)
Layer Relevance Propagation (LRP) [START_REF] Bach | On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation[END_REF]: It attributes recursively to each neuron's input relevance proportional to its contribution of the neuron output.

CHAPTER 2. STATE OF THE ART

The magnitude of the contribution of each pixel or intermediate neuron is called

"relevance" values R. R j = k a j w j k 0, j a j w j k R k
Here, j and k are two neurons of any consecutive layers. We already know the relevance R in the output layer, so we'll start from there and use this formula iteratively to calculate R for every neuron of the previous layer. a denotes the activation of the respective neuron, and w is the weight between the two neurons.

Perturbation based methods

Local Interpretable Model-Agnostic Explanations (LIME) [Ribeiro et al., 2016]:

The principle of LIME method is to perturb the input and see how the predictions change. This turns out to be a benefit in terms of interpretability, because it can perturb the input by changing components that make sense to humans (e.g., words or parts of an image), even if the model is using much more complicated components as features (e.g., word embeddings).

Meaningful Perturbation [START_REF] Fong | Interpretable Explanations of Black Boxes by Meaningful Perturbation[END_REF]]: Similar to LIME method, in the work of meaningful perturbation, they delete some areas of the image and observe the influence to model's prediction. Their method is faster to converge than LIME. So far, most of explanation methods are based on post-hoc methods. In this section, we list and introduce classic explanation methods. In the end, Table 2.4

Miscellaneous

summarizes the main high cited explanation methods that are used in different areas.

Applications of explainable AI

According to the research of the existing literature, the XAI technology are used in various media or data such as text, image, graph, audio, electrocardiogram (ECG) etc. See Table 2.5. Moreover, explanation techniques would show up in each field such as medical, transportation, finance etc as long as they apply deep learning methods in these solutions.

Table 2.5 -Applications and medium of explanation methods

Media Applications

Text NLP [Liu et al., 2019], Finance, Social media, Sales, Human resources, Energy

Image Medical images [START_REF] Tjoa | A survey on explainable artificial intelligence (xai): Toward medical xai[END_REF], Natural images Graph GNN explainer [START_REF] Ying | Gnnexplainer: Generating explanations for graph neural networks[END_REF] Audio Speech recognition [START_REF] Becker | Interpreting and explaining deep neural networks for classification of audio signals[END_REF] ECG DeepExplain ECG [START_REF] Raghunath | Prediction of mortality from 12-lead electrocardiogram voltage data using a deep neural network[END_REF]

Evaluation of explanation methods

With widely used of explanation techniques on deep learning architectures, the evaluation standards of XAI should be established. In [START_REF] Arya | One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques[END_REF], they state the evaluations of XAI should be of: Competence, Fairness, Safety, Usability,Human-AI collaboration, Accountability, Privacy, in [START_REF] Goebel | Explainable ai: the new 42? In International cross-domain conference for machine learning and knowledge extraction[END_REF], they propose:

Comprehensibility, Succinctness, Actionability, Reusability, Accuracy, Completeness. In [START_REF] Samek | Explaining deep neural networks and beyond: A review of methods and applications[END_REF] NC TotalConductance [START_REF] Dhamdhere | How important is a neuron? In International Conference on Learning Representations[END_REF] TC DeepDreams [START_REF] Mordvintsev | Inceptionism: Going deeper into neural networks[END_REF] DD TCAV [START_REF] Kim | Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV)[END_REF] TCAV TCAV with RCV [START_REF] Graziani | Regression concept vectors for bidirectional explanations in histopathology[END_REF] TCAVR UBS [START_REF] Yeche | Ubs: A dimension-agnostic metric for concept vector interpretability applied to radiomics[END_REF] UBS SENN [Adebayo et al., 2018a] SENN CHAPTER 2. STATE OF THE ART that they are still abstract and not so mature to perform. In this section, we list 3 executable methods, they are:

• Sanity checks [START_REF] Adebayo | Sanity checks for saliency maps[END_REF] • Area over Perturbation Curve [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF] • Input variant [START_REF] Kindermans | The (un) reliability of saliency methods[END_REF]

Sanity checks

In the method of sanity checks [START_REF] Adebayo | Sanity checks for saliency maps[END_REF], they propose an idea to explain a deep neural networks model, that is to perform sanity checks on a certain model in both input data and model parameter two aspects to see the change of saliency maps.

The model parameter randomization test:

Comparing the trained and untrained two models, if there is no difference in the saliency maps, it indicates that the saliency map method is not sensitive to the inspection of the model parameters and is not helpful.

The data randomization test: Compare the data with labels and the data with replacement labels on the same trained model. If there is no difference in the saliency map, it means that the saliency map method does not depend on the relationship between the image and the label.

Another of their finding is that the image processing algorithm edge detection can also have a visual effect similar to saliency map, because it can extract the edge where the gradient is significant. Meanwhile, it does not rely on deep learning models or training data. This comparison indicates that visual analysis is not so sufficient and effective in judging whether an interpretation method is sensitive to models or data. The quantitative methods should be applied in the evaluation.

Therefore, the intention of so-called sanity check is to remove some explanation methods that are not sensitive to changes in models and data before implementing a specific method.

Area over Perturbation Curve

Explanation methods (also called analyzers, methods that analyzes the model) perform differently depending on the model, the task at hand, the data, etc. In order to quantitatively evaluate those analyzers, [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF] build upon the perturbation analysis, originally designed to assess explanation methods in classification networks. Let us first describe the perturbation process and then the evaluation metric.

First, the input image to be analyzed is subsampled by a grid. Each subwindow of the grid is ranked according to its importance w.r.t. to the pixel-wise saliency scores assigned by the analyzers. Then, the information content of the image is gradually corrupted by adding perturbation (Gaussian noise) to each subwindow, starting with the most relevant subwindow, w.r.t. the ranking just mentioned. The effect of this perturbation on the model performance is measured with the prediction error. This procedure is repeated for each subwindow.

Generally, the accuracy of model will drop quickly when important information is removed and remains largely unaffected when perturbing unimportant regions.

Thus, the analyzers can be compared by measuring how quickly their performance drops. That is to say, the quicker the model performance drops after introducing perturbation, the better the analyzer is capable of identifying the input components responsible for the output of the model.

The quantitative evaluation proposed in [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF] for classification network, consists in computing the difference between the score f (x) indicating the certainty of the presence of an object in the image x, in the presence and in the absence of perturbation. This difference is called Area over Perturbation Curve (AOPC) and defined more precisely defined in in [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF] as:

AOPC Anal y zer = 1 N N n=0 ( f (x n ) (0) - 1 K K k=0 f (x n ) (k) ) (2.35)
where N is the number of images, K is the number of perturbation steps, x is the input image.

Input variant [START_REF] Kindermans | The (un) reliability of saliency methods[END_REF] checks the reliability of saliency methods by preprocessing the input images, and they found that saliency methods that do not satisfy input invariance so that result in misleading attribution. This indicates that the saliency methods sometimes are not one hundred percent reliable so that inspire people to either abandon this method or seeks for other methods instead of saliency methods.

Libraries and tools of XAI

Perspectives

In some sensitive areas, such as autonomous driving and smart healthcare, the interpretability of deep models is essential because it is about whether the model's decisions are safe and reliable or explainable. Explainable AI validates the interpretability of existing deep learning models only in one aspect; other techniques are still needed to validate the robustness of the models.

Besides saliency methods, graph knowledge can be used in machine learning which can augment (intermediate) features with more semantics [Lecue, 2020]. The mentioned above explanation methods are mainly post-hoc methods, that is to say the deep learning methods themselves are not explainable. There is one voice claiming that the deep learning models should be self-explainable [Rudin, 2019].

Conclusion

In this chapter, we present related work in three major research areas, starting with medical image segmentation methods, where we introduce traditional image segmentation methods as well as various models and methods based on machine (deep) learning, where the loss function also plays a very important role in deep learning. Some of these loss functions are evolved from evaluation criteria in segmentation. Secondly, we present the method of direct biomarker prediction. In these methods, there is also a transition from traditional machine learning methods to deep learning regression models among others. Finally, we present in detail the concepts, methods, tools and application areas of explainable artificial intelligence.

The

Motivation

Today, CNN are the state-of-the-art in medical image segmentation. One key component of CNN is the loss function, that drives the backpropagation of the error between the predicted value and and the reference label. Cross Entropy is a widely used, standard loss function. However, as mentioned in previous chapter (Chapter 2.1), class imbalance problem is prone to happen especially in skin images, in which the lesion only takes a small proportion in the whole image. In order to handle class imbalance, weights can be assigned to samples of different classes. The Dice loss function [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF], a soft approximation of the well-known Dice metric, is specifically designed for image segmentation. However, the Dice loss only considers foreground (i.e. object) pixels, and does not take into account the background pixels in the image.

Therefore, in this work, we propose a loss function called Kappa loss, based on the Kappa index, that can not only deal with class imbalance problems in medical image segmentation, but also considers the whole information of an image. The motivating factor spurring our approach is rooted in the fact that all pixels should be taken into account, since a large part of the image is occupied by object (or in our case melanoma) pixels. We believe that by using the Kappa loss, we will enforce the constrain on the true negative pixels in addition to the true positive ones, just reaching a better balance between the two classes. On the other hand, we segment the skin lesion images using deep convolutional neural networks. We demonstrate the efficiency of the proposed loss on images of skin lesions or moles, which are typical cases where the lesion takes a significant part of the image.

The clinical problem: skin cancer detection from lesion segmentation

Diagnosis of skin lesion

Melanoma is a type of skin cancer, that, if not detected and treated within limited time, may be fatal, since it can spread to other organs quickly. According to a survey [Allan, 2019], in 2019, about 7230 people (4740 men and 2490 women) will lose their lives because of melanoma in the USA. Dermatologists establish their diagnosis by visual inspections of moles, and by extracting texture, size and shape analysis infor-56 CHAPTER 3. KAPPA LOSS FOR SKIN LESION SEGMENTATION mation. One first step is often the segmentation of the mole. Manual segmentation by dermatologists as described in [START_REF] Jafari | Skin lesion segmentation in clinical images using deep learning[END_REF]] is a time-consuming process, hardly compatible with the usual workload of medical experts, and that can be subjective.

Figure 3.1 -Samples of skin images: 1st column is from dataset Skin-Cancer-Detection, 2nd column from ISIC 2017, 3rd&4th column from ISIC 2018. In the skin image of 2-4th column, the object is large with respect to the image.

Related works in skin lesion segmentation

For the skin lesion image segmentation, early works utilized computer vision based methods. Computer vision based image segmentation methods have been thoroughly investigated before the advent of deep learning. [START_REF] Yuan | A narrow band graph partitioning method for skin lesion segmentation[END_REF] uses active contour to detect border of skin lesion. In [START_REF] Zhou | Gradient vector flow with mean shift for skin lesion segmentation[END_REF], the authors combine gradient vector flow with mean shift to segment the skin lesion images. Preprocessing is performed in [START_REF] Schaefer | Colour and contrast enhancement for improved skin lesion segmentation[END_REF] to tackle the problem of low contrast and color between background and object. [START_REF] Pennisi | Skin lesion image segmentation using delaunay triangulation for melanoma detection[END_REF] presents Delaunay

Triangulation to extract the contour of skin lesion image. Region merging based approach is used in [START_REF] Wong | Automatic skin lesion segmentation via iterative stochastic region merging[END_REF]. In [START_REF] Jain | Computer aided melanoma skin cancer detection using image processing[END_REF], authors apply image processing tools to extract features (Asymmetry, Border, Color, Diameter) of skin lesion in order to classify the image as melanoma or not. Later on, deep learning based skin image segmentation methods were proposed. The work of [START_REF] Jafari | Skin lesion segmentation in clinical images using deep learning[END_REF] includes 3 steps: preprocessing (image filtering), CNN, selection of largest area. In [START_REF] Attia | Skin melanoma segmentation using recurrent and convolutional neural networks[END_REF], the authors spend effort on aggregating convolutional and recurrent neural networks. In [START_REF] Yuan | Automatic skin lesion segmentation using deep fully convolutional networks with jaccard distance[END_REF], authors implement a CNN with a loss function based on Jaccard distance that can deal with class imbalance problem.

[ Xie et al., 2020a] design a deep learning architecture using attention mechanism that can generate high-resolution feature maps to preserve spatial details.

The Kappa loss

From metrics to loss

The Dice index (DI) is widely known as an overlap measure in binary image segmentation, defined as the ratio between twice the intersection of two regions over their union. The DI values range from 0 to 1, 0 meaning empty overlap while 1 indicates perfect match. The Dice Index was originally designed to be an inter-rater agreement [Dice, 1945], independently from the pixel labeling problem.

+ a b a + b - c d c + d Total a + c b + d N
Let us define as a the number of counts where raters agree positively, d the number of counts where raters agree negatively, and b and c where the two raters disagree with each other, N is the sum of a, b, c and d (see Table 3.1).

In the image segmentation, we regard the two raters as ground truth and the predicted image, Figure 3. account these true negative. In particular, the Kappa coefficient [Hubert, 1977], a chance-corrected measure of agreement voted by two raters, is defined as:

Kappa = 2(ad -bc) (a + b)(b + d ) + (a + c)(c + d ) (3.2)
As recalled in the pioneering paper [Zijdenbos et al., 1994] that first uses the Dice index as a metric to evaluate segmentation quality, the Dice index is a limit case of the Kappa index when d a, b, c:

lim d →∞ Kappa = 2a 2a + b + c = DI, (3.3)
Thus, that is to say, the Dice index only considers the foreground pixels to compute the overlap of the predicted region and the ground truth, based on the assumption that region or object pixels are small compared to the background area. However in some cases, especially in medical skin images, this assumption does not hold. We show examples of such cases, in the 3 images on the right in Figure 3.1. This is the rationale behind the use of the kappa index as loss function: all pixels in the image are taken into account, and not only the foreground pixels. Note that a weighted version of the kappa index has shown to be a loss of choice for ordinal classification, in comparison to logarithmic loss [de La Torre et al., 2018]. However, it was not introduced in the context of image segmentation.

Definition of the Kappa loss

In order for the Kappa coefficient to be used as a loss function in a CNN, it has to be differentiable so that its gradient may be computed. Thus the probabilities (i.e. output values of the last layer of the networks) have to be used, instead of hard labels, in the definition of the Kappa loss. We rewrite elements a, b, c and d by taking into account the predicted segmentation (or probability) at pixel i , denoted as p i , and

the ground truth at this same pixel, denoted as g i . The pixel-wise representation of these elements is shown in Formula 3.4, where N is the number of pixels in the image.

a = N i =1 (p i g i ), b = N i =1 (1 -p i )g i , c = N i =1 (1 -g i )p i , d = N i =1 (1 -p i )(1 -g i ) (3.4)
We obtain the soft approximation of the Kappa loss by replacing the affectations from Formula (3.4) in Formula (3.2).

Kappa

l oss = 1 - 2(ad -bc) (a + b)(b + d ) + (a + c)(c + d ) (3.5)
Then, substituting the soft proxy of Equation 3.4 into Kappa loss (Formula 3.5)

and simplifying it, we get:

Kappa l oss = 1 - 2 N i =1 p i g i -N i =1 p i • N i =1 g i /N N i =1 p i + N i =1 g i -2 N i =1 p i g i /N (3.6)
Deriving the Kappa loss with respect to predicted probabilities at pixel j , the gradient of Kappa loss is as Formula 3.7:

∂Kappa ∂p j = -2 g j ( g i + p i -2 p i g i /N) ( p i + g i -2 p i g i /N) 2 + p i g i (1 -2 g i /N 2 ) + ( g i ) 2 /N) ( p i + g i -2 p i g i /N) 2
(3.7)

In the case of Dice loss, Formula (3.6) of Kappa loss boils down to Dice loss [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF]:

Dice loss = 1 - 2 N i =1 p i g i N i =1 p i + N i =1 g i (3.8)
For this Dice loss, variants of this definition may have p 2 i + g 2 i instead of p i + g i in the denominator, or include a smoothness term (a small value) added to the denominator and the numerator [START_REF] Sudre | Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations[END_REF], Wong et al., 2018, Pedemonte et al., 2018], but that only helps in case of missing labels and is not critical. At this point, we have verified the theoretical derivation of Kappa loss and its derivability. Next, we will verify its feasibility and performance in combination with U-Net in our experiments. We have used the well-known U-Net architecture, one of the most popular CNN for medical image segmentation [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], to implement the Kappa CHAPTER 3. KAPPA LOSS FOR SKIN LESION SEGMENTATION loss function. The U-Net is a fully convolutional network with an encoder-decoder architecture and skip connections. Compared to the original architecture, we simplified the network, given the limited amount of images. The original U-Net includes 64 filters at the first level, for a total of 31,031,685 parameters. We set the initial number of filters to be 16 (3×3), so the number of parameters is 1,946,449. We also add a batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] operation after each ReLU activation function to avoid gradient vanishing. In order to avoid overfitting, we use drop out to reduce parameters in the U-Net with a rate of 0.5.

CNN for image segmentation

Experiments and results

Datasets

We use 6 publicly available datasets of skin images with mole or melanoma, to assess the proposed Kappa-based loss function. They are: Skin-Cancer-Detection (SCD, 206 images, supplied by Vision and Image Processing Lab, University of Waterloo), split into two subsets which are melanoma (Mel, 119 images) and notmelanoma (Non-mel, 87 images), and 3 datasets from International Skin Imaging Collaboration (ISIC) [START_REF] Codella | Skin lesion analysis toward melanoma detection: A challenge at 2017 isbi, hosted by the international skin imaging collaboration (isic)[END_REF], Tschandl et al., 2018, Codella et al., 2017, Gutman et al., 2016] which have 2594, 2000 and 900 images respectively. In the latter, images not only include a lesion part but also present noise such as hair, which increases the difficulty of segmentation. Moreover, the object (lesion area) in the image of dataset SCD or ISIC 2018 varies from one to another in size. We split each dataset into training set, validation set and test set, respectively, with the same amount of images in each set. Images are resized to 256 × 256.

Experimental settings

Because the amount of medical data is limited, we use data augmentation to increase the number of it. Data augmentation including rotation, shifting, shearing, zooming and flipping is used in training. Protocol is a 3-fold cross validation. The optimizer is Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with a learning rate of 1e -4 . The batch size is 4. The model is trained for 100 epochs. Evaluation metrics are the Dice index (DI) and the Hausdorff distance (HD), which is the maximum point-to-point distance between two contours. The implementation tool is based on Keras and Tensorflow 1.0. Tesla P100 GPU server is used in the experiments.

Results

Quantitative analysis

We trained the U-Net described in the previous section from scratch on the 6 datasets independently, with two different loss functions, the Dice loss that will be the baseline, and the Kappa loss. Results are shown in Table 3.2. The Hausdorff distances between Kappa and Dice losses are similar, which means that Kappa is not correcting distant, false positive pixels, except for the first dataset (Non-mel), where HD drops by 7%. However, substantial improvement on the Dice Index (DI) is obtained for the Kappa loss, in comparison to the Dice loss, for several datasets.

Qualitative analysis

Some segmentation results are shown in Figure 3.4, which shows that in some cases, the Kappa loss can help to make the segmentation more accurate. Looking at Figure 3.5, we can also observe that the Kappa loss converges faster than Dice loss under the same U-Net model settings.

Key feature maps of Kappa loss

According to example images in Figure 3.6, we extract several feature maps of U-Net from different layers with Dice loss and Kappa loss respectively (Figure 3.7-3.12).

Such that we can know the intermediate inference process. These three skin lesion images are with noise, small lesion and large lesion area from dataset ISIC 2016.

Those feature maps are selected from encoder and decoder of U-Net. One can find

Conclusion

In this work, we have proposed a new loss function, based on the Kappa index, to be used in CNN for medical image segmentation. Different from the Dice loss (which is as a baseline in this study), this loss function considers all pixels (background pixels included) in the evaluation of the predicted segmentation, i.e., the Kappa loss is a generalization of Dice loss. We believe that by enforcing constraint on both positive and negative pixels, segmentation accuracy or convergence may be improved. We have shown the Kappa loss differentiability and used the state-of-the-art U-Net architecture to implement it. We compared the Kappa loss quantitatively to the Dice loss on several public datasets of melanoma and skin segmentation. Promising results were obtained, showing the potential of the Kappa loss.

However, our work only compares the proposed Kappa loss with Dice loss, future work involves extending our benchmarking experiments to other loss functions, to further investigate the behavior of Kappa loss with respect to other loss functions; and generalizing the Kappa loss to multi-label image segmentation, as was proposed for the generalized Dice loss in [START_REF] Sudre | Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations[END_REF]. Kappa loss could also be used in multi-scale approaches, when segmentation is required inside a region of interest (e.g. bounding box), where there is a balance between positive and negative pixels. On the other hand, the proposed Kappa loss could also be used in other target segmentation tasks beside skin lesion segmentation, which is also part of author's work in the future. Meanwhile, the skin lesion segmentation is just an intermediate step for skin cancer diagnose, in other words, the specific indices such as ABCDEs [START_REF] Rigel | Abcde-an evolving concept in the early detection of melanoma[END_REF] need to be further quantified according to the skin lesion segmentation results.

Motivation

Very often, medical image segmentation is the first step to compute parameters from the image, such as volume: for example, the cardiac ventricles are segmented in magnetic resonance images in order to estimate the cardiac contractile function via some indices (e.g. ejection fraction) [START_REF] Petitjean | A review of segmentation methods in short axis cardiac mr images[END_REF]. Another example is anthropometry, where measuring the skeletal muscle body mass and fat body mass, which is a significant pronostic factors in cancer, are estimated from the segmentation of muscle and fat in CT images [START_REF] Zhen | Direct volume estimation without segmentation[END_REF], Hussain et al., 2016, Pang et al., 2018, Luo et al., 2020a, Zhang et al., 2020a[START_REF] Zhang | Direct estimation of fetal head circumference from ultrasound images based on regression cnn[END_REF].

Instead of resorting to segmentation, which is a costly and error-prone process, one can attempt to estimate the (single or multiple) characteristics or biomarkers, directly. Works on this topic have gotten a second wind with the breakthrough of deep learning, that allows to take advantage of the power of feature representation and to perform an end-to-end regression. However direct, "segmentationfree" approaches rely on much less information to estimate the biomarker, and it is not clear yet if segmentation-free approaches can reach the level of accuracy of segmentation-based approaches. To our knowledge, there is no study that rigorously compares segmentation based methods and segmentation-free methods for a given application of biomarker estimation, and quantifies the gap between them.

This observation motivates the present contribution, where we propose a fair, quantitative comparison of segmentation-based and segmentation-free (i.e. regression) approaches to estimate how far regression-based approaches stand from segmentation approaches, for a problem that has a major clinical impact: the estimation of the head circumference in US images. This estimation is important to accurately assess the growth of the fetus.

In this chapter, we investigate several settings, i.e. state of the art segmentation models and various backbones for the regression CNN architectures, to obtain the best of both worlds, and investigate also time and memory consumption in addition to estimation accuracy. To make the segmentation-free approaches more convincible, we adapt explanation methods in regression CNN and provide an interpretation of what a saliency map is, in the regression case. We are thus able to gain insight into the CNN regression model for our HC prediction problem, and see what pixels contribute the most to the estimation of the HC: we expect them to be those of the head contour. We also address the problem of evaluating the explanation methods, in the regression case. Adebayo's sanity checks consist in performing randomization tests, in the data or in the model, and evaluate the changes in the produced saliency maps [START_REF] Adebayo | Sanity checks for saliency maps[END_REF]. Another example is Samek's proposal, that has particularly inspired us [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF], to compare and assess different explanation methods. The principle is to inject noise gradually in the image, in locations that have been highlighted by the saliency maps, and see how the prediction is affected by this perturbation. However, the method is designed for classification networks and requires some adaptation.

HC measurement from US images 4.2.1 Background

Automated measurement of fetus head circumference (HC) is performed throughout the pregnancy as a key biometric to monitor fetus growth and estimate gestational age. In clinical routine, this measurement is performed on ultrasound (US) images, via manually tracing of the skull contour, along to fitting it to an ellipse, this being done by sonographers. Figure 4.1 is one sample of fetus head of ultrasound (US) image, from the HC18 public dataset [START_REF] Van Den | Automated measurement of fetal head circumference using 2d ultrasound images[END_REF] used in this paper. Identifying the head contour is challenging due to low signalto-noise ratio in US images, and also because the contours have fuzzy (and sometimes missing) borders (Figure 4.1). Manual contouring is an operator-dependant operation, which is measured by experienced sonographers by calipers, subject to intra and inter-variability, which yields inaccurate measurements, as measured in [START_REF] Sarris | Intraand interobserver variability in fetal ultrasound measurements[END_REF]: the 95% limits of agreement have been measured to ±7mm for the intra-operator variability and ±12mm for the inter-operator variability. Another study concluded that the sonographic measured HC consistently underestimates the actual postpartum HC by an average of 13.5 mm or 4% [START_REF] Melamed | Sonographic estimation of fetal head circumference: how accurate are we[END_REF].

Usually, automating the measurement of fetus head circumference in US images is achieved through automatic segmentation methodology. Segmentation methods typically involves image-processing or machine learning based approaches, some post-processing of the result, so as to fit it into an ellipse. This process involves multiple steps, is adhoc, and can be prone to error. Let us emphasize on the fact that here, the segmentation is just an intermediate step to compute a characteristic

Related works on head circumference estimation

Several approaches have been proposed in the literature to measure the head circumference in US images, based on image segmentation [START_REF] Li | Automatic fetal head circumference measurement in ultrasound using random forest and fast ellipse fitting[END_REF], Lu et al., 2005, Jardim and Figueiredo, 2005]. Some follow a two-step approach, namely fetus head localization and segmentation refinement. For example, in [van den Heuvel et al., 2018a], the first step consists in locating the fetus head with Haar-like features used to train a random forest classifier; and the second step consists in the measurement of the HC, via ellipse fitting and Hough transform. Similar method is used in [START_REF] Li | Automatic fetal head circumference measurement in ultrasound using random forest and fast ellipse fitting[END_REF].The above segmentation method is based on traditional machine learning. In recent years, deep learning-based head circumference segmentation algorithms have improved in terms of performance and efficiency.

These approaches build upon deep segmentation models also in a two-step process, contour prediction and ellipse fitting [START_REF] Kim | Automatic evaluation of fetal head biometry from ultrasound images using machine learning[END_REF]. In [START_REF] Budd | Confident head circumference measurement from ultrasound with real-time feedback for sonographers[END_REF], the standard segmentation model U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] is trained using manually labeled images, and segmentation results are fitted to ellipses. The mean absolute error (MAE) tested on HC18 dataset in [START_REF] Budd | Confident head circumference measurement from ultrasound with real-time feedback for sonographers[END_REF] is 1.90 mm, the Dice accuracy is 0.982, the Hausdorff distance (HD) is 1.292 mm. In [START_REF] Sobhaninia | Fetal ultrasound image segmentation for measuring biometric parameters using multi-task deep learning[END_REF], authors build upon the same idea, combining image segmentation and ellipse tuning together in a multi-task learning network. Their segmentation accuracy is 0.968 in Dice score, 1.72 mm in HD, 2.12 mm in MAE. In [START_REF] Fiorentino | A regression framework to headcircumference delineation from us fetal images[END_REF], the authors use first a region-proposal CNN for head localization, and a regression CNN trained on distance fields to segment the HC. [START_REF] Moccia | Mask-r 2 cnn: a distance-field regression version of mask-rcnn for fetal-head delineation in ultrasound images[END_REF] advances the work [START_REF] Fiorentino | A regression framework to headcircumference delineation from us fetal images[END_REF] since they propose CHAPTER 4. FETUS HEAD CIRCUMFERENCE PREDICTION Mask-R 2 CNN neural network to perform HC distance-field regression for head delineation in an end-to-end way, which does not need prior HC localization or postprocessing for outlier removal. All these methods rely on a segmentation of the fetus head as a prerequisite to estimating the HC.

The segmentation free approaches for biomarker estimation have been introduced in Chapter 2. Throughout the above literature, there are no studies based on ultrasound images and there are no methods to directly measure fetus head circumference. Therefore, based on the above studies that have been successfully implemented for objects such as the heart, one of the core tasks of this thesis is to find a scheme to directly predict fetus head circumference, to explore the explainability of the method, and to compare it with segmentation-based methods.

Methodological framework

Head circumference estimation based on segmentation CNN segmentation model

We investigate several segmentation architectures which are the state of the art network in medical image segmentation, to segment the contour of fetus head: the well-known U-Net model [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], U-Net++ [Zhou et al., 2018], DoubleU-Net [START_REF] Jha | Doubleu-net: A deep convolutional neural network for medical image segmentation[END_REF], FPN [Lin et al., 2017a], LinkNet [START_REF] Chaurasia | Linknet: Exploiting encoder representations for efficient semantic segmentation[END_REF], PSPNet [START_REF] Zhao | Pyramid scene parsing network[END_REF]. We trained these architectures from scratch but also investigate transfer learning as a way to mitigate the limited number of images in the HC18 dataset. Even though the natural images from ImageNet1 [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] and US images have obvious dissimilarities, some generic representations can be learnt from a large-scale dataset, that might be beneficial to other types of images, and they have proven so in the context of MR images [START_REF] Wacker | Transfer learning for brain tumor segmentation[END_REF]. Thus we have used various backbone models, namely VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], ResNet50 [He et al., 2016a], Effi-cientNet [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF], pre-trained on the ImageNet dataset, for all architectures mentioned above. For the loss function, we use the Dice loss, highlighted by [START_REF] Ma | Loss odyssey in medical image segmentation[END_REF] to be one of the best loss function for medical image segmentation.

We also used Kappa loss [START_REF] Zhang | Kappa loss for skin lesion segmentation in fully convolutional network[END_REF] in our experiments for fetus head segmentation, the experimental results demonstrate that both Dice loss and Kappa loss performs equally well. Therefore, in the following segmentation experiments, we show the results with Dice loss.

Post-processing of segmentation results

It can happen that the segmentation results have some noise or incomplete part such as holes, which can cause inaccurate ellipse fitting. Thus some postprocessing is applied on the segmentation results: contours are detected from the segmentation map by Canny filter [Canny, 1986], then the largest connected component is kept when several contours are detected. Generally, the shape of the maximum contour is irregular and this randomly shaped contour needs to be fitted to an ellipse before obtaining the ellipse parameters.

HC computation based on segmentation results

To my knowledge, there are three ways to measure the length of an ellipse based on a given binary image.

1. Counts the number of pixel points of the ellipse outline in the image.

2. The Euclidean distance between the locations of each contour pixel point is calculated and then accumulated to obtain the arc length of the ellipse2 .

3. Apply the formula for calculating the circumference of an ellipse.

The common drawback of the first two methods is that the calculated ellipse perimeter is larger than the actual ellipse perimeter when there are duplicate pixel points on the ellipse contour. So in this work, we use the third method of calculating the elliptical perimeter.

After post-processing the segmented results, the next step is to perform ellipse fitting in order to get the parameters (long axis, short axis, center points, angle) of the ellipse to compute its length. The length of an ellipse denoted HC is approximated by Ramanujan approximation method (Equation 4.1) [START_REF] Barnard | Inequalities for the perimeter of an ellipse[END_REF] in which h = (a-b) 2 (a+b) 2 , a and b being the long and short axis of the ellipse:

HC = π(a + b)(1 + 3h 10 + 4 -3h ) (4.1)

Head circumference estimation using regression CNN Regression CNN model

As shown in from b0 to b7.), DenseNet121 [START_REF] Huang | Densely connected convolutional networks[END_REF] (The number 121 corresponds to the number of layers with trainable weights (exclude batch norm)), Xception [Chollet, 2017], MobileNet [START_REF] Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF], InceptionV3 [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF] (The versions including Inception V1, V2, V3, V4 and Inception-ResNet.). In order to improve model convergence, and for the reasons stated above in the previous section, these models have been pretrained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF], and we fine-tune3 them for the task at hand.

Loss functions MAE loss

The loss functions commonly used in regression CNN include the Mean Absolute Error (MAE) loss (also called L1 loss), defined as:

MAE loss = 1 N N i =1 |p i -g i | (4.2)
Where p i is the probability of predicted pixels, g i the real value of head circumference in pixels, and N the number of pixels in an image. MAE loss function is more stable when dealing with outliers.

However, MAE has a serious problem (when used for neural networks): the gradient of the update is always the same, i.e., the gradient is large even for small values of loss. This is detrimental to the learning of the model. To solve this drawback, we can use a varying learning rate (e.g. Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]) that reduces the learning rate when the loss is close to a minimum.

MSE loss

Mean Square Error (MSE) loss (L2 loss) is defined in Equation 4.3. The MSE performs well and converges effectively even with a fixed learning rate. Because the gradient of the MSE loss increases as the loss increases and decreases as the loss tends to 0. The gradient of the MSE loss increases as the loss increases and decreases as the loss tends to 0. This leads to more accurate results at the end of training using the MSE model. But, since the MSE loss takes the square of the error, so if the error > 1, then the MSE will further increase the error. If there are outliers in the data, then the error value will be large, so a model using MSE will give greater weight to the outliers compared to using MAE to calculate the loss.

MSE loss

= 1 N N i =1 (p i -g i ) 2 (4.

3)

Huber loss There is another loss function called the Huber loss (HL), in which it combines the MSE loss and MAE loss through a hyper parameter δ. Here δ = 1, that is to say, when the error is less than 1, the Huber loss becomes MSE loss, otherwise, it becomes parametric MAE loss. Thus, the benefit of Huber loss can dynamically call the loss function according the change of error so that utilizing the strengths of MAE and MSE losses in suitable case.

HL =            1 N N i =1 1 2 (p i -g i ) 2 , for |p i -g i | < δ 1 N N i =1 δ * (|p i -g i | - δ 2
), otherwise (4.4) We will investigate all three of them in our experiments, as there is no heuristic to choose one loss over the other, as highlighted in [START_REF] Lathuilière | A comprehensive analysis of deep regression[END_REF].

Model configuration

Segmentation models # param (M) Regression models # param (M)

Original

Explainability of regression CNN

If deep learning methods are the gold standard in most image processing tasks, they are often considered as black boxes and fails to provide explainable decisions. In this work, we investigate various saliency maps methods, to leverage their ability at explaining the predicted value of the regression CNN. Since saliency maps methods have been developed for classification CNN mostly, we provide an interpretation for regression saliency maps, as well as an adaptation of a perturbation-based quantitative evaluation of explanation methods.

Explanation methods for CNN

In Chapter 2, we have reviewed plenty of explanation methods. In this study, we use several post-hoc explanation methods to investigate or valid the explainability of regression CNNs. Two categories of explanation methods are generally considered, which yields a saliency map that estimates how much each pixel contributes to the prediction. They are perturbations-based or propagation-based. In perturbation-based approaches, the goal is to estimate how perturbation applied to the input image, such as blurring or injecting noise, changes the predicted class [Fong andVedaldi, 2017, Zintgraf et al., 2017]. In propagation-based techniques, the idea is to backpropagate a relevance signal from the output to the input. Both of these two types explanation methods can valid the model's explainability. In this work, for convenience, we use the latter category (propagation-based) of methods that actually encompass three classes, which can be directly called via a public Python package Innvestigate [START_REF] Alber | innvestigate neural networks![END_REF]:

• (i) Sensitivity (gradient-based) methods: The Gradient [START_REF] Simonyan | Deep inside convolutional networks: Visualising image classification models and saliency maps[END_REF] method;

the SmoothGrad [START_REF] Smilkov | Smoothgrad: removing noise by adding noise[END_REF] method; the Input*Gradient [START_REF] Shrikumar | Not just a black box: Learning important features through propagating activation differences[END_REF] method; and the Integrated Gradients [START_REF] Sundararajan | Axiomatic attribution for deep networks[END_REF].

• (ii) Deconvolution methods.

The DeConvNet [START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF] method; the Guided BackProp [START_REF] Springenberg | Striving for simplicity: The all convolutional net[END_REF] method.

• (iii) Layer-wise Relevance Propagation (LRP) variants: LRP [START_REF] Bach | On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation[END_REF] method; DeepTaylor [START_REF] Montavon | Explaining nonlinear classification decisions with deep taylor decomposition[END_REF] method.

In the classification setting, a saliency map provides an estimation of how much each pixel contributes to the class prediction. In the regression setting, the saliency map will provide an estimation of how much each pixel is impacting the model, and is contributing to decrease the prediction error, as measured by the loss function, that is in general the MAE or MSE.

Evaluation of explanation methods based on perturbation

Only using saliency maps for visualizing the highlight area that the regression CNN models learn from input images are not convincible enough. Because there is no evidence which saliency map is suitable for our proposed regression CNN. Therefore, in this study, we use a perturbation-based method to quantitatively evaluate different explanation methods. In Chapter 2, we introduced some evaluation methods on various explanation methods, and Area over Perturbation Curve (AOPC) [START_REF] Samek | Evaluating the visualization of what a deep neural network has learned[END_REF] is one of evaluation methods that build upon the perturbation analysis in classification tasks. Here, we propose to adapt the AOPC to the regression case. Generally, in classification or segmentation tasks, the evaluation metrics is accuracy, while in regression CNN model, the metrics is loss value between true value and predicted value. If we denote by (x) (0) the prediction error of initial image evaluated by the analyzer and (x n ) (k) (1 ≤ k ≤ K) the prediction error of the perturbed image (x n ) (k) at step k, we can define the AOPC regression Analyzer as:

AOPC r eg r essi on

Anal y zer = 1 N N n=0 ( (x n ) (0) - 1 K K k=0 (x n ) (k) ) (4.5)
A larger AOPC score in absolute value means that an analyzer has a steep decrease when the perturbation steps is increasing. It means on the one hand, the regression CNN models are sensitive to the noise, which demonstrates that the models are capable to learn the key features from input images; on the other hand, the high AOPC score means that an explanation method can well detect the key features that the models learned from input images.

Experiment configuration

In order to create a fair experimental environment, both approaches, segmentation or regression, are evaluated with the same protocol, namely with 5-fold cross validation, the folds being identical for all the methods. We set the optimizer as Adam with a learning rate of 10 -4 . The batch size is 16. The training takes 100 epochs.

The implementation is based on deep learning framework Keras. In the segmentation experiments, we use the exsisted public Python library Segmentation Models [Yakubovskiy, 2019]. The programs are executed on Tesla P100 GPU server with 16 GB memory.

Evaluation metrics

Evaluation metrics for the segmentation results are the Dice index (DI), the Hausdorff distance (HD), and the Average symmetric surface distance (ASSD). The mean absolute error (MAE) and the percentage MAE (PMAE) are used to compare the predicted and the ground truth HC values.

HC estimation based on segmentation CNN

We train and test 6 different segmentation architectures (U-Net, U-Net++, DoubleU-Net, FPN, LinkNet, PSPNet) with three pretrained CNN backbones (VGG16, ResNet50, EfficientNet). Besides, we added the original U-Net architecture [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] that does not have any backbone. We found that the segmentation models pretrained on ResNet50 outperformed the other two CNN backbones. As shown in Table 4.2, that contains both the segmentation accuracy and the HC estimation MAE. From this table, one can gather that:

Segmentation-wise, all segmentation models obtained similar scores, as shown by values un columns DI, HD and ASSD in the Table 4.2. And these segmentation accuracy have outperformed that in the literature [START_REF] Budd | Confident head circumference measurement from ultrasound with real-time feedback for sonographers[END_REF], Sobhaninia et al., 2019]. However, when it comes to the estimation error of the HC, the U-Net-B2 and LinkNet-B2 are the best architectures, as assessed by a two-sided, paired Student's t-test between pair of method scores, that resulted in a p-value inferior to 0.05 for these 2 networks. Both networks achieve an MAE value (after postprocessing) of 1.08 mm and 1.15 mm respectively. Post-processing allows indeed to obtain a small enhancement in the MAE value compared to the results without post-processing. Transfer learning techniques help to improve the segmentation accuracy when comparing U-Net with pretrained ResNet50 and U-Net with initial ResNet50.

We also analysed some segmentation results (Figure 4.3) on some vague US fetus head images, the influence of noise and artifacts of images in segmentation-based methods is less than that in the segmentation-free methods (presented in Figure 4.10).

HC estimation based on regression CNN

We train and test regression CNN architectures with 7 different pretrained CNN backbones, experimented with 3 regression loss functions (MAE loss, MSE loss and Huber loss) on the HC18 dataset. The evaluations of direct HC estimation are given in Table 4.3. One can find that the Regression EfficientNet (Reg-B3-L1) in conjunction with the MAE loss, performs better than the other CNN models: the resulting

Bland-Altman plot analysis

The Bland-Altman plot is another way to analyze the agreement between two measurements, by plotting the difference between the measurements vs their mean, that makes it easy to spot a bias between the measurements. From the Bland-Altman plot in Figure 4.6, obtained on a fold of 200 test images, we observe that regression approaches struggle with larger fetus head images, which is interesting since segmentation approaches usually fail on small structures. One can also see that for the segmentation models, 8 out of 200 points are outside the 95% agreement limit; for regression models, there are 12 outliers out of 200, mostly distributed in larger HC values. Unsurprisingly, room for improvement is left for regression-based approaches. One can also identify the 95% agreement limits: for the best segmentation model, they are [-3.12mm, 0.7mm], and for the best regression model, they are [-3.25mm, 2.92mm]. We can compare these limits to the 95% agreement limits on inter-operator variability, which is ±12 mm [START_REF] Sarris | Intraand interobserver variability in fetal ultrasound measurements[END_REF], Table 1 page 272]: the fact that they are greatly smaller highlights the high relevance of both of segmentation-based and segmentation-free approaches as alternative to automatically estimate the HC from US images. Figure 4.6 -Bland-Altman plots of the segmentation and regression CNN models. The x-axis represents the average value of ground truth and predicted HC; the y-axis, the difference between ground truth and predicted HC (in mm). The horizontal red solid lines represent the upper and lower limits of 95% consistency. The middle dotted green line represents the mean of the difference.

Memory usage and computational efficiency

In addition to prediction accuracy, we also compared the memory usage and computational efficiency of both segmentation-based and segmentation-free approaches. In the aspect of memory usage of a CNN model in theory, Algorithm 4.1 4gives the pseudo code of estimating memory cost of a model. It simply consists of three parts: the memory of embedded model, the memory of model layers as well as the memory of model weights.

Theoretical memory usage of CNN models

The theoretical memory usage of a CNN during training requires to store the network parameters and the activation outputs of every layer, used to compute the gradients, for each batch. As show in Table 4.5, as one could expect, regression CNN models requires less memory storage in general, than the segmentation-based approaches, see column Mem-M. However in practice, the gap between regresssion and segmentation models is not so large, as shown by the actual memory cost in the prediction stage, defined as the maximum used memory when the inference is stable (computed using Python library Memory Profiler). In particular, the best regression method (Reg-B3-L1) is even requiring more memory than segmentation methods.

Computational efficiency

As Table 4. The maximum used memory of models is after the red vertical line.

Comparison of HC estimation with state-of-the-art

Table 4.6 -Comparison of HC estimation with state-of-the-art on HC18 dataset. B2=ResNet50, B3=EfficientNetb2, L1=MAE loss, DI=Dice Index, N/A=Not applicable.

Metrics MAE(mm) DI(%)

Segmentation-based methods [START_REF] Sobhaninia | Fetal ultrasound image segmentation for measuring biometric parameters using multi-task deep learning[END_REF] 2.12±1.87 96.84±2.89 [START_REF] Budd | Confident head circumference measurement from ultrasound with real-time feedback for sonographers[END_REF] 1.81±1.65 98.20±0.80 [START_REF] Fiorentino | A regression framework to headcircumference delineation from us fetal images[END_REF] 1.90±1.76 97.75±1.32 [START_REF] Moccia | Mask-r 2 cnn: a distance-field regression version of mask-rcnn for fetal-head delineation in ultrasound images[END_REF] 1.95±1.92 97.90±1.11 U-Net-B2(Proposed) 1.08±1.25 98.80±0.9 Segmentation-free methods Reg-B3-L1(Proposed) 1.83±2.11 N/A At last, the proposed segmentation-based methods and segmentation-free methods are compared with state-of-the-art (SotA) methods (Table 4.6). In the SotA solutions, segmentation intervention are still needed although their models are fancy. For example, in [START_REF] Fiorentino | A regression framework to headcircumference delineation from us fetal images[END_REF], 3 steps including fetus head localization, segmentation, ellipse fitting are performed for computing HC, which seems to be cumbersome. While the proposed two kinds of methods can effectively estimate HC through simple architectures such as U-Net with ResNet or regression EfficientNet with the benefit of transfer learning. One should also note that the sota methods are not that comparable with each other due to their different experiment protocols. For instance, in [START_REF] Budd | Confident head circumference measurement from ultrasound with real-time feedback for sonographers[END_REF], they trained HC18 dataset combined with other fetus head US images.

Explainability of regression CNN

Because we can see the segmented results directly from segmentation models, and the HC is calculated according to the fitted ellipse, thus the results are trustable.

However, contrary to segmentation models, regression models come at a cost of low explainability, i.e. the model is not providing explicit explanations along with the HC prediction.

In order to shed the light on what is indeed learnt by the regression CNN, we use a post-hoc explanation method to analyse the regression model. In our previous work [START_REF] Zhang | Explainability for regression cnn in fetal head circumference estimation from ultrasound images[END_REF], we showed that the Layer-wise Relevance Propagation (LRP) method [START_REF] Bach | On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation[END_REF] was appropriate to explain CNN regression models for this application. The idea of LRP is to compute a relevance score for each input pixel layer by layer in backward direction. It first forward-passes the image so as to collect activation maps and backpropagates the error taking into account the network weights and activations, yielding saliency maps [START_REF] Morch | Visualization of neural networks using saliency maps[END_REF],

in which the areas that most contributed to a decision are highlighted. Note that in [START_REF] Dobrescu | Understanding deep neural networks for regression in leaf counting[END_REF], authors also used LRP method to explain the results of a regression CNN that aims at counting leaf on plant photographs. We agree that Class Activation Map methods such as Grad CAM [START_REF] Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF] may be interesting since they provide promising human-interpretable visual explanations for a given CNN architecture. Their principle is to using a global average pooling layer, and to compute the saliency map as the weighted combination of the resulting feature maps at the second last (before softmax) layer. Since we do not have classes here but regressed values, it might be interesting to explore "regression activation map" as defined in [START_REF] Wang | Diabetic retinopathy detection via deep convolutional networks for discriminative localization and visual BIBLIOGRAPHY explanation[END_REF].

One can discover from Figure 4.9 that the regression CNN can indeed find the key features from head contour on the input US images and relies on, to some extent, on many contour pixel to make the HC estimation. This indicates that the predictions of regression CNN are reliable to some extent. On the other hand, we can tell from Figure 4.9 that the feature extraction capability of each regression CNN is different according to the red contribution points. (1)

(2)

(3) (4)

(5) 12 is an example of the perturbation process of Gradient analyzer.

Step 0

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6 Ste

Step 8

Step 9

Step 10

Step 11

Step 12

Step 13

Step 14 Ste

Step 15 Step 14

Step 13

Step 12

Step 11

Step 10

Step 9

Step 8

Step 7

Step 6

Step 5

Step 4

Step 3

Step 2

Step 1

Step 0 Saliency map Image

Step 15

Step 14

Step 13

Step 12

Step 11

Step 10

Step 9

Step 8

Step 7

Step 6

Step 5

Step 4

Step 3

Step 2

Step 1

Step 0 Saliency map Image Figure 4.12 -Perturbation process for the saliency map produced by the Gradient method.

Step 0 is the original input image. From step 1 to step 15, Gaussian noise is added gradually on the image subwindows. The perturbation order of these subwindows corresponds to the saliency scores assigned by the Gradient method analysis, i.e. the most contributing pixels are perturbed first. Red: noise, blue: original image pixels.

In Figure 4.13, we show the evolution of the prediction error w.r.t. the quantity of noise added at each perturbation steps, on first the most significant subwindow in the analyzer's sense, to the least significant one. One can observe that consistently, the prediction error is increasing, as the level of noise increases. Methods with the steepest curve, LRP and Input*gradient, exhibit the largest sensitivity to perturbations, and as such, should highlight the contributing pixels, in the sense of this criterion. Interestingly the Integrated gradient analyzer seems to be relevant for VGG16, but not for Reg-ResNet50. In the future, it will be interesting to vary the subwindow size to see if results are affected. We expect that a finer grid will be better suited to a thin structure like the head skull. We adapted the evaluation metric of the regression CNN model from accuracy to predicted error, thus, with the noise added gradually according to the importance, the loss increases, the faster the loss increase, which means the better analyzer it is.

If the relevant features are blocked by Gaussian noise, the model can not predict well, then the analyzer can not detect the relevant feature neither. Therefore, the sensitive analyser will drop steeply, while the insensitive analyser does not change a lot. To quantify this criteria, the area of perturbation curve (AOPC, Equation 4.5) is regarded as score of one explanation method, for instance, in Figure 4.12, the AOPC value is the difference between accuracy of input image and the average accuracy of 15 perturbed images. Different from classification or segmentation CNN models, the evaluation metrics of regression CNN model is the loss between ground truth and predicted value, not the accuracy, thus we convert the loss by adding a negative sign. In Table 4.7, we compared AOPC scores on regression VGG16 and regression ResNet50 models respectively. Since the AOPC is the difference between the prediction error with and without perturbation, we expect that the analyzer that are indeed perturbed by the noise will return a large AOPC score, in absolute value. We can see that the regression ResNet50 has higher AOPC score than regression VGG16 model. Again we can gather from this table that both the LRP and Input*Gradient methods perform well in those two models. Note that other explanation methods have inconsistent performance depending on the model. This highlights the necessity to choose the proper explanation method before analyzing a specific model. In our head circumference estimation study, we use the selected explanation method to analyse different regression CNNs. On the one hand, the model's explainability can be proved that the direct HC prediction using regression CNN is 99 CHAPTER 4. FETUS HEAD CIRCUMFERENCE PREDICTION reasonable and effective. On the other hand, different regression CNN models can be compared in the aspect of feature extraction ability. For more information about the experiments of explainability of regression CNNs, please refer to Appendix A.

Conclusion

In this work, we have addressed the problem of HC estimation from US images via both a conventional segmentation approach with post-processing and ellipse fitting, and a regression-based approach that can directly predict HC without segmentation intervention. Our idea was to quantify how far regression-based approaches stand from segmentation approaches, when the final task is to estimate a parameter, i.e. a biomarker, from the image. Although segmentation-based methods provide explainable results for the HC estimation because the segmentation result is visible, they often require dedicated post-processing steps. On the other hand, regression approaches based on CNN are end-to-end, less costly and prone to error and even though they do not offer explicit explainability, this aspect can be explored using saliency maps for example [START_REF] Zhang | Explainability for regression cnn in fetal head circumference estimation from ultrasound images[END_REF]. In our study, we have explored both segmentation and segmentation-free approaches with state-of-the-art CNN architectures and backbones. By setting the same experimental conditions, we have proposed a fair, quantitative comparison of these two approaches, in order to assess if the direct estimation approach is viable for this task. Even though the estimation error is much higher with the regression networks, the results are still promising and in line with inter-operator variability. Trade-off between prediction error and computation efficiency of two types of methods are existed. Therefore, direct estimation, regression-based approaches have a high potential that should be deepened in the future. Whereas we used general-purpose architectures for our regression methods, it will be interesting to investigate customized architecture for this task, and that include attention mechanisms.

In the future work, we will assess the generic regression CNNs on other medical datasets to estimate multiple biomarkers. Besides, we plan to investigate the segmentation-free approaches with other, recent CNN architectures that have higher ability of feature representation, e.g transformer architectures, as well as multi-task learning which combines segmentation branch and regression branch.

For the regression loss functions, in this work, we explored MAE, MSE as well as Huber loss (with a fixed hyper parameter δ = 1), the performance of these three losses are various with different regression CNN models and input data. There is no heuristic which one is better. Therefore, in the future work, we will explore the other loss function such as Robust loss [Barron, 2019] which is a generalization loss of different regression loss functions.

Motivation

As we have seen in the previous chapter, medical image segmentation is often a prerequisite step toward the computation of biomarkers. In Chapter 4, we tackled the problem of head circumference estimation. In this chapter we focus on another problem that could benefit from direct estimation: estimating the volume of cardiac ventricular cavities and left myocardium from MR images [START_REF] Petitjean | A review of segmentation methods in short axis cardiac mr images[END_REF]]. This problem is more complex since US images are 2D in the case of head circumference estimation and here MR images are 3D. Morever, in US images, the zone to be segmented, i.e. the skull, is dense and thus appears directly as a contour, whereas in the MR images, the cardiac ventricles and myocardium are in the mediastinum, in the middle of other organs, and their respective boundaries are not especially highlighted.

In this work, we investigate how a vanilla regression CNN can perform to estimate automatic multi-structure cardiac volume without segmentation. The method is performed on public "Automatic Cardiac Diagnosis Challenge" dataset (ACDC), the predicting targets are the volume of RV, LV, and MYO, respectively.

This chapter is organized as follows: in Section 5.2 we present some background in MR image processing for cardiac function evaluation. Section 5.3 we introduce the dataset preprocessing and regression models. Experimental results and limitations are discussed and presented in Section 5.4. And conclusions are drawn in Section 5.5.

Background on cardiac function evaluation

The cardiovascular diseases (CVDs) are one of most common diseases in the world, which is the leading cause of death globally, taking an estimated 17.9 million lives each year. CVDs are a group of disorders of the heart and blood vessels and in- or 3D kernels [START_REF] Jang | Automatic segmentation of lv and rv in cardiac mri[END_REF], Isensee et al., 2017, Yang et al., 2017], generative adversarial networks (GAN) [START_REF] Savioli | A generative adversarial model for right ventricle segmentation[END_REF], and recent transformers as encoders [START_REF]Transunet: Transformers make strong encoders for medical image segmentation[END_REF] has further improved the segmentation accuracy, etc. However, for quantifying the segmented results, it still needs two steps (segmenta-CHAPTER 5. CARDIAC MULTI-STRUCTURE VOLUME PREDICTION tion+quantification) to obtain the volume of cardiac structures.

Based on the above literature we know that segmentation-based methods are still more popular for the current prediction of the structural volume of the cardiac, but later direct prediction methods have also emerged (See Chapter 2), and among these direct prediction methods for certain biomarkers, there are gradually beginning to be studies for cardiac biomarkers, but the methods for data preprocessing and data augmentation are not very transparent, and there is a lack of interpretable studies of the models, so we have further supplemented and improved this work.

Methodology

ACDC dataset and preprocessing

The ACDC dataset

The public "Automatic Cardiac Diagnosis Challenge" dataset (ACDC) dataset [START_REF] Bernard | Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved[END_REF] is used in this study. These data are obtained on two different MRI scanners with different magnetic strengths. Generally, the image quality is better when the magnetic strength is higher. The dataset contains 100 magnetic For the ground truth, there are masks of three cardiac structures annotated by experts, and the volumes of RV, MYO and LV. The volume of the cardiac is calculated according to the following formula. In Formula 5.1, N is the number of slice in each subject, S_RV is the summary of pixels (area) belong to RV class in one slice (same as S_MYO, S_LV ), px_x (mm) is pixel size of x dimension, px_y (mm) is pixel size of y dimenstion, space_z (mm) is the slice thickness. The principle of this formula is to superimpose the area of each slice, so that the cumulative calculated area is the volume of the different structures of the cardiac. For uniformity of units (millimeters and milliliters), the result of the calculation is divided by 1000 to turn it into a CHAPTER 5. CARDIAC MULTI-STRUCTURE VOLUME PREDICTION volume in milliliters. (5.1)

Volume

=                      RV = N i S_RV *

Data cleaning

Data cleaning is to remove some data that are outliers from the main part, which is necessary before performing certain methods on specific dataset. Otherwise, it will affect the experimental results and accuracy. In ACDC dataset, in order to make sure the ground truth volumes of each subject offered by the authors of ACDC dataset are correct, we check the consistence of given volumes and volumes computed through Formula 5.1. We found that there are 6 patients out of 100 pairs subjects (including ED and ES) whose volumes of cardiac are seriously deviated from the computed volumes. And the difference are up to hundred level which can not be ignored. See Table 5.1. The left subjects in given volumes are consistent with computed volumes (because their differences are less than 1). Thus, we will remove these 6 patients in the experiments. Uniforming number of slices After completing the fixation of the cardiac target in the two-dimensional direction, the number of slices should also be consistent for each MRI data, i.e., we added or removed cardiac slices at minimal cost in order to satisfy the principle of constant input data size. Specifically, we select the median of all data depths in the ACDC dataset as the uniform number of slices, and for data above that number, we remove them from the bottom of the cardiac (because the area of the slices at the bottom of the cardiac is the smallest), and for data below that number, we duplicate the slices at the bottom of the cardiac and accumulate them until we reach that number. At last, the shape of a 3D MR image is (100 × 100 × 9).

Data augmentation based on grid search

After the above data preprocessing steps, there are 182 valid images whose shapes are (100 × 100 × 9). However, this number of data sets is far from sufficient for a deep neural network model with a large number of learnable parameters and can easily lead to overfitting. Usually, data augmentation is the common way to increase the number of images, specifically through image processing techniques (rotation, translation, etc.) to increase the diversity of images.

In this study, given the small amount of raw data, we develop an efficient automatic data augmentation algorithm. The algorithm is based on a backtracking method to find a subset without duplicates, i.e., this is a grid search to enumerate different combinations of image processing algorithms to add a custom number of images.

First of all, we list 10 different basic image processing algorithms. Then, the index of these functions are sent into the Algorithm 5.32 .

x.'aug_rotate'; y.'aug_rotate_r'; z. end while 21: end for method to find all the subsets of these 10 methods, then 2 10 = 1024 different combinations (subsets) will be generated. For example, the original dataset has 182 subjects, if the user wants to generate 5000 subjects, then we take 5000/182 = 28 combinations from 1024. This ensures each subject is different from the others. In this work, we generate 28*182 = 5096 synthetic images.

Regression CNN 2D convolution for 3D data

Given that MRI or CT images are in a 3-dimensional format, the third (z direction) dimension is the depth of the organ scan or the number of slices. From the point of view of the image, it can also be considered to be composed of multiple channels, see Figure 5 return result 17: end procedure would be natural to think of using a 3-dimensional convolution kernel to learn the neighborhood features and spatial information of that data, for example, the segmentation models 3D U-Net [Çiçek et al., 2016], V-Net [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF], etc. In addition to 3D convolution, it is possible to utilize 2D convolutional neural networks on 3D images, in which each slice is regarded as one input channel. The literature [START_REF] Yang | Reinventing 2d convolutions for 3d images[END_REF], [START_REF] Hassanzadeh | 2d to 3d evolutionary deep convolutional neural networks for medical image segmentation[END_REF], [START_REF] Vu | Evaluation of multislice inputs to convolutional neural networks for medical image segmentation[END_REF] has proved the feasibility of 2D CNN on multiple image slices.

For 2D convolution, the input layer and the filter have the same depth, in other words, the number of image channels is the same with the number of convolutional kernels/filters. The filter slides in 2D direction. Then the input and the filter are summed together into one feature map, each element is a pixel. Iteratively, the feature map goes deeper with more filters, which depends on the architecture of CNN model.

For 3D convolution, the filter is a 3D kernel, which is generally (3 × 3 × 3), the filter moves on 2D channel first, then moves in z direction. The output is a 3D matrix. That is to say, each element is a voxel. Afterwards, the 3D feature map goes into next layers with more 3D filters. Because 3D convolutions can describe the spa- tial relationships of objects in the 3D space. It is beneficial for some applications, such as 3D segmentation/reconstruction of biomedical imagining, which is related to voxel-wise classification. Figure 5.4 shows the 2D and 3D convolution process on 3D data.

In our study, our task is image-wise regression. That is, the prediction results of the model are determined based on the most important feature of the whole image.

Thus, this feature can be learned either by 2D or 3D convolutional kernels. However, 2D convolution and 3D convolution models are much different in terms of time and space complexity, and when combined with the above figure (Figure 5.4), 3D convolution models require a large number of training parameters, and this huge number of parameters requires large memory space and computational power. VGG is (3 × 3), (3 × 3 × 3) in 3D regression VGG. One can find that the 2D regression VGG takes less time and memory than that with 3D regression VGG. Therefore, considering the above analysis, we will choose 2D regression CNN to predict the volume of different structures of 3D cardiac data.

The architecture of regression CNN Regression CNNs

We design a deep regression CNN architecture shown in Figure 5.5. Any CNN model can be a backbone to learn the feature from training dataset, for instance, VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], ResNetV2 [START_REF] He | Identity mappings in deep residual networks[END_REF] or EfficientNet [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF]. For the regression part, after the feature maps were flatten into fully connected layer, we simply use linear regression (224,224,9), the ground truth are the volume of 3 structures of cardiac. FC is fully connected layer after feature representation, the FC layer goes through linear regression layer, the output is the predicted volumes (vol_RV, vol_MYO, vol_LV).

Loss functions Same as previous work, the regression CNN model is optimized by regression loss function such as mean absolute error (MAE) loss or mean square error (MSE) loss or Huber loss (HL) [START_REF] Esmaeili | A novel approach to quantized matrix completion using huber loss measure[END_REF].

Explainability of regression CNNs

Because we can see the segmented results directly from segmentation models, and the cardiac volume is calculated according to the segmented areas, thus the results are trustable. However, the regression CNN models come at the cost of a low interpretability, i.e. the model is seen as a black box, which does not provide explanations along with the cardiac volume prediction. To this end, we use a post-hoc explanation method to analyse the regression model, in our previous work [START_REF] Zhang | Explainability for regression cnn in fetal head circumference estimation from ultrasound images[END_REF], we validated that the method Layer-wise Relevance Propagation (LRP) [START_REF] Bach | On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation[END_REF] can well explain the regression CNN models in the form of saliency maps [START_REF] Morch | Visualization of neural networks using saliency maps[END_REF].

Transfer Learning from RGB to multi-channel cardiac MRI data

In order to further improve the model's performance, we use transfer learning strategy to load CNN backbones that are pretrained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. Although ImageNet images and MR images have obvious dissimilarities, some generic representations can be learnt from a large-scale dataset, that might be beneficial to other types of images, and they have proven so in the context of brain MR images [START_REF] Wacker | Transfer learning for brain tumor segmentation[END_REF]. Since these pre-trained models are trained on natural RGB images, the input depth of the models is 3 channels, which will not work in multislice data. To solve this problem, we loop through the layers of the pre-trained CNN model and replicate the average of the existed weights to new channels in each layer so that it can ensure the input layer matches the subsequent layers.

Experiments and results

Experiment protocol

The ACDC [START_REF] Bernard | Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved[END_REF] dataset in this study has 182 subjects after data preprocessing. We split it into the training set (100), the validation set (32), the test set (50). The total training set has 2900 3D MR images including data augmentation. The optimizer is Adam. The learning rate is 1e -4 , the batch size is 16. The algorithm is completed using Python and Keras library with GPU p1003 . The training epoch is 100. 5-fold cross validation is performed. The structure of regression CNN is: 2D convolutional kernel with multi-class volume prediction. For example, the input shape is (N,224,224,9), N is the number of input images, the image is resized to 224*224, one image has 9 slices.

In the experiments, we train three different regression CNN backbones, which are VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], ResNet50V2 [START_REF] He | Identity mappings in deep residual networks[END_REF] and Ef-ficientNet [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] respectively. These pretrained models originally only had three channels, in this work, we expand these three channels into 9 channels in order to match the data shape. We perform data normalization both in MRI images ((img -µ)/σ) and ground truth volumes (gt/ max(gt)). The evaluation metrics are mean absolute error (MAE) and percentage MAE (PMAE). We also conducted plenty of additional experiments including cardiac data scale, cardiac slice selec-CHAPTER 5. CARDIAC MULTI-STRUCTURE VOLUME PREDICTION dataset [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. From this table one can find that the prediction errors (MAE) are large, and the RV structure is the most difficult to predict. The regression ResNet with MAE loss has lower prediction error than the other models. So we take this model as a analysis example in the following sections.

Comparison with state-of-the-art

We compared our method with the state-of-the-art (SotA) on the same ACDC dataset. In the Table 5.5, many of their results have separate EDV (left) and ESV (right). The SotA is based on direct prediction (segmentation-free) methods and segmentation methods. One can find that out method has a large gap with the already existing methods, especially segmentation ones, which is a little bit disappointing and shows that at this point, direct estimation of cardiac structures volum with vanilla CNN is a bit early. However, the comparison should be handled with care. In this table, the segmentation-based or segmentation-free methods predict the area of the cardiac structures slice by slice and then accumulate them to obtain the volume of the cardiac structures. Another point is that the experimental protocol and the test set are not the same. Therefore, there may be some bias to compare those results with the our method.

CHAPTER 5. CARDIAC MULTI-STRUCTURE VOLUME PREDICTION

We performed a statistical analysis of the prediction results for a test set with 50 cardiac data. Among three cardiac structures (RV, MYO, LV), RV has the highest mean absolute error predicted by regression CNN model (Figure 5.8). This is expected as it is known in the clinic to be the most difficult to estimate. The Bland-Altman plot (Figure 5.10) of predicted cardiac structure volumes also demonstrates that the large bias of three cardiac structures compared to ground truth values, especially in RV.

From a pathological point of view, in order to ensure the generalization of the model, the model is trained with the data evenly distributed according to the pathology. Based on this fact, the prediction of RV volume is also difficult in data with abnormal RV cardiac disease (Figure 5.8). Besides RV disease, the prediction bias is also large to the patients with dilated cardiomyopathy (DCM) and hypertrophic cardiomyopathy (HCM) (Figure 5.9).

R V M Y O LV

Saliency maps of Regression CNN

We generated and analyzed saliency maps of the regression CNN model for cardiac structure volume prediction using the LRP algorithm based on iNNvestigate library [START_REF] Alber | innvestigate neural networks![END_REF], see

Conclusion

In the general practice, the ventricle cavities and the myocardium are first segmented, and then the areas and volumes are calculated based on the result of the segmentation. In this study, we investigated how regression-based CNN models can directly predict the volume of cardiac structures (RV, MYO, LV) without segmentation intermediate steps. Our method was validated on the ACDC dataset. We first preprocessed the ACDC data set by cropping and unifying the slice number to 9.

Data augmentation is used, based on grid search method, to increase the amount of data. Transfer learning is applied in this study: the CNN backbones are pretrained on ImageNet. The predicted results of cardiac structures were analyzed and discussed that large bias exists, especially in RV structure. We also analysed the model's interpretability through a post-hoc explaining method. The saliency maps tells that this regression-based methods are reliable to some extent. Although the idea of direct estimation has big potential in a clinical setting, results are not fully convincing yet and the prediction error need to be further reduced. We could first explore the estimation of the area from single slice, then computing the volume. We also believe that with more adhoc or specific architectures, that would be better adapted to the data (3D, noise), could allow to enhance the results.

to assist physicians can greatly reduce the amount of effort physicians spend on an individual patient, allowing for early prevention or treatment of the patient.

In this study, we utilized regression CNNs to directly predict the cardiac structure (RV, MYO, LV) volumes. Before training the the models on ACDC dataset [START_REF] Bernard | Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved[END_REF], we performed preprocessing the data in the aspects of cardiac area cropping, slice number uniforming as well as data augmentation given that each subject has different shape in the original dataset and the amount of the data is small. For the regression CNN model, transfer learning is also used in our method, for which the CNN backbones are pretrained on the natural images dataset (ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]). But in this study, we adapted the proposed regression CNN models from RGB channel to multi-channel to fit in the training data in order to copy the weights from pretrained CNN backbones. Several experiments have been conducted and analyzed. The experiments have promising results except for the volume of RV structure which is difficult to estimate. So far, we have extended the modality of medical images from 2D ultrasound images to 3D MRI images using our regression CNNs. And the prediction targets/biomarkers are also proceeded from single head circumference to multi-structure cardiac volume.

Perspectives

In this part of work, our proposed regression CNN models can predict the fetus head or cardiac structure volumes directly from medical images. The other measurable biomarkers can also be estimated via this method without segmentation. Besides, since our proposed model are simple which consists of only CNN backbones and a regression layer in the last layer, thus more advanced architecture can be used on this method. As for regression loss functions, other loss functions can also be explored besides MAE, MSE and Huber loss. For the cardiac structure volume prediction, room for improvement is left, one possible reason is the low quality of the cardiac images. The other MRI cardiac datasets will be validated by our models with high ability of feature extraction.

Explainable AI in medical imaging

Saliency maps of regression CNN models in medical images

In this thesis, we made a survey about explainable AI (XAI). In the specific application of XAI, we generated saliency maps from several post-hoc explaining methods on the regression CNNs. In the HC prediction problem, we utilized explaining methods to valid the interpretability of regression CNN models in the form of saliency maps. The experiment results indicated that the highlighted areas of saliency maps match the contours of the fetus head as observed by the human eye, and these highlighted areas are the main contribution to the predictions made by the model.

Therefore, we can know that the regression CNN has the ability to learn features and to make predictions based on that feature. We also validated the explainability of regression CNNs on 3D cardiac imaging using one explaining method. We can know the flaws and bias of input images from saliency maps.

Evaluation metrics of explaining methods in regression CNNs

Besides the saliency maps that can visually show the highlighted features learned by deep learning models generated by certain explaining method. There is another method that can quantitatively evaluate each explaining method based on the perturbation method. In our study, we adapted the criteria (AOPC score) from classification CNN to regression CNN. On the one hand, we used this criteria to evaluate if an explaining method is effective. On the other hand, it can be used to evaluate of one regression CNN model is better than others.

Perspectives

In this part of work, we valid the explainability of regression CNN models using post-hoc propagation-based methods via saliency maps. On the other hand, the perturbation-based methods can also be used to valid the feature learning ability of deep learning models.

Future work

In the future work, we will continue to do further exploration and optimization along the existing research. we will explore more possibilities in two levels, which are technical level and medical imaging level respectively.

Technology innovation

Geometric deep learning

Most of current deep learning models are based on data-driven, in other words, the model's performance is excellent as long as the data has huge amount and good quality so that the model can learn various features/information from them. Geometric deep learning is intended to avoid The Curse of Dimensionality [START_REF] Indyk | Approximate nearest neighbors: towards removing the curse of dimensionality[END_REF]] by the idea of symmetry prior to keep the data invariance in the forms of graphs or grids or other mathematical representations. Thus, this technology will be a study topic in the future.

Attention mechanisms in computer vision

Attention mechanisms are originated in the field of natural language processing.

It was later applied to the field of computer vision and achieved remarkable results. In particular, the Transformer model [START_REF] Vaswani | Attention is all you need[END_REF] and Vision Transformer [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] of recent years, which has only attention structures, completely replaces the convolutional neural layer. Their performances surpass the CNN based deep learning models. In addition, to some extent, the attention mechanism model also carries a self-explanatory property, which can improve the credibility of the model. Therefore, its application to medical image analysis is of great interest.

Making the deep learning models explainable

In deep learning-based medical image analysis, the prediction results made by the model are required to be as accurate and trustworthy as possible, otherwise serious medical incidents may occur. This requires us to interpret the model posthoc on the one hand, and on the other hand to make the model capable of selfinterpretation.

Medical imaging problems in practice

Verify the other medical datasets

In future works, we plan to verify the proposed segmentation-free (regression based) methods on the other medical datasets. Despite the success of this method CHAPTER 6. CONCLUSIONS AND FUTURE WORK in fetal head circumference prediction, we hope that it can obtain similar results to the segmentation-based method on more other organs or tissues of medical images. At that time, new problems and challenges may be encountered, such as the pre-processing of images, or the generalization ability of the model, which we will analyze specifically based on the specific problem.

Making the methods applicable in clinical application

Because advanced technologies are created for practical problems, we im in future work, to investigate the needs of real medical problems, for example, from the problem of pre-processing medical images generated in the machine, to the physician's expectation to get specialized medical images. This will enable doctors to focus on the patient itself and alleviate the time and effort spent on the other tasks. Another cates that the two vectors are highly correlated. Thus, to put it bluntly, the attention mechanism measures the similarity of two matrices.

Multi-Head Attention mechanism

In Transformer model, Multi-Head Attention is used, which is adding all the heads (h) together, each head is an Attention. In order to fit/optimize the model, trainable weights matrices are multiplied with each head as well as the whole Multihead.

Multihead = Conc at (head 1 , head We split an image into fixed-size patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder. In order to perform classification, we use the standard approach of adding an extra learnable "classification token" to the sequence. The illustration of the Transformer encoder was inspired by [START_REF] Vaswani | Attention is all you need[END_REF].

METHOD

In model design we follow the original Transformer [START_REF] Vaswani | Attention is all you need[END_REF] as closely as possible.

An advantage of this intentionally simple setup is that scalable NLP Transformer architectures -and their efficient implementations -can be used almost out of the box. The position information is added in each patch. A learnable classification matrix is also added. The figure is obtained from [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF].

With the fiery success of Transformer in text dealing, Vision Transformer (ViT) [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] is proposed in computer vision. The ViT model is inspired from Transformer [START_REF] Vaswani | Attention is all you need[END_REF], which is actually a Multilayer perceptron

My

  PhD work was made possible thanks to the financial support from CSC (China Scholarship Council) and the convenient computational support from CRI-ANN (Le Centre Régional Informatique et d'Applications Numériques de Normandie) for research. Finally, I would like to thank the Chinese and French organizations and people who made this UT-INSA project possible. Through this project, I had the opportunity to open a new world and experience the technology, humanities and cuisine (Les baguettes et les croissants sont mes préférés!) of France, as well as the surrounding European countries, which is a valuable spiritual treasure for me. There are many prejudices and misunderstandings in this world, and you can only gain your own insights if you experience and witness them yourself.

  (a) US of a fetus head [van den Heuvel et al., 2018b] (b)

Figure 1 . 2 -

 12 Figure 1.2 -Skin lesion images. The top images are skin photos, the bottom images are ground truth. The images are from Public ISIC 2018 dataset [Codella et al., 2018].

  Figure 1.3 is an example of fetus head in the form of ultrasound images. The fetus head is approximated as an ellipse annotated by experienced sonographers. Generally, a fetus growth is CHAPTER 1. INTRODUCTION divided into three trimesters [van den Heuvel et al., 2018a] according to the length of head circumference. With the aid of deep learning techniques, the segmentation of head circumference becomes efficient and accurate, but post-processing of the segmentation results, i.e., ellipse fitting and perimeter calculation, is still required.

Figure 1

 1 Figure 1.3 -US images of fetus head, the red ellipses are ground truth annotated by sonographers, below the images are the values of head circumferences in millimeter (mm) and pixels. The images are from Public HC18 dataset [van den Heuvel et al., 2018b].

Figure 1 . 4 -

 14 Figure 1.4 -Cardiac structure MR images. The first two images are one slice of a patient in ED stage and its ground truth. the last two images are one slice of the same patient in ES stage and its ground truth. The images are from Public ACDC dataset [Bernard et al., 2018].

Figure 1

 1 Figure 1.5 -Contributions of this thesis. Three techniques with respect to three kinds of medical image data (Application cases).

Figure 2 . 1 -

 21 Figure 2.1 -Review of deep learning based image segmentation (Please start reading from 12 o'clock and in clockwise direction).

Fig. 1 .

 1 Fig.1. U-net architecture (example for 32x32 pixels in the lowest resolution). Each blue box corresponds to a multi-channel feature map. The number of channels is denoted on top of the box. The x-y-size is provided at the lower left edge of the box. White boxes represent copied feature maps. The arrows denote the different operations.

Figure 2 . 2 -

 22 Figure 2.2 -Original U-Net architecture. The number of channels is denoted above the box, different colors mean different operation.

Fig. 1 .

 1 Fig. 1. Overview of 20 loss functions for medical image segmentation.

Figure 2 .

 2 Figure 2.3 -A review of loss functions. The figure is abtained from [Ma et al., 2021].

CHAPTER 2 .

 2 STATE OF THE ARTCross Entropy based lossCross Entropy loss It describes the distance between two distributions. The smaller the Cross Entropy, the closer the two are. Taking binary pixel/image classification as an example, one distribution is the class prediction probability, which is the output

Focalloss

  Focal loss (FL) [Lin et al., 2017b] is the variant of Cross Entropy loss. It solved the extreme object-background class imbalance problem by adding two coefficients α and γ to balance the weight of one-class examples, and adjust the rate to increase the importance of correcting mis-classified examples. In the original paper, the best performance was when the γ value was set to an empirical value of 2,

Figure 2 . 4 -

 24 Figure 2.4 -A diagram of Ground truth image (a), predicted image (b) and the overlap between two masks (c). In (c), green pixels are TP, blue ones are FP, red ones are FN, grey ones are TN. The figure is adapted from[START_REF] Taghanaki | Deep semantic segmentation of natural and medical images: a review[END_REF].

  It is defined as the maximum surface distance (Di st ) between the segmentation results and ground truth. HD = Max(Max(Di st (Seg , GT)), Max(Di st (GT, Seg ))) (2.26) Average symmetric surface distance (ASSD) It computes the average surface distance between the segmentation results and ground truth. ASSD = Mean(Mean(Di st (Seg , GT)), Mean(Di st (GT, Seg ))) (2.27)

CHAPTER 2 .

 2 STATE OF THE ART broadly classify the studies of deep learning-based direct prediction of biomarker from medical images into the following categories and will describe it in detail The so-called multi-scale is actually sampling the signals/images at different scales, and usually at different scales we can observe different features to accomplish different tasks. Then the intention of multi-scale learning is to enlarge the reception field in the networks. The specific network structure can be classified as follows: (1) Multi-scale input. (2) Multi-scale feature fusion. (3) Multi-scale model fusion. (4) Combination of the above methods.

Figure 2 . 5 -

 25 Figure 2.5 -Multi view/channel fusion strategy, the input is cardiac top slice, middle slice and bottom slice, the figure is obtained from [Luo et al., 2017].

XFig. 2 .

 2 Fig. 2. The flowchart of the proposed unsupervised feature learning and random forest regression. [Left block]: unsupervised cardiac image representation learning by multi-scale deep networks from a unlabeled dataset. [Right block]: training regression forests and on labeled data. [Bottom block]: joint bi-ventricular volume estimation with the trained regressors.

Fig. 3 .

 3 Fig. 3. The schematic diagram of unsupervised feature learning with the proposed multi-scale deep networks. The three blocks from bottom to top are the input MR images, a multi-scale convolutional RBM and an RBM.

  contrast to the original CRBM, we propose multi-scale CRBM (MCRBM) with filters of different sizes, which means we have S × K filters with S the number of scales.By stacking an RBM on top of the proposed MCRBM, we obtain a three-layer network, i.e. , the multi-scale convolutional deep belief network (MCDBN). Totally unlabeled cardiac MR images are fed into the MCRBM to learn a set of multi-scale filters, i.e. , feature detectors.

Figure 2 . 6 -

 26 Figure 2.6 -Multi-feature fusion strategy, the input is cardiac images, the feature maps are in different size because of different kernel/filter size, the figure is obtained from [Zhen et al., 2016a].

CHAPTER 2 .

 2 STATE OF THE ARTMulti-task learningW.Xue et al. / Medical Image Analysis 43 (2018) 54-65 2. Overview of DMTRL, which combines a deep convolution neural network (CNN) for cardiac image representation, two parallel recurrent neural network (RN poral dynamic modeling of cardiac sequences, a Bayesian based multitask relationship learning module for LV indices estimation, and a softmax classifier for se identification.3. Architecture of the newly designed CNN for cardiac MR images. The opion type of each layer is encoded in colors. The size and number of convoon kernel are shown above the diagram, while the dimensionality of output is wn below. Our CNN is adequate for robust representation of cardiac images with all size of network parameters.

Fig. 4 .

 4 Fig. 4. Visualization of feature maps obtained by our CNN for an example image.

Figure 2 .

 2 Figure 2.7 -Multi-task learning with different neural networks, the figure is obtained from[START_REF] Xue | Full left ventricle quantification via deep multitask relationships learning[END_REF]

  set up a encoder (convolution) and decoder (deconvolution) networks to reconstruct the input medical data, then the multiple indices of cardiac are estimated from the reconstructed images, see Figure 2.8. W. Wang et al.: Quantification of Full LV Metrics via Deep Regression Learning

FIGURE 4 .

 4 FIGURE 4. The overview of the segmentation component. (a) The DenseNet architecture for segmentation. Features of cardiac MR images are extracted mainly through three dense blocks and three transition blocks. Each ''deconv'' here corresponds to the sequence ''deconv-BN-ReLU'' and there are two different segmentation results from the DenseNet: the left ''true'' segmentation result is generated from the predicted category labels while the right ''soft'' one is produced by weighting three probability maps. Details of the dense block and the transition block are illustrated in (b) and (c), where k (= 16 in our network) represents the growth rate of feature channels, and θ (= 0.5 in our network) determines the output number of channels. (a) The DenseNet architecture for segmentation. (b) Dense block. (c) Transition block.

FIGURE 5 .

 5 FIGURE 5. The CNN for regression component. The CNN consists of three convolution layers and two fully connected layers, each convolution contains convolution and ReLU operations. The soft segmentation results of cardiac MR image from the DenseNet are as input for the CNN.

  where X = x s,f are the input cardiac images, and Y = ŷc s,f ,(m,n) are the category labels for each pixel. s = 1 • • • S denotes diverse subjects, and f = 1 • • • F represents frame sequence, c ∈ {background, myocardium, cavity}. (m, n) denotes the pixel index, and w Dense is the parameters set of DenseNet. The DenseNet is trained by minimizing the mean log-likelihood cost, and the loss for category prediction is

Figure 2 . 8 -

 28 Figure 2.8 -Segmentation results as input training data, the outputs are indices of cardiac, the figure is obtained from [Wang et al., 2019].

Fig. 2 .

 2 Fig. 2. The flowchart of the proposed unsupervised feature learning and random forest regression. [Left block]: unsupervised cardiac image representation learning by multi-scale deep networks from a unlabeled dataset. [Right block]: training regression forests and on labeled data. [Bottom block]: joint bi-ventricular volume estimation with the trained regressors.

Fig. 3 .

 3 Fig. 3. The schematic diagram of unsupervised feature learning with the proposed multi-scale deep networks. The three blocks from bottom to top are the input MR images, a multi-scale convolutional RBM and an RBM.

  the original CRBM, we propose multi-scale CRBM (MCRBM) with filters of different sizes, which means we have S × K filters with S the number of scales.By stacking an RBM on top of the proposed MCRBM, we obtain a three-layer network, i.e. , the multi-scale convolutional deep belief network (MCDBN). Totally unlabeled cardiac MR images are fed into the MCRBM to learn a set of multi-scale filters, i.e. , feature detectors. The feature maps from CRBM go further through an RBM to obtain more compact representations.

Figure 2

 2 Figure2.9 -The flowchart of the feature learning and random forest regression, the figure is obtained from[Zhen et al., 2016a].

Fig. 2 .

 2 Fig. 2. An overview of the proposed framework, temporal regression network (TempReg-Net) Note that only convolutional layers are shown and Conv1 and Conv2 and Conv5 layers are followed by Pooling layers of size 3x3 and stride 2.

Fig. 2 Fig. 2 :

 22 Fig. 2 shows an overview of the proposed TempReg-Net framework, combining CNN and RNN (more specifically, the Long Short Term Memory (LSTM)). First, a feature

Figure 2 .

 2 Figure 2.11 -Attention mechanism in computer vision. The figure is obtained from[START_REF] Woo | Cbam: Convolutional block attention module[END_REF].

Figure 2 .

 2 Figure 2.12 -A test image (a cat) from CIFAR dataset and the gradient-based saliency map of the test image predicted on a customize CNN model. A saliency map in which pixels are colored by their contribution to the classification.

Figure 2 .

 2 13 shows feature maps of different layers. The output of each feature map is the weight value of each neuron.

Figure 2 .

 2 Figure 2.13 -A test image (a cat) from CIFAR dataset and feature maps of different layers of a customize CNN model.

Figure 2 .

 2 Figure 2.14 -Comparison of gradients, deconvnet, guided backpropagation methods (Figure is adapted from [Springenberg et al., 2015].

Anchors[

  Ribeiro et al., 2018]: It's a method of rule-based, model-agnostic explanations called anchors, designed to exhibit both these properties. Anchors highlight the part of the input that is sufficient for the classifier to make the prediction, making them intuitive and easy to understand. SHapley Additive exPlanations (SHAP) [Castro et al., 2009]: SHAP assigns each feature an importance value for a particular prediction. It calculates the marginal contribution of features to the model output, and then explain the "black box model" from both global and local levels. SHAP constructs an additive explanatory model, and all features are regarded as "contributors". RISE [Petsiuk et al., 2018]:It estimates importance empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs.DeepDream [Mordvintsev et al., 2015]:The idea in DeepDream is to choose a layer (or layers) and maximize the "loss" in a way that the image increasingly "excites" CHAPTER 2. STATE OF THE ART the layers. The complexity of the features incorporated depends on layers chosen by users, i.e, lower layers produce strokes or simple patterns, while deeper layers give sophisticated features in images, or even whole objects.Testing with Concept Activation Vector (TCAV)[START_REF] Kim | Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV)[END_REF]: Unlike saliency explaining a single example, TCAV tries to explain a concept in terms of humanfriendly and find the corresponding visual pattern.

  above three research areas correspond to the research themes of this doctoral dissertation, i.e., in the next chapters we present several contributions for our own methods and experimental results around the problems in segmentation, the problem of biomarker estimation, and the problem of explainability of deep learn-of skin lesion . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Related works in skin lesion segmentation . . . . . . . . . . . 3.3 The Kappa loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 From metrics to loss . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Definition of the Kappa loss . . . . . . . . . . . . . . . . . . . . 3.3.3 CNN for image segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Experimental settings . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 2 -

 32 Figure 3.2 -Venn diagram of ground truth and predicted area. Ground truth contour vs predicted contour, with a, b, c the number of pixels included in both contours, only in the predicted area, only in the ground truth, respectively.

  Figure 3.3 -Architecture of customized U-Net. Each box corresponds to a multi-channel feature map. The number of channels is denoted above the box, different colors mean different operation.

Figure 3 . 4 -

 34 Figure 3.4 -Examples of segmentation results. From up to down: skin lesion image, ground truth, segmentation result with Dice and Kappa loss.

Figure 3 Figure 3 . 6 -Figure 3 Figure 3

 33633 Figure 3.5 -Loss function (left) and DI metric (right) during the 100 epochs of training process on the dataset ISIC 2016.

Figure 3 Figure 3

 33 Figure 3.9 -Feature maps of U-Net with Dice loss on a skin image with small lesion.

Figure 3 Figure 3

 33 Figure 3.11 -Feature maps of U-Net with Dice loss on a skin image with large lesion.

Figure 4 .

 4 Figure 4.1 -US images of fetus head from HC18 dataset [van den Heuvel et al., 2018b]. Red ellipses are head contours. Below the image is given the corresponding head circumference (HC). Images may have different pixel size.

Figure 4 .

 4 Figure 4.2 shows both architectures of segmentation-based and segmentation-free (regression-based) approaches. We will describe these two models in detail in the following sections.

Figure 4 . 2 -

 42 Figure 4.2 -Overview of head circumference estimation process based on either deep segmentation-based method or deep regression-based method. HC: Head circumference, pp: post-processing (The dotted box means it is optioanal), EF: Ellipse fitting.

Figure 4 . 2 ,

 42 the regression CNN are composed of a CNN backbone and regression layer (linear activation function), which can learn the features of input fetus head to estimate HC value directly. The function of CNN backbone is to extract key features from input training data. Afterwards, these feature maps are flattened into one long feature vector. The feature vector are activated by linear function (Actually keep unchanged). The backbone CNN that we experimented are state-of-theart architectures: VGG16 [Simonyan and Zisserman, 2015] (16 in VGG16 refers to it has 16 layers that have weights.), ResNet50 [He et al., 2016a] (It is a variant of ResNet model which has 48 convolution layers along with 1 MaxPooling and 1 Average Pooling layer.), EfficientNetb2 [Tan and Le, 2019] (It has various architecture versions CHAPTER 4. FETUS HEAD CIRCUMFERENCE PREDICTION

Table 4 . 1 -

 41 Number of trainable parameters (#) of segmentation and regression CNN models. M = million. Backbone names: B1 = VGG16, B2 = ResNet50, B3 = EfficientNetb2, B4 = DenseNet121, B5 = Xception, B6 = MobileNet, B7 = InceptionV3. Reg=Regression.

  [START_REF] Van Den | Automated measurement of fetal head circumference using 2d ultrasound images[END_REF] contains 999 US images acquired during the various trimesters of the pregnancy, along with the corresponding ground truth of the skull contour map and HC values. The reference contour of fetus head is annotated as ellipse shape by professional sonographer and the HC value as well as pixel size of each image is given in a text file. The gestational age range of this dataset is 10-40 weeks[START_REF] Van Den | Automated measurement of fetal head circumference using 2d ultrasound images[END_REF].Data pre-processingImage preprocessing includes a resizing from 800×540 pixels to 224×224, and normalization by subtracting the mean and dividing by standard deviation. The HC values are normalized by dividing by the maximum value of HC, in order to improve convergence. We split the dataset into training set (600 images), validation set (199 images) and test set (200 images) in random order. We augment the data of the training set by performing horizontal flipping, and rotation with 10 degrees, the amount of training data is 1800 images.

Table 4 . 2 -

 42 Segmentation accuracy of the 17 segmentation models and HC estimation accuracy with (w) and without (w/o) post-processing (pp). The results are mean and ± standard deviation. DI = Dice Index, HD = Hausdorff Distance, ASSD = Average symmetric surface distance (mm), MAE = Mean Absolute Error (mm, pixel), PMAE = Percentage MAE. B1 = VGG16, B2 = ResNet50, B3 = EfficientNetb2. Best results are in bold.

Figure 4 . 5 -

 45 Figure 4.5 -Scatter plots of the 2 best segmentation models U-Net-B2 and LinkNet-B2, and regression models (L1 = MAE loss, L2 = MSE loss). The x-axis represents the ground truth HC and the y-axis the predicted HC (in mm).

Algorithm 4 . 1

 41 The estimated memory cost of a model Input: Regression CNN Model, batch_size. Output: Total memory (gigabytes). 1: procedure MEMORY_USAGE(Model, batch_size)

Table 4 .

 4 5 -Training and predicting time and memory cost of segmentation vs. segmentation-free models on test set (200 images). B1 = VGG16, B2 = ResNet50, B3 = Ef-ficientNetb2, B4 = DenseNet121, B5 = Xception, B6 = MobileNet, B7 = InceptionV3, L1 = MAE loss, Mem-M= theoretical memory of model, Mem-P= memory in prediction stage, GB = gigabyte.

  5 shows, the training time per epoch over 1800 training US images for the segmentation method U-Net-B2 (U-Net with ResNet50), takes 29 seconds on a Tesla P100 GPU. For the best regression model Reg-B3-L1 (EfficientNet), it takes 20 seconds. In the prediction stage with a Intel Core i7 CPU, 32 GB RAM, the Regression Reg-B3-L1 only takes 36.95 seconds over 200 test images; in other words,predicting one image requires 0.18 second, to be compared to 0.35 seconds of the U-Net-B2. Segmentation-based methods require longer time at training but also at inference time, than segmentation-free methods. As a conclusion, whereas the advantage of using regression-based approach is clear computationwise, there is no clear evidence that regression models are less memory greedy, in the experimental conditions we set up. It's worthy to note that with the continuous progress of hardware and computing power, such time error between segmentation-based and segmentation-free methods may be ignored in clinical practice.

Figure 4 .

 4 Figure 4.7 compare the best segmentation-based and segmentation-free methods in terms of memory cost during the model prediction stage. From the figure, we can find that predicting the same number of images, the segmentation-free method takes less time than the segmentation method. This is because there are two processes in the segmentation model, image feature extraction (Encoder) and feature upsampling (Decoder).

Figure 4 .

 4 Figure 4.7 -Memory cost of U-Net, B2=ResNet during prediction stage. The maximum used memory of models (when the inference is stable) is after the red vertical line.

Figure 4 .

 4 Figure 4.8 -Memory cost of Regression B3=EfficientNet, L1=MAE during prediction stage.The maximum used memory of models is after the red vertical line.

Figure 4 . 9 -

 49 Figure 4.9 -Saliency maps of different regression CNNs explained by LRP method. Row (1), the input US fetus images (numbers are the ground truth HC values.); (2), Regression VGG16 (numbers are the predicted HC values.); (3), Regression ResNet50; (4), Regression EfficientNetb2; (5), Regression Densenet121; (6), Regression Xception; (7), Regression Mobilenet; (8), Regression InceptionV3. The best predicted results are in bold. The red points in saliency maps are positive values, the blue points are negative values.

CHAPTER 4 .

 4 FETUS HEAD CIRCUMFERENCE PREDICTION viation 0.3 is added to each subwindow, according to their importance assigned by analyzers during the 16 steps. The input data is the test data set (200 images) and corresponding ground truth. Thus it is slightly time consuming than the saliency map.Figure 4.

Figure 4 .

 4 Figure 4.13 -Prediction error (in pixels) of different analyzers during each perturbation step based on Regression VGG16 and Regression ResNet50 model. The horizontal axis is the perturbation steps.

  dataset and preprocessing . . . . . . . . . . . . . . . . . 5.3.2 Regression CNN . . . . . . . . . . . . . . . . . . . . . . . . . . . protocol . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.3 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Figure 5.1 -The structures of a human cardiac. (a) Diagram of a human heart (The figure is taken from Wikimedia Commons); (b) 3 structures (RV, MYO, LV) of cardiac in short axis view (The figure is taken from[Ibrahim, 2011]).

  resonance images (MRI) subjects in training set, each subject has 3 manual annotated labels, i.e., left ventricular (LV), myocardium (MYO), right ventricle (RV). Each subject has end diastolic (ED) and end systolic (ES) phase. Because the machine acquires consecutive frames within one heartbeat cycle. The ED stage and ES stage are selected by an experienced specialist or physician by observing changes in the size of the heart chambers. These subjects are divided into 4 types of disease, myocardial infarction (MINF), dilated cardiomyopathy (DCM), hypertrophic cardiomyopathy (HCM), abnormal right ventricle (ARV), and patients with normal cardiac (NOR).

  px_x * px_y * space_z/1000 MYO = N i S_MYO * px_x * px_y * space_z/1000 LV = N i S_LV * px_x * px_y * space_z/1000

From

  Figure 5.2, we can see that the cropped MI image maximizes the retention of the cardiac target and removes other organs or noise from the image, which facilitates the learning of features in the image by the deep model.

Figure 5 . 2 -

 52 Figure 5.2 -One slice of original ACDC data and ground truth (gt), the size is 182*216, the right column is the cropped MRI cardiac subject and its ground truth, the size is 100*100.

  Figure 5.3 -One of preprocessed 3D cardiac data from ACDC dataset. The cardiac slices can be regarded as image channels.

  Figure5.4 -2D (a) and 3D (b) convolution on 3D image. For 2D convolution, the input layer and the filter have the same depth (channel number = kernel/filter number), the output is a one-layer matrix. For 3D convolution, the filter moves on 2D channel then moves in z direction, the output is a 3D matrix. The figure is taken from Towardsdatascience.

CHAPTER 5 .

 5 Figure 5.5 -The architecture of regression CNN for predicting the volume of RV, MYO, LV directly. The input training data are preprocessed MRI images(224,224,9), the ground truth are the volume of 3 structures of cardiac. FC is fully connected layer after feature representation, the FC layer goes through linear regression layer, the output is the predicted volumes(vol_RV, vol_MYO, vol_LV).

Figure 5 Figure 5 Figure 5

 555 Figure 5.8 -The mean absolute error of three cardiac structures (RV, MYO, LV) according to different pathologies. The prediction results are from Regression ResNet.

  Figure 5.11 -Saliency maps of regression CNN models on cardiac images. The first image is one single input cardiac slice and its saliency map (SA) of regression EfficientNet (reg_efn) and regression VGG (reg_VGG). The second image is the 9-slice input cardiac and their saliency maps of the regression ResNet model. Red color means positive contributions, blue color means negative contributions.

Figure A. 2 -

 2 Figure A.2 -Saliency maps of different explanation methods under 3 different loss functions and regression CNN model VGG16 (V) and ResNet50 (R).

Figure 1 :

 1 Figure1: Model overview. We split an image into fixed-size patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder. In order to perform classification, we use the standard approach of adding an extra learnable "classification token" to the sequence. The illustration of the Transformer encoder was inspired by[START_REF] Vaswani | Attention is all you need[END_REF].

Figure B. 3 -

 3 Figure B.3 -Vision Transformer model. An image is split into certain number of patches.The position information is added in each patch. A learnable classification matrix is also added. The figure is obtained from[START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF].

Table 1 .

 1 1 -Five common modalities of medical imaging.

	Modality	Principle		Manner& Duration	Usage
					• bone fractures
	X-Ray	X-rays use ionizing radiation which are quick, painless tests that produce images of structures inside one's body, especially bones.	One will lie, sit, or stand while the x-ray machine takes images. One may be asked to move into several positions. 10-15 minutes.	• arthritis • osteoporosis • infections • breast cancer • swallowed items • digestive tract problems
	CT Scan	CT scans use a series of x-rays to create cross sections of the inside of the body, including bones, blood vessels, and soft tissues.	One will lie on a table that slides into the scanner. The x-ray tube rotates around one to take images. 10-15 minutes.	• injuries from trauma • bone fractures • tumors and cancers • vascular disease • heart disease
					• infections
					• guide biopsies
				One will lie on a table that	• aneurysms
		MRIs use magnetic	slides into the MRI machine,	• Multiple Sclerosis (MS)
		fields and radio waves	which is deeper and nar-	• stroke
	MRI	to create detailed im-	rower than a CT scanner.	• spinal cord disorders
		ages of organs and	The MRI magnets create	• tumors
		tissues in the body.	loud tapping or thumping	• blood vessel issues
				noises. 45 minutes-1 hour.	• joint or tendon injuries
					• gallbladder disease
		Ultrasound uses high-	A technician applies gel to	• breast lumps
		frequency	sound	one's skin, then presses a	• genital/prostate issues
	Ultrasound	waves to produce images of organs and	small probe against it, mov-ing it to capture images of	• joint inflammation • blood flow problems
		structures within the	the inside of one's body. 30
		body.		minutes-1 hour.

State of the Art Chpt.3 Kappa loss Chpt.4 Fetus Head Circumference Prediction Chpt.5 Cardiac Multi-structure Volume Prediction Chpt.6 Conclusion

  6. This chapter presents three states of the art in three domains of interest: medical image segmentation, direct biomarker estimation from medical images, and explainable AI for computer vision models in deep learning. For each of this field,

		CHAPTER 1. INTRODUCTION CHAPTER 2. STATE OF THE ART
	Chapter 2	Sec.1.1 Backgournd
	Chtp.1 Introduction	Sec.1.2 Motivation Sec.1.3 Contributions
	State of the Art	Sec.1.4 Thesis structure
		Sec.2.1 Medical image segmentation methods
	Sec.2.3 Explainable Artificial Intelligence (XAI) Sec.2.2 Direct biomarker estimation methods Sec.3.4 Experiments and results Sec.3.3 Methodology Sec.3.1 Motivation Sec.3.2 Background & Related works Chtp.2 Sec.3.5 Conclusion Contents
		Sec.4.1 Motivation
		Sec.4.2 Background & Related works
		Sec.4.3 Methodology
		Sec.4.4 Experiments and results
		Sec.4.5 Conclusion
		Sec.5.1 Motivation
		Sec.5.2 Background & Related works
		Sec.5.3 Methodology
		Sec.5.4 Experiments and results
		Sec.5.5 Conclusion
		Sec.6.1 Conclusion
		Sec.6.2 Perspective
	Figure 1.6 -Thesis structure.

  Di ce , w CE and γ. In this exponential logarithmic Cross Entropy item, w is the weight inside of the Cross Entropy loss to reduce the influences of more frequently

	this loss simply summarize the Cross
	Entropy loss and Dice loss together.
	Di ceCE = Cr ossEnt r op yl oss + Di cel oss	(2.17)
	Dice+Focal loss it [Zhu et al., 2019] combines Dice loss with Focal loss.
	Di ceFoc al = Di cel oss + Foc al l oss	(2.18)
	Exponential Logarithmic loss (ELL) [Wong et al., 2018] combines Dice loss and
	Cross Entropy loss in the exponential logarithmic way with respective weighting fac-
	tors w seen labels.
	ELL = w Di ce E[(-l n(Di ce)) γ ] + w CE E[w(-l n(p i )) γ ]	(2.19)

Table 2

 2

	Datasets	Year	Number of subjects	Ground truth	Type
			training set test set
	HC18 1	2018 999	335	head circumference	US
	Sunnybrook 2 2009 45	-	LV, MYO, Pathology	MRI
	LVSC 3	2011 100	100	LV, Pathology	MRI
	MICCAI RV 4	2012 16	32	RV	MRI
	Kaggle 5	2015 500	300	Cardiac Volumes	MRI
	ACDC 6	2017 100	50	LV, RV, MYO, Pathology MRI
	LVQUAN18 7	2018 145	30	Cardiac indices	MRI
	LVQUAN19 8	2019 56	30	Cardiac indices	MRI

.3. These challenges are oriented to segmentation tasks at first, and mainly focused on cardiac images. The other dataset of different organs or tissues mentioned in this survey were not publicly accessible. Table 2.3 -Public medical image datasets used in direct estimation.

  prediction, spine Cobb angle prediction, kidney disease diagnose by volume prediction, and cardiovascular disease diagnose by some medical indices prediction.Especially in cardiac problem, great efforts are put by plenty of researchers through various traditional machine learning or deep learning based methods. However, due to the quality and types of medical images as well as deep learning models are various, thus the author have the following perspectives.

	Therefore, when the data is appropriate and the model is designed reasonably, sat-
	isfactory results can generally be obtained. However, interpretability is quite impor-
	tant in medical imaging analysis and disease diagnose. If human being can under-
	stand and trust the explainable unseenable deep learning black box, then this type

Data type: Currently, the medical images cover MRI, CT, Ultrasound, X-Rays, they are 2D or 3D formats. But in this survey, almost all the studies convert the 3D images into 2D slices. For example, in cardiac indices or volume estimation problem, they input 2D slice or 2D+Time slice, but not the whole 3D scanned data, that is to say, they predict the area first then sum up the area of each slice.

Preprocessing: Because of the complexity of dataset, data preprocessing is necessary in every research. If the data amount is limited, then data augmentation should be performed before or during model training process. Moreover, data resizing, cropping and normalization are common operation in deep learning. If the target is small in the whole image, then ROI detection can largely reduce processing time and improve efficiency, for instance in cardiac data, it just needs to focus on two or four chambers but not the other parts of body. Also, certain slice selection can be done depend on specific demand which may improve the performance because of clear features. However, because this step (data and preprocessing strategy) in each experiment is very different from the other papers, which leads to the results hard to be compared.

Methods:

We can clearly see that the deep learning methods in recent years dominate in various applications. In deep learning methods, they are divided into several sub branches such as multi-scale learning, multi-task learning, attention mechanism, combining traditional machine learning with deep learning methods, etc. of deep learning methods are reliable to applied in clinical medical applications and become a right-hand man.

Table 2 .

 2 4 -Summary of explanation methods and their abbreviations.

	Abbreviation Method
	Method

, they propose Faithfulness/Sufficiency, Human Interpretability, Applicability and Runtime. From these evaluation critics We can see CHAPTER 2. STATE OF THE ART

Table 2 .

 2 6 -Explanation tools through different platforms.

	Tools	Category	Tools	Category
	Heatmapping 3	web	CNN Explainer 4	web
	Explainable AI Demos 5	web	A Neural Network Playground 6 web
	Summit 7	web	NeuralDivergence 8	web
	SCIN 9	web	Neuroscope	free software
	LUCID 10	library	Keras-vis 11	library
	DeepExplain 12	library	iNNvestigate 13	library
	TensorFlow Graph Visualizer 14 library	tf-explain 15	library
	TorchRay 16	library	Captum 17	library
	What-If Tool 18	library	SHAP 19	library
	Interpret 20	library	Eli5 21	library
	Skater 22	library	GANDissect 23	library
	Yellowbrick 24	library	AIF360 25	library
	Alibi Explain 26	library	AIX360 27	library
	Explainable AI 28	commercial exAID 29	commercial
	H2o 30	commercial DASL 31	commercial
	SCOPA 32	commercial

indicates that the explainable AI is everywhere and of much necessity in different areas and quite a lot researcher are dedicated in making the deep learning technol-CHAPTER 2. STATE OF THE ART ogy understandable and trustable.

Table 3

 3

		Ground Truth (Rater 1)
	+	-	Total
	Predicted
	Results
	(Rater 2)

.1 -Counts of agreement and disagreement from two raters. a + d is the number of targets for which two raters agreed, b + c is the number of targets for which they disagreed, N = a + b + c + d .

Table 3 .

 3 2 -Averaged Dice index (DI) and Hausdorff distance (HD) values (± standard deviation), for Dice and Kappa losses on 6 different datasets(87, 119, 206, 900, 2000, 2594 images respectively).

			Dice loss	Kappa loss
	dataset	DI ↑	HD(mm) ↓	DI ↑	HD(mm) ↓
	Non-mel	0.65±0.11	5.06±1.79	0.73±0.11	4.70±2.02
	Mel	0.80±0.06	6.70±1.93	0.81±0.03	6.59±1.88
	SCD	0.82±0.04	7.94±1.72	0.83±0.03	7.91±1.68
	ISIC-16	0.80±0.05	8.42±2.19	0.84±0.01	8.41±2.25
	ISIC-17	0.80±0.05	8.07±1.93	0.84±0.05	8.03±1.94
	ISIC-18	0.81±0.03	7.59±2.60	0.82±0.04	7.52±2.66

  function because of binary pixel-wise classification. For regression models, both the weights of CNN feature extractor part and regression layer are trainable.

	Because the number of training data of HC18 is limited, to avoid over-fitting, we
	set the dropout rate as 0.7; in other words, 30% of parameters in regression CNN
	models are kept. The activation function of last layer in Regression CNN is linear
	function.
	U-Net	31.06	Reg-B1	15.15
	U-Net-B1, B2, B3	23.75, 32.51, 14.23	Reg-B2	23.63
	DoubleU-Net	29.29	Reg-B3	76.73
	U-Net++ B1, B2, B3	24.15, 34.34, 16.03	Reg-B4	70.04
	FPN-B1, B2, B3	17.59, 26.89, 10.77	Reg-B5	20.91
	LinkNet-B1, B2, B3	20.32, 28.73, 10.15	Reg-B6	3.26
	PSPNet-B1, B2, B3	21.55, 17.99, 9.41	Reg-B7	21.82
	The number of trainable parameters of each model is listed in Table 4.1. Due to
	space limitations, we cannot list all the CNN models, instead we list several models
	with outstanding performance in recent years. In this work, several CNN backbone
	models in segmentation and regression respectively are utilized in order to fine tune

the networks with the HC18 dataset. For segmentation models, the weights of encoder and decoder are set to be trainable, the activation function of last layer is Sig-moid

Table 4 .

 4 3 -Average performance of 21 regression CNN models over 5 fold cross validation. The results are mean and ± standard deviation. MAE = Mean Absolute Error, PMAE = Per-

	1.23±1.49 11.83±38.75 9.11±10.70 1.04±4.13 0.75±1.04	2.59±1.88 18.93±11.53 18.76±10.96 1.57±1.19 1.56±1.17	1.21±1.29 9.99±18.72 8.98±8.48 0.85±1.98 0.74±0.75	1.439±1.70 13.95±21.89 10.49±11.22 0.85±1.98 0.74±0.75	1.08±1.25 8.69±14.40 7.86±7.51 0.74±1.46 0.65±0.68	1.32±1.67 10.23±16.98 9.94±13.58 0.86±1.62 0.83±1.32	1.3±2.12 16.95±77.93 9.91±18.52 1.51±7.73 0.87±2.33	1.15±1.59 12.65±41.16 8.63±12.24 1.16±4.65 0.72±1.13	1.19±1.44 19.08±108.01 8.92±11.01 1.57±9.02 0.76±1.21	1.29±1.61 11.17±22.67 9.70±12.84 0.99±2.58 0.80±1.16	1.26±1.33 10.35±17.99 9.18±8.58 1.90±9.68 0.77±0.87	1.39±1.5 11.09±16.01 10.33±10.53 0.94±1.58 0.86±1.06	1.32±1.44 11.32±16.14 9.91±10.23 0.98±1.70 0.83±1.02	1.15±1.32 8.86±11.83 8.45±8.38 0.74±1.08 0.69±0.77	1.29±1.51 10.55±15.97 9.70±9.84 0.89±1.62 0.79±0.85	1.33±1.38 22.39±79.36 9.84±9.03 2.21±8.94 0.81±0.81	1.20±1.34 11.98±22.70 8.75±7.98 1.07±2.47 0.72±0.68	1.29±1.35 10.59±16.40 9.64±9.14 0.93±1.94 0.81±0.86
	1.55±4.41	2.60±1.88	1.31±2.07	1.82±2.90	1.16±1.78	1.34±1.97	2.03±8.39	1.74±6.38	2.32±11.80	1.44±2.43	1.38±2.16	1.46±1.92	1.46±1.914	1.19±1.56	1.37±1.94	3.07±12.89	1.66±3.62	1.38±1.95
	3 1.56±2.67 0.35±0.29	98.7±1.4 1.14±0.85 0.29±0.26	98.6±1.3 1.16±1.24 0.31±0.26	98.3±1.7 1.79±2.58 0.39±0.35	98.8±0.9 1.09±1.11 0.28±0.22	98.7±1.1 1.10±1.03 0.29±0.24	98.5±2.4 1.29±1.46 0.31±0.25	98.7±1 1.24±1.66 0.29±0.22	98.7±1.2 1.17±1.28 0.29±0.25	98.6±1.1 1.28±1.68 0.32±0.28	98.7±0.9 1.18±1.18 0.30±0.23	98.7±1 1.19±1.52 0.30±0.25	98.6±1.2 1.31±1.53 0.33±0.25	98.7±1.1 1.12±0.99 0.30±0.22	98.6±1 1.15±1.04 0.31±0.26	98.6±1.4 2.00±3.88 0.38±0.44	98.8±0.9 1.42±2.31 0.31±0.27	98.7±1.1 1.12±1.13 0.32±0.25
		DoubleU-Net	U-Net(B1)	U-Net-B2-no *	U-Net(B2)	U-Net(B3)	U-Net++(B1)	U-Net++(B2)	U-Net++(B3)	FPN(B1)	FPN(B2)	FPN(B3)	LinkNet(B1)	LinkNet(B2)	LinkNet(B3)	PSPNet(B1)	PSPNet(B2)	PSPNet(B3)

Table 4 .

 4 4 -Comparison of HC estimation for the 2 best segmentation and regression models. B2: Resnet50. B3: EfficientNet, L1 = MAE loss, L2 = MSE loss. The results are mean and ± standard deviation. MAE = Mean absolute error, PMAE = Percentage MAE. The best results are in bold. (p value<0.05)

	Metrics	MAE(mm)	MAE(px)	PMAE(%)
	Methods		Segmentation-based methods
	U-Net-B2	1.08±1.25	7.87±7.51	0.65±0.68
	LinkNet-B2	1.15±1.32	8.45±8.39	0.69±0.77
			Segmentation-free methods
	Reg-B3-L1	1.83±2.11	13.57±13.53	1.17±1.43
	Reg-B3-L2	2.35±2.74	17.32 ±17.95	1.53 ±2.02

Table 4 .

 4 7 -Performance (AOPC scores) of different explanation methods after perturbation, with two regression models. G: Gradient, SG: SmoothGrad, DCN: DeConvNet, DT: Deep-Taylor, GB: GuidedBackprop, I*G: Input*Gradient, IG: IntegratedGradients. Lower is better. Best scores in bold.

	Model	G	SG	DCN	DT	GB	I*G	IG	LRP
	Reg_VGG16	-7.31	-7.39	-2.87	-7.40	-1.66	-9.19	-9.49	-9.17
	Reg_ResNet50	-11.53 -11.84 -9.25	-9.89	-9.72	-14.75 -5.60	-14.58

Table 5 .

 5 1 -Abnormal samples in ACDC dataset, S is the total pixel numbers of all sllices, V_c is volume from calculation (see Formula 5.1), V_gt is volume from given data, Diff is the difference between V_g and V_c.

Patient spacing(x,y,z) shape(z,y,x) S(pixel) V_c(ml) V_gt(ml) Diff(ml)

  CHAPTER 5. CARDIAC MULTI-STRUCTURE VOLUME PREDICTIONportion in each slice (See Figure5.2). And this is not conducive to the training of convolutional neural networks. Because the model requires the fixed shape of input data and the target/feature should be as clear as possible.

	Therefore, we perform data cropping to uniform the shape of 3D MR images and
	find the region of interest (ROI) of cardiac. We first find the maximum bounding
	box of the cardiac from ground truth images 1 . Then we crop the MR images based
	on the maximum bounding box (See Algorithm 5.1).
	Algorithm 5.1 Cardiac MRI cropping algorithm
	Input: Original Cardiac MRI and ground truth (GT).
	Output: Cropped Cardiac MRI.
	P019	(1.445,1.445,10) (11, 256, 216) 32269	673.78	868.59	194.81
	P078	(1.367,1.367,10) (8, 256, 216)	25813	482.36	630.20	147.84
	P079	(1.367,1.367,10) (9, 256, 216)	18667	348.83	455.74	106.91
	P080	(1.758,1.758,10) (6, 256, 216)	9173	283.50	223.95	59.55
	P093	(1.563,1.563,7)	(10, 224, 180) 23491	401.71	57.35	344.35
	P099	(1.786,1.786,5)	(16, 224, 154) 27180	433.49	866.71	433.21
	Data preprocessing

Image cropping and ROI detection From Table

5

.1 we can also see that each original data has different sizes in 3 dimensions. Moreover, the cardiac takes small pro-1: for subject_i in GT_Dataset do Finding max Bounding box in ground truth dataset.

  .3.

	CHAPTER 5. CARDIAC MULTI-STRUCTURE VOLUME PREDICTION
	Algorithm 5.3 Data augmentation based on grid search
		Input: Index of function list.	Functions of data augmentation
		Output: Subsets without duplicate.
	1: Initialize result	Store all the subsets
	2: Initialize temp_set	Temporal set
	3: procedure BACKTRACKING(nums, startIndex)
	4:	if startIndex >= nums.size then
	5:	return
	6:	end if
	7:	for startIndex in nums.size do	Horizontal traversal
	8:	temp_set.append(nums[startIndex])
	9:	result.append(temp_set)
	10:	BACKTRACKING(nums, startIndex+1)	Vertical traversal
	11:	temp_set = temp_set[-1]
	12:	end for
	13: end procedure
	14: procedure SUBSETS(Index of function list)
	15:	BACKTRACKING(Index of function list,0)
	16:
	When dealing these 3D cardiac data with deep convolutional neural networks, it
	110

Table 5

 5

	.2 compares 2D regression CNN with 3D regression CNN in the aspects of number
	of parameters, model memory and training time, etc. The backbone is based on
	VGG16 [Simonyan and Zisserman, 2015]. The convolution kernel of 2D regression

Table 5 .

 5

	2 -Comparison of 2D regression VGG and 3D regression VGG. The training time is
	on the GPU server; the model memory is theoretical requirement; the inferencing memory
	is actual memory cost during the prediction stage of a model; M=Million; GB=Gigabytes;
	N/A=Not applicable.

Model type # of param (M) Training time s/epoch Model memory (GB) Inferencing memory (GB)

	RegVGG_2D	14.72	21	0.518	0.83
	RegVGG_3D	44.93	719	3.724	N/A*
	* The 5-fold training time of regression VGG 3D (around 100 hours) exceeds the maximum GPU
	time (48 hours).

Table 5 .

 5 4 -Prediction error on volume of 3 cardiac structures using regression CNN models with 3 different loss functions, MAE loss, MSE loss and Huber loss (HL). The 3 CNN backbones are VGG16, ResNetV2, EfficientNetb2 (efn). The models are trained on 2900 training images. The results are average results of 5-fold cross validation.

	Model	MAE_RV(ml) PMAE(%)	MAE_MYO(ml) PMAE(%)	MAE_LV(ml) PMAE(%)
	MAE loss
	Reg_VGG	50.51±39.81	65.64±92.96	41.58±34.38	36.43±37.59 35.29±29.49 40.20±52.63
	Reg_ResNet 43.11±36.57	51.63±69.46	36.98±29.25	31.96±29.44 33.19±26.48 39.09±47.45
	Reg_efn	49.55±40.88	60.76±83.83	36.70±32.26	33.50±36.88 33.51±26.93 39.78±52.15
	MSE loss
	Reg_VGG	50.65±40.57	67.29±94.96	42.40±36.86	38.05±41.40 36.02±28.39 42.90±57.95
	Reg_ResNet 43.82±34.47	56.33±79.52	38.11±29.74	34.63±33.99 33.28±26.27 42.01±53.51
	Reg_efn	49.24±40.20	64.68±103.61 36.10±31.35	31.88±34.57 33.03±27.80 38.31±49.99
	HL loss
	Reg_VGG	49.02±38.38	62.88±86.09	40.14±35.26	35.07±37.47 33.11±27.63 37.30±52.79
	Reg_ResNet 49.29±38.12	66.20±93.30	35.13±31.27	30.99±32.11 32.75±28.30 41.57±56.69
	Reg_efn	49.11±39.99	65.08±103.74 36.79±31.14	32.62±34.80 33.28±28.10 37.29±46.31

Table B .

 B 5 -Prediction error (mean absolute error in milliliter, MAE) on volume of 3 cardiac structures from different data splitting settings using regression VGG model with/without data augmentation. The data is split into (train, valid, test). ± is standard deviation. ,50) 45.0±36.0 36.0±30.8 32.6±25.1 48.8±34.4 32.4±28.1 40.0±30.7 (110,32,40) 43.1±32.0 28.3±24.9 29.2±21.5 48.4±32.8 31.9±26.1 40.3±29.9 (120,32,30) 35.7±31.5 29.0±27.6 28.2±22.6 51.6±36.5 35.5±35.0 41.9±34.0 (130,32,20) 37.5±33.1 28.8±23.0 27.2±23.0 41.6±32.9 36.9±33.5 36.0±27.2 (140,32,10) 38.3±28.2 30.8±23.7 26.9±17.5 45.1±34.2 33.5±26.3 39.1±26.4 try to take the same ratio of patients in each type in training and test set. So that the model can be trained in a relatively generic mode. Table B.6 summarize 182 subjects in ACDC dataset. Table B.6 -Data distribution in ACDC dataset based on pathology.

		with data augmentation	without data augmentation
	Data split	RV	MYO	LV	RV	MYO	LV
	(70,52,60)	51.9±41.8 34.9±32.4 36.4±27.3 52.6±41.4 40.6±36.2 43.9±35.7
	(100,32Number	Pathology	Train	Valid	Test
	36	DCM	20	6		10
	38	HCM	20	8		10
	38	MINF	20	8		10
	34	NOR	20	4		10
	36	RV	20	6		10
	Total
	182	5	100	32		50

Table B .

 B 7 compares the prediction error between the evenly distributed training and test data and randomly distributed training and test data with respect of 5 types of pathologies. From this table one can see that the performance of regression CNN models have a little improvement when the training data have the same amount of each disease type. The experimental results demonstrate that the idea of making training and test data evenly distributed in various data types is good for model's generalization ability.

2 Vision Transformer The architecture of Vision Transformer Published

  2 , • • • , head h )W o (B.2)where head i = At t ent i on(QW as a conference paper at ICLR 2021

										Q i , KW K i , VW V i )
	MLP Head B.3.Transformer Encoder Class Bird Ball Car ... Vision Tr ansfor mer (ViT)				MLP Norm + Tr ansfor mer Encoder L x
												+
	Patch + Position Embedding	0	1	2	3	4	5	6	7	8	9	Multi-Head Attention
	* Extra learnable
	[ c l as s ] embedding
												Norm
												Embedded
												Patches

*

Linear Projection of Flattened Patches

The information is gathered from https://blog.radiology.virginia.edu/different-imaging-tests-explained/

Please refer to scipy.ndimage.morphology.distance_transform_edt.

The Hadamard product operates on identically shaped matrices and produces a third matrix of the same dimensions.

https://hc18.grand-challenge.org/

http://smial.sri.utoronto.ca/LVChallenge/Home.html

www.cardiacatlas.org/challenges/lv-segmentation-challenge/

http://rvsc.projets.litislab.fr/

www.kaggle.com/c/second-annual-data-science-bowl

https://www.creatis.insa-lyon.fr/Challenge/acdc/index.html

https://lvquan18.github.io/

https://lvquan19.github.io/ Due to these original datasets are not suitable for directly use in different methods, for instance, the amount, the size, the dimension, etc. Thus it's necessary to apply preprocessing before using them on specific tasks and solutions. In general, data augmentation, normalization, resizing and cropping, Region of Interest (ROI)

http://www.heatmapping.org/

https://poloclub.github.io/cnn-explainer/

https://lrpserver.hhi.fraunhofer.de/

https://playground.tensorflow.org/

https://fredhohman.com/summit/

http://haekyu.com/neural-divergence/

https://www.dfki.de/skincare/classify.html

The most highly-used subset of ImageNet is the ImageNet Large Scale Visual Recognition Challenge (

ILSVRC) 2012-2017 image classification and localization dataset. This dataset spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images and 100,000 test images.

OpenCV library function arcLength is used.

Fine-tuning is arguably the most widely used approach for transfer learning when working with deep learning models. Generally, it starts with a pre-trained model on the source task and trains it further on the target task[START_REF] Guo | Spottune: transfer learning through adaptive fine-tuning[END_REF]

* U-Net-B2-no: not pre-trained on ImageNet.

The source code reference of computing the theoretical memory of a model: https://gist.github.com/jizhang02/ef8eb45450f3d943fea37c6544d3808c

In this step, we remove 3 extreme examples because the size of bounding box are even larger than the size of images of others.

The source code is at https://gist.github.com/jizhang02/4f4a08aa54fe39e4a0ac9b272562bde4

The server is supplied by Centre Régional Informatique et d'Applications Numériques de Normandie (https://www.criann.fr)

The names of Q, K, V are based on the concept of information retrieval system, where Q means Query, K means Key, V means Value.

Acknowledgements

I would like to thank my dissertation reviewers for reading my PhD thesis and providing valuable comments and suggestions.

My 42-month PhD is coming to an end, and I could not have succeeded without the people who have helped me along the way.

List of Publications List of Publications

MAE for this regression network is 1.83 mm. This error is not only smaller than the error (1.90mm) based on segmentation methods in the literature [START_REF] Budd | Confident head circumference measurement from ultrasound with real-time feedback for sonographers[END_REF],

but also much smaller than the error in manual measurements intra (7 mm) and inter-variability (12 mm) of sonographers. Therefore, our proposed method have promising potential for the clinical practice. However, the comparison to manual variability should be handled with care as these results have not been obtained on the same dataset.

Agreement analysis of segmentation CNN vs. regression CNN Comparison of HC estimation accuracy

To compare the performance of the segmentation-free vs. the segmentation approaches, we have gathered the 2 best results from Tables 4.2 and 4.3 into Table 4.4.

From this table, one can see that the best segmentation approach (U-Net-B2: U-Net with pretrained ResNet50 with post-processed segmentation results) is better than the best regression approach (Reg-B3-L1) by 40.7%.

Comparison of learning curves

The learning curves during training and validation stages are generated. We can also notice from

Agreement analysis of prediction results

We also analyse the agreement between the estimated HC values by both types of methods against the real HC values via linear regression. From Figure 4.5, one can first observe a remarkable linear correlation between the prediction and the reference values, for all 4 models, whether it is segmentation or regression models. There

CHAPTER 4. FETUS HEAD CIRCUMFERENCE PREDICTION

We also display some saliency maps where regression models fail to make an accurate estimation (see Figure 4.10). We observe that the features extracted by regression CNN models are fooled by hypersignal (i.e. high intensity pixels) above the head, which leads to increased predicted HC values. This illustrates the case where the background is heterogeneous and makes it difficult for the network to distinguish the head contour and thus to accurately estimate the head circumference.

For more analysis on explainability of regression CNNs, please refer to Appendix A.

(3)

(1)

Evaluation of explanation methods

Qualitative evaluation of explanation methods

We visualize the saliency maps provided by the 8 selected explanation methods in Figure 4.11. From these images, we can barely see the features retrieved by explanation method DeConvNet and Gradient in both models, that is to say these two methods seem somehow insensitive to the models. This may be explained by the gradient shattering problem [START_REF] Balduzzi | Neural taylor approximations: Convergence and exploration in rectifier networks[END_REF] for the gradient method. Regarding DeConvNet's saliency map, it may be due to the the architecture of deconvolution network which reconstructs the convolution networks reversely. In addition, for Reg-ResNet50, methods Gradient, GuidedBackprop and SmoothGrad fail to highlight the head contour. We will see that these observations are confirmed by the quantitative evaluation.

Quantitative evaluation of explanation methods

Here, we compare the explanation methods through perturbation analysis. In this experiment, the input image of size 128×128 pixels is divided into a grid of 4×4 subwindows of size 32×32 pixels. Gaussian noise with mean value 0 and standard de-CHAPTER 5. CARDIAC MULTI-STRUCTURE VOLUME PREDICTION tion, the influence of data augmentation, the hyper parameters selection etc., for more details please refer to Appendix B.

Results

Prediction error on 2D regression VGG16 vs. 3D regression VGG16

In the previous section (Section 5.3.2), we theoretically compare the difference of 2D convolution and 3D convolution. In this experiment, we train the 3D cardiac MRI data using 2D regression VGG and 3D regression VGG, respectively. For the sake of fairness, both these two models are trained from scratch. In Table 5.3, we compares the prediction error on test set between the 2D and 3D regression VGG16. One can find that the 2D regression VGG has smaller prediction error than the 3D ones. Thus, through this results combined with the theoretical analysis based on the previous 2D regression VGG, including training parameters, memory, and training time, we conclude that the 2D regression VGG on multi-slice data offers the best compromise. Therefore, in the following experiments, we use all 2D regression CNNs to train and predict the data in order to save time and memory.

Prediction results of cardiac structure volumes

The following experiments are based on the results of 2D regression CNNs. We use regression VGG16, regression ResNetV2 and regression EfficientNet to separately train the multi-slice data to predict the volumes of three structures of the cardiac simultaneously. In the supervised learning mode, the input ground truth is the volumes of each cardiac structure. We trained the cardiac data of end diastolic (ED) and end systolic (ES) phase together, in other word, the ED and ES of the one patient are in the same fold (training, validation and test)

Discussions

The

Loss functions

The prediction error of regression CNN model with three regression loss functions respectively doesn't have significant difference or pattern (See

Prediction results analysis

The quality of the data is uneven, which may hinder the prediction of the model.

Conclusions

In this thesis, we have proposed some contributions in medical image segmentation, biomarker estimation through regression CNN and explainability in regresssion CNN, that we summarize below and give our perspectives.

A loss function based on the Kappa index Summary

The class imbalance problem cannot be ignored in image segmentation when using supervised deep learning techniques. One loss function that is already well known for solving the class imbalance is Dice loss [START_REF] Milletari | V-net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF], which is based on Dice index. The Dice loss calculates the overlap area between predicted positive area and ground truth positive area. That is to say, the Dice loss does not take the background pixels into account.

We proposed a loss function which is based on Kappa index, called Kappa loss.

Different from Dice loss, we consider all the pixels including the background information (negative area in prediction and ground truth). The skin lesion segmentation experiments results showed that our proposed Kappa loss can not only surpass the Dice loss by a small margin but also the model has better convergence than the U-Net with Dice loss. At this point, we have added a new member to the family of loss functions, namely the Kappa loss function.

Perspective

In this part of work, we have proposed Kappa loss and proved the generalization of it compared to Dice loss. However, more experiments can be performed in the coming days. Firstly, we can valid more medical image dataset besides the skin lesion images using Kappa loss in segmentation. Secondly, since our proposed Kappa loss can deal with both class imbalance and class balanced problems, therefore, we can divide the dataset into different sub-datasets according to the size of the image targets to verify this loss function. Thirdly, we will compare the other loss functions with proposed Kappa loss on more different imaging datasets.

Biomarker prediction

Fetus head circumference prediction

The fetus head circumference (HC) is one of key biomarkers for monitoring a fetus growing stage. Conventional fetus head circumference prediction is performed by segmentation methods. However, the segmentation-based methods require more than one step: contour segmentation and ellipse fitting, then the head circumference calculation.

In this work, a direct HC prediction approach was proposed. We utilize regression CNN model to directly predict fetus HC from ultrasound images without intermediate segmentation steps. The regression CNN is composed of a CNN backbone and a regression layer. Transfer learning strategy is used in order to improve the prediction accuracy. The loss function is regression loss (MAE loss, MSE loss or Huber loss).

Another contribution of this work is that we compared the proposed segmentation-free (regression CNNs) with segmentation-based methods in a fair experimental environment from several aspects. We used the same dataset (HC18 [START_REF] Van Den | Automated measurement of fetal head circumference using 2d ultrasound images[END_REF]) including data preprocessing and dataset split and GPU server to train and estimate the HC value. We evaluated the explainability of regression CNNs, the prediction error of two approaches, the theoretical memory as well as practical computation efficiency of two models, the learning curves of two models during training, and agreement analysis of two prediction results.

The experiments results of segmentation-free methods are comparable to that of segmentation-based methods although improvement room is left. Nevertheless, the HC prediction error of both segmentation-based and segmentation-free methods are smaller than the manual variability. Moreover, trade-off is existed between prediction error and computation efficiency of segmentation-based and our segmentation-free models.

Cardiac structure volume prediction

There is a more complicated case about direct biomarker prediction is cardiac multi-structure volume prediction from 3D MR imaging. With many lives lost each year due to cardiovascular disease, a quick and effective examination of the patient's heart is critical, but the patient to doctor ratio varies from region to region and hospital to hospital. Therefore, designing automated and effective diagnostic methods

Appendix A The explainability of regression CNNs

A.1 The explainability of regression models

A.1.1 The explainability of regression VGG and regression ResNet

Last section describes the performance of each explanation methods. Now, we can utilize these explanation methods to compare different regression CNN models.

As shown in Figure 4.11, both regression VGG16 and regression ResNet50 are successful in learning the features from ultrasound images to assess the HC. From In this experiment, we arbitrarily pick one of the best performing methods from the previous results, and thus the use Input*Gradient explanation method to generate saliency maps from images with small prediction error ( The models are able to learn the features from these images, therefore the saliency maps show key features. However, it is not always the case: the first row shows a small prediction error, and the head contour are not specifically highlighted. For the badly predicted images, the saliency maps highlight features that are spread and not localized into meaningful segments. The models can not learn the features from these images. However, beyond the score, it seems to be related to the quality of the images: This probably due to irregular and blurry head features, and fan-shaped areas existed in the images which could affect the decision of model. Therefore, performing image preprocessing before training is an effective way to improve the performance of models.

A.1.3 Comparison of saliency maps for different loss functions

In addition to comparing the saliency maps of different regression CNN models as well as the saliency maps on good/bad prediction results. We further compare the performance of different regression loss functions in regression CNNs through different saliency maps. We use 8 different explanation methods to generate saliency . This is due to the square item in the MSE loss, which will change obviously than MAE loss with absolute item.

• Because the Huber loss is a compound loss of MAE loss and MSE loss with a weight value between them. It performs alike with MAE loss and MSE loss.

• As has been discussed before, the regression CNN models and explanation methods have different performance in each saliency maps, which can help to select the better ones. This groups of figures follow the same rule.

Quantitative analysis

Table A.1 is the experiment of adding perturbation on test images (200 images), then the regression CNN models go through each explanation method. For instance, in Gradient explanation method (The second column), the Regression ResNet with Huber loss has the lowest AOPC value (-24.17), which means this model is the most sensitive than the others. That is to say, the Regression ResNet with Huber loss identify the right feature from test images. Therefore, one can known which loss function is more suitable in this way.

A.2 Conclusion

Understanding whether the model can learn the relevant features in images and take the right decision is crucial in the medical domain. Whereas there have been a

wealth of works in classification networks, there is a void for interpreting regression networks.

In this study, we address the problem of estimating the head circumference in fetal head directly from US images. We use several post-hoc explanation techniques that produce saliency maps and adapt a perturbation based quantitative evaluation method, to assess the relevance of the saliency maps. We also investigate the explainability of regression losses including the MAE loss, MSE loss and Huber loss.

The experimental results proved that the regression CNN models are able to learn the key features from the input ultrasound fetus images, and in particular, the head circumference. One finding is that for this application, Gradient and De-ConvNet method are particularly insensitive to different CNN models or data, and that ResNet50 seem to have better learnt the head features. Thus so far, we have extended the model property from classification to regression and explored a specific regression task.

Moreover, we should not only explain the model but also get some feedback according to explanation results, for example, in our case, the content of images can also affect the model's decision, because for those images that the model have bad predictions, the explanation methods cannot show clear features, neither. Finally, the performance of explanation methods used in this work are different from each to others. However, relying only on the saliency maps or on the perturbation methods is far from being enough to get insights from a black box.

B.1 The influence of data modality

The data modality in this study refers to the different slices of cardiac data, the different training scale. In this section, we explore the influence of different data modalities. Besides, we respectively valid the prediction ability of regression CNN on single cardiac structure and multi cardiac structures.

B.1.1 Selection of cardiac slices Motivation

In short axis view of cardiac MR images, each slice represents a part of cardiac. All the slices stacked up is a complete cardiac. However, when using deep learning models to learn relevant features from these slices, it is not necessarily the case that the more complete the information is, the better; instead, some redundant slices can lead to prediction errors caused by noise. For example, in the ACDC dataset, some of the slices of the subject contain little or no information about the cardiac structure. For this reason, feature extraction may benefit from appropriate streamlining of the cardiac slices. In related works, [START_REF] Luo | Multi-views fusion cnn for left ventricular volumes estimation on cardiac mr images[END_REF] explored various combinations of slices from single image at different position to two images, and then three images etc. Their experiments results showed that when the input view is the combination of Top+Middle+Bottom slice from a cardiac, the model has the best performance.

Experiments on different slice combinations

In this experiment, we explored different slice combinations on ACDC dataset inspired by [START_REF] Luo | Multi-views fusion cnn for left ventricular volumes estimation on cardiac mr images[END_REF]. We respectively take the Top slice, Top+Middle+Bottom slices (the 2 th , 3 th , 4 th slice of cardiac), Top three slices (2 th , 3 th , 4 th ), Middle three slices (4 th , 5 th , 6 th ), Bottom three slices (6 th , 7 th , 8 th ), and the entire 9 slices. Experimental results (Table B.1) showed that the model's performance is better when the input training data has three slices of a cardiac, which implies that too few slice (one slice only) or too many slices (9 slices) does not bring effective information to the regression CNN models. As the augmented data gradually increased, the prediction error of the model decreased until it reached a very small value, and then the error began to increase again. This indicates that on the one hand, training a CNN model requires sufficient amount of data, and on the other hand, it also points out that negative effects may occur when there is too much homogeneous augmented data. Same observation results and conclusion is given in [Huang et al., 2021a].

B.1.3 Single cardiac structure prediction vs. Multi-structure

In the above experiments, the three structures of cardiac are estimated simultaneously by regression CNNs. In this experiment, we explore the performance on single cardiac structure prediction. See the table below (Table B.3). One can find that the model's prediction error on single structure is lower than the multi-structure prediction. That makes sense because predicting one target once is easier than predicting multi-targets at the same time for a regression CNN model. Despite that, in the cardiac multi-structure estimation case, it's more practical for applying multi-structure prediction model.

B.2 Determination of hyper parameters

In this section, we explore the hyper parameters in three aspects.

B.2.1 Batchsize and learning rate

When training a dataset, one need to specify how many epochs this model is to be trained on the training set. In general, the epoch is chosen to be between 100 and 200, depending on experience. If it is too small, the model is not trained sufficiently, and if it is too large, it will take too long and the loss will no longer decrease. In practical situations, when the computing power of the device is average, the model cannot train all the data at once in one epoch. Therefore, the dataset can be divided into small batches to be trained one by one according to the computing power, the batchsize is the number of images in one batch.

In deep learning techniques, updating the weights of each neuron of the neural networks is achieved by means of a specified optimization algorithm, such as Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. The weight update also has a rate, i.e., a learning rate. The learning rate is as important as the optimization algorithm. If it is too large, the optimization will diverge; if it is too small, the training will take too long or we will end up with sub-optimal results.

In this experiment, we explored the different learning rates and batchsize (Table B.4). When the learning rate is set large, the prediction error on volume of cardiac structures is also larger than the other two groups. One reason could be the model cannot optimize well if the model learns too rush. Another reason could be the model is pretrained on ImageNet, in that case the model is trained with the learning rate 1e-4, then the pretrained weights in the model become confused with the larger learning rate during training. For batch size, frankly, no clear pattern can be found from this table. It depends on the learning rate actually. Furthermore, we describe the learning curves with respect to different learning rates and batchsize. See Figure B.2. One can find that when the learning rate is small (1e-5), the model learns slowly and doesn't converge yet within 100 epochs. For the batchsize, it seems that a larger batchsize can reduce the gap between training loss and valid loss. The reason for this phenomenon is to be demonstrated by further research.

B.2.2 Dataset splitting

In order to train any machine learning model, no matter what type of dataset is used, one must split the dataset into training data and test data, and a small part of data for validation. When splitting a dataset there are two competing concerns:

-If the training data is less, the model's performance may have greater variance.

Because the model does not recognize new and unseen data very well.

-If the testing data is less, the model's performance statistic will have greater variance.

Thus, the data should be split in such a way that neither is too high, it depends more on the amount of data at hand. Because the number of medical images is limited, in addition to do cross-validation to the data, it is crucial to choose the appropriate splitting ratio.

In this experiment, we test several data splitting ways. The prediction error of the model is shown in Table B.5. When the training data is sufficient, the prediction error is small, that is with data augmentation, the prediction error is decreased with larger ratio on training data, which is because the model is generalized very well through a great deal of training data. While when the training data is insufficient, that is without data augmentation, the prediction error is large and unstable, which is as result of not well generalized model and less statistics samples.

B.2.3 Data type distribution

During the experiment, we found that the RV structure is difficult to predict than the other two structures (MYO and LV). This may due to the training data and test data are not evenly distributed with respect to the each disease type. Since in ACDC dataset, there are 5 types of cardiac, i.e. 4 diseases and 1 normal type. For this, we

Self-Attention mechanism

The attention mechanism in deep learning can be broadly interpreted as a vector of importance weights: to predict or infer an element, such as a pixel in an image or a word in a sentence, we use an attention vector to estimate the degree of its association with other elements and use the weighted sum of their values as an approximation of the target. Attention mechanisms have evolved to the point where there are many categories [Weng, 2018]. In Transformer model [START_REF] Vaswani | Attention is all you need[END_REF], the authors use Self-Attention mechanism, whose mathematical definition is as below:

A common understanding is that the same matrix is given 3 names Q, K, V 1 . Two of the matrices (Matrix Q and Transposition of matrix K) do the dot product, then normalized (Softmax), and then multiplied with the third matrix (V). d k is the dimension of matrix K. The dot product is divided by the scaling factor d k so that the gradient value remains stable during the training process (avoid gradient vanishing). The geometric meaning of dot product is the angle between two vectors, the projection of one vector onto the other vector. A large value of the projection indi-(MLP) with attention blocks. There are no recurrence and convolutions but attention mechanism in this model. The core idea of ViT is Self Attention and Multi-Head Attention mechanism. The principle of them has been explained in the last section.

Different from Transformer, the input of ViT is patches of images. An image is split into certain number of patches. The position information of each patch is added in each patch. A learnable matrix is also added for final classification. In every Transformer Encoder, the occurrence of Layer Normalization is to normalize the optimization space and accelerate convergence. Besides, residual Networks are added in the encoder to prevent from network degradation problems [He et al., 2016a].

MLP unit includes linear transformation with ReLU activation functions.

Regression ViT

In this work, we use ViT to directly predict the volume of cardiac structures, which is a regression task. To achieve this goal, we change the last activation function from

ReLU in MLP Head into linear activation function. We use regression loss function such as MAE, MSE or Huber loss instead of loss functions for classification.

B.3.3 Experiments and analysis

Experiment results

In the computation efficiency aspect, the regression ViT model just took around 1 hour and half to train, which is faster than the regression CNNs (4 hours in Regression VGG16, 10 hours in Regression ResNet50). We tested the prediction error of ViT model under different loss functions separately. It can be found from Table B.8 that the prediction error of the three structures are average, and the all the prediction errors are slightly lower compared to regression CNNs. The author believes it has great potential to obtain better performance in this application. Because the original ViT paper [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF]] also points out that the performance of ViT is higher when the dataset is of great deal. In addition of data amount, there are many hyper-parameters in the ViT model as described in the previous section which are factors to optimize the model. Thus it is necessary to keep experimenting which combination of hyper-parameters in ViT will achieve the best performance in the future works.

B.4 Conclusion

In this appendix, we added extra experiments on ACDC dataset in different aspects.

For the data, we validated the input view, i.e. different number of slices, we found that the regression CNN model can not well learn the feature comprehensively from the entire slices of a cardiac, which implies that too much information may mean disruptions for a model. We also explored the influence of data augmentation at [ Lin et al., 2017b] Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017b).

Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.