Keywords: segmentation

I am especially grateful to Prof. Su Ruan and Prof. Yuemin Zhu for reviewing my thesis. I appreciate their interest in my work as well as all of their insightful comments and suggestions.

I would like to thank M. Vincent Noblet, Prof. Valérie Burdin and Prof. Ronan Fablet for taking part in my PhD defense.

In particular, I would especially like to express my special thanks of gratitude to my supervisor Prof. François Rousseau who gave me the golden opportunity to do this wonderful thesis, which leverage me to learn a great amount of knowledge and experience in the fields of computer vision, image processing as well as machine learning and deep learning. His patience, motivation, immense knowledge guided me in all the time of research. "Merci beaucoup François !" I am really thankful to Prof. Valérie Burdin (IMT Atlantique) et Prof. Mireille Gareau (Université de Rennes 1) who have gave a chance to study in France. I am thankful to Prof.

Jean-Louis Dillenseger (Université de Rennes 1) for guiding me during my master internship. I also would like to thank all my professors and my colleges from IMT Atlantique, LaTIM and "Domino Reading Group" as well as Ho Chi Minh University of Technology.

A very special thank you goes to my friends in "4 bis rue de Kérangoff", "Bờ zet hội"and "Hội cầu lông Brest"for the moment we share together.

Last but not the least, thanks to "Pa", "Mẹ", "Ông Bà", "Vợ, "Bi Bô"and my big family for always being with me through every step of my life.

iv

Résumé étendu

La modélisation des images est un point clef important pour de nombreuses tâches en traitement d'images, comme la super-résolution, la segmentation ou la synthèse de texture. L'analyse et le traitement des différentes caractéristiques composant une image, nécessite la mise en place d'approches adaptatives locales. Dans ce contexte, la définition de représentations locales efficaces est dédiée pour une application visée. Les approches par l'apprentissage profond ont permis des avancées significatives en terme de performance de traitement. D'un côté, cette approche se fond sur la notion de relation entre les patches, et plus particulièrement en analysant automatiquement les caractéristiques des patches pour les agréger par la suite. D'un autre côté, les méthodes d'apprentissage profond reposent sur une hypothèse de fonctionnement des neurones biologiques d'une représentation de l'image par un ensemble de filtrage. Ces représentations ont été introduites dans des modèles a priori dans le cadre de résolution de problèmes inverses.

L'objectif de cette thèse est d'étudier le comportement de différentes représentations d'images, notamment par apprentissage profond, dans le contexte d'application en imagerie médicale. Le but est de développer une méthode unifiée efficace pour les applications visées que sont la super-résolution, la segmentation et la synthèse. La super-résolution est un processus v d'estimation d'une image haute-résolution à partir d'une ou plusieurs images basses-résolutions. Dans cette thèse, nous nous concentrons sur la super-résolution unique, c'est-à-dire que l'image haute-résolution (HR) est estimée par une image basse-résolution (LR) correspondante. Augmenter la résolution de l'image grâce à la super-résolution est la clé d'une compréhension plus précise de l'anatomie. L'application de la super-résolution permet d'obtenir des cartes de segmentation plus précises. Étant donné que deux bases de données qui contiennent les images différentes (par exemple, les images d'IRM et les images de CT), la synthèse est un processus d'estimation d'une image qui est présentée dans la base de données de cible à partir d'une image de la base de données de source. Parfois, certains contrastes tissulaires ne peuvent pas être acquis pendant la séance d'imagerie en raison du temps et des coûts élevés ou de l'absence d'appareils. Une solution possible est à utiliser des méthodes de synthèse d'images médicales pour générer les images avec le contraste différent qui est manquée dans le domaine à cible à partir de l'image du domaine donnée. L'objectif des images synthétiques est d'améliorer d'autres étapes du traitement automatique des images médicales telles que la segmentation, la super-résolution ou l'enregistrement. Dans cette thèse, nous proposons les réseaux neurones pour la super-résolution et la synthèse d'image médicale. Les résultats démontrent le potentiel de la méthode que nous proposons en ce qui concerne les applications médicales pratiques.

Un réseau de neurones convolutifs (en anglais CNN -Convolutional Neural Networks) est un type de réseau de neurones artificiels. Une architecture de réseau de neurones convolutifs est structurée par un ensemble de couches de traitement, particulièrement les couches convolutives et les fonctions d'activation. En outre, selon une application visée, nous pouvons vi rajouter les autre éléments comme la couche de pooling, la couche entièrement connectée (fully connected) ou la couche convolutifs transposée. Le réseau de neurones convolutives, qui fut présenté il y a longtemps [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]] a vraiment reçu l'attention de la communauté de recherche à partir de 2012 par une méthode ayant gagné un challenge de classification dans une conférence de vision par ordinateur. Ce réseau appelé Alexnet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] contient huit couches: cinq couches convolutives et trois couches entièrement connectées. Ensuite, les architectures de CNN sont devenues l'état de l'art pour de nombreuses tâches en traitement d'images comme la super-résolution [Dong et al., 2016a, Kim et al., 2016a], la segmentation [START_REF] Kamnitsas | Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation[END_REF][START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] ou la classification [He et al., 2016a, Simonyan and[START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. Par la suite, plusieurs réseaux CNNs ont été améliorés afin d'augmenter leur performance pour la classification, par exemple, en augmentant le nombre de couches (e.g. VGGnet avec 19 couches [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF],

Resnet avec 152 couches [He et al., 2016a]), en concaténant les filtres en un bloc (e.g. GoogLenet [START_REF] Szegedy | Going deeper with convolutions[END_REF]), ou par l'apprentissage résiduel en bloc (e.g. Resnet [START_REF] He | Identity mappings in deep residual networks[END_REF]), ou en connectant tous les couches à forte densité (e.g. Densenet [Huang et al., 2017a]). Afin de détecter l'objet dans les images avec CNNs, nous pouvons attacher la boîte de délimitation parallèle à sa classification d'objet (R-CNN) [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] et sa segmentation (Mask R-CNN) [START_REF] He | Mask r-cnn[END_REF].

Plusieurs méthodes de CNNs ont été proposées pour la segmentation de l'imagerie médicale. On peut notamment citer U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] et DeepMedic [START_REF] Kamnitsas | Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation[END_REF]. U-net a une forme de la lettre U avec les skip-connections entre les couches. DeepMedic combine deux réseaux de CNNs afin d'augmenter la performance de segmentation cérébrale: un chemin pour l'image original et une autre pour sa version de vii basse-résolution.

Les architectures CNN sont devenues l'état de l'art en super-résolution (SR). Initialement, [Dong et al., 2016a] a proposé une architecture CNN à trois couches. La première couche convolutionnelle extrait implicitement un ensemble des caractéristiques pour l'image LR d'entrée, la deuxième couche représente non-linéairement des caractéristiques de l'image basseresolution aux patches haute-résolution et la troisième couche reconstruit l'image HR à partir de ces représentations de patchs. Et puis, les caractéristiques suivantes ont été rapportées pour améliorer la performance SR tel que un réseau plus profond [Kim et al., 2016a], bloc résiduel [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF], couche de sous-pixel [START_REF] Shi | Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[END_REF], fonction de coût perceptuelle (au lieu de fonctions de coût quadratiques moyennes) [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], Ledig et al., 2017[START_REF] Zhao | Loss functions for neural networks for image processing[END_REF], réseaux récurrents [Kim et al., 2016b], le réseau contradictoire générateur [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF], Très récemment, [Chen et al., 2018b] ont proposé une version 3D de densenet pour la SR des image IRM. Inspiré du travail de [START_REF] Jog | Self super-resolution for magnetic resonance images[END_REF], [START_REF] Zhao | Self super-resolution for magnetic resonance images using deep networks[END_REF] a étudié la super-résolution automatique pour l'IRM en utilisant des réseaux résiduels profonds [START_REF] Lim | Enhanced deep residual networks for single image super-resolution[END_REF]. Récemment, un réseau plus profond avec 20 couches [Kim et al., 2016a] inspiré par VG-Gnet [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] est devenu une basé pour les méthodes suivantes [START_REF] Timofte | Ntire 2017 challenge on single image super-resolution: Methods and results[END_REF]. Cependant, en raison de la variété des méthodes proposées et du nombre de paramètres pour l'architecture des réseaux, il est actuellement difficile d'identifier les componants clés de l'architecture CNN pour obtenir des bonnes performances pour la SR et évaluer leur applicabilité dans le contexte de l'image IRM cérébrale 3D. De plus, l'extension des architectures CNN aux images 3D, en tenant compte des facteurs de mise à l'échelle anisotropes peut être intéressante pour viii s'adresser aux nombreux paramètres d'acquisition clinique possibles, tandis que les architectures CNN classiques n'adressent qu'un facteur d'échelle prédéfini. La disponibilité de l'imagerie multimodale pose également la question sur la capacité des architectures de CNN à exploiter de telles données multimodales pour améliorer la SR d'une modalité donnée.

Ce manuscrit est rédigé en anglais et structuré en cinq chapitres et une annexe.

Le chapitre 1 correspond à une introduction générale où sont décrits le contexte, la motivation, l'objectif et la méthodologie de cette thèse.

Le chapitre 2 décrit notre méthode de super-résolution en imagerie cérébrale en utilisant les réseaux CNNs (convolutional neural networks). D'abord, nous allons passer une bibliographie qui contient différents types de méthodes de super-résolution tel que la méthode basée sur les modèles et celle basée sur l'apprentissage comprenant les réseaux de neurones convolutifs.

Et puis, la méthode de super-résolution est consacrée à l'imagerie médicale.

Ensuite, nous proposon l'application de la méthode de super-résolution basée sur CNNs aux images cérébrales d'IRM. Il s'agit de l'application des CNNs 3D afin d'obtenir la super-résolution à partir d'une seule image.

Huit paramètres principales du réseau 3D sont étudiés en détail avec des expérimentations pour améliorer sa performance: méthodes d'optimisation, initialisation des poids, apprentissage résiduel, profondeur du réseau, taille du filtre, nombre de filtres, taille de patch d'apprentissage, nombre de sujets pour l'apprentissage. Pour exploiter la capacité du réseau, deux autres applications sont proposées. Le premier est à mélanger plusieurs facteurs échelles (par exemple, mis à échelle deux fois et trois fois par rapport d'une basse résolution) dans le même ensemble de données d'apprentissage. Le ix réseau appris avec plusieur facteurs peut être appliqué pour des échelles arbitraires tandis que celui appris une facteur est seulement utilisé pour une résolution désirée. Le deuxième application vise à concaténer les images haute-résolution référence pondérée et l'image bass-résolution interpolée à l'entrée du réseau CNN. Par rapport à ces deux applications, la diversité des bases de données d'apprentissage est également abordée. Enfin, nous appliquons notre méthode de super-résolution aux images cérébrales d'IRM des nouveau-nés et puist segmenter les images de haute résolution obtenues afin d'évaluer la contribution de la méthode proposée. Nous montons les résultats visuelles. Les illustrations contribute que la superrésolution peut aider la segmentation d'image.

Le chapitre 3 décrit une approche pour une réalisation simultanée de la superrésolution et de la segmentation à partir d'une seule image. Elle est basée sur le réseau de neurones génératives contradictoires dit generative adversarial network (GAN). La superrésolution et la segmentation sont souvent effectuées de manière séparée comme la section dernière du chapitre 2. Dans cet chapitre, nous proposons réaliser ces deux opérations en même temps. L'application est focalisée seulement sur des images IRM néonatales du cerveau en T2 qui ont les résolutions basses, car les nouveaunés ne peuvent pas patienter allonger sur une machine d'acquisition dans plusieurs cases clinques. Les résultats de la super-résolution sont comparés avec la méthode proposée dans le chapitre précédent. Les images hauteresolution estimées semblent légèrement inférieurs en termes des métriques de qualité mais meilleurs visuellement. Concernant les résultats de la segmentation qui sont évalués par DICE, notre méthode montre les meilleurs résultats comparés avec deux méthodes de segmentation de littérature.

Le chapitre 4 introduit la synthèse d'images médicale. Une synthèse des

x méthodes existantes basées principalement sur l'apprentissage et sur le réseau CNN est fait dans la première section. Ensuite, nous proposons deux approches basées respectivement sur le réseau CNNs et sur le réseau GAN. La première est directement appliquée du principe qui a été utilisé pour la super-résolution décrit dans le deuxième chapitre pour synthétiser des images couplées, c'est-à-dire, dans la base d'apprentissage, les deux séquences d'images sont toutes appairées (paired cross-modal synthesis).

Les résultats rassemblent aux images vérité-terrain mais avec un peu de floue et de bruit. En plus, nous considérons le deuxième cas plus difficile: les deux séquences d'images sont toutes non appairées (unpaired crossmodal synthesis). Cette méthode de synthèse d'images est basée sur le réseau GAN. Afin de resoudre le problème de synthèse d'image non appariée, nous proposons utiliser trois fonctions de coût: adversarial loss, cycle consistency loss et total variation. Cependant, il reste la difficulté pour choisir un coefficient optimal de pondération pour total variation qui controle le compromis entre la réduction des artéfacts de

Context and motivation

MRI is a medical imaging technique used to visualize the anatomy and the physiological processes of the body. MRI scanners are based on the interaction of a nuclear spin with an external magnetic field. The rotation of a particle around some axis as an intrinsic form of angular momentum is called spin. An MRI scanner forms a strong magnetic field around the area of a subject to be imaged. The protons of hydrogen atoms from biological organisms are excited by a radio frequency (RF) pulse and then emit energy in the form of RF signal when returning to the original state. By applying different types of the sequence of RF pulses, different types of modality are created. Two important terms of the acquisition process are repetition time (TR) and time to echo (TE). TR denotes the period between successive pulse sequences at the same slice, TE denotes the period between the emission of the RF pulse and the reception of the echo signal. Common anatomical MRI sequences are T1-weighted (T1w) images, T2-weighted (T2w) images and fluid-attenuated inversion recovery (FLAIR).

T1w, T2w MRIs and FLAIR are generated by using respectively short TE and TR, long TR and TE, and very long TR and TE times. A modality shows up different physical properties of tissue, that induces different contrasts between them. Thus, each modality has a specific range of applications in medical diagnosis.

1.1.1 Medical single image super-resolution: an approach to generate high-resolution images

Acquisition time of MRI data and signal-to-noise ratio are two parameters that drive the choice of an appropriate image resolution for a given study. The accuracy of further analysis such as brain morphometry can be highly dependent on image resolution. A typical image resolution of a current MRI is desired for greater than or equal to 1mm. However, imaging with desired resolutions costs of low signal to noise ratio and long scan time. For example, MR images with an isotropic resolution of 1mm and 0.7mm are respectively shown in Figure Super-Resolution (SR) aims to enhance the image resolution using single or multiple data acquisitions [START_REF] Milanfar | Super-resolution imaging[END_REF]. Increasing image resolution through super-resolution is a key to more accurate understanding of the anatomy [START_REF] Greenspan | Super-resolution in medical imaging[END_REF]. The applications of superresolution have been shown that applying super-resolution techniques leads to more accurate segmentation maps of brain MRI data [START_REF] Rueda | Single-image super-resolution of brain mr images using overcomplete dictionaries[END_REF] or cardiac data [START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF].

Recently, a series of papers suggested the successful application of deep learning, leading to state-of-the-art results in many practically tasks of computer vision [START_REF] Dong | Learning a deep convolutional network for image super-resolution[END_REF], Kim et al., 2016a[START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Shi | Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[END_REF] and medical image processing [START_REF] Charron | Automatic detection and segmentation of brain metastases on multimodal mr images with a deep convolutional neural network[END_REF], Chen et al., 2018b[START_REF] Kamnitsas | Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation[END_REF], Meyer et al., 2018[START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF], Pham et al., 2017a[START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. In this thesis, the architectures of convolutional neural networks are investigated for MRI super-resolution. The performance of a given architecture depends on several parameters such as the filter size, the number of filters, the number of layers, etc. Understanding how these parameters affect the reconstruction of the HR image with respect to the considered application setting (e.g., number of training samples, image size, scaling factor) is a key issue, which remains poorly explored. For instance, regarding the number of layers, it is commonly believed that the deeper the better [Kim et al., 2016a, Simonyan and[START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. However, adding layers increases the number of parameters and can lead to overfitting. Previous works [Dong et al., 2016a[START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF], have shown that "a deeper structure does not always lead to better results" [Dong et al., 2016a].

Specifically focusing on MRI data, the specific objectives of this study are:

• Are 2D or 3D networks relevant to brain MRI SR ?

• the evaluation and understanding of the effect of key elements of CNN for brain MRI

SR

• How can networks handle different scaling factors ?

• Investigating multimodality-guided SR using CNN ?

• Can a pre-trained model apply to different data ?

• How do networks apply to a real data ?

• Does the application of super-resolution improve automatic segmentation algorithms ?

• Two steps for the 3D isotropic segmentation of anisotropic MRI images are: increasing the image resolution using interpolation techniques or SR and then isotropic image segmentation. Do we have a method for simultaneous super-resolution and segmentation ?

1.1.2 Medical image cross-modal synthesis: an approach to generate synthesized images

The pulse sequences in the acquisition process influence strongly the performance of MRI analysis algorithms. Medical image analysis techniques, which optimally learned with data from one specific modality, could not apply to data of a different modality because each modality expresses particular tissue contrast of the body anatomy. For example, neonatal brain T2w MRIs are appropriate to reconstruct brain surface while the T1w scans lack sufficient tissue contrast [START_REF] Leroy | Atlas-free surface reconstruction of the cortical grey-white interface in infants[END_REF]. Sometimes, certain tissue contrasts may not be acquired during the imaging session because of time-consuming, expensive cost or lacking of devices. One possible solution is to use medical image cross-modal synthesis methods to generate the missing subject-specific scans in the desired target domain from the given source image domain. The objective of synthetic images is to improve other automatic medical image processing steps such as segmentation, super-resolution or registration.

In this thesis, convolutional neural networks are applied to cross-modal synthesis in the context of supervised learning. In addition, an attempt to apply generative adversarial networks for unpaired cross-modal synthesis brain MRI is described. The specific objectives of this study are:

• Can CNN-based methods be applied to solve cross-modal synthesis problem ?

• Is there a method which can generate the synthetic image of a specific subject given unpaired training dataset ?

Thesis overview

In this thesis, our motivation is dedicated to studying the behaviours of different image representations and developing a method for super-resolution, cross-modal synthesis and segmentation of medical imaging.

Chapter 2 introduces single image super-resolution. Firstly, single image super-resolution is first modelled by the image acquisition process. Several methods for super-resolution of natural images are discussed from model-based to learning-based approaches. Since the observation model is assumed unknown or hard-defined, "blind super-resolution" is then In addition, two datasets are used to verify the transferable ability of the pre-trained networks.

Furthermore, our method is applied to low-resolution in-vivo neonatal brain MR images so as demonstrates the qualitative performance.

Chapter 3 introduces an approach to simultaneous super-resolution and segmentation using a generative adversarial network. Generative adversarial networks have been investigated to estimate realistic super-resolved images and efficient semantic segmentation. However, superresolution and segmentation are usually processed separately. Firsly, an end-to-end generative adversarial network for simultaneous high-resolution reconstruction and segmentation of brain MRI data is proposed. This joint approach is first assessed on the simulated low-resolution images of the high-resolution neonatal dataset. Then, the learned model is used to enhance and segment real clinical low-resolution images.

Chapter 4 introduces cross-modal medical image synthesis. Two main approaches of medical image synthesis are summed up relied on the property of training dataset: paired and unpaired images. Next, two approaches for brain MRI synthesis are proposed. The first approach applies the most performing convolutional neural networks in Chapter 2 for paired dataset.

The second approach lies on the application of generative adversarial networks for unpaired image synthesis.

Chapter 5 concludes the thesis and draws future works.

Appendix A brings up a brief introduction of deep learning. This part consists of the definition of a neural network, the different architectures of neural networks such as convolutional neural networks, activation functions, residual networks and densely connected networks. Next, an application of convolutional neural networks to style transfer and generative adversarial networks are described in detail. The last section introduces the optimization methods of neural networks.

Chapter 2

Brain MRI super-resolution using 3D convolutional neural networks [START_REF] Borman | Super-resolution from image sequences-a review[END_REF][START_REF] Park | Super-resolution image reconstruction: a technical overview[END_REF] since the first work by [START_REF] Tsai | Multiple frame image restoration and registration[END_REF].

A digital image is composed of elements called pixels. Image spatial resolution, which refers to line pairs per unit distance or pixels (dots) per unit distance, describes the details contained in an image [START_REF] Gonzalez | Digital Image Processing[END_REF]. For example, a two-dimensional (2D) image with the resolution of 0.1 × 0.1 mm 2 has 5 line pairs per unit distance (mm) for each direction.

High-resolution image can improve the quality of image for human interpretation and machine perception due to the representation of more details. However, imaging acquisition device, which consists of imaging sensors, or imaging acquisition procedure (e.g. the purpose of users) can limit the image resolution. Theoretically, the higher density of the sensors in a digital imaging device may induce higher resolution image. In fact, it is not easy to increase the number of the sensors on a fixed area of the device because of the increase in cost of products and the limitations of current integrated circuit. The post-processing approaches as superresolution (SR) can overcome physical constraints and also improve the image resolution.

Super-resolution is the process of estimating high-resolution (HR) images from one or several low-resolution (LR) images. The unknown HR image can be reconstructed by multi-image super-resolution methods using several interrelated LR images involved with a determined equation set (e.g. linear constraints) [START_REF] Milanfar | Super-resolution imaging[END_REF]. In this work, we focus on single image super-resolution (SR) that estimates the HR image from one corresponding LR image. A closely related method with SR to address this problem is to use the single-image interpolation [START_REF] Hou | Cubic splines for image interpolation and digital filtering[END_REF]Andrews, 1978, Thévenaz et al., 2000] as a weighted average of the LR pixels y j :

   x i = y j i = j x i = 1 M M i=1 w ij y j otherwise (2.1)
where x i is HR pixels, the weights w are calculated as a function that changes over the distance between the new pixel and M LR ones. The result of the single-image interpolation approach is too smooth because there is no additional information that compensates for the lost of high-frequency components [START_REF] Milanfar | Super-resolution imaging[END_REF]. An example of SR results is illustrated in Figure 2.1. The most common up-sampling method, which is image interpolation in Figure 2.1 (b), shows a blurred reconstruction, while the SR start-of-the-art methods such as A+ [START_REF] Timofte | A+: Adjusted anchored neighborhood regression for fast super-resolution[END_REF] and SRCNN [Dong et al., 2016a] preserve edges and provide higher visual quality. A+ [START_REF] Timofte | A+: Adjusted anchored neighborhood regression for fast super-resolution[END_REF], (d) SRCNN [Dong et al., 2016a] using the available code from authors.

Image observation model

The image acquisition device can be affected by various factors such as: digital sampling, the relative motion of scene and the camera, optical blur, decimation and noise. Mathematically, let X and Y denote the desired HR and the observed LR image, the acquisition process can be modelled as follows: there may be many solutions (i.e. not unique) for one observed input, expressing that the dimension of the observed data always is less than those of the latent HR image. In fact, the observation matrix may be unknown due to the complexity of real imaging systems. Even if the matrix is known, SR is still ill-posed. Thus, many solutions from two main categories:

Y = HX + N = D ↓ BF X + N (2.2) where Y ∈ R n , X ∈ R m , H ∈ R
model-based and learning-based methods that can be proposed for this problem. In the next sections, we will introduce some basic techniques proposed in the literature.

Model-based methods

Given an observation model as Equation (2.2), the SR image can be estimated by minimizing a least-square cost function as:

X = argmin X φ(X, Y) = argmin X Y -HX 2 . (2.3)
where φ(X, Y) denotes the fidelity term. The linear least squares method gives the solution of this equation as:

X = (H T H) -1 H T Y (2.4)
However, there are many possible solutions since H is ill-conditioned. Based on the observation model, the iterative back-projection (IBP) method [START_REF] Irani | Improving resolution by image registration[END_REF] proposes to calculate the residual between a simulated LR image with the LR observation Y and then sum the reconstruction error back to the estimated HR image X as:

   X0 = S ↑ Y Xt+1 = Xt + S ↑ (H Xt -Y) (2.5)
where t is the current iteration, S ↑ is a upscaling operation (e.g. nearest-neighbor interpolation). The contrast along edges is better recovered than interpolation method. However, the IBP technique which depends the initialized results, is highly sensitive to noise and outliers.

In these cases, the result may contain high frequency artifacts because of ignoring the visual complexity of the ill-posed problem [START_REF] Milanfar | Super-resolution imaging[END_REF], Rousseau et al., 2010c]. Thus, this limitations raise the importance of regularizations. A regularizer can be added into the cost function to stabilize the problem as:

X = argmin X Y -HX 2 + λR(X) (2.6)
where λ is a global weight and R(X) is a regularization term. The most commonly used method for regularization of ill-posed problems is Tikhonov regularization as:

R(X) = p Ω C p | X (p) | 2 (2.7)
where C p is a positive parameter, Ω is the searching zone and X (p) denotes the p th order derivative of X. Another regularization is total variation as:

R(X) = p Ω C p | X (p) | (2.8)
These approaches assume smooth regions of natural images separated by sharp edges. [START_REF] Sun | Image super-resolution using gradient profile prior[END_REF] propose gradient profile prior which is fitted by a general exponential generalized Gaussian distribution as:

R(X) = λα(λ) 2σΓ( 1 λ ) exp -α(λ) X σ λ (2.9)
where Γ denotes Gamma function and α(λ) = Γ( 3 λ ) Γ( 1 λ ) denotes the scaling factor which makes the second moment of the distribution equal to σ 2 . λ is the trade-off parameter. [Kim andKwon, 2010, Tappen et al., 2003] propose natural image prior as Markov random field model:

P r(x | y) = s φ(x s ) r φ(x r , y) (2.10)
where φ() can be a function (e.g. ℓ 2 -norm). x and y denote the HR and LR patches.

Each regularizer assumes a specific image model as data distribution. The minimization of the equation (2.6) with different regularizations on X usually leads to different solutions.

The choice of image prior is crucial for solving the SR problem. In addition, adding prior knowledge on the image solution (such as piecewise smooth image) may lead to unrealistic solution. The work in [START_REF] Efrat | Accurate blur models vs. image priors in single image super-resolution[END_REF] investigates that an accurate estimate of the PSF is more influenced than a sophisticated prior. Thus, the parameterized prior of the model-based methods is inadequate for the general solution of the SR problem, that requires an approach can learn locally the prior by samples.

Learning-based methods

Learning methods for SR

Another approach is to find out the relationship of HR images and corresponding LR images by assuming available external data. Given a set of extracted patch pairs {(x i , y i )} N i=1 , a method is to learn a direct mapping [START_REF] Freeman | Example-based super-resolution[END_REF] from the LR patches y i to the HR patches x i , connected by the observation model as Equation (2.2):

y i = Hx i + N (2.11)
The relationship between these training pairs is denoted as a mapping φ(x i , y i ). The HR patches x of a testing LR patch y are reconstructed based on Markov random field by counting the neighbour searching zone and the trained mapping as:

P (x | y) = 1 Z m,n∈Ω I θ mn (x m , xn ) m∈Ω I φ(x m , y m ) (2.12)
where Z is a normalization constant, Ω I denotes the image space and the node matrix θ mn (x m , xn ) is calculated as:

θ mn (x m , xn ) = exp - j (x m,j -xn,j ) 2 2σ (2.13)
where j denotes the pixel of patches and σ is a noise parameter. This method is impacted by the patch size. Small patches infer the mapping very fragile but larger patches need large training images. The assumption of two corresponding manifolds of paired patches called neighbour embedding for SR [START_REF] Chang | Super-resolution through neighbor embedding[END_REF], which can be used to decrease the amount of training pairs thanks to nearest neighbours search. The method estimates an HR patch x from k-nearest neighbours Ω k in the training set of LR testing patches y:

αi = argmin α i y - y i ∈Ω k α i y i s.t. y i ∈Ω k α i = 1 x = y i ∈Ω k αi x i (2.14)
One disadvantage of this method is difficult to choose an effective number of k, for example, a large k can lead to overfitting. An effective method is based on the assumption of an over-complete dictionary and searching for sparse representation which can combine linearly the atoms of the dictionary (called the sparse coding method). For the SR problem, the sparse-coding-based method [START_REF] Yang | Image super-resolution as sparse representation of raw image patches[END_REF] proposes to train dictionaries between HR patches and LR patches. The objective is to find the coefficients α as:

α = argmin α α 1 s.t. F D l α -F y 2 2 ≤ ǫ (2.15)
where the LR dictionary D l and the HR dictionary D h consist of training LR and correspond HR patches respectively. F denotes feature extractions as follows:

               F 1 = [-1, 0, 1] F 2 = F T 1 F 3 = [1, 0, -2, 0, 1] F 4 = F T 3 (2.16)
where T denotes transpose. After finding the optimal coefficients, the HR patches are estimated as x = D h α. An improved version of this work [START_REF] Yang | Image super-resolution via sparse representation[END_REF] proposes to train joint dictionaries D l and D h instead of one single constraint on the LR dictionary to enforce the similarity of the representation of image pairs as:

α = argmin α α 1 s.t. F D l α -F y 2 2 ≤ ǫ 1 , P D h α -m 2 2 ≤ ǫ 2 (2.17)
where P denotes the overlapped patch extraction and m denotes overlapped reconstructed HR values. The sparse representation method for SR is extended in many works by different training approaches and dimensionality reduction [START_REF] Zeyde | On single image scale-up using sparse-representations[END_REF], anchored neighborhood embedding [START_REF] Timofte | Anchored neighborhood regression for fast examplebased super-resolution[END_REF][START_REF] Timofte | A+: Adjusted anchored neighborhood regression for fast super-resolution[END_REF] or network-based approximation [START_REF] Wang | Deep networks for image superresolution with sparse prior[END_REF]. A sparse-coding-based network for SR is proposed in [START_REF] Wang | Deep networks for image superresolution with sparse prior[END_REF] (SCN) by using the learned iterative shrinkage and thresholding algorithm (LISTA) [START_REF] Gregor | Learning fast approximations of sparse coding[END_REF]. The method SCN approximates the coefficients by using a multi-layer network Φ as: α = Φ(y, W ), where W denotes network parameters. The HR dictionary D h and network parameters is optimized by minimizing the loss function as:

L(W, D h ) = i D h Φ(y i , W ) -x i 2 (2.18)
where the training pairs (y i , x i ). While the sparse-coding method in [START_REF] Yang | Image super-resolution via sparse representation[END_REF] proposes to use first-and second-order derivatives per one image dimension as the feature (i.e.

operators

). The method in [START_REF] Gu | Convolutional sparse coding for image super-resolution[END_REF] decomposes the whole image into several features by learned convolutional filters (more than 4 as in [START_REF] Yang | Image super-resolution via sparse representation[END_REF]) and then uses the sparse representation to match the LR-HR patches of each feature. In order to accelerate the speed of sparse representations (searching the coefficients), anchored neighborhood regression (ANR) [START_REF] Timofte | Anchored neighborhood regression for fast examplebased super-resolution[END_REF] proposes to use nearest neighbours of dictionaries. The ANR method replaces ℓ 1 -norm by ℓ 2 -norm in Equation 2.15 in order to take advantage of a least squares regression as:

min α F N l α -F y 2 2 + λ α 2 (2.19)
where N l , which corresponds to local neighbourhood of LR dictionary D l , can be computed as in the case neighbour embedding [START_REF] Chang | Super-resolution through neighbor embedding[END_REF]. The solution of α is now given by:

α = (N T l N l + λI) -1 N T l F y (2.20)
where I denotes identity matrix and λ is a constant. The testing HR patches is then calculated through the neighborhood N h of the HR dictionary as:

x = N h α = N h (N T l N l + λI) -1 N T l F y = P j F y = P j y F (2.21)
Here, P j is called stored projection matrix and y F denotes the input feature. The work of [START_REF] Timofte | A+: Adjusted anchored neighborhood regression for fast super-resolution[END_REF] (A+), which develops ANR, proposes to finding K training samples which have the same cluster with the input patch y instead of the nearest neighbors of LR space, leading to more accurate results and faster estimation. In order to cluster LR patches before the dictionary training, A+ adopts the method of regressions in [START_REF] Yang | Fast direct super-resolution by simple functions[END_REF] as:

x = Ĉk y 1 , Ĉk = argmin C k y k -C k x k 1 2 2 (2.22)
where the transformation matrix C k is found by training patch pairs (y k , x k ) for k th cluster and 1 denotes the vector filled with ones. However, each group of patches learns a single regressor where the estimation hardly yields satisfactory results. [START_REF] Dai | Jointly optimized regressors for image superresolution[END_REF] proposes to optimize jointly regressors as:

Ĉk , Pj = argmax C k ,P j K k=1 M j=1 C k,j x k -P j F y k 2 (2.23)
Then, the expectation-maximization (EM) algorithm is used to optimize the cost function.

Several approaches have been investigated to improve the sparse-coding-based method by analysing the sensitivity of principal components such as dictionary size, augmentation of data or combining other techniques [START_REF] Timofte | Seven ways to improve example-based single image super resolution[END_REF].

Other family of learning-based approaches relies on clustering the patches by feature extracting and then matching HR-LR patches by random forest [Huang et al., 2015b[START_REF] Salvador | Naive bayes super-resolution forest[END_REF][START_REF] Schulter | Fast and accurate image upscaling with superresolution forests[END_REF]. Instead of implicitly figuring out patch regressions based on the dictionaries as Equation (2.17), the method in [START_REF] Schulter | Fast and accurate image upscaling with superresolution forests[END_REF] (RFL)

proposes to use a random forest regressor:

x = P j (y F )y F = T (y F ) (2.24)
where P j (y F ) denotes locally linear regressions and T is tree ensembles. By averaging the linear model T r of each tree j, the estimated HR image is modelled as:

x = 1 J J j=1 T r (j) (y F ) (2.25)
where r represents the leaf in tree T r (j) aligned to the feature input y F . RFL uses 4 filters for the feature extractions as ANR and A+ (shown in Equation (2.16)). The work in [Huang et al., 2015b] finds that there are four main edge-based patterns in which patches are grouped.

Then, four random forest can be used to train the linear regressions of each pattern class. For fast inference and adaptively feature extraction, Local Naive Bayes framework is propose for random-forest-based SR in [START_REF] Salvador | Naive bayes super-resolution forest[END_REF]. The optimal regressor from tree T r (j) for a patch x is estimated by Naive Bayes derivation as:

T r

(j ⋆ ) i = argmax T r (j) i p(T r (j) i | x) = argmax T r (j) i log p(x | T r (j) i ) (2.26)
Assuming that we have M features on which the clusters are grouped as

y F i (1 ≤ i ≤ M ),
feature independence results in the log likelihoods as:

T r

(j ⋆ ) i = argmax T r (j) i log 2 (M ) i log p(y F i | T r (j ) i ) (2.27)
However, these approaches depend crucially on the feature extractions based on pre-defined filters. In addition, because of patch regressions, these methods need optimally global optimization when applying on a testing image, that takes computation costs for each patch reconstruction. In the next section, the methods, which use convolutional neural networks, attempt to learn implicitly necessary features in the networks.

Another approach for SR problem defines matrix H -1 as a combination of a restoration matrix R ∈ R m×m and a upscaling interpolation operator S ↑ : R n → R m with respect to the interpolated LR (ILR) image Z ∈ R m (Z = S ↑ Y). Given a set of HR images X i and their corresponding LR images Y i , the restoration operator R can be estimated by minimizing the following loss function:

R = argmin R k i X i -R(S ↑ Y i ) 2 = argmin R k i X i -R(Z i ) 2 (2.28)
Once R is estimated, given a LR image Y, the computation of an HR image X is straightforward: X = R(S ↑ Y). In order to model the restoration operation R, the first deep learning method (SRCNN) proposes to use 3 convolutional layers [START_REF] Dong | Learning a deep convolutional network for image super-resolution[END_REF] for LR feature representation, LR-HR feature matching and image reconstruction. This method does not require any feature descriptions and outperforms the previous hand-crafted methods. The first convolutional layer called R 1 implicitly extracts a set of feature maps for the input LR image as:

R 1 (Z) = max(0, W 1 * Z + B 1 ) (2.29)
where W 1 and B 1 represent the filters and biases respectively, and " * " denotes the convolution operation. A rectified linear unit (ReLU) is applied on the filter responses. The second layer maps these feature maps nonlinearly to HR patch representations:

R 2 (Z) = max(0, W L-1 * R 1 (Z) + B 2 ) (2.30)
Finally, the third layer reconstruct the HR image from these patch representations:

R L (Z) = W L * R 2 (Z) + B L (2.31)
where L denotes the number of weight layers of networks (i.e. L = 3 with SRCNN). In order to optimize the network, SRCNN uses the stochastic gradient descent with momentum algorithm. However, SRCNN attempts to add more than 4 weighted layers but deeper models give lower performance. An illustration of SRCNN is shown in Figure 2

(Z) = max(0, W j * R j-1 (Z) + B j ) j ∈ [2, L -1] (2.32)
In this case, L is equal to 20. The networks of VDSR are proposed to learn the mapping from the interpolation LR images to the residual between the interpolation LR images and the corresponding HR images as:

R = argmin R k i (X i -Z i ) -R(Z i ) 2 (2.33)
Due to residual learning, effective weight initialization and gradient-clipping optimization scheme, VDSR can build more layers than SRCNN, leading to more accurate performance.

Recursive neural networks are first proposed in [Kim et al., 2016b] (DRCN). This network replaces the mapping function as the series of convolutional layers in Equation 2.32 by the recursive convolutional layers as:

R 2 (H) = (g • g • ...•)g(H) = g D (H) (2.34)
where • denotes a function composition and g D denotes the D-fold product of g. Assuming

H 0 = R 1 (Z)
, a recurrent relation g as:

H d = g(H d-1 ) = max(0, W * H d-1 + b) (2.35)
DRCN can improve performance by increasing recursion depth, that does not add new parameters for additional convolution layers. A common point of these methods is that they use interpolated images as the input of the networks. The use of interpolation operator consumes of memory (i.e. larger weights storage of each filter per layer). A new layer called sub-pixel layer proposed in [START_REF] Shi | Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[END_REF] or a deconvolution layer in [Dong et al., 2016b], inside which the LR image is upscaled, allows the networks independent of interpolation techniques as:

R = argmin R k i X i -R(Y i ) 2
(2.36) These layers are proposed to be attached at the end of the networks:

R L (Y) = W L * S ↑ R L-1 (Y) (2.37)
Instead of learning one scale factor, laplacian pyramid networks in [START_REF] Lai | Deep laplacian pyramid networks for fast and accurate super-resolution[END_REF] propose to train simultaneous several factors through a set of progressive upscaling layers. A network with more than 16 residual blocks (a block consists of two convolutional layers with batch normalization, ReLU and skip connection) is proposed in [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF] (SRResnet).

The recursive blocks and the residual blocks are then combined in the work of [START_REF] Tai | Image super-resolution via deep recursive residual network[END_REF] to build more layers but still maintain the efficiency. Although, the deeper networks (more than 20 weight layers) such as SRResnet have very accurate quantitative metrics, the methods give less perceptual reconstructions [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF]. The investigation of other effective functions instead of mean squared error-based cost functions has been proposed such as ℓ 1 -norm loss [START_REF] Zhao | Loss functions for neural networks for image processing[END_REF], Charbonnier loss [START_REF] Lai | Deep laplacian pyramid networks for fast and accurate super-resolution[END_REF], perceptual loss [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], Ledig et al., 2017]. The objective function in Equation 2.36 can be rewritten as a ℓ 1 -norm:

R = argmin R k i X i -R(Y i ) (2.38)
or a robust Charbonnier loss function as:

R = argmin R k i X i -R(Y i ) 2 + ǫ 2 ρ (2.39)
where ǫ ρ is set to 1e -3. [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] propose to train the restoration network R to generate the output R(Y) which has the perceptual content of the HR image X based on the perceptual loss as:

L perceptual (X, R(Y)) = k (F l k (R(Y)) -F l k (X)) 2 (2.40)
where F l is the feature maps of the l th layer of a pre-trained network (e.g. VGG-net [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]). Generative adversarial networks [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF] (SRGAN) improve the idea of the perceptual loss by adding an adversarial loss. SRGAN consists of two networks: a network called the generator R generates super-resolved images and another discriminates the generated images and the true ones as the discriminator D. The adversarial objective of SRGAN can be described as:

L adversarial = min R max D E X∼P X [logD(X)] + E Y∼P Y [log(1 -D(R(Y)))]
(2.41)

The total objective of SRGAN is formulated as the weighted losses:

L SRGAN = L perceptual + 10 -3 L adversarial (2.42)
We have reviewed several learning-based image super-resolution algorithms using statistical approaches, sparse coding, random forest and CNNs. One of the main benefits of learningbased methods is the potential for non linear representation between HR and LR pairs.

Furthermore, the methods are capable to learn a substantial amount of regressors which observes and generalizes relationships inside the data. However, not all the information in the training set may be relevant to observed LR images. The feature extractions are crucial to some learning-based methods such as sparse coding or random forest. Since the first success of SRCNN, the number of convolutional neural networks based methods are numerous, thanks to the ability of feature learning inside the networks and the support of GPU computational power (that reduces the training time). On the other hand, the CNN-based techniques are lacking of mathematical theories because they can optimize considered non-convex functions.

But we can not deny that SR methods using CNNs work extremely well.

Blind super-resolution

Most methods assume a known PSF of imaging systems for the observation model. Then, the models are trained based on this assumption. However, the pre-trained model significantly decreases the quality of results when applying to real LR images acquired with a different PSF. Blind super-resolution methods attempt to estimate the appropriate PSF of the observed LR image instead of using a pre-defined kernel. In order to estimate the PSF, these approaches assume stochastic reconstruction steps initialized by a random PSF for an optimal reconstruction. Every patch y i extracted from the LR image (i = 1, ..., M ) can be expressed from the observation model as:

y i = Hx i + N (2.43)
where x i denotes the patches of the HR image, H is the observation matrix. Some methods [START_REF] He | A soft map framework for blind superresolution image reconstruction[END_REF][START_REF] Wang | Patch based blind image super resolution[END_REF] propose to simultaneously estimate the HR image and the PSF parameter using the a joint maximum a posteriori (MAP) of probabilistic combination models p as:

p(x i , h | y i ) ∝ p(y i | x i , h)p(x i )p(h) (2.44)
where p(x i ) and p(h) are prior terms and p(y i | x i , h) is the data likelihood. Assuming that the term N of Equation 2.43 stands for a white Gaussian noise with a zero-mean and the standard deviation of σ, the data likelihood can be expressed by image formation model as:

p(y i | x i , h) = M i=1 1 N M i=1 exp - y i -H(h) i x i 2 2σ 2 (2.45)
Here, the HR image prior p(x i ) can be computed by the learning methods [START_REF] Freeman | Example-based super-resolution[END_REF] and the PSF prior p(h) can be assumed to be a uniform distribution over a predefined range because of no prior knowledge on it [START_REF] Wang | Patch based blind image super resolution[END_REF]]. H(h) i is the estimated observation model during the generation of y i . However, these assumptions may lead to inaccurate estimation [START_REF] Michaeli | Nonparametric blind super-resolution[END_REF] because the methods attempt to estimate simultaneously the prior of the HR image x i and the kernel h. Instead, [START_REF] Michaeli | Nonparametric blind super-resolution[END_REF] only computes the MAP estimate of the kernel h:

ĥ = argmax h p(h) M i=1 p(y i | h) = argmax h p(h) M i=1 x i p(y i | x i , h)p(x i )dx i (2.46)
where p(x i ) is a prior term. Similarly, we can express the estimation as :

ĥ = argmax h p(h) M i=1 x i exp - y i -Hx i 2 2σ 2 p(x i )dx i (2.47)
Given N HR training patches x i , the prior term can be approximated by empirical mean as:

ĥ = argmax h p(h) M i=1 1 N N j=1 exp - y i -Hx j 2 2σ 2 (2.48)
where p(h) is a nonparametric prior. This is in contrast to [START_REF] Wang | Patch based blind image super resolution[END_REF] which assumes a parametric prior. [START_REF] Michaeli | Nonparametric blind super-resolution[END_REF] emphasize that the term Hx j can be equivalently written as X j h because of the dependence of Equation (2.48) on h, where X j is a matrix corresponding to convolution with x i and a down-sampling operator. Equation (2.48) can be solved by taking the log as:

ĥ = argmin h 1 2 Ch - M i=1 log   N j=1 exp - y i -X j h 2 2σ 2   (2.49)
where C can be a chosen matrix to penalize for non-smooth kernels. The blind SR methods can be used to approximate the real PSF of observed LR images using principled MAP estimations. The use of blind SR based on learning methods is very potential for real applications such as enhancing the historical image. However, the current algorithms for blind SR are based on several assumptions, that may reduce the generalization of the observation model.

Zero-shot learning

If external training datasets are not available, one approach called zero-shot learning proposes to exploit the similarity of patches inside the image. Assuming that the observation model in Equation 2.2 with noise free as:

Y = D ↓ BX (2.50)
The method in [START_REF] Glasner | Super-resolution from a single image[END_REF] attempts to find the HR of a LR image by exploiting cross-scale patch redundancy called internal examples. A set of several downscaled versions from the LR one Y can be generated as

I -i = D ×-i ↓ BY.
The strategy first finds the nearest neighbours of a patch y in the LR image from several downscaled versions and then copies to upscaled versions I i . Then, the method combines these upscaled versions to reconstruct the HR image by the multi-image methods as [START_REF] Milanfar | Super-resolution imaging[END_REF]. Instead of 2D transformation as in [START_REF] Glasner | Super-resolution from a single image[END_REF] (i.e. translation), [Huang et al., 2015a] propose a transform matrix to find the self-similarity between internal recurrence of patches inside the testing image. The method in [START_REF] Shocher | Zero-shot" super-resolution using deep internal learning[END_REF] exploits the kernel estimation in [START_REF] Michaeli | Nonparametric blind super-resolution[END_REF] and the powerful representation of CNN-based technique for training the internal example patches by assuming the testing LR image as HR patches and its lower-resolution versions as LR cross-scale patches.

Zero-shot learning is used to overcome difficulties where the external dataset is lacking. In addition, these methods are very useful for LR images which contain redundant patches. However, since one LR image patch can construct several HR image patches, zero-shot learning may ignore details which are missed in the testing image.

Applications of super-resolution in medical imaging

Previously, the techniques for 2D natural images have been reviewed. However, photo-realistic images can not model specific 3D organs or the human body. In addition, medical imaging modalities are very diverse. Each modality has specific features which can be used to medical image analysis. Thus, SR methods for specific medical imaging are also studied. Besides, a set of different 2D images could only represent the slices of 3D architectures, not connections in 3D space, that raises the need of 3D models for 3D medical images. The study focused on medical imaging supports better for other practical applications.

Moreover, higher resolution medical image is the key to early detection of abnormalities or pathologies. One of the tasks of medical imaging is to increase and to extent the possible resolution so as achieve true isotropic 3-D images. In practice, the maximal sampling frequency of the imaging device detectors limits the captured range of radio frequencies from the imaged object. A solution to increase resolution is to reduce detectors size, however, this increases the noise, thus reduces SNR. Increasing image resolution through super-resolution is a key to better understanding of the anatomy [START_REF] Greenspan | Super-resolution in medical imaging[END_REF]. Medical image SR can be used to improve the performance of image segmentation and image registration methods. A better quality of an image can result more accurate segmentation and registration. Previous works have shown that applying super-resolution techniques leads to more accurate segmentation maps of brain MRI data [START_REF] Jog | Self super-resolution for magnetic resonance images[END_REF][START_REF] Rueda | Single-image super-resolution of brain mr images using overcomplete dictionaries[END_REF] or cardiac data [START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF].

The use of SR techniques has been studied in the context of medical analysis, specially of brain images: anatomical MRI [START_REF] Luo | Fast single image super-resolution using estimated low-frequency k-space data in mri[END_REF], Manjón et al., 2010a,b, Rousseau, 2008, Rousseau et al., 2010a,b, Rueda et al., 2013[START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF], diffusion MRI [START_REF] Fogtmann | A unified approach to diffusion direction sensitive slice registration 105 and 3-d dti reconstruction from moving fetal brain anatomy[END_REF][START_REF] Poot | Super-resolution for multislice diffusion tensor imaging[END_REF][START_REF] Scherrer | Super-resolution reconstruction to increase the spatial resolution of diffusion weighted images from orthogonal anisotropic acquisitions[END_REF][START_REF] Steenkiste | Super-resolution reconstruction of diffusion parameters from diffusionweighted images with different slice orientations[END_REF], spectroscopy MRI [START_REF] Jain | Patch-based super-resolution of mr spectroscopic images: Application to multiple sclerosis[END_REF], quantitative T 1 mapping [START_REF] Ramos-Llordén | A unified maximum likelihood framework for simultaneous motion and t_{1} estimation in quantitative mr t_{1} mapping[END_REF], Van Steenkiste et al., 2017], fusion of orthogonal scans of moving subjects [START_REF] Gholipour | Robust super-resolution volume reconstruction from slice acquisitions: application to fetal brain mri[END_REF][START_REF] Jia | A new sparse representation framework for reconstruction of an isotropic high spatial resolution mr volume from orthogonal anisotropic resolution scans[END_REF][START_REF] Kainz | Fast volume reconstruction from motion corrupted stacks of 2d slices[END_REF], Rousseau et al., 2010c]. In the next sections, we will focus on two families of medical image SR: model-based methods and learning-based methods.

Applications of model-based methods

The non-local mean upsampling [Manjón et al., 2010b] (NMU) method performs first high quality reconstructed image via the iteration patch-based filtering as:

xt+1 = 1 C ∀k∈Ω w(x t , xt k )x t k (2.51)
where xt is the voxel of the reconstructed HR image at the current iteration t, C is a constant

and Ω is the searching zone. The initialized image is supposed as X 0 = S ↑ Y. The weighted coefficient w is calculated based on the non-local mean (NLM) filter [START_REF] Coupé | An optimized blockwise nonlocal means denoising filter for 3-d magnetic resonance images[END_REF] as:

w(x t , xt k ) =    e |x t -x t k | 2 h 2 if | µ t -µ t k |< 3h/ √ N 0 otherwise (2.52)
where µ is the average of 3D patches x around the voxel x, h denotes a filtering parameter and N is the number of voxel in the 3D patch. The second step of NMU exploits the IBP method [START_REF] Irani | Improving resolution by image registration[END_REF] for ensuring consistency between the observation model and the estimated high resolution.

The medical image SR problem could be also solved by minimizing a objective function with a regularization term as Equation 2.6. The objective function with a ℓ 2 -norm regularization, which is proposed in [START_REF] Gholipour | Robust super-resolution volume reconstruction from slice acquisitions: application to fetal brain mri[END_REF], Rousseau et al., 2010c], can be written as:

X = argmin X Y -HX 2 + λ CX 2 2 (2.53)
where C is a positive definite matrix. A combination of low-rank regularization [START_REF] Liu | Tensor completion for estimating missing values in visual data[END_REF] and total-variation regularization [START_REF] Rudin | Nonlinear total variation based noise removal algorithms[END_REF] proposed in [START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF] (LRTV) transforms the SR problem as:

X = argmin X Y -HX 2 + λ rank Rank(X) + λ tv T V (X) (2.54)
where Rank is the weighted sum of trace norms of all slices along each dimension of an image and T V (total-variation) denotes the integral of the absolute gradients of data. The regularizer Rank takes advantages of the similarity between the slices in different directions, that can not happen in the 2D image cases. However, these priors assume that the image is too smooth leading to lack of the details of the true image representation.

Applications of learning-based methods

The learning-based methods can not only find implicitly the parameters of prior energy function via examples but also define a specific regularization expression. The work in [START_REF] Rousseau | A supervised patch-based image reconstruction technique: Application to brain mri super-resolution[END_REF] extends the NMU method for capturing more information of a training dataset. Given a training dataset which consists of paired HR-LR images

D = {(X i , Y i ) | i = 1, ..., N }, we can reconstruct the HR image X of the testing LR image Y as X(x) = N i=1 ∀k∈Ω w i (x, x k )X i (x k ) N i=1 ∀k∈Ω w i (x, x k ) (2.55)
where x is the current voxel with the neighbour searching zone Ω and w i (x, x k ), which denotes the weighted coefficients, is calculated by the similarity between Y and each LR sample from the external set as:

w i (x, x k ) =    e |x-x k,i | 2 h 2 if | µ -µ k |< 3h/ √ N 0 otherwise (2.56)
where the parameters of this equation are similar to Equation 2.52. This method also needs a correction step in order to improve the robustness. Instead of using IBP as in [Manjón et al., 2010b], the 3D patches x of the HR reconstruction are calibrated by the HR samples as:

x(x) = N i=1 ∀k∈Ω w i (x, x k )x i (x k ) (2.57)
The extension of sparse representation methods [START_REF] Yang | Image super-resolution via sparse representation[END_REF][START_REF] Zeyde | On single image scale-up using sparse-representations[END_REF] is proposed in [START_REF] Rueda | Single-image super-resolution of brain mr images using overcomplete dictionaries[END_REF]] by using multi-scale Sobel filters. In this work, the authors demonstrated the importance of 3D feature detectors within brain MRI data. The filter set, which is proposed to analyse multi-scale edges of interpolated testing LR images, consists of 2 high-frequency filters with the patch size of 3 and 5 for each direction. Then, the HR image is reconstructed by finding in the LR-HR sparse dictionaries and is then corrected by the IBP method as [START_REF] Irani | Improving resolution by image registration[END_REF].

Recently, 3D convolutional neural networks for MRI SR, which have been investigated in [Pham et al., 2017a], learn the feature representation automatically inside the networks. We will discuss this approach in the Section 2.2. Later, [Chen et al., 2018b] proposed a 3D version of densely connected networks (DenseNet) [Huang et al., 2017a] for brain MRI SR. Before DenseNet, the residual networks in [He et al., 2016a] (ResNet) achieved the most performance in image classification. ResNet can build up to 1000 convolution layers thanks to the residual blocks [START_REF] He | Identity mappings in deep residual networks[END_REF], that is impossible to the previous networks [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]Zisserman, 2014, Szegedy et al., 2015]. However, ResNet takes a lot of memory training. Densely connected networks [Huang et al., 2017a] can achieve performance as good as deep networks (e.g. ResNet [He et al., 2016a]) but reduces memory training thanks to feature concatenations through all layers. Assuming that the external dataset is not available, inspired by the work of [START_REF] Jog | Self super-resolution for magnetic resonance images[END_REF], [START_REF] Zhao | Self super-resolution for magnetic resonance images using deep networks[END_REF] investigated self super-resolution for MRI using enhanced deep residual networks [START_REF] Lim | Enhanced deep residual networks for single image super-resolution[END_REF]. [START_REF] Zhao | Self super-resolution for magnetic resonance images using deep networks[END_REF] relies on the fact that a LR anisotropic 3D image has a in-plane high resolution (e.g. axial slice). Then, the LR image is interpolated to generate an interpolated isotropic image as a HR reference image.

A simulated LR image is then generated from the HR reference. The deep network in [START_REF] Lim | Enhanced deep residual networks for single image super-resolution[END_REF] is trained with the patches of simulated pairs and finally applied to the original LR image.

Evaluation

For quantitative comparison, the peak signal to noise ratio (PSNR) in decibels (dB) and

Structural Similarity Index (SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF] are commonly used to evaluate the performance of image reconstruction algorithms. Given a dynamic range d, the PSNR is defined as:

P SN R = 10 log 10 ( d 2 M SE ) (2.58)
where the mean squared error (MSE) is defined as:

M SE = iǫΩ (X(i) -X(i)) 2 (2.59)
where X is the reconstructed image with respected to the ground truth X, Ω is the number of pixels or voxels of images.

SSIM is used for measuring the image quality based on perceived similarity. SSIM is calculated as:

SSIM = (2µ X µ X + c 1 )(2σ X X + c 2 ) (µ 2 X + µ 2 X + c 1 )(σ 2 X + σ 2 X + c 2 ) (2.60)
where µ X , σ 2 X are respectively the average and the variance of window X of image X, similarly for X and σ X X denotes the covariance of these windows. c 1 and c 2 are two constants.

However, in some cases, a higher PSNR or SSIM does not indicate that the reconstruction is of higher quality because they do not correlate with human assessment of visual quality [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], Ledig et al., 2017[START_REF] Wang | Mean squared error: Love it or leave it? a new look at signal fidelity measures[END_REF]. PSNR and SSIM rely only differences between pixels which may not describe the high-level human visual perception via feature representation. Thus, when comparing methods, the need of qualitative results should be shown to have a general assessment.

Discussion

Major advances in the domains of computer vision indicated the ability of SR methods. The most popular approach is based on solving the observation model. In order to constraint the ill-posedness of model-based methods, the adding prior can bound the conditions of the solution. The capacity of non linear representations which is used in the learning-based methods helps to capture the relationships of low-resolution images and high-resolution ones. In contrast to other learning methods which strictly depend on the feature extractions, convolutional neural networks with implicitly feature representation have become the state-of-the-art models for SR. However, SR algorithms may face the fact that the point spread function of observation model is not always ideally. Thus, several methods attempt to solve SR which does not assume a fixed blurring function (blind SR). In addition, many techniques exploit the redundant information of internal patches to increase the solution of low-resolution images.

The methods of two dimensional natural image SR, which have also mentioned, can be expanded to 3D images. However, the medical image SR can not be viewed inseparable from 2D photo-like techniques. In application of SR in medical imaging, we have introduced two main categories: model-based and learning-based methods. The techniques based on the observation models, which depend on the assumptions of image priors, do not need to collect other external data. However, they can lead to too smooth results due to crucial prior. In order to exploit the missing information which can provided by a training set, the learning-based methods can be used. Several learning-based techniques require feature extractions that can reduce the information of image representation. In addition, the better learning algorithms will help us to better performance, augment the capacity of feature learning storage or faster convergence. Convolutional neural networks is one of the methods which do not depend on feature extraction because they can learn representation filters implicitly inside their layers.

Learning-based single super-resolution using convolutional neural networks

CNN architectures have become the state-of-the-art for image SR. However, due to the variety of the proposed methods and the high number of parameters for the networks architecture design, it is currently difficult to identify the key elements of CNN architecture to achieve good performance for image SR and assess their applicability in the context of 3D brain MRI.

In addition the extension of CNN architectures to 3D images, taking into account floating and possibly anisotropic scaling factors may be of interest to address the wide range of possible clinical acquisition settings, whereas classical CNN architectures only address a predefined (integer) scaling factor. The availability of multimodal imaging setting also questions the ability of CNN architectures to benefit from such multimodal data to improve the SR of a given modality.

First of all, our work verifies the need of fitting data and network parameters for 3D brain MRI. Then, this work presents a comprehensive review of deep 3D convolutional neural networks, and associated key elements, for brain MRI SR. Following [START_REF] Timofte | Seven ways to improve example-based single image super resolution[END_REF], who have experimentally showed several ways to improve SR techniques from a baseline architecture, we study the impact of eight key elements on the performance of convolutional neural networks for 3D brain MRI SR. We demonstrate empirically that residual learning associated with appropriate optimization methods can significantly reduce the time of the training step and fast convergence can be achieved in 3D SR context. Overall, we report better performance when learning deeper fully 3D convolution neural networks and using larger filters. Interestingly, we demonstrate that a single network can handle multiple arbitrary scale factors efficiently, for example, from 2 × 2 × 2 mm to 2 × 2 × 1 mm or 1 × 1 × 1 mm, by learning multiscale residuals from spline-interpolated image. We also report significant improvement using a multimodal architecture, where a HR reference image can guide the CNN-based SR of a given MRI volume.

Recall that single image SR is a typically ill-posed inverse problem that can be stated according to the following linear formulation:

Y = HX + N = D ↓ BX + N (2.61)
where Y ∈ R n is the LR observed image, X ∈ R m is the HR image, H ∈ R m×n is the observation matrix (m > n) and N denotes an additive noise. D ↓ represents the downsampling operator and B is the PSF. In a learning-based context where a set of image pairs

(X i , Y i )
is available, the objective is to learn the mapping H -1 from the LR images Y i to the HR images X i , leading to the following formulation:

X = arg min X X -H -1 Y 2 .
(2.62)

In this setting, the matrix H -1 can be modeled as a combination of a restoration matrix F ∈ R m×m and an upscaling interpolation operator S ↑ : R n → R m . Given a set of K HR images X i and their corresponding LR images Y i , the restoration operator F can be estimated as follows:

F = arg min F K i=1 X i -F (S ↑ Y i ) 2 = arg min F K i=1 X i -F (Z i ) 2 (2.63) where Z ∈ R m is the interpolated LR (ILR) version of Y (i.e. Z = S ↑ Y).
F is then a mapping from the ILR image space to the HR image space.

Methodology

2.2.1.1 Restoration by convolutional neural networks : 2D or 3D models for 3D data ?

The restoration matrix F corresponds to the mapping from Z to X. In SRCNN [Dong et al., 2016a], this mapping is decomposed into three operations, described as follows:

     F 1 (Z) = max(0, W 1 * Z + B 1 ) F 2 (Z) = max(0, W 2 * F 1 (Z) + B 2 ) F 3 (Z) = W 3 * F 2 (Z) + B 3
(2.64)

where:

• W i and B i are the convolution parameters to learn, where i ∈ {1, 2, 3}.

W i corresponds to n i convolution filters of support c × f i × f i × f i ,
where c is the number of channels in the input of layer i, f i and n i are respectively the spatial size of the filters and the number of filters of layer i,

• max(0, •) refers to a ReLU applied to the filter responses.

Each of these operations is designed using one layer of the neural network. The first step, called F 1 , extracts overlapping patches of the LR image and computes a set of feature maps.

F 1 is similar to a popular strategy in image restoration by representing patches by a set of pre-trained bases (such as PCA or DCT). In SRCNN, this step is performed by convolving the image by a set of learned filters. The second operation, F 2 , which is mathematically very close to F 1 , is a non-linear mapping from the LR feature maps to HR feature maps. Finally, the third operation, F 3 , is a convolutional layer corresponding to the image reconstruction.

W 3 can be seen as the projection of HR feature maps onto the image domain and then patches averaging.

SRCNN has been originally designed for 2D natural image processing. In [Dong et al., 2016a],

W 1 , W 2 and W 3 consist of n 1 filters with 2D patch size f 1 ×f 1 , n 2 filters with patch size f 2 ×f 2 and one filter with patch size f 3 × f 3 respectively. In order to apply this restoration operator called F 2D , we propose first a straightforward strategy consisting in averaging restored versions of 3D ILR images Z 3D for each direction to estimate a 3D HR image X 3D :

X 3D = F axial 2D (Z 3D ) + F coronal 2D (Z 3D ) + F sagittal 2D (Z 3D ) (2.65)
Using this strategy, it is possible to apply the model learned with natural images [Dong et al., 2016a] (called here SRCNNF-Nat). In addition, a network is trained with a dedicated learning image dataset (called SRCNNF-Brain).

In addition, we investigate the use of a 3D network which consists of n 1 filters with voxel size

f 1 × f 1 × f 1 , n 2 filters with voxel size f 2 × f 2 × f 2 and one filter with voxel size f 3 × f 3 × f 3
in Section 2.2.2.2. The 3D HR image is then computed as follows: Instead of learning the mapping directly from the LR space to the HR one, it might be easier to estimate a mapping from the LR space to the missing high-frequency components, also called the residual between HR and LR data: R = X -Z or equivalently X = Z + R. This approach can be modeled by a skip connection in the network. In such a residual-based modeling, one typically assumes that R is a function of Z. The computation of HR data is then expressed as follows: X = Z + F (Z) where F can be learned using the following equation:

X 3D = F 3D (Z 3D ).

Restoration by 3D residual-learning convolutional neural networks

F = arg min F K i=1 (X i -Z i ) -F (Z i ) 2 .
(2.66)

Following [Kim et al., 2016a], mapping F from Z to (X -Z) is decomposed into nonlinear operations corresponding to the combination of convolution-based and rectified linear unit (ReLU) layers. The baseline deeper architecture used in this work can be described as follows:

     F 1 (Z) = max(0, W 1 * Z + B 1 ) F i (Z) = max(0, W i * F i-1 (Z) + B i ) f or 1 < i < L F L (Z) = W L * F L-1 (Z) + B L (2.67)
where L is the number of layers. This network architecture is depicted in Figure 2.3. Please note that, for instance, the SRCNN model [Dong et al., 2016a] corresponds to a specific parameterization of this baseline architecture (f 1 = 9, f 2 = 1, f 3 = 5, n 1 = 64, n 2 = 32 and with no skip connection).

Experimental setting

MRI dataset and LR simulation

To evaluate SR performances of CNN-based architectures, we have used two MRI datasets:

the Kirby 21 dataset and the NAMIC Brain Multimodality dataset.

The Kirby 21 dataset [START_REF] Landman | Multi-parametric neuroimaging reproducibility: a 3-t resource study[END_REF] As in [START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF] and [START_REF] Rueda | Single-image super-resolution of brain mr images using overcomplete dictionaries[END_REF] Spline Interpolation SRCNNF-Brain (9-1-5) SRCNN3D First, we studied the impact of the number of epochs used for training for both SRCNNF-Brain and SRCNN3D networks (see Figure 2.4). A strong improvement with respect to spline interpolation can be noted with few epochs (less than 500). Then, the mean PSNR increases slowly to reach substantial improvements around 2500 epochs. SRCNN3D seems to lead to better performances than SRCNNF-Brain no matter what the number of epochs used.

Table 3.1 provides a summary of quantitative evaluation within isotropic scale factor 2 for the following methods: cubic spline interpolation, non-local means upsampling (NMU) [Manjón et al., 2010b], Low-rank total variation (LRTV) [START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF], SRCNNF-Nat [Dong et al., 2016a], SRCNNF-Brain and SRCNN3D. The reported mean gain tends to show that CNNbased approaches achieve better performance than spline interpolation, NMU or LRTV. For NMU and LRTV, we used the code provided by the authors. Our experiments show that the use of CNN-based approaches can lead to significant improvement over spline interpolation.

More specifically, it can be seen that training the networks using specific data provides better results than using models trained over natural images. Moreover, the use of a that 3D architecture directly learns the 3D structure of MRI volumetric images. In the next sections, we will focus on improving the performance of 3D networks based on the sensitivity analysis of baseline 3D architectures.

Baseline and benchmarked for 3D architectures

The network architecture that is used as a baseline approach in this study is illustrated in Figure 2.3. The baseline network is a 10 blocks (convolution+ReLU) network with the following parameters: 64 convolution filters of size (3 × 3 × 3) at each layer, mean squared error (MSE) as loss function, weight initialization by [START_REF] He | Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification[END_REF] (MSRA filler), Adam (adaptive moment estimation) method for optimization [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], 20 epochs on Nvidia GPU and using Caffe package [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF], batch size of 64, learning rate set to 0.0001, no regularization or drop out has been used. The learning rate multipliers of weights and biases are respectively 1 and 0.1. For benchmarking purposes, we consider two other state-of-the-art SR models: low-rank total variation (LRTV) [START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF] and SRCNN3D [Pham et al., 2017a]. SRCNN3D [Pham et al., 2017a], which is an extension in 3D of the method described in [Dong et al., 2016a], has 3 convolutional layers with the size of 9 3 , 1 3 and 5 3 respectively. The layers of SRCNN3D consist respectively of 64 filters, 32 filters and one filter.

θ = {W i , B i } i=1,...,L .
Most optimization methods for CNNs are based on gradient descent. A classic method applies a mini-batch stochastic gradient descent with momentum (SGD) [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] as used in [Dong et al., 2016a, Pham et al., 2017a]. However, the use of fixed momentum causes numerical instabilities around the minimum. Nesterov's accelerated gradient (NAG) [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate o (1/k2)[END_REF] was proposed to cope with this issued.

The use of small learning rates induces slow convergence. By contrast, high learning rates may lead to exploding gradients [Bengio et al., 1994, Glorot and[START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]. In order to address this issue, [Kim et al., 2016a] proposed the stochastic gradient descent method with an adjustable gradient clipping (SGD-GC) [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF] to achieve an optimization with high learning rates (e.g. α = 0.1). The predefined range over which gradient clipping is applied may still cause SGD-GC not to converge quickly or make difficult the tuning of a global learning rate. Recently, methods have been proposed to address this issue through an automatic adaption of the learning rate for each parameter to be learned. RMSProp (root-mean-square propagation) [START_REF] Tieleman | Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF] and Adam (adaptive moment estimation) [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] are the two most popular models in this category.

The results of four optimization methods (NAG, SGD-GC, RMSProp and Adam) for the baseline network are illustrated in Figure 2.6. Firstly, regardless the method used, the baseline network shows better performance than LRTV [START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF] and SRCNN3D [Pham et al., 2017a]. Secondly, it can be observed that the baseline network can converge very fast and stably. Concretely, the proposed optimization scheme needs only 20 epochs with small learning rate of 0.0001 to converge while the SRCNN3D shown in Figure 2.4 takes 2500 epochs. Finally, in these experiments, the most efficient and effective optimization method is Adam as regards both PSNR metric and convergence speed. Hence, in the next sections, we use Adam method with β 1 = 0.9 and β 2 = 0.999 to train our networks with 20 epochs.

Weight initialization

The optimization algorithms for training a CNN are typically initialized randomly. Inappropriate initialization can lead to long time convergence or even divergence. Several studies [Dong et al., 2016a[START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF], Pham et al., 2017a] used a normal distribution N (0, 0.001) to initialize the weights of convolutional filters. However, because of too small initial weights, the optimizer can be stuck into a local minimum especially when building deeper networks. Both [Dong et al., 2016a] concluded that deeper networks do not lead to better performance, and [START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF] confirmed that the addition of extra convolutional layers to the 7-layer model is found to be ineffective. Uniform distribution U (-3/(nf 3 ), 3/(nf 3 ))

(called Xavier filler) [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] was also proposed to initialize the weights of deeper networks. In order to add more layers to networks, [START_REF] He | Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification[END_REF] suggested an initial training stage by sampling from the normal distribution N (0, 2/(nf 3 )) (called here Microsoft Research Asia -MSRA filler).

Overall, we evaluate here the weight initialization schemes described in [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] and [START_REF] He | Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification[END_REF], a normal distribution N (0, 0.001) as in [Dong et al., 2016a[START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF] and a normal distribution N (0, 0.01) for the considered SR architecture.

Experiments with a deeper architecture were also performed, more precisely for a 20-layer architecture, which is the deepest architecture that could be implemented for the considered experimental setup due to GPU memory setting. As shown in Figure 2.7, the initialization with normal distributions N (0, 0.001) failed to make the training of both 10-layer and 20-layer residual-learning networks converge. In addition, the 20-layer network also does not converge when initialized with normal distributions N (0, 0.01). By contrast, MSRA and Xavier filler schemes make the networks converge and reach similar reconstruction performance. For the rest of this chapter, we use MSRA weight filler as initialization scheme.

Residual learning

The CNN methods in [Dong et al., 2016a,b, Shi et al., 2016] use the LR image as input and outputs the HR one. We refer to such approach as a non-residual learning. Within these approaches, low-frequency features are propagated through the layers of networks, which may increase the representation of redundant features in each layer and in turn the computational efficiency of the training stage. By contrast, one may consider residual learning or normalized HR patch prediction as pointed out by several learning-based SR methods [Kim et al., 2016a[START_REF] Timofte | Anchored neighborhood regression for fast examplebased super-resolution[END_REF], 2014[START_REF] Zeyde | On single image scale-up using sparse-representations[END_REF]. When considering CNN methods, one may design a network which predicts the residual between the HR image and the output of the first transposed convolutional layer [START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF]. Using residual blocks, a CNN architecture may implicitly embed residual learning while still predicting the HR image [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF]. Here, we perform a comparative evaluation of non-residual learning vs. residual learning strategies. Figure 2.8 depicts PSNR values and convergence speed of residual vs non-residual network structures with 10 and 20 convolutional layers. The residual-learning networks converge faster than the non-residual-learning ones. In addition, residual learning leads to improvements in PSNR (+0.4dB for 10 layers and +1.2dB for 20 layers). It might be noted that these experiments do not support the common statement that the deeper, the better for CNNs. Here, the use of additional layers is only beneficial when using residual modeling.

Deeper architectures even lower the reconstruction performance with non-residual learning.

Depth, filter size and number of filters

As shown by the previous experiment, the link between network depth and performance remains unclear. Besides, it is hard to train deeper networks because gradient computation can be unstable when adding layers [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]. For instance, [START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF] tested extra convolutional layers to a 7-layer model but achieved negligible performance improvement. As mentioned above, SRCNN [Dong et al., 2016a] was also tested with deeper architectures but no improvement was reported. However, [Kim et al., 2016a] argue that the performance of CNNs for SR could be improved by increasing the depth of network compared to neural network architectures in [Dong et al., 2016a[START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF].

The previous section supports that deeper architectures may be beneficial when considering a residual learning. We further evaluate here the reconstruction performance as a function of the number of layers. Results are reported in Figure 2.9. They stress that increasing network depth with residual learning improves the quality of the estimated HR image (e.g. +1.6dB increasing of the depth from 3 to 20 or +0.5dB increasing of the depth from 7 to 20). The parameterization of the convolutional filters is also of key interest. Inspired by the VGG network designed for classification [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], previous CNN methods for SR mostly focused on small convolutional filters of size (3×3×3) in [START_REF] Kamnitsas | Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation[END_REF], Kim et al., 2016a[START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF]. Small filter size can build deeper networks but reduces the memory for computation cost [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. [START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF] even argued that such architecture can lead to better non-linear estimations. Regarding the number of filters for each layer, [Dong et al., 2016a] reported greater reconstruction performance when increasing the number of filters. But these experiences were not reported in other CNN-based SR studies [Kim et al., 2016a[START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF]. Here, we both evaluate the effect of the filter size and of the number of filters. filters) involves a smaller number of parameters, such that it took us only 24 hours to train.

These experiments suggest that deeper architectures with small filters can replace shallower networks with larger filters both in terms of computational complexity and of reconstruction performance. In addition, the increase in the number of filters within networks can increase the performance. However, we were not able to use 128 filters with the baseline architecture due to the limited amount of memory. This stresses out the need to design memory efficient architectures for 3D image processing using deeper CNNs with more filters.

Training patch size and subject number

In the context of brain MRI SR, the acquisition and collection of large datasets with ho- The size of training patches should be larger or equal to the size of the receptive field (the region of the input space affects a particular layer) of the considered network [Kim et al., 2016a, Simonyan and[START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], which is given by ((f -1)D+1) 3 for a D-layer network with filter size f 3 . Figure 2.11 confirms that better performance can be achieved using larger training patches (from 11 3 to 31 3 with the 10-layer network and from 11 3 to 29 3 with the 12-layer network). However, if the patch size is larger than the receptive field (e.g. 21 3 within the 10-layers network and 25 3 within the 12-layers network), the improvement is very little while we consume considerably more GPU memory and training time.

mogeneous
We stressed previously that the selection of the network depth involves a trade-off between reconstruction performance and GPU memory requirement and training time increase. A similar result can be drawn with respect to the patch size. Regarding the number of training subjects, Figure 2.12 points out that a single subject is enough to reach better performance than spline interpolation. This has also been discovered in the work of [START_REF] Shocher | Zero-shot" super-resolution using deep internal learning[END_REF], Zhao et al., 2018] in which a super-resolution pipeline using the right testing image (self SR) is proposed. Interestingly, reconstruction performance increases slightly when more subjects are considered, which appears appropriate for realworld applications. However, in fact, more training dataset takes more time within the same experience settings. In the next sections, for saving training time, we propose to use 10 subjects for learning.

Handling arbitrary scales

In some CNN-based SR approaches, the networks are learned for a fixed and specified scaling factor. Thus, a network built for one scaling factor cannot deal with any other scale. In medical imaging, [START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF] 2,2,2) ×(3,3,3) ×(2,2,2),(3,3,3) ×(2,2,2),(3,3,3

Multimodality-guided SR

In some clinical cases, it is common to acquire one isotropic HR image and LR images with different modalities (different contrasts) in order to limit the acquisition time. Hence, a coplanar isotropic HR image might be considered as a complementary information source to reconstruct HR MRI images from LR ones [Rousseau et al., 2010a]. To address this multimodality-guided SR problem, we add a concatenation layer as the first layer of the network as illustrated in Figure 2.13. This layer concatenates the ILR image and a registered We experimentally evaluate the relevance of the proposed multimodality-guided SR model according to the following setting. We investigate whether the complementary use of a Flair or a T2-weighted MRI image might be beneficial to improve the resolution of a LR T1-weighted MRI image. Concerning the Kirby dataset, we apply an affine transform estimated using FSL [Jenkinson et al., 2012] to register images from the same subject into a common coordinate space. We assume here that the affine registration can compensate motion between two scans acquired during the same acquisition session since here an organ does not undergo significant deformation between two acquisitions. The registration step has been checked visually for all the images. Data of the NAMIC dataset are already in the same coordinate space so no registration step is required. It can be seen that multimodality driven approach can lead to improved reconstruction results. In these experiments, the overall upsampling result depends on the quality of the HR image used to drive the reconstruction process. Thus, adding high resolution information containing artifacts limits reconstruction performance. This is especially the case for the Kirby dataset. For instance, when considering T2w images, no improvement is observed for Kirby dataset and an improvement greater than 1dB is reported for NAMIC dataset. As the T2w image resolution is lower than T1w modality in Kirby dataset, these results may emphasize the requirement for HR information source to expect significant gain with respect to the monomodal model. Figure 2.16 shows visually that edges in the residual image between the ground truth and the reconstruction by the multimodal approach are reduced significantly compared to interpolation and monomodal methods (e.g. the regions of lateral ventricles). This means that the multimodal approach brings the reconstructions which are the most similar to the ground truth. These qualitative results highlight the fact that the proposed multimodal method provides a more favorable performance than other compared methods.

In addition, we explore the impact of the network depth augmentation with regard to the performance of multimodal SR approach. The experiments shown in Figure 2.15 indicate that the deeper structures do not lead to better results within the multimodal method.

How transferable are learned features?

Training a CNN from scratch requires an amount of training data and may take a long time.

Moreover, to avoid overfitting, the training dataset has to reflect the appearance variability of the images to reconstruct. In the context of brain MRI, part of image variability comes from acquisition systems. Hence, we investigate the impact of such image variability onto SR performance by evaluating transfer learning skills among different datasets corresponding to the same imaging modality.

In order to characterize such generalization skills, we evaluate the extent to which the selection of a given training dataset affects the reconstruction performance of the network. We proceed as follows: We train from scratch two 20L-ReCNN networks separately for a 10-image NAMIC T1-weighted dataset and a 10-image Kirby T1-weighted dataset, and we test the trained models for the remaining 10-image NAMIC and Kirby T1-weighted datasets. The considered case-study involves a scaling factor of (2 × 2 × 2). For quantitative comparison, the PSNR and the structural similarity (SSIM) (the definition of SSIM can be found in [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF]) are used to evaluate the performance of each model in Table 2.3. For benchmarking purposes, we also include a comparison with the following methods: cubic spline interpolation, low-rank total variation (LRTV) [START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF], SRCNN3D [Pham et al., 2017a]. The use of 20-layer CNN-based approaches for each training dataset can lead to improvements over spline interpolation, LRTV method and SRCNN3D (with respect to both PSNR and SSIM).

Although, we lose a little gain (e.g. PSNR: 0.55dB for testing Kirby and 0.74dB for NAMIC, SSIM: 0.003 for Kirby and 0.0019 for NAMIC) when using different training and testing dataset (i.e. different resolution), our proposed networks have better results than compared methods.

For qualitative comparison, Figures 2.17 In addition, the HR reconstruction of the 20L-ReCNN model shows that its differences from the ground truth are less than other methods (i.e. the contours of the residual image of the 20L-ReCNN method are less occurrences than those of others). Hence, we can infer that our proposed method best preserves contours, geometrical structures and better recovers the image contrast compared with the other methods.

Practical applications of super-resolution

There are many practical situations, including infant brain MRI scans [START_REF] Makropoulos | The developing human connectome project: A minimal processing pipeline for neonatal cortical surface reconstruction[END_REF], rapid emergency scans [START_REF] Walter | Imaging of renal lesions: evaluation of fast mri and helical ct[END_REF], where the LR images with an anisotropic voxel size are typically acquired due to patient comfort (e.g. infants can not lie on bed for a long time, emergency). These images usually have a high in-plane resolution and a low through-plane resolution. Interpolation is commonly used to upsampled these LR image to isotropic digital resolution. However, interpolated LR images may lead partial volume artifacts that affect segmentation [START_REF] Ballester | Estimation of the partial volume effect in mri[END_REF]. In such cases, motion correction and multi-image super-resolution can be used to achieve HR isotropic images [START_REF] Makropoulos | The developing human connectome project: A minimal processing pipeline for neonatal cortical surface reconstruction[END_REF]. If these methods are not always available, investigators and clinicians have no choice to process these LR images. For example, the MAIA dataset has the T2w images which acquired with the voxel size of 0.4464 × 0.4464 × 3 mm. In this section, we attempt to use our single image SR method to enhance the resolution of these clinical data and improve the segmentation methods applied to these images.

Super-resolution of clinical neonatal data

The idea is to use convolutional neural networks to transfer the rich information available from high-resolution experimental dataset to lower-quality image data. The procedure first uses CNNs to learn mappings between real HR images and their corresponding simulated LR images with the same resolution of real data. The LR data is generated by the observation model decomposed into a space-invariant blurring model and a downsampling operator. The two most popular choices for MRI PSF approximation for SR evaluation are a rectangular pulse Box-PSF with the box width of slice width [Manjón et al., 2010b], a Gaussian kernel [START_REF] Greenspan | Super-resolution in medical imaging[END_REF][START_REF] Rueda | Single-image super-resolution of brain mr images using overcomplete dictionaries[END_REF][START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF]. However, the most accurate representation is the use of a Gaussian kernel with the full-width-at-half-maximum (FWHM) equal to slice thickness [START_REF] Greenspan | Super-resolution in medical imaging[END_REF]. Once models learned, these mappings enhance the LR resolution of unseen low quality images.

In order to verify the applicability of our CNN-based methods, we have used two neonatal brain MRI dataset: the dHCP dataset [START_REF] Hughes | The developing human connectome: announcing the first release of open access neonatal brain imaging[END_REF] and the MAIA dataset. The HR images are T2-weighted MRIs of the Developing Human Connectome Project (dHCP) [START_REF] Makropoulos | The developing human connectome project: A minimal processing pipeline for neonatal cortical surface reconstruction[END_REF], and provided by the Evelina Neonatal Imaging Centre, London, UK. 40 neonatal data were acquired on a 3T Achieva scanner with the repetition (TR) of 12 000 ms and the echo times (TE) of 156 ms respectively. The size of voxels is 0.5 × 0.5 × 0.5 mm3. In-vivo neonatal LR images has a voxel size of 0.4464 × 0.4464 × 3 mm3.

The pipeline of this application is described as follows: • The HR T2w images of the dHCP dataset are first filtered by a 3D Gaussian kernel with the standard deviation (σ x , σ y , σ z ) calculated as :

         F W HW x = 2 √ 2 ln 2σ x = ST x F W HW y = 2 √ 2 ln 2σ y = ST y F W HW z = 2 √ 2 ln 2σ z = ST z (2.68)
where (x, y, z) is image coordinates, ST denotes the slice thickness of new images.

Concretely, in this case, the slice thickness is calculated as SW x = 0.4464mm, SW y = 0.4464mm, SW z = 3mm. Then, the blurred HR images are downscaled by nearestneighbour interpolation to generate simulated LR images.

• The simulated LR images are then upscaled by the spline interpolation. HR and corresponding interpolated LR patches with the size of 25 in cube are cropped randomly from 40 pairs of the HR and the interpolated LR images with 3200 patches per image.

• A convolutional neural networks with 20 layers, in which the parameters are described in previous sections, learns the mapping between interpolated LR and HR patches.

Once the network learned, the model is stored for the next step. We also test LRTV [START_REF] Shi | Lrtv: Mr image super-resolution with low-rank and total variation regularizations[END_REF] but do not achieve good reconstructions (shown in Figure 2.20 (c)). Note that we do not have the ground truth of real LR data for calculating quantitative metrics. The comparison reveals that the 20-layers CNNs-based proposed method recovers shaper images and better defined boundaries. For example, the cerebrospinal fluid (CSF) of the cerebellum of proposed method in Figure 2.19 is more visible than compared methods. The cortex of 20L-ReCNN method is less blurry than others in Figure 2.21. The ventricle These results confirm qualitatively the efficacy of the approach. In addition, these results could support cortex segmentation due to the visibility of cortex boundaries.

Super-resolution for segmentation

In this section, we would like to verify the contribution of SR to medical image segmentation.

"SR cannot be viewed as an isolated domain." [START_REF] Greenspan | Super-resolution in medical imaging[END_REF]. SR has a strong relationship with image segmentation. Indeed, super-resolution techniques are used to achieve more accurate segmentation maps of brain MRI data [START_REF] Jog | Self super-resolution for magnetic resonance images[END_REF][START_REF] Rueda | Single-image super-resolution of brain mr images using overcomplete dictionaries[END_REF]. In order to evaluate state-of-the-art segmentation algorithms in actual clinical settings with respect to our SR results, we use morphologically adaptive neonatal tissue segmentation (MANTIS)

toolbox [START_REF] Beare | Neonatal brain tissue classification with morphological adaptation and unified segmentation[END_REF] to segment the cortex of MAIA SR T2w images. MANTIS proposes a pipeline which combines unified tissue segmentation and morphological adaptation to segment the neonatal brain. BET method of FSL toolbox [Jenkinson et al., 2012] is used to strip skull before applying MANTIS. Figure 2.22 shows the result of segmentation method MANTIS for spline interpolation and two SR technique: NMU [Manjón et al., 2010b] and our proposed method (20L-SRReCNN). The cortex segmentation within our 20L-SRReCNN is more fully connected than others. The outer boundary of cortex segmentation map of our method is smoother than compared methods.

Although, we do not have the ground truth segmentation maps (with the resolution of 0.5 × 0.5 × 0.5 mm 3 ) of the clinical T2w images, there are the manual segmentations of these subjects with respect to higher-resolution T1w images with voxel size of 0.268 × 0.268 × 1.2 mm 3 from a radiologist. We would like to evaluate the segmentation results with respect to upsampling methods by these higher-resolution manual segmentation maps. Because these T1w images and T2w images are not paired, the estimated segmentation maps are then mapped onto the original T1w images by a rigid registration between HR T2w and T1w data.

A threshold of 0.5 is applied to generate binary segmentation maps. Table 2.4 shows the dice scores of the segmentation method MANTIS on the 2 images of the MAIA testing dataset with respect to different approaches: original T1w images with voxel size of 0.268×0.268×1.2 mm 3 , interpolated T1w images with voxel size of 0.5×0.5×0.5 mm 3 , original T2w images with voxel size of 0.4464 × 0.4464 × 3 mm 3 , upsampling T2 images with voxel size of 0.5 × 0.5 × 0.5 mm 3 using interpolation, NMU and 20L-SRReCNN. The Dice index is described as:

Dice = 2T P 2T P + F P + F N (2.69)
where T P ,F P and F N denote true positive, false positive and false negative between the estimated and the original segmentation. First, the segmentation results from isotropic-resolution T2w images are better than higher-resolution T1w images and isotropic T1w images. Secondly, super-resolution methods, which generate better reconstructions, support more accurate segmentation results. Finally, the segmentation method MANTIS for our estimated HR images shows the best results compared to other approaches. These results come from the fact that our SR method estimates more accurate HR reconstructions.

Conclusion

The section 2.2 investigates CNN-based models for 3D brain MR image SR. Based on a comprehensive experimental evaluation, we would like to draw the following conclusions and recommendations regarding the setup to be considered. We highlight that eight complementary factors may drive the reconstruction performance of CNN-based models. The combination of 1) appropriate optimization with 2) weight initialization and 3) residual learning is a key to exploit deeper networks with a faster and effective convergence. The choice of an appropriate optimization method can lead to a PSNR improvement of (at least) 1dB. In this study, it has appeared that Adam method [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] provides significantly better reconstruction results than other classic techniques such as SGD, and a faster convergence.

Moreover, weights initialization is a very important step. Indeed, some approaches simply do not achieve convergence in the learning phase. This study has also shown that residual modeling for single image SR is a straightforward technique to improve the reconstruction performances (+0.4dB) without requiring major changes in the network architecture. Appropriate weight initialization methods described in [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]Bengio, 2010, He et al., 2015] allow us to build deeper residual-learning networks. From our point of view, these three aspects of SR algorithm are the first to require special attention for the implementation of a SR technique based on CNN.

Overall, we show that better performance can be achieved by learning a 4) deeper fully 3D convolution neural network, 5) adding more filters and 6) increasing filter size. In addition, using 7) larger training patch size and 8) augmentation of training subject lead to increase the performance of the networks. The adjustment of these 5 elements provides a similar improvement (about 0.5dB). Although it seems natural to implement the deepest possible network, this parameter is not always the key to obtaining a better estimate of a highresolution image. Our study shows that, depending on the type of input data (monomodal or multimodal), network depth is not necessarily the main parameter leading to better image reconstruction. In addition, it is necessary to take into account the time of the learning phase as well as the maximum memory available in the GPU in order to choose the best architecture of the network. For instance, for the monomodal SR case based on the simulations of Kirby dataset, we suggest using 20-layer networks with 64 small filters with size of 3 3 regarding 10 training subjects of size 25 3 to achieve practicable results.

In CNN-based approaches, the upscaling operation can be performed by using transposed convolution (so-called fractionally strided convolutional) layers in [Dong et al., 2016b[START_REF] Oktay | Multi-input cardiac image super-resolution using convolutional neural networks[END_REF] or sub-pixel layers [START_REF] Shi | Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[END_REF]. However, the pre-trained weights of these networks are totally optimized for a specified scale factor. This is a limiting aspect of CNNbased SR for MR data since a fixed upscaling factor is not appropriate in this context. In Evaluation of SR techniques is done on simulated LR images. However, one potential use of SR techniques would be to improve the resolution of isotropic data acquired in clinical routine. The Figure 2.23 shows upsampling results on isotropic T1-weighted MR images (the resolution was increased from 1 × 1 × 1mm 3 to 0.5 × 0.5 × 0.5mm 3 ). In this experiment, the applied network has been trained to increase image resolution from 2 × 2 × 2mm 3 to 1 × 1 × 1mm 3 . Although quantitative results cannot be computed, visual inspection of reconstructed upsampled images tend to show the potential of this SR method. No external dataset has been used for these experiences. Thus, features learned at a lower scale (2mm in this experiment) may be used to compute high-resolution images that could be used for fine studies of thin brain structures such as the cortex. Further work is required to investigate this aspect or self-super-resolution [START_REF] Jog | Self super-resolution for magnetic resonance images[END_REF], Zhao et al., 2018] and more particularly the link with self-similarity based approaches [Huang et al., 2015a]. In this thesis, we have proposed a multimodal method for brain MRI SR using CNNs where a HR reference image of the same subject can drive the reconstruction process of the LR image.

By concatenating these HR and LR images, the reconstruction of the LR one can be enhanced by exploiting the multimodality feature of MR data. As shown in previous works [Manjón et al., 2010a[START_REF] Rousseau | Brain hallucination[END_REF], Rousseau et al., 2010a], the use of HR reference can lead to significant improvements of the reconstruction process. However, unlike the monomodal setup, a deeper network does not lead to better performance within the experiments on NAMIC dataset. Experiments from our study show that future work is needed to understand the relationship between network depth and the quality of HR image estimation.

Moreover, we have experimentally investigated the performances of CNN for generalizing on a different dataset ("i.e. how a learned network can be used in another context"). More specifically, our study illustrates how knowledge learned from one MR dataset is transferred to another one (different acquisition protocol and different scales). We have used Kirby and NAMIC datasets for this purpose. Although a slight decrease in performance can be observed, CNN-based approach can still achieve better performance than existing methods.

These results tend to demonstrate the potential applications of CNN-based techniques for MRI SR. Further investigations are required to fully assess the possibilities of transfer learning in medical imaging context, and the contributions of fine-tuning technique [START_REF] Tajbakhsh | Convolutional neural networks for medical image analysis: full training or fine tuning[END_REF].

Finally, future research directions for CNN-based SR techniques could focus on other elements of the network architecture or the learning procedure. For instance, batch normalization (BN)

step has been proposed by [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF]. The purpose of a BN layer is to normalize the data through the entire network, rather than just performing normalization once in the beginning. Although BN has been shown to improve classification accuracy and decrease training time [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], we attempt to include BN layers into CNN for image SR but they do not lead to performance increase. Similar observations have been made in a recent SR challenge [START_REF] Timofte | Ntire 2017 challenge on single image super-resolution: Methods and results[END_REF]. From a geometrical point of view, BN does not appear as an important "operation" for regression [START_REF] Rousseau | Residual networks as geodesic flows of diffeomorphisms[END_REF]. Moreover, while the classical MSE-based loss attempts to recover the smooth component, perceptual losses [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], Ledig et al., 2017[START_REF] Zhao | Loss functions for neural networks for image processing[END_REF] are proposed for natural image SR to better reconstruct fine details and edges. Thus, adding this type of layer (residual block) or defining new loss functions may be beneficial for MRI SR and may provide new directions for research.

In this study, we have investigated the impact of adding data (about 3200 patches per added subject of Kirby dataset) on SR performances through PSNR computation. It appeared that using more subjects sightly improves the reconstruction results in this experimental setting.

However, further work could focus on SR-specific data augmentation by rotation and flipping, which is usually used in many works [Kim et al., 2016a[START_REF] Timofte | Seven ways to improve example-based single image super resolution[END_REF], and intensity variation to handle different contrast and bias field for improving algorithm generalization.

The practical applications of SR are demonstrated in the studies presented: image quality transfer from high-resolution experimental dataset to clinical neonatal low-resolution images and augmenting the performance of segmentation methods. Our CNN-based SR method

shows clear improvements over interpolation, which is the standard technique to enhance image quality from visualisation by a radiologist. SR method is therefore an ideal replacement for interpolation.

Chapter 3

Simultaneous super-resolution and segmentation using a generative adversarial network: Application to neonatal brain MRI

Introduction

Long-term studies of the outcome of prematurely born infants have clearly documented that the majority of such infants may have significant motor, cognitive, and behavioral deficits.

However, there is a limited understanding of the nature of the cerebral abnormality underlying these adverse neurologic outcomes. Magnetic Resonance Imaging (MRI) provides unique opportunities for in vivo investigation of the early developing human brain. However, the analysis of clinical neonatal brain MRI data remains challenging mainly due to low anisotropic image resolution. Improving morphological data processing such as image resolution enhancement and brain segmentation, is a key-point to provide robust morphometry analysis tools to the community.

One of the first key components of the processing pipeline of clinical MRI data is the upsampling image estimation. Super-resolution (SR) is a post-processing technique that aims at enhancing the resolution of an imaging system [START_REF] Greenspan | Super-resolution in medical imaging[END_REF]. SR is a challenging inverse problem; in particular the estimation of texture and details remains difficult. Recently, supervised deep learning-based techniques have shown great improvement over modelbased approaches. In this context, applying 3D convolutional neural networks (CNNs) yields promising results for MRI data [Chen et al., 2018b, Pham et al., 2017a]. However, the use of ℓ 2 -norm loss leads to smooth, unrealistic high resolution images [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], Ledig et al., 2017]. Generative adversarial networks (GANs) have thus been proposed to estimate textured and sharper images [Chen et al., 2018a, Ledig et al., 2017].

Once the high resolution image reconstruction is performed, the implementation of an automatic segmentation robust approach is crucial for fine brain structure analysis [START_REF] Makropoulos | A review on automatic fetal and neonatal brain MRI segmentation[END_REF]. Segmenting thin structures such as the neonatal cortical gray matter remains difficult and the segmentation step is always considered separately from image reconstruction.

In this chapter, we propose an end-to-end GAN-based approach which can generate both the perceptually super-resolved image and a cortical segmentation map from a single lowresolution (LR) image. The proposed approach called SegSRGAN is both assessed on simulated data and real clinical data.

Method

Formulation of single image super-resolution

The objective of single image SR is to estimate a high-resolution (HR) image X ∈ R m from one observed LR image Y ∈ R n . SR problem can be formulated using the following linear observation model:

Y = H ↓ BX + N = ΘX + N (3.1)
where N is the additive noise, B ∈ R m×m is a blur matrix (depending on the point spread function), H ↓ : R m → R n is a downsampling decimation and

Θ = H ↓ B ∈ R n×m (m > n).
A popular approach that solves SR problem defines the matrix Θ -1 as the combination of a restoration operator F ∈ R m×m and an upscaling interpolation operator

S ↑ : R n → R m computing the interpolated LR image Z ∈ R m (Z = S ↑ Y).
In the context of supervised learning, given a set of HR images X i and their corresponding LR images Y i , the restoration operator F can be estimated by minimizing the following loss function:

F = arg min F i X i -F (Z i ) 2 2 . (3.2)
However, it is known that the use of ℓ 2 -norm may lead to oversmoothing high resolution images. In order to provide realistic HR images, perceptual loss function [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] have been used in a GAN [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF]. This is a paradigm shift since it is no longer a question of minimizing only the reconstruction error but of estimating a realistic image, i.e. a high resolution image that corresponds to the observation model with a realistic texture aspect.

A perceptual loss can be formulated as the weighted sum of the content loss (based, e.g., on pixel-wise mean squared error loss) and an adversarial loss component. In GAN-based approaches, the purpose is to train a generating network G that estimates for a given LR input image Y a corresponding HR image G(Y). The goal of the discriminator network D is to classify real images X and simulated HR images G(Y). The game between the generator G and the discriminator D is expressed as an adversarial loss:

L adv = min G max D E X∼P X [logD(X)] + E Y∼P Y [log(1 -D(G (Y)))] (3.3)
where P X and P Y denote the data distribution of X and Y respectively.

Formulation of image segmentation

In this work, image segmentation is viewed as a supervised regression problem:

S X = R (X) (3.4)
where R denotes a non-linear mapping from the upscaled image X to the segmentation map S X . Similarly to the SR problem, assuming that we have a set of images X i and corresponding segmentation maps S X i , a general approach for solving this segmentation problem is to find the mapping R by minimizing the following loss function:

R = arg min R i S X i -R(X i ) 2 2 . (3.5)
Unlike the SR problem, the use of ℓ 2 -norm is less critical as it is expected to estimate smooth segmentation maps.

Joint mapping by generative adversarial networks

We propose the use of a GAN-based approach to estimate jointly a HR image and its corresponding segmentation map from one LR image. To this end, a convolution-based generator network G takes as input an interpolated LR image Z and computes a HR image X and a segmentation map S X by minimizing the following reconstruction loss:

L rec = min G i ρ ((X, S X ) i -G(Z i )) (3.6)
where (X, S X ) i are concatenated along the feature channel. In this work, we use a robust loss as Charbonnier loss [START_REF] Charbonnier | Deterministic edgepreserving regularization in computed imaging[END_REF][START_REF] Lai | Deep laplacian pyramid networks for fast and accurate super-resolution[END_REF] :

ρ(x) = x 2 + ν 2 (3.7)
where ν is set to 10 -3 .

The discriminator network D attempts to distinguish the real data (X, S X ) and the generated ones G (Z). The game between the generator G and the discriminator D is usually modeled with a minimax objective as Equation (3.3).

However, using such loss function, GAN may be unstable or can suffer from mode collapse during training. Thus, in this work, we propose to use Wasserstein GAN loss described in [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF]:

L adv = min G max D E X∼P X ,S X ∼P S X [D((X, S X ))]- E Z∼P Z [D(G (Z))] + λ gp E XS [( (∇ XS D( XS) 2 -1) 2 ] (3.8)
where XS is the interpolation of the true data and the generated one as (1ǫ)(X, S X ) + ǫG (Z), ǫ ∼ U[0, 1]. λ gp and ∇ denote the gradient penalty coefficient and gradient operator, respectively. The images X, S X and Z are extracted randomly from the data distributions of HR images P X , HR segmentation maps P S X and LR images P Z . The terms D((X, S X )), D(G (Z)) and D( XS) are the responses of the discriminator with respect to the real data, the generated data and the interpolated data, respectively. The full objective function is expressed as:

L total = L rec + λ adv L adv (3.9)
where λ adv is a trade-off parameter between reconstruction loss and adversarial loss. Figure 3.1 illustrates our proposed GAN-based method for joint mapping of SR and cortex segmentation.

Architecture of generator and discriminator networks

The generator network (see Figure 3.2 (a)) is a convolution-based network with residual blocks. Let C i j -S k be a block consisting of the following layers: a convolution layer of j filters of size i 3 with stride of k, an instance normalization layer (InsNorm) [START_REF] Ulyanov | Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis[END_REF] and a rectified linear unit (ReLU).

R k denotes a residual block as Conv-InsNorm-ReLU-Conv-InsNorm that contains 3 3 convolution layers with k filters. U k denotes layers as Upsampling-Conv-InsNorm-ReLU layer with k filters of 3 3 and stride of 1. After the last layer, we apply a sigmoid activation for the channel of segmentation map and an element-wise sum of the channel of reconstruction and the interpolated LR image (residual-learning as in [Kim et al., 2016a[START_REF] Pham | Multi-scale brain mri super-resolution using deep 3d convolutional networks[END_REF]).

The generator architecture is:

C 7 16 -S 1 , C 3 32 -S 2 , C 3 64 -S 2 , R 64 , R 64 , R 64 , R 64 , R 64 , R 64 , U 32 , U 16 , C 7 2 -S 1 .
The discriminator network (see Figure 3.2 (b)) contains five convolutional layers with an increasing number of filter kernels, increasing by a factor of 2 from 32 to 512 kernels. Let C k be a block consisting of the following layers: a convolution layer of k filters of size 4 3

with stride of 2 and a Leaky ReLU with a negative slope of 0.01. The last layer C 2 1 is a 2 3 convolution filter with stride of 1. No activation layer is used after the last layer. The

discriminator consists of C 32 , C 64 , C 128 , C 256 , C 512 , C 2 1 .

Experiments and Results

Datasets and network training

To assess the ability to reconstruct HR volume and segment the cerebral cortex , we applied the proposed method on T2-weighted (T2w) MR images of the developing Human Connectome Project1 (dHCP). 40 T2w images were acquired using a 3T Achieva scanner with a 0.5 × 0.5 × 0.5 mm 3 resolution with TR = 12 000 ms TE = 156 ms, respectively. 30 images were used for training networks, whereas the other 10 were used as testing images. As in [START_REF] Greenspan | Super-resolution in medical imaging[END_REF], LR images were generated by using a Gaussian blur with the full-widthat-half-maximum (FWHM) set to slice thickness before a downsampling step to obtain a 0.5 × 0.5 × 1.5 mm 3 resolution.

We have also applied the proposed method onto clinical neonatal MRI data acquired in the neonatology service of Reims Hospital. These LR images have a resolution of 0.446×0.446×3 mm 3 . 40 HR images of the dataset dHCP were filtered and downsampled as in [START_REF] Greenspan | Super-resolution in medical imaging[END_REF] in order to generate LR images with a same resolution as clinical data. The network was trained using 40 pairs of simulated data and then applied to real LR images for visual evaluations. All data had bias correction and for network training, they were normalized between 0 and 1. No subjects nor image patches appear twice in the different subsets.

The 3D network was trained over 200 epochs with batch size of 16, using Adam method with learning rate of 0.0001 and updates the discriminator 5 times before training the generator as in [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF]. The parameters λ adv and λ gp were set to 0.001 and 10 respectively.

The training patch size is 64 3 . At test time, the whole HR image and segmentation volume were reconstructed by the weighted predictions of patches. A thresholding at 0.5 has been performed to obtain binary segmentation maps.

Results

Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) have been used to evaluate the performance of SR reconstructions. Table 3.1 provides a summary of quantitative evaluations for the following methods: cubic spline interpolation, a 20-layers CNN-based SR approach (20L-SRReCNN) [START_REF] Pham | Multi-scale brain mri super-resolution using deep 3d convolutional networks[END_REF] (described in Chapter 2) and our proposed SegSRGAN. It can be seen that 20L-SRReCNN provides highest PSNRs as in [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], Ledig et al., 2017] since this approach minimizes a ℓ 2 -norm-based loss. However, while the two CNN-based approaches (20L-SRReCNN and SegSRGAN) lead qualitatively to similar realistic results on dHCP dataset (see Figures 3.3 and 3.4), the proposed approach provides best reconstructed HR images on clinical data with better contrast on cortical gray matter (see Figure 3.5).

The Dice index is used to evaluate the cortical segmentation maps obtained by the following state-of-the-art methods: iterative multi-atlas patch-based approach (IMAPA) [START_REF] Tor Díez | An iterative multi-atlas patch-based approach for cortex segmentation from neonatal MRI[END_REF], DrawEM [START_REF] Makropoulos | Automatic whole brain MRI segmentation of the developing neonatal brain[END_REF] and the proposed SegSRGAN. As in a typical clinical setting, the three methods have been applied on interpolated images. Table 3.2 shows that quantitatively the proposed approach lead to the best cortical segmentation results with significant improvement with respect to the two other methods. Moreover, as mentioned in [START_REF] Tor Díez | An iterative multi-atlas patch-based approach for cortex segmentation from neonatal MRI[END_REF], the use of IMAPA applied on original HR dHCP images leads to a mean DICE of 0.887 (standard deviation of 0.011) that is very similar to the results obtained with SegSRGAN (applied on interpolated images).

As indicated in Section 2.2.3.2, we would like to evaluate the impact of upsampling methods for clinical LR T2w images with respect to segmentation methods. There are the manual HR segmentations of T1w images (ground truths). The estimated segmentation maps applied to SR results are mapped onto the original T1w images by a rigid registration between estimated HR T2w and original T1w data. Table 3.3 shows the segmentation results of the method MANTIS [START_REF] Beare | Neonatal brain tissue classification with morphological adaptation and unified segmentation[END_REF] for HR reconstructions of upsampling methods following: interpolation, NMU, 20L-SRReCNN and our SR results of SegSRGAN. The mean dice of the segmentation maps of MANTIS for our estimated HR image is better than the ones of compared upsampling methods. Moreover, we apply the supervised segmentation method (IMAPA) [START_REF] Tor Díez | An iterative multi-atlas patch-based approach for cortex segmentation from neonatal MRI[END_REF] for the estimated isotropic T2w images using above upsampling methods (show in Table 3.3). Our proposed method uses the same training dataset of segmentation atlases as IMAPA. Table 3.3 shows that our SR results support other segmentation methods better than compared SR methods. In addition, our segmentation results also give comparable dice scores as the pipeline of 20L-SRReCNN and IMAPA. Results on real LR data (see Figures 3.5 and 3.6,Tables 3.3 and 3.4) confirm the potential of the proposed approach for fine analysis of clinical neonatal brain MRI. 3.4: Dice scores of the supervised segmentation method IMAPA (using the same training dataset with our method, the same segmentation protocol) on the 2 images of the MAIA testing dataset with respect to different approaches (columns): interpolated T2w images, upsampling T2 images using NMU, 20L-SRReCNN and our SR results (SegSR-GAN), and our proposed segmentation map of interpolated T2w images (SegSRGAN).

Discussion

In this chapter, we have presented a simultaneous super-resolution and segmentation method for 3D brain MR images using a generative adversarial network. Our experiments on both simulated and clinical data have shown that better performance can be achieved by this joint approach compared to state-of-the-art techniques, opening up new perspectives in the processing of clinical LR neonatal brain MRI data.

We have investigated that our proposed GAN-based method is more robust than the CNNbased approach. The CNN-based method achieves the highest PSNR/SSIM because it attempts to minimize the pixel-wise difference between super-resolved images and reference HR images using ℓ 2 -norm cost function. This is reasonable as we have presented in Chapter 2. However, CNN-based methods are restricted to the predetermined condition of specific training data and their performance is then decreased when testing real images, where these conditions are not satisfied (also mentioned in [START_REF] Shocher | Zero-shot" super-resolution using deep internal learning[END_REF]). Meanwhile, the SR method using GAN attempts to minimize the difference of the texture between generated images and ground truth HR counterparts using the adversarial loss. This loss makes networks more robust to simulated training data. Future work is required to explore new quality metrics to evaluate better the performance of SR methods.

Our proposed method illustrated that the learned model from high-resolution experimental dataset can be transferred successfully to another low-resolution clinical dataset in order to enhance the image quality. We have used dHCP and MAIA dataset for this purpose. These results demonstrate the potential of GAN-based techniques for practical applications of medical image processing. We believe that our proposed approach can be used to another tasks such as medical image synthesis or other types of segmentation maps such as cerebrospinal fluid or ventricles in brain MRI.

In this study, we assume the paired training dataset, where input images have output counterparts (e.g. LR images and corresponding HR images). In some clinical cases, paired couples are not always available (e.g. T2w images with a specific resolution and the segmentation maps of T1w images with another resolution), that raises the question of self-supervised techniques for mapping of unpaired training dataset.

Introduction

There are many medical imaging modalities in the clinical context such as: radiography, magnetic resonance imaging (MRI), computed tomography (CT) scan, ultrasound. Each modality shows up the physical properties of tissue in organs and special abnormalities for detecting different diseases. The diversity of medical image modalities is useful for diagnosticians but can be a challenge for automated image analysis. In clinical scenarios, the number of tissue contrasts that can be acquired is limited because of time consuming or expensive cost. Collecting all medical images of one subject is impractical. Cross-modal synthesis without real acquisition is considered as an intensity transformation applied to given input images of a source modality to generate new images with a specific tissue contrast. Synthetic images are not intended to be used for diagnostic purposes. Synthesis of a medical image can be used for a preprocessing step before applying more complex image processing algorithms. The objective of cross-modal synthesis is to generate images that are close enough approximations to real images so as to improve automated image processing. Cross-modality synthesis of medical images is proposed for many application such as segmentation [START_REF] Iglesias | Is synthesizing mri contrast useful for inter-modality analysis?[END_REF][START_REF] Roy | Mr contrast synthesis for lesion segmentation[END_REF], super-resolution [START_REF] Pham | Multi-scale brain mri super-resolution using deep 3d convolutional networks[END_REF][START_REF] Rousseau | Brain hallucination[END_REF][START_REF] Rueda | Single-image super-resolution of brain mr images using overcomplete dictionaries[END_REF],

and multimodal registration [START_REF] Roy | Magnetic resonance image example-based contrast synthesis[END_REF][START_REF] Wein | Automatic ctultrasound registration for diagnostic imaging and image-guided intervention[END_REF]. The thesis in [START_REF] Cordier | Multi-atlas patch-based segmentation and synthesis of brain tumor MR images[END_REF] shows a review of the annotated data, which can be used to augment the performance of medical image analysis methods for pathological cases. A statistical model of cross-modal synthesis can be expressed as: where R is a mapping, Y and X denote images of source and target domains and N is an additive noise. Despite of paired images, the relation between the T1w and T2w tissue contrasts is totally non-linear as several regions share the opposite gradients but some regions are otherwise.

Y = RX + N (4.1)
One T2w intensity can be transformed from multiple T1w intensities and vice versa.

Paired cross-modal synthesis

The synthesis techniques have been studied in the context of medical imaging analysis using joint histogram [START_REF] Kroon | Mri modalitiy transformation in demon registration[END_REF]. Given a dataset with the coupled images of source domain and target domain {(Y i , X i )}, the patch-based synthesis method [START_REF] Iglesias | Is synthesizing mri contrast useful for inter-modality analysis?[END_REF] finds k-nearest neighbor patches y k in the training base of the patch y of observed image Y as:

( k, ŷk ) = argmin k,y k ∈{Y i } y k -y 2 (4.2)
When the paired set {ŷ k , xk } is found, the synthesized patch is the average of k optimal patches xk as: x = k xk . The patch-based method is improved by the iterative approach [START_REF] Ye | Modality propagation: coherent synthesis of subject-specific scans with data-driven regularization[END_REF] as:

( kt+1 , ŷt+1 k ) = argmin k,y t k ∈{Y i },x t k ∈{X i } (1 -α) y t k -y t 2 + α xt k -xt 2 (4.3)
where, xt is the synthesized image by the optimal corresponding patches xt k at the t th iteration, α denotes the trade-off between two terms. Instead of ℓ 2 -norm patch-based approaches, the regression tree method is proposed for synthesis MRI contrasts in [START_REF] Jog | Magnetic resonance image synthesis through patch regression[END_REF] to find the complex mapping between modalities. Similarly, the improved versions of this technique as random forest decision can be found in [START_REF] Huynh | Estimating ct image from mri data using structured random forest and auto-context model[END_REF][START_REF] Jog | Random forest flair reconstruction from t 1, t 2, and p d-weighted mri[END_REF][START_REF] Han | Mr-based synthetic ct generation using a deep convolutional neural network method[END_REF]. In parallel, [START_REF] Roy | Magnetic resonance image example-based contrast synthesis[END_REF], Ye et al., 2013] adapts the sparse-coding-based methods for SR as in [START_REF] Yang | Image super-resolution via sparse representation[END_REF] for synthesis MRI contrasts, assuming jointly dictionaries for T1w and T2w MR images. The multi-layer neural network for cross-domain synthesis is first proposed in [START_REF] Van Nguyen | Cross-domain synthesis of medical images using efficient location-sensitive deep network[END_REF] (LSDN) for mapping the intensity feature and the spatial coordinates from the input domain to the intensity of target domain as:

min Φ(F Y i , P Y i ) -X i 2 (4.4)
where Φ represents the network, F and P denotes intensity-based feature extractions and spatial informations respectively. Instead of pooling layers for spatial-based voxel connections as CNNs, the method LSDN proposes multiplication operations between layers and the shrinking connection at each layer for reduced the computation cost. The 2D U-net architecture [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] (shown in Figure 4.3) is applied to generate CT from discontinuous MRI slices in [START_REF] Han | Mr-based synthetic ct generation using a deep convolutional neural network method[END_REF]. [START_REF] Nie | Estimating ct image from mri data using 3d fully convolutional networks[END_REF] proposes to use fully 3D fully convolutional neural networks inspired by [Dong et al., 2016a] for reconstructing CT scans from MRI volumes. The improved versions of this network in [START_REF] Nie | Medical image synthesis with context-aware generative adversarial networks[END_REF][START_REF] Nie | Medical image synthesis with deep convolutional adversarial networks[END_REF] exploit image gradient difference loss and adversarial loss with auto-context model. The process of auto-context model (shown in Figure 4.4) is to refine the synthesized images via an iterative process between input images and the estimated images at each iteration using different training models. Recently, the work in [START_REF] Xiang | Deep embedding convolutional neural network for synthesizing ct image from t1-weighted mr image[END_REF] synthesizes the MRI consecutive axial slices into CT scans using 2D embedding CNNs in which the reconstruction stage and the transform stage are concatenated. The stages are expressed as:

   F tran,i = P ReLU (W tran,i ⋆ F tran,i-1 + B tran,i )

F rec = W rec ⋆ F tran,j + B rec (4.5)
where F rec and F tran,i denotes the estimated synthesis and the response of the i th layer respectively. W tran,i ,B tran,i , W rec and B rec are network parameters and ⋆ denotes convolution operation. Then, an embedding block is defined as a concatenation of these two stages before a transform stage which maintains the number of response at each layer. et al., 2018] 4.1.2 Unpaired cross-modal synthesis

Coupled training set of one subject is not always available. Unpaired synthesis methods have recently investigated in [Huang et al., 2017b, 2018, Vemulapalli et al., 2015]. The method in [START_REF] Vemulapalli | Unsupervised cross-modal synthesis of subject-specific scans[END_REF] proposes 3 steps to handle unpaired data synthesis. Firstly, a set of patch-based nearest neighbour candidates of the source image is generated using mutual information M I() of the source patches y and the target patches x as:

M I(y; x) = H(x) -H(x) + H(y | x) (4.6)
where H(x) and H(y) are the marginal entropies and H(y | x) denotes the conditional entropy. The second step attempts to synthesis the source image Y using best candidates by maximizing the cost function as:

max w vk H(X) -H(X) + H(Y | X) + λSC(X, Y) s.t. k w vk = 1, v ∈ V (4.7)
where SC(X, Y) is a regularization term that promote spatial consistency between the neighbour candidates and V denotes two neighboring voxels. Finally, coupled sparse representation of source modality image and the synthesized target modality image is calculated to refine the result of the preceding steps as super-resolution problems [START_REF] Wang | Deep networks for image superresolution with sparse prior[END_REF][START_REF] Yang | Image super-resolution via sparse representation[END_REF][START_REF] Zeyde | On single image scale-up using sparse-representations[END_REF] as:

α = argmin α α 1 s.t. D y α -y 2 2 ≤ ǫ 1 , D xα -x 2 2 ≤ ǫ 2 (4.8)
where D x and D y denote the joint dictionaries for the patches of synthesized and source domains and α is the sparse support. Instead of learning dictionaries of synthesized and source images as [START_REF] Vemulapalli | Unsupervised cross-modal synthesis of subject-specific scans[END_REF], improved sparse coding methods based on target/source images for synthesis with convolution representation and regularizations can be found in [Huang et al., 2017b[START_REF] Huang | Cross-modality image synthesis via weakly coupled and geometry co-regularized joint dictionary learning[END_REF]. The work in [START_REF] Huang | Cross-modality image synthesis via weakly coupled and geometry co-regularized joint dictionary learning[END_REF] proposes to synthesize MRI contrasts using sparse representations and two regularizers as maximum mean discrepancy and geometry preservation based on a few pairs of data. The unpaired couple dictionaries of target and source domain are learned from the sparse representation as:

min α,D L(α, D, y, x) = min α,D D y α y -y 2 2 + D x α x -x 2 2 + λ α y 1 + λ α x 1 +τ F(α y , α x ) + γM M D(α y , α x ) + µGeo(α y , α x ) (4.9)
where D x is now the dictionary of target domain, α = {α y , α x } denotes the sparse code, and M M D and Geo denotes maximum mean discrepancy regularization and geometry coregularization. In order to ensure the identity of the sparse codes from the source to the target, we assume the linear projection in the common feature space via a mapping function as F(α y , α x ). However, the method needs the pair training data to constraint the unpaired image data by the fact that they must share the same high-frequency features. Equation where F HF is the high-frequency feature extractor and T denotes the binary matrix which consists of one element of 1 and other set to be 0. The 1 element is set to the maximum value of an affinity matrix which consists of measured distances of paired patches.

Discussion

A brief review of cross-modal synthesis for medical imaging has been described. The learningbased methods such as patch-based techniques, sparse coding, random forest and CNN-based are commonly used for paired cross-modal synthesis. In the context of supervised learning, techniques proposed for image synthesis have the same point of view as example-based learning SR methods. The availability of paired modalities of the same subject is sometimes lacking. Unpaired cross-modal synthesis are proposed to overcome this disadvantage.

However, the need of few paired training images is inevitable for the refinement of synthetic results. In the next sections, our CNN-based methods for SR is applied to paired image synthesis. Mostly, we attempt to propose an approach to totally unpaired MRI synthesis using generative adversarial networks.

4.2 Supervised synthesis with convolutional neural networks

Mathematical formulation

In the context of supervised learning, assuming that a training data set which consists of pairs of images in a source modality Y j (e.g. T1w images) and the corresponding images in the target modality X j (e.g. T2w images), our objective is to find the mapping f that optimize the cost function as:

f = arg min f j ρ(f (Y j ) -X j ) (4.11)
where ρ can be ℓ 1 or ℓ 2 -norm for instance. The convolution neural networks, which are described in the chapter 2, are directly applied to solve our synthesis problem. The mapping f from Y i to the residual (X j -Y j ) is decomposed into nonlinear operations as the combination of convolutional layers with the ReLU activation as:

f = arg min f j f (Y j ) -(X j -Y j ) 2 (4.12)
Residual learning strategies make the convergence of CNNs faster. In principle, residual connections induce the responses of layers to be close to zeros, making the network easier to train. The interest of residual learning is also proposed in [START_REF] Nie | Medical image synthesis with deep convolutional adversarial networks[END_REF]. The architecture of our networks can be described as follows:

     f 1 (Y) = max(0, W 1 * Y + B 1 ) f i (Y) = max(0, W i * F i-1 (Y) + B i ) f or 1 < i < L f L (Y) = W L * F L-1 (Y) + B L (4.13)
where L is the number of layers. Once the training step is done, the synthesized image of a given image is estimated as X = f (Y) + Y.

Dataset and training parameters

We use T1w and T2w MR images of NAMIC Brain Multimodality 1 to assess the ability of our CNN-based method (20L-SRReCNN). These data have been acquired using a 3T GE.

The T1w images are acquired in contiguous spoiled gradient-recalled acquisition (fastSPGR)

with the following parameters: TR=7.4ms, TE=3ms, TI=600, 10 degree flip angle, 25.6cm 2 field of view, matrix=256×256. The contiguous T2-weighted images are acquired with the following parameters: TR=2500ms, TE=80ms, 25.6 cm 2 field of view, 1 mm slice thickness.

Voxel dimensions of these images are 1 × 1 × 1mm 3 .

We use a series of 19 convolution layers of 3 × 3 × 3 with 64 filters and the ReLU activations.

The last layer is a 3 × 3 × 3 convolution layer with one filter. ADAM method is used to optimize the network with 20 epochs (batch size of 64).

Experimental results

REPLICA [START_REF] Jog | Random forest regression for magnetic resonance image synthesis[END_REF] In this section, we study performances of the proposed CNN architecture of SR for supervised synthesis of subject-specific scans. The baseline methods for comparison are random forest regression for synthesis [START_REF] Jog | Random forest regression for magnetic resonance image synthesis[END_REF] (REPLICA). The metrics PSNR and SSIM with respect to normalized results between 0 and 1 are used to evaluate the methods. The quantitative results are shown in Table 4.1. Our method has a gain of about 2.3dB (synthesizing T1w images from T2w images) and 2.7dB (synthesizing T2w images from T1w images) with respect to PSNR compared to REPLICA. Although, our CNN-based approach has a greater SSIM when synthesizing T1w images from T2w images but lower SSIM when synthesizing T2w images from T1w images than those of the random forest-based method.

The qualitative results are shown in Figures 4.6

Discussion

We have introduced an approach to synthesize one image of a target modality from an observed image of a source modality. Although the results are sensible to the noise of data, our proposed method could also generate synthesized images which are comparable to the baseline method. Our approach shows the potential of CNN-based technique for cross-modal medical image synthesis problem. Evaluation of cross-modal techniques is done on the original images. However, the results

show blurry and noisy. The pre-processing steps such as pre-denoising and bias correction should be applied before training networks. In addition, further works is required to investigate the principal elements of networks with respect to the performance of networks such as:

depth of networks, non-residual-learning, the size of filters. Thus, a general pipeline will be drawn based on these works.

Future research direction for supervised learning CNN-based cross-modal techniques could focus on the other networks such as generative adversarial network [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], Nie et al., 2018[START_REF] Yang | Mri image-toimage translation for cross-modality image registration and segmentation[END_REF] or on other elements of the network architecture as the embedded networks [START_REF] Xiang | Deep embedding convolutional neural network for synthesizing ct image from t1-weighted mr image[END_REF]. Moreover, the perceptual losses of SR problem as in [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], Ledig et al., 2017] can be applied to reduce the noisy data and generate the naturally synthesized images.

Unpaired synthesis with generative adversarial networks

Recently, image-to-image translation using GANs have been receiving significant attention from research community [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], Kim et al., 2017[START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017]. Recent work learns this task in an unpaired learning manner [START_REF] Kim | Learning to discover cross-domain relations with generative adversarial networks[END_REF][START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017]. For instance, an architecture with two-block GANs and a connection based ℓ 1 -norm cycle-consistent loss has been investigated for translating unpaired images [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF]. Another work similar to [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] but with ℓ 2 -norm cycle-consistent loss has also proposed in [START_REF] Kim | Learning to discover cross-domain relations with generative adversarial networks[END_REF]. A concurrent work [START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF] with the same approach as cycleGAN has improved the stability of GANs but using

Wasserstein GAN [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] instead of sigmoid cross-entropy loss used in the original GANs. However, all these frameworks do not take advantage of the shared features between modalities. In this section, we propose an approach to use GANs for unpaired medical image synthesis.

Mathematical formulation

In this work, we propose an unsupervised learning technique for cross-modal synthesis of brain MRI scans using generative adversarial networks. Our GAN consists of one single discriminator and one single generator. Given the training dataset of unpaired images, we assume the "class" of each tissue contrast corresponding to a non-negative integer number (e.g. T1w images are the class "1" and T2w images are the class "2"). Our networks use the embedding techniques described in [START_REF] Gal | A theoretically grounded application of dropout in recurrent neural networks[END_REF] to turn integer indexes into dense vectors of the fixed size of class numbers : class → c. The embedding classes make networks easier to train [START_REF] Gal | A theoretically grounded application of dropout in recurrent neural networks[END_REF]. In addition, they can separate different classes but learn joint mappings into the same network. The training step of the discriminator is independent to those of the generator. Thus, we need one embedding layer for each network in order to it can minimize its weights independently.

We express the objective of this adversarial learning as: Here, the generator G tries to minimize this cost function while the discriminator D tries to maximize it. When the discriminator gets to local optimal, the use of logarithm term in the adversarial loss is equivalent to minimizing the Jenson-Shannon divergence [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] between the real images and the synthetic images. If the real images and the synthetic images share no support, Jenson-Shannon divergence implied by Equation (4.14) becomes saturated (i.e. a constant). Optimizing Equation 4.14 suffers from due to the gradientvanishing effect. And sometimes, the model tends to the collapsing mode [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF][START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Salimans | Improved techniques for training gans[END_REF]. In order to effectively avoiding the mode collapsing problem, [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] propose to replace the logarithm cross-entropy loss by Wasserstein distance (WGAN) as:

L adv = E xtar,c D,
L adv = -E xtar,c D,tar [D(x tar , c D,tar )] + E xsrc,c D,tar ,c G,tar [D(G(x src , c G,tar ), c D,tar ))] s.t. D L ≤ 1 (4.15)
where D L ≤ 1 denotes 1-Lipschitz constraint. Here, the discriminator D becomes a "critic" because it does not distinguish the true and the generated but attempts to minimize the differences between them. In order to implement this constraint, [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] use the weight clipping method for the discriminator. However, the value of the clipping threshold, which affects the interactions between the weight constraint and the cost function, is hard to choose. An inappropriate value can induce either vanishing or exploding gradients.

An improved version of WGAN proposes to use gradient penalty [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] for Equation 4.14 as:

L adv = -E xtar,c D,tar [D(x tar , c D,tar )] + E xsrc,c D,tar ,c G,tar [D(G(x src , c G,tar ), c D,tar ))] + λ gp E x[ (∇ xD(x, c D,tar ) 2 -1) 2 ] (4.16)
where x = ǫx src +(1-ǫ)G(x src , c tar ) denotes the interpolation between the real image and the generated image with a random number ǫ ∼ U [0, 1], and λ gp controls the importance between the objectives. [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] investigate that the optimal critic has unit gradient norm almost everywhere under the data distribution of the true images and the generated images. Intuitively, the true image and the optimally generated image should share the same gradient. Thus, this procedure makes the networks easily to train.

Cycle Consistency Loss

The adversarial loss does not guarantee that learned mappings can induce a generated image that matches exactly the target image because the networks map the same set of input images to any random images in the target domain [START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017]. Equation (4.16) only optimizes the mapping between the domain part between domains. Thus, a cycle consistence loss [START_REF] Kim | Learning to discover cross-domain relations with generative adversarial networks[END_REF][START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017] is applied to the generator to preserve the identical content of the input images and the translated one, described as:

L cyc = E xsrc,c G,tar ,c G,src [ x G,src -G(G(x src , c G,tar ), c G,src ) l ] (4.17)
where c G,src is the embedded source domain label and l is a norm. The generator translates the input images into the output images and then reconstructs from translated ones to the original images. [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] investigate that the use of ℓ 1 -norm is better than ℓ 2norm for training GANs because ℓ 1 -norm encourages less blurring. Thus, [START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF][START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017] propose the cycle consistence loss with ℓ 1 -norm so as to encourage low-frequency correctness. In this work, we propose to use a robust Charbonnier loss function (a differentiable variant of ℓ 1 -norm) ρ(x) = x 2 + ǫ 2 ρ (ǫ ρ is set to 1e -3), which achieves a better high-quality reconstruction than ℓ 2 -norm such as in the SR problem [START_REF] Lai | Deep laplacian pyramid networks for fast and accurate super-resolution[END_REF]. The cycle loss can be now expressed as:

L cyc = E xsrc,c G,tar ,c G,src [ρ (x src -G(G(x src , c G,tar ), c G,src ))] (4.18)

Total Variation Loss

The output image y tar may be generated with high-frequency artefacts, which are remarked in several GAN methods [START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF][START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017]. The TV regularizer has been investigated in the neural style transfer domain [START_REF] Gatys | Image style transfer using convolutional neural networks[END_REF][START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], super-resolution [START_REF] Gatys | Image style transfer using convolutional neural networks[END_REF][START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] and feature inversion [START_REF] Mahendran | Understanding deep image representations by inverting them[END_REF]. Thus, we apply total variation (TV) regularization to encourage spatial smoothness of the output. The TV loss for a 3D output is described as:

L T V (y) = i,j,k (y i,j,k+1 -y i,j,k ) 2 + (y i,j+1,k -y i,j,k ) 2 + (y i+1,j,k -y i,j,k ) 2 β 2 (4.19) 4.3.1.4 Full Objective
Our full objective is described as:

L total = L adv + λ cycle L cycle + λ T V L T V = -E xtar,c D,tar [D(x tar , c D,tar )] + E xsrc,c D,tar ,c G,tar [D(G(x src , c G,tar ), c D,tar ))] + λ gp E x[ (∇ xD(x, c D,tar ) 2 -1) 2 ] + λ cycle E xsrc,c G,tar ,c G,src [ρ (x src -G(G(x src , c G,tar ), c G,src ))] + λ T V L T V (G(x src , c G,tar )) (4.20)
where λ cycle and λ T V is parameters which denote the significance of the reconstruction process and the TV regularization. In summary, the generator G is aimed at generating the synthetic images G(x src , c G,tar ), which are as similar as possible to the images of a target domain

Network training

The 3D network is trained over 20 epochs on GPU Titan X using Keras with batch size of 1.

Training uses Adam method [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with learning rate of 0.0001 and updates the discriminator 5 times before training the generator as in [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF]. When updating the generator, we freeze the weights of the discriminator. In our experiments, we set λ gp = 10, λ cyc = 5000. At test time, the whole synthesized image is reconstructed by the weighted predictions of patches of the testing image.

Experimental results

We show our qualitative results in Figures 4.10 and 4.11. Although, our approach is an unsupervised learning method, it can generate the synthetic images which have the same contrast to the ground truth images. The CSF, white matter and gray matter regions are reconstructed as close as the ground truth. Synthesized images using our proposed unsupervised approach capture most of the structural information. The quantitative results are illustrated in Table 4.2. We use PSNR/SSIM metrics to evaluate our method with respect to TV regularization (λ T V ∈ {0, 0.001, 0.005}) and compared to the supervised learning method 20L-SRReCNN (described in Section 4.2). When synthesising T1w images from T2w images, the proposed GAN-based methods have comparable results as 20L-SRReCNN. However, our unsupervised method shows worse PSNR/SSIM than the supervised method. The reason relies on the fact that the GAN-based method attempts to optimize two mappings inside one single network at the same time while two networks 20L-SRReCNN are trained for one mapping.

The role of TV regularization is crucial to our method. TV regularization induces better results than whose of no TV parameter. The lower λ T V = 0.001 leads to higher PSNR but lower SSIM than λ T V = 0.005. The illustration of the role of TV regularization is shown in

Discussion

In this section, we proposed a general unsupervised GAN-based method for cross-modal synthesis of subject-specific scans. Our method works without paired training data from source and target domains. Although the results are a little blurry, the technique shows an approach to solve our cross-modal synthesis problem without the paired dataset. Our method is effective for joint training between different domains thanks to the embedded labels of the target/source domains. We demonstrate that our conditional GAN can learn mappings between multiple MRI tissue contrasts.

We investigate that better performance can be achieved by adding total variation regularization. Our study shows that a small trade-off parameter for this regularizer is enough to generate better results. A higher parameter can induce too smooth reconstructions. The investigation of other regularizations will be considered in the future works such as ℓ 2 -norm or Gaussian kernel.

The future research directions for GAN-based unpaired cross-modal synthesis could focus on applying the synthetic images for CNN-based SR techniques. As shown in [START_REF] Pham | Multi-scale brain mri super-resolution using deep 3d convolutional networks[END_REF]] that the multimodal method for brain MRI where a HR reference image can leverage the SR results. We believe that the synthetic HR images instead of using the original HR contrast can lead to better results.

Another application of cross-modal synthesis is to support segmentation methods. [START_REF] Leroy | Atlas-free surface reconstruction of the cortical grey-white interface in infants[END_REF] shows that the cortex segmentation of neonatal brain can be drawn from T2w images. However, the resolution of T2w images is usually lower than the corresponding T1w images of the same subject. Unpaired cross-modal synthesis can generate the T2w synthetic images from the HR T1 images. Then, segmentation methods may apply to these HR synthetic T2w images for cortex segmentation.

"La prochaine révolution de l'IA sera non-supervisée"-Yann LeCun (2 ). More improved GANs technique are growing up. In the future work, we will study more GAN-based techniques (e.g.

Fisher GAN [START_REF] Mroueh | Fisher gan[END_REF]) or other new layers of networks for better synthesis reconstruction. Kernel methods for GAN in [START_REF] Zhang | Adversarial feature matching for text generation[END_REF] would allow to improve the performance of adversarial networks.

Chapter 5 The first contribution presented, relies on the investigated of 3D convolutional neural networks for brain MRI super-resolution instead of classic 2D networks. Then, several principal elements of networks are analysed to improve the performance such as the optimization methods, the depth of networks, weight initialization schemes, residual learning, multiscale learning. Next, an approach to take advantage of another HR reference image for improve the MRI super-resolution process is proposed. Finally, the application of super-resolution for enhancing the real clinical neonatal brain MRI and supporting segmentation methods is investigated, which demonstrates our proposed networks with respect to practical medical imaging applications.

Conclusions and Perspectives

The second contribution described an approach for joint mappings of high-resolution reconstruction and segmentation using 3D generative adversarial networks. This method is not only assessed on the simulated low-resolution images of the high-resolution neonatal dataset, but also used to enhance and segment real clinical anisotropic low-resolution images. Our results demonstrate the potential of our GAN-based method with respect to practical medical applications.

The third contribution proposes 3D convolutional neural networks for paired cross-modal synthesis and 3D generative adversarial networks for unpaired cross-modal synthesis. Our CNN-based network for SR applied directly to cross-modal synthesis shows comparable results

to the start-of-the-art methods. Moreover, we propose an approach to exploit 3D generative adversarial networks for unpaired cross-modal synthesis. The results of our unsupervised method are encourage. Further improvements of generative adversarial networks are required to improve the performance.

Although, several researchers have proposed many methods to solve these two problems, many challenges still constraint these techniques from wide applications. Firstly, handling a huge number of training examples or complicated models can be induce computational cost. The methods such as CNNs depend on GPU for accelerating the intensive computation. Secondly, the observation model with a given point spread function can not be estimated perfectly, leading the sensitivity of techniques with outliers or the dissatisfaction of ideal conditions.

Finally, the metrics such as PNSR or SSIM may not induce more appealing results. Better measures or qualitative results are still needed for performance evaluation. Thus, future researches continue to investigate better methods and more performance evaluation for the developments of these two applications.

Perspectives

In this thesis, we considered MRI contrasts (T1w and T2w). The addition of other contrasts such as FLAIR or dMRI or other modalities such as CT would be used to investigate the robustness of neural networks-based techniques. In [START_REF] Alexander | Image quality transfer and applications in diffusion mri[END_REF], image quality transfer (IQT) propagates information from rare or expensive high quality dMRI images to abundant or cheap low quality dMRI images by machine learning technique. The method raises the question of the potential of CNN in dMRI SR. We believe that our proposed cross-modal synthesis can be used to generate MRI brain scans from CT and vice verso, or diffusion-weight MRI or from low dose to high dose CT scan.

When applying CNN-based methods in a realistic setting, the choice of PSF is crucial. Thus, the second future direction would involve blind SR [START_REF] Michaeli | Nonparametric blind super-resolution[END_REF]Irani, 2013, Wang et al., 2005] instead of a simulated PSF so as to approximate better the PSF of observed LR images. On the other hand, the perceptual approaches can also be used to make the network independent from the PSF model.

The objective function of neural networks is based on the differences between pixel-wise or voxel-wise. Thus, this may lead to lack of texture information inside images. The future work where x, f neu (x) ∈ R, g is an activation function, W is a weight and B is a bias. W • x denotes the dot product. For sake of clarity, we denote x and f (x) as the argument (or input) and a model (or a function) respectively. The perceptron attempts to find the weight and bias, which approximate the relation of given input and correspond output. The space of solution of a perceptron is limited in the set of linear separability. In order to extend the solution space, a multilayer perceptron (MLP) connects several perceptrons for higher dimension of x and f (x) with hidden layers. A MLP with one hidden layer can be denoted as:

f M LP (x) = g 2 (W 2 • g 1 (W 1 • x + B 1 ) + B 2 ) (A.
f CN N (x) = g 2 (W 2 ⋆ g 1 (W 1 ⋆ x + B 1 ) + B 2 ) (A.3)
where ⋆ denote a convolution operation, W i , B i denote the weight and the bias of the i th convolution layer. Mathematically, a convolution, which is a weighted sum of each element of the input to its local neighbors by filters, between an image I and a filter F can be written as: .4) where I ∈ R M,N , F has a size of m × n and I F denotes the convoluted image. Depending on our tasks, we could add more types of layers such as pooling layers, transposed convolution layers, embedding layers [START_REF] Gal | A theoretically grounded application of dropout in recurrent neural networks[END_REF], batch normalization layers [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] or sub-pixel layers [START_REF] Shi | Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[END_REF], etc.

I F (k, l) = m i=1 n j=1 I(i + k -1, j + l -1)F (i, j) (A
CNNs have been first studied in [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition[END_REF]Miyake, 1982, LeCun et al., 1998]. Never- has improved the performance of the predecessor. In parallel, InceptionNet, which consists of several blocks of the concatenation of filters and pool layers [START_REF] Szegedy | Going deeper with convolutions[END_REF], has more performance than VGG-nets. Several architectures of CNNs have been proposed to many computer vision tasks. In order to generalize CNNs to object detection, the work in [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Ren | Faster r-cnn: towards real-time object detection with region proposal networks[END_REF] (R-CNN) aims to identify objects via a bounding box in the image.

Intuitively, the hyperbolic tangent function maintains the linear relationship of the argument in the range of [-1, 1] and it saturates if otherwise. In addition, the property of this function makes gradient-based optimization methods easily calculate the derivatives. Afterward, an architecture CNN up to 100 layers has been proposed to use the residual blocs (Resnet) [He et al., 2016a]. A residual bloc [START_REF] He | Identity mappings in deep residual networks[END_REF] draw in Figure A.4, of the i th and (i + 1) th convolution layer can be : .11) where BN i is the batch normalization (BN) function [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] of i th layer.

f Res (x) = g i+1 (BN i+1 {W i+1 ⋆ g i [BN i (W i ⋆ x)]} + x) (A
When the parameters of training networks are optimized, the distribution of activation functions is changed, leading to internal covariate shift. BN layers propose to normalize the response of convolution layers to produce activations with a stable distribution as: .12) where x i denotes values of i th batch of input x over the mini-batch m (i = 1, ..., m), µ B and σ B are respectively the average and variance of the mini-batch and ǫ is a constant. β and γ are respectively learned scale and shift parameters of the layer to ensure network to avoid forward or backward signals vanish [He et al., 2016a]. Later, Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] supplements R-CNN the possibility of object segmentation with Resnet. Recently, [Huang et al., 2017a] have proposed to connect all layers in a block or in the networks called Densenet. These latest networks not only decrease number of parameters but also show good performance as the "ultra deep" Resnet. An example of densely connected block is illustrated in Figure A.5. Densenet proposes to concatenate all preceding layers

BN (x i ) = γ x i -µ B σ 2 B + ǫ + β (A
x 1 , x 2 , ..., x i for the (i + 1) th layer as: .13) where Concat denotes the concatenation of feature-maps of all preceding layers. W i+1 and B i+1 are the parameters of the (i + 1) th layer. The intuition of this approach is that each layers share all feature maps as "collective knowledge" [Huang et al., 2017a].

f den,i+1 (x) = g i+1 (BN i+1 [W i+1 ⋆ Concat(x 1 , x 2 , ..., x i ) + B i+1 ]) (A

1. 1 Figure 1 . 1 :

 111 Figure 1.1: Adult brain MRI (Subject: 01011-t1w of the dataset NAMIC). The voxel size of the images is 1 × 1 × 1mm.

Figure 1 . 2 :

 12 Figure 1.2: Adult brain MRI (Subject: 100307 of the dataset HCP100). The voxel size of the images is 0.7 × 0.7 × 0.7mm.

Figure 1 . 3 :

 13 Figure 1.3: Neonatal brain MRI (Subject: S00007 of the dataset MAIA). The voxel sizes of the T1w image and the T2 image are respectively about 0.2679 × 0.2679 × 1.2mm and 0.4464 × 0.4464 × 3mm.

  A+[START_REF] Timofte | A+: Adjusted anchored neighborhood regression for fast super-resolution[END_REF] (d) SRCNN[Dong et al., 2016a]

Figure 2 . 1 :

 21 Figure 2.1:The examples of single SR methods for a LR image of dataset Set5. LR image "bird" is reconstructed using the following methods: (b) bicubic interpolation, (c) A+[START_REF] Timofte | A+: Adjusted anchored neighborhood regression for fast super-resolution[END_REF], (d) SRCNN[Dong et al., 2016a] using the available code from authors.

  m×n is the observation matrix (m > n) and N denotes an additive noise. D ↓ represents the downsampling operator, B is the blur matrix and F encodes the motion information. B is also called the point spread function (PSF). The purpose of SR methods is to estimate X from the observations Y. SR is an ill-posed inverse problem where

Figure 2 . 2 :

 22 Figure 2.2: Pipeline of the method SRCNN [Dong et al., 2016a].

Figure 2 . 3 :

 23 Figure 2.3: 3D residual-learning convolutional neural networks for single brain MRI super-resolution.

  consists of MRI scans of twenty-one healthy volunteers with no history of neurological conditions. Magnetization prepared gradient echo (MPRAGE, T1-weighted) scans were acquired using a 3-T MR scanner (Achieva, Philips Healthcare, The Netherlands) with a 1.0 × 1.0 × 1.2 mm 3 resolution over an FOV of 240 × 204 × 256 mm acquired in the sagittal plane. Flair data were acquired using 1.1 × 1.1 × 1.1 mm 3 resolution over an FOV of 242 × 180 × 200 mm acquired in the sagittal plane. The T2-weighted volumes were acquired using a 3D multi-shot turbo-spin echo (TSE) with a TSE factor of 100 with over an FOV of 200 × 242 × 180 mm including a sagittal slice thickness of 1 mm.MR images of NAMIC Brain Multimodality 1 dataset have been acquired using a 3T GE at BWH in Boston, MA. An 8 Channel coil was used in order to perform parallel imaging using ASSET (Array Spatial Sensitivity Encoding techniques, GE) with a SENSE-factor (speedup) of 2. The structural MRI acquisition protocol included two MRI pulse sequences. The first results in contiguous spoiled gradient-recalled acquisition (fastSPGR) with the following parameters; TR=7.4ms, TE=3ms, TI=600, 10 degree flip angle, 25.6cm 2 field of view, matrix=256×256. The voxel dimensions are 1 × 1 × 1mm 3 . The second-XETA (eXtended Echo Train Acquisition) produces a series of contiguous T2-weighted images (TR=2500ms, TE=80ms, 25.6 cm 2 field of view, 1 mm slice thickness). Voxel dimensions are 1 × 1 × 1mm 3 .

  , LR images have been generated from a Gaussian blur and a down-sampling by isotropic scaling factors. In the training phase, a set of patches of training images is randomly extracted. The training dataset comprises 10 subjects (3200 patches 25 × 25 × 25 per subject randomly sampled) and the testing dataset is composed of 5 subjects. During the testing step, the network is applied on the whole images.

Figure 2 . 4 :

 24 Figure 2.4: The evolution of the mean PSNR of SRCNNF-Brain and SRCNN3D with respect to the number of epochs c [2017] IEEE.

Figure 2 . 5 :

 25 Figure 2.5: Illustration of SR results (KKI2009-02-MPRAGE) with isotropic voxel upsampling (scale factor is 2). LR data (b) with voxel size 2.4 × 2 × 2mm 3 is up sampled to size 1.2 × 1 × 1mm 3 c [2017] IEEE.

Figure 2 . 6 :

 26 Figure2.6: Impact of the optimization methods onto SR performance: SGD-GC, NAG, RMSProp and Adam optimisation of a 10L-ReCNN (10-layer residual-learning network with f = 3 and n = 64). We used Kirby 21 for training and testing with isotropic scaling factor ×2. The initial learning rates of SGC-GC, NAG, RMSProp and Adam are set respectively to 0.1, 0.0001, 0.0001 and 0.0001. These learning rates are decreased by a factor of 10 every 20 epochs. The momentum of these methods, except RMSProp, is set to 0.9. All optimization methods use the same weight initialization described in[START_REF] He | Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification[END_REF].

  Figure 2.7: Weight Initialization Scheme vs Performance (residual-learning networks with the same filter numbers n = 64 and filter size f = 3 using Adam optimization and tested with isotropic scaling factor ×2 using Kirby 21 for training and testing, 32000 patches with size 25 3 for training).

Figure 2

 2 Figure 2.8: Non-residual-learning vs Residual-learning networks with the same n = 64 and f 3 = 3 3 and the depths of 10 and 20 (called here 10L-CNN vs 10L-ReCNN and 20L-CNN vs 20L-ReCNN) over 20 training epochs using Adam optimization with the same training strategy and tested with isotropic scale factor ×2 using Kirby 21 for training and testing.

Figure 2 . 9 :

 29 Figure 2.9: Depth vs Performance (residual-learning networks with the same filter numbers n = 64 and filter size f = 3 over 20 training epochs using Adam optimization and tested with isotropic scale factor ×2 using Kirby 21 for training and testing, 32000 patches with size 25 3 for training).

  with filter size f 3 = 3 3 20L-ReCNN with filter size f 3 = 33

Figure 2 . 10 :

 210 Figure 2.10: Impact of convolution filter parameters (sizes f × f × f = f 3 with n filters) on PSNR and computation time. These 10-layers residual-learning networks are trained from scratch using Kirby 21 with Adam optimization over 20 epochs and tested with the testing images of the same dataset for isotropic scale factor ×2.

Figure 2 .

 2 Figure2.10 shows that a 10-layer network with a filter size of 5 3 shows results as well as a 20-layer network with 3 3 filters. Besides reconstruction performance, the use of a larger

Figure 2 . 11 :

 211 Figure 2.11: First row: Training patch size vs Performance. Second row: Patch size vs Training Time. Third row: Patch size vs Training GPU Memory Requirement. These networks with the same n = 64 and f 3 = 3 3 are trained from scratch using Kirby 21 with batch of 64 and tested with the testing images of the same dataset for isotropic scale factor ×2.

Figure 2 . 12 :

 212 Figure 2.12: Number of Subjects vs Performance (10-layer residual-learning networks with the same filter numbers n = 64 and filter size f = 3 over 20 training epochs using Adam optimization and tested with isotropic scale factor ×2 using Kirby 21 for training and testing, 3200 patches per subject with size 25 3 for training).

  have applied CNNs for upscaling cardiac image slices with the scale of 5 (e.g. upscaling the voxel size from 1.25 × 1.25 × 10.00mm to 1.25 × 1.25 × 2.00mm). Typically, their network is not capable of handling other scales due to the use of fixed deconvolutional layers. In brain MRI imaging, the variety of the possible acquisition settings motivates us to explore multi-scale settings.

Following[

  Kim et al., 2016a], we investigate how we may embed multiple scales in a single network. It consists in creating a training dataset within which we consider LR and HR image pairs corresponding to different scaling factors. We test two cases: the first condition where the learning dataset for combined scale factors (×2, ×3) has the same number as a single scale factor and the second one where we double the learning dataset for multiple scale factors. To avoid a convergence towards a local minimum of one of the scaling factors, we learn network parameters on randomly shuffled dataset.Table2.2 summarizes experimental results. First, when the training is achieved for the scaling from (2 × 2 × 2) on a dataset of (2 × 2 × 2) scale, it can be noticed that reconstruction performances decrease significantly when applied to other scaling factors (there is a drop from 39.01dB to 33.43dB when testing with(3 × 3 × 3)). Second, it can be noticed that when the training is performed on multi-scale data within the same training samples, there is no significant performance change compared to training from a single-scale dataset. Third, the more training dataset leads to a better performance. Training from multiple scaling factors leads to the estimation of a more versatile network. Overall, these results tend to show that one single network can handle multiple arbitrary scaling factors.

Figure 2 .

 2 Figure 2.13: 3D deep neural network for multimodal brain MRI super-resolution using intermodality priors. Skip connection computes the residual between ILR image and HR image.

Figure 2 .

 2 Figure 2.14: Multimodality-guided SR experiments. The LR T1-weighted images are upscaled with isotropic scale factor ×2 using respectively monomodal network (10L-ReCNN for LR T1w), HR T2w multimodal network, HR Flair multimodal network and both HR Flair and T2w multimodal images.

Figure 2 . 15 :

 215 Figure 2.15: Depth vs Performance (multimodal SR using residual-learning networks with the same filter numbers n = 64 and filter size f = 3 over 20 training epochs using Adam optimization and tested with isotropic scale factor ×2 using NAMIC for training and testing).

Figure 2 .

 2 Figure 2.14 shows the results of the multimodality-guided SR compared to the monomodal SR for both Kirby dataset (a) and NAMIC datasets (b). It can be seen that multimodality

  Figure 2.16: Illustration of the axial slices of monomodal and multimodal SR results (01018, pathological case) with isotropic voxel upsampling using NAMIC for training and testing. LR T1-weighted image (b) with voxel size 2 × 2 × 2mm 3 is upsampled to size 1 × 1 × 1mm 3 . Multimodal network 10L-ReCNN uses the HR T2-weighted reference (c) to upscale LR image. The different between ground truth image and reconstruction results are at the bottom. Their zoom version are at the right.

  and 2.18 show the results of reconstructed 3D images obtained from all the compared techniques. The zoom version of the reconstructions 20L-ReCNN shows sharpen edges and a grayscale intensity which are closest to the ground truth.

Figure 2 . 17 :

 217 Figure 2.17: Illustration of SR results (KKI2009-02-MPRAGE, non-pathological case, of dataset Kirby) with isotropic voxel upsampling. LR data (b) with voxel size 2×2×2.4mm 3 is upsampled to size 1 × 1 × 1.2mm 3 . The difference between the ground truth image and the reconstruction results are in the right bottom corners. Both network SRCNN3D and network 20L-ReCNN are trained with the 10 last images of Kirby.

Figure 2 . 18 :

 218 Figure 2.18: Illustration of SR results (01011-t1w, pathological case, of dataset NAMIC) with isotropic voxel upsampling. LR data (b) with voxel size 2 × 2 × 2mm 3 is upsampled to size 1 × 1 × 1mm 3 . The zoom versions of the axial slices are in the right bottom corners.

Figure 2 . 19 :

 219 Figure 2.19: Illustration of coronal SR results with isotropic voxel upsampling. Original data with voxel size of 0.4464 × 0.4464 × 3 is resampled to size 0.5 × 0.5 × 0.5 mm 3 . 20L-ReCNN is trained with the dHCP dataset

  Figure 2.20: Illustration of sagittal SR results with isotropic voxel upsampling. Original data with voxel size of 0.4464 × 0.4464 × 3 is resampled to size 0.5 × 0.5 × 0.5 mm 3 .20L-ReCNN is trained with the dHCP dataset.

  Figure 2.21: Illustration of sagittal SR results with isotropic voxel upsampling. Original data with voxel size of 0.4464 × 0.4464 × 3 is resampled to size 0.5 × 0.5 × 0.5 mm 3 .20L-ReCNN is trained with the dHCP dataset.

  MANTIS for NMU [Manjón et al., 2010b] (d) MANTIS for 20L-ReCNN

Figure 2 . 22 :

 222 Figure 2.22: Illustration of coronal cortex segmentation results (red color) using MAN-TIS toolbox [Beare et al., 2016] with isotropic voxel upsampling. Original data (a) with voxel size of 0.4464 × 0.4464 × 3 is resampled to size 0.5 × 0.5 × 0.5 mm 3 . 20L-ReCNN is trained with the dHCP dataset.

  this study, we have presented a multi-scale CNN-based SR method for single 3D brain MRI that is capable of learning multiple scales by training full all isotropic scale factors due to an independent upsampling technique such as spline interpolation. Handling multiple scales is related to multi-task learning. The lack of flexibility of learned network architecture raises an open issue requiring further studies: how can we build a network that can deal with a set of observation models (i.e. multiple scales, arbitrary point spread functions, non uniform sampling, etc.)? For instance, when applying SR techniques in a realistic setting, the choice of the PSF is indeed a key element for SR methods and it depends on the type of MRI sequence. The shape of the PSF also depends on the trajectory in the k-space (cartesian, radial, spiral). Making the network independent from the PSF model (i.e. blind SR) would be a major step for its use in routine protocol. Further research directions could focus on making more flexible CNN-based SR methods for greater use of these techniques in human brain mapping studies.

  Figure 2.23: Illustration of SR results (01018-t1w of dataset NAMIC) with isotropic voxel upsampling. Original data with voxel size of 1 × 1 × 1 mm 3 is upsampled to size 0.5 × 0.5 × 0.5 mm 3 . 20L-ReCNN is trained with the NAMIC dataset.

Figure 3 . 1 :

 31 Figure 3.1: The illustration of our proposed 3D SegSRGAN for joint mapping of SR and segmentation.

Figure 3 . 2 :

 32 Figure 3.2: The architecture of our proposed 3D SegSRGAN for joint mapping of SR and segmentation.

Figure 3 . 3 :

 33 Figure 3.3: SR results for one dHCP subject: (a) original HR image; (b-d) SR reconstruction of the LR image generated from (a) c [2019] IEEE

Figure 3 . 4 :

 34 Figure 3.4: Segmentation results for one dHCP subject: (a) segmentation ground-truth of Figure 3.3 (a); (b,c) segmentation of Figure 3.3 (b); (d) HR segmentation from the LR image using the joint SegSR-GAN method c [2019] IEEE.

Figure 3 . 5 :

 35 Figure 3.5: Reconstruction (b-d) and segmentation results (e) on a real LR neonatal brain image (a) (Subject S00059 of MAIA dataset) with voxel size of 0.446 × 0.446 × 3 mm 3 , re-sampled to 0.5 × 0.5 × 0.5 mm 3 .

Figure 3 . 6 :

 36 Figure 3.6: Reconstruction (b-d) and segmentation results (e) on a real LR neonatal brain image (a) (Subject S00096 of MAIA dataset) with voxel size of 0.446 × 0.446 × 3 mm 3 , re-sampled to 0.5 × 0.5 × 0.5 mm 3 .

Figure 4 . 1 :

 41 Figure 4.1: 2D histogram of intensity correspondences between paired T1w and T2w MRI over an entire image of the same subject form dataset NAMIC. Higher density regions is indicated by brighter color. The figure shows that the relationship between two modalities is not only non-linear but also not unique. It does not exist a function to transform from one T1w image to one T2w image and vice versa.

  (a) Input T2w MRI image (b) Ground truth T1w MRI image (c) REPLICA[START_REF] Jog | Random forest regression for magnetic resonance image synthesis[END_REF] (d) 20-layers SRReCNN[START_REF] Pham | Multi-scale brain mri super-resolution using deep 3d convolutional networks[END_REF]

Figure 4 . 2 :

 42 Figure 4.2: The examples (i.e. the axial slices of a brain MRI) of cross-modal synthesis methods. The input T1w MRI image (a) is synthesized by the random-forest MRIsynthesis method REPLICA[START_REF] Jog | Random forest regression for magnetic resonance image synthesis[END_REF] and SRReCNN[START_REF] Pham | Multi-scale brain mri super-resolution using deep 3d convolutional networks[END_REF]].

Figure 4 .

 4 1 illustrates the intensity of a T1-weighted MR (T1w) image and the corresponding T2-weighted MR (T1w) image (shown Figure 4.2 (a) and (b)) of the same subject. These paired images of the same subject are acquired by the same imaging system with the share the same resolution, orientation, coordinate and the same number of voxels.

Figure

4. 2

 2 shows synthesized T2 weighted MR (T2w) images (Figure 4.2 (c) and (d)) from a T1 weighted MR (T1w) image (Figure 4.2 (a)). The synthetic images are estimated as closely as possible to their ground truth.

Figure 4 . 3 :Figure 4 . 4 :

 4344 Figure 4.3: U-net architecture[START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]

Figure 4 .

 4 5 shows an example where the T1w and T2w images are unpaired. These unpaired images of different subjects (of different datasets) are acquired with different resolutions. Thus, they do not share the same general structure. This raises a question about the ability of synthesizing the T2w image of the observed T1w image (e.g. shown in Figure 4.5 (a)) given an unpaired T2w image (e.g. shown in Figure 4.5 (b)).

  (a) T1w axial slice (Subject: 01011-t1w of the dataset NAMIC). The voxel size of the images is 1 × 1 × 1mm.

  (a) T2w axial slice (Subject: 100307 of the dataset HCP100). The voxel size of the images is 0.7 × 0.7 × 0.7mm.

Figure 4 . 5 :

 45 Figure 4.5: Adult brain MRIs of different subjects

( 4 .

 4 9) is rewritten by adding the objective function on these few pairs: min α,D L(α, D, y, x) + F HF x -T F HF y

  and 4.7. Visually, our proposed method reconstructs better contours and sharpener edges than the compared method. The cortex (e.g. gray matter) of the result of 20L-SRReCNN shown in 4.6 is more visible and more curvature than those of REPLICA. However, the white matter regions of this CNN-based method is too smooth. In the case of synthesized T2w images of Figure4.7, the result of our CNN-based technique has salt-and-pepper noise, leading to lower SSIM than REPLICA. The problem of noisy synthesis T2w images comes from the property that we use the residual for training our networks and T1w images are not pre-denoised. The networks attempt to find the mapping for the voxel-wise differences between paired training images without considering their structure as in Equation (4.12).

Figure 4 . 6 :

 46 Figure 4.6: The examples (i.e. the coronal slices of a brain MRI) of cross-modal synthesis methods. The input T2w MRI image (a) is synthesized by the method REPLICA [Jog et al., 2017] and our proposed 20L-SRReCNN. The zoom versions are at the upper corners.

Figure 4 . 7 :

 47 Figure 4.7: The examples (i.e. the sagittal slices of a brain MRI) of cross-modal synthesis methods. The input T2w MRI image (a) is synthesized by the method REPLICA [Jog et al., 2017] and our proposed 20L-SRReCNN.

4. 3

 3 .1.1 Adversarial Loss A generator G is trained to learn functions between multiple domains. The objective of the generator is to fool the discriminator D by generating the images y tar , which are indistinguishable from real images of the target source x tar . The generator learns a mapping from input images of a source domain x src with a target label c G,tar (embedded by G) to the generated images as G : {x src , c G,tar } → y tar . We use c G,tar to control the desired mapping between different domains. For example, G(x T 1w , c G,T 2w ) estimates the synthesized T2w image y T 2w of the T1w image x T 1w and G(x T 1w , c G,F lair ) estimates the synthesized Flair image y F lair of this T1w input. Meanwhile the discriminator D is trained to distinguish the generated images and the real image x tar of the target domain. In order to support the discriminator to discriminate different target domains, the discriminator is conditioned by the label c D,tar , which is embedded by D, as D : {y tar , c D,tar } → D(y tar , c D,tar ) and D : {x tar , c D,tar } → D(x tar , c D,tar ).

  tar [log(1 -D(x tar , c D,tar ))] + E xsrc,c D,tar ,c G,tar [log(D(y tar , c D,tar ))] = E xtar,c D,tar [log(1 -D(x tar , c D,tar ))] + E xsrc,c D,tar ,c G,tar [log(D(G(x src , c G,tar ), c D,tar ))] (4.14)

x

  tar , from the images of a source domain x src . The images x tar and x src are unpaired. The discriminator D distinguishes the true image x tar and the generated image G(x src , c G,tar ), which are conditioned by the embedded target domain c D,tar . Meanwhile, the generator G attempts to fool the discriminator by setting the generated image as the true images of the adversarial loss. In addition, the synthetic images are mapped backward to the source images as G(G(x src , c G,tar ), c G,src ). Intuitively, the generated images of the synthetic images back to the source domain must be the source images x src . The cycle consistency loss guarantees this property of images. Besides, the TV regularizer is applied to the synthetic images G(x src , c G,tar ) so as to ensure their smoothness.

Figure 4 .

 4 8 illustrates our proposed GANbased method that generates synthetic T2w images from real T1w images of a specific subject using other the T2w images of other subjects.

Figure 4 . 8 :

 48 Figure 4.8: Illustration of our proposed 3D GANs for unpaired cross-modal synthesis so as to generate synthetic T2w images from T1w images

Figure 4 . 9 :

 49 Figure 4.9: The architecture of our proposed 3D GANs for unpaired cross-modal synthesis

  Our proposed GANs with λT V = 0.001 (d) Ground truth T2w MRI image

Figure 4 .

 4 Figure 4.10: The examples (i.e. the axial slices of a brain MRI) of cross-modal synthesis methods. The input T2w MRI image (a) is synthesized by the supervised method 20L-SRReCNN and our unsupervised method GAN. 20L-SRReCNN our GAN λ T V = 0.005 λ T V = 0.001 λ T V = 0 PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM Synthesized T1w 15.6848 0.9584 15.8333 0.9562 16.9957 0.9528 14.0984 0.8971 Synthesized T2w 20.5420 0.9528 15.3921 0.9419 16.3176 0.9401 14.6016 0.8925 Table4.2: The results of PSNR/SSIM for our GAN-based cross-modal synthesis methods with respect to the parameter λ T V . All methods using the training and testing images of NAMIC with λ gp = 10, λ cyc = 5000.

  Our proposed GANs with λT V = 0.001 (d) Ground truth T1w MRI image

Figure 4 .

 4 Figure 4.11: The examples (i.e. the axial slices of a brain MRI) of cross-modal synthesis methods. The input T1w MRI image (a) is synthesized by the supervised method 20L-SRReCNN and our unsupervised method GAN.

Figure 4 .

 4 Figure 4.12. No TV regularizer induces high-frequency artefacts. The reconstruction with λ T V = 0.005 leads to more smoothed results.

  Our proposed GANs with λT V = 0.005 (c) Our proposed GANs with λT V = 0.001 (d) Our proposed GANs with λT V = 0

Figure 4 .

 4 Figure 4.12: The sensibility of TV regularization within our GAN-based method. The zoom versions of ventricle regions are at the lower corners.

  2)where now x ∈ R n and f M LP (x) ∈ R m (n, m ≥ 1). g i , W i , B i denote the activation function, weight and bias of the i th layer. The units of MLPs are fully connected, each node in one layer connects every node in the following layer. A MLP consists of several fully-connected layers, activation layers and a cost function (so-called an objective function). However, MLPs are restricted for one-dimensional training set. In order to better represent higher dimensional patterns (e.g. edges, contours), we can supplement our neural networks with the convolution operation. In the next section, we will go in detail of convolutional neural networks, their characteristics and how to optimize the training of networks.

Figure A. 1 :

 1 Figure A.1: The example of a computing neuron with input x and output f (x) as Equation A.1 and a MLP with hidden layers

  theless, these networks have been received the most attention from research community since 2014. A CNN called Alexnet has won a challenge of image classification [Krizhevsky et al., 2012]. This network consists of eight layers (convolution and fully-connected layers which need to be trained) and other in-place layers such as Maxpool (i.e. selecting the maximum value of a pooling window), activation functions (e.g. rectified linear unit (ReLU), Softmax). The architecture of AlexNet is drawn in Figure A.2 (a).

  Figure A.2: The architecture of AlexNet[START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] VGG-net [Simonyan and[START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] for image classification (Recreating from[Krizhevsky et al., 2012, Simonyan and[START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]). Conv and Dense are convolution and fullyconnected layers respectively. The block Conv/ReLU and Dense/ReLU denote respectivelt a convolution layer and a fully-connected layer before a ReLU layer.

Figure A. 3 :

 3 Figure A.3: Some activation functions

Figure A. 4 :

 4 Figure A.4: The architecture of a residual block [He et al., 2016b]. BN denotes a batch normalization layer

Figure A. 5 :

 5 Figure A.5: The architecture of a densely connected block [Huang et al., 2017a]. A conv block may consist of convolutional layers, padding layers, BN and ReLU layers

  haute fréquence et le flou induit. Malgré cela, nous croyons que notre nouvelle approche peut amener de nouvelles perspectives très intéressantes pour beaucoup d'applications. Le chapitre 5 conclut notre thèse et ouvre des perspectives.

	MRI	Magnetic Resonance Imaging
	MSE	Mean Squared Error
	NAG	Nesterov's Accelerated Gradient
	Acronyms Net Network NMU Non-local Means Upsampling Chapter 1
	PCA	Principal Component Analysis
	2D PSNR Introduction Two dimensions Peak Signal to Noise Ratio
	3D ReLU	Three dimensions Rectified Linear Unit
	3T RMSProp Root-Mean-Square Propagation 3 Tesla Machine
	Adam SGD	Adaptive Moment Estimation Stochastic Gradient Descent
	BN SNR	Batch Normalization Signal to Noise Ratio
	CNN SSIM	Convolutional Neural Network Structural Similarity index
	CSF SR	Cerebrospinal fluid Super Resolution
	Conv T1w	Convolution T1-weighted
	dB T2w	decibel T2-weighted
	FLAIR	Fluid-Attenuated Inversion Recovery
	FOV	Field Of View
	GAN	Generative Adversarial Network
	GC	Gradient Clipping
	GM	Gray matter
	GPU	Graphics Processing Unit
	HR	High Resolution
	InsNorm Instance Normalization
	ILR	Interpolated Low Resolution
	LR	Low Resolution
	LRTV	Low Rank Total Variation
	MAE	Mean Absolute Error
	MPRAGE Magnetization-Prepared Rapid Gradient-Echo
		xi xii

Contents 1.1 Context and motivation . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Medical single image super-resolution: an approach to generate high-resolution images . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.2 Medical image cross-modal synthesis: an approach to generate synthesized images . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Thesis overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

  No denoising or bias correction algorithms were applied to the data. Image intensity has been normalized between 0 and 1. The following figures are drawn based on the average PSNR over all test images.

	2.2.2.2 Results with respect to 2D and 3D networks
		37
		36
	PSNR(dB)	35
		34
		33
		0 32	500	1000	Epochs	1500	2000	2500

The peak signal-to-noise ratio (PSNR) in decibels (dB) is used to evaluate the SR results with respect to the original HR images.

  SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM KKI2009-01-MPRAGE 33.42 0.9234 33.60 0.9303 34.29 0.9458 34.47 0.9366 36.16 0.9616 36.61 0.9656 KKI2009-02-MPRAGE 31.27 0.9402 31.37 0.9465 31.88 0.9597 32.40 0.9500 34.19 0.9700 34.60 0.9727 KKI2009-03-MPRAGE 35.88 0.9541 36.19 0.9596 36.88 0.9688 37.11 0.9608 38.93 0.9783 39.57 0.9808 KKI2009-04-MPRAGE 34.49 0.9441 34.73 0.9499 35.48 0.9617 35.59 0.9526 37.43 0.9728 37.91 0.9756 KKI2009-05-MPRAGE 35.72 0.9392 36.08 0.9458 36.86 0.9583 36.72 0.9482 38.40

	Image	Cubic Spline PSNR SSIM PSNR 0.9695 NMU LRTV SRCNNF-Nat SRCNNF-Brain	SRCNN3D 38.88 0.9728
	Mean	34.16 0.9402 34.40 0.9464 35.08 0.9585 35.26 0.9496 37.02	0.9704	37.51	0.9735
	Standard deviation	1.90	0.0111	2.00	0.0106	2.09	0.0083	1.90	0.0088	1.90	0.0060	1.97	0.0053
	Gain	-	-	0.24	0.0063	0.92	0.0183	1.10	0.0094	2.87	0.0302	3.36	0.0333
											3D CNN-based
	model achieves better performance than averaging 2D model outputs. Figure 2.5 shows
	examples of reconstructed 3D images obtained from all the compared techniques. Visually,
	HR estimation of SRCNN3D best preserves contours and has the best contrast compared
	with the results of other methods.

Our experiments shows that better performance can be achieved by learning model parameters on adequate data. 3D SR models for 3D data outperforms 2D counterparts thanks to the fact

Table 2 .

 2

1: The results of PSNR/SSIM for isotropic scale factor ×2 with the gain between compared methods and spline interpolation c [2017] IEEE.

  acquisition settings is a critical issue. We here evaluate the extent to which the number of training subjects affects SR reconstruction performance. As the training samples are extracted as patches of brain MRI images, we also evaluate the impact of the training patch size onto learning and reconstruction performance.

	PSNR(dB)	38.3 38.4 38.5 38.6 38.7 38.8 38.9 39.0	11	13	15 10L-ReCNN 17	19	21 12L-ReCNN	23	25 20L-ReCNN	27	29	31
	(hours) Training time Training GPU Memory (GB)	5 10 15 20 25 0 0 2 4 6 8 10 12	11 11	13 13	15 15	17 17	19 19	21 21 Patch size τ 3	23 23	25 25	27 27	29 29	31 31

  Table 2.2: Experiments with multiple isotropic scaling factors with the 20-layers network using the training and testing images of Kirby 21. Bold numbers indicate that the tested scaling factor is present in the training dataset. We test two conditions of same training data and double training data

		PSNR	PSNR	PSNR	PSNR	)
	×(2,2,2) ×(2,2,3) ×(2,2.5,2) ×(2,3,3) ×(2.5,2.5,2.5) ×(3,3,3)	39.01 36.80 37.71 35.23 35.47 33.43	35.25 35.11 35.41 35.13 35.52 35.01	37.35 36.47 36.91 35.75 36.09 34.89	38.80 37.24 37.93 36.20 36.63 35.20

Table 3 .

 3

		Interpolation 20L-SRReCNN SegSRGAN
	PSNR	30.70	35.84		31.75
	SSIM	0.9492	0.9739		0.9624
	Table 3.2: Quantitative evaluation of segmentation methods on dHCP dataset c [2019]
			IEEE.
			IMAPA DrawEM SegSRGAN
	Dice (mean)		0.788	0.818	0.886
	Dice (standard deviation)	0.061	0.014	0.011

1: Quantitative evaluation of SR methods on dHCP dataset c [2019] IEEE.

Table 3 . 3 :

 33 Dice scores of the segmentation method MANTiS on the 2 images of the MAIA testing dataset with respect to different approaches (columns): original T1w images, interpolated (Interp.) T1w images, original T2w images, upsampling T2 images using interpolation, NMU, 20L-SRReCNN and our SR results of the proposed SegSRGAN

	Subject	T1w			T2w
		Original Interp. Original Interp. NMU 20L-SRReCNN Our SR results
	MAIA #1	0.6215	0.6205	0.7090	0.7052 0.7190	0.7330	0.7480
	MAIA #2	0.6746	0.6802	0.6694	0.7118 0.7182	0.7333	0.7333

Table 4 .

 4 1:The results of PSNR/SSIM for cross-modal synthesis methods of subjectspecific scans. All methods using the training and testing images of NAMIC.

	our 20L-SRReCNN

  Contents Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Conclusions

In this work, two intended applications of medical image representations have been presented: single image super-resolution (SR) and cross-modal synthesis. SR and cross-modal synthesis have been receiving attention from the research community for recent years. The desire for SR in medical imaging stems from many applications: understanding of the anatomy, helping accurate segmentation and registration, and overcoming the hardware limitations of medical imaging devices. The motivation for cross-modal synthesis raises from many aspects: the mutual support between multi-modality medical imaging, helping accurate segmentation and super-resolution. Several methods for these problems have been introduced: patch-based method, sparse coding, random forest and neural networks.

NAMIC : http://hdl.handle.net/1926/1687

http://www.developingconnectome.org

RFIAP2018 : https://rfiap2018.ign.fr/

Acknowledgements

Part of this work was funded by ANR MAIA project, grant ANR-15-CE23-0009 of the French National Research Agency, INSERM and Institut Mines Télécom Atlantique (Chaire "Imagerie médicale en thérapie interventionnelle").

Network architectures and training

The works in [START_REF] Choi | Stargan: Unified generative adversarial networks for multi-domain image-to-image translation[END_REF][START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017] use instance normalization (InsNorm) layers [START_REF] Ulyanov | Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis[END_REF] for their networks as:

where x i denotes values of i th batch of input x over the mini-batch m (i = 1, ..., m), µ B and σ B are respectively the average and variance of i th batch and ǫ is a constant. β and γ are respectively learned scale and shift parameters of the layer. However, the use of one β and one γ for all domain-to-domain mappings limits the representation of a rich visual vocabulary for the construction. Instead, we use conditional instance normalization (CondInsNorm) layers [START_REF] Dumoulin | A learned representation for artistic style[END_REF] as:

where β ctar and γ ctar are respectively learned scale and shift parameters of the layer for the mapping from the input image x src to the target domain c tar and × denotes multiplication.

The embedding layer in [START_REF] Gal | A theoretically grounded application of dropout in recurrent neural networks[END_REF] is used to create the embedded domains.

All our networks are based on 3D layers with training patch size of 128 × 128 × 128.

Generator architectures

We denote c7s1-k as 7 × 7 × 7 Convolution-CondInsNorm-ReLU layer and c7-k as 7 × 7 × 7

Convolution layer with k filters and stride 1. Let c3s2-k denotes a 3 × 3 × 3 Convolution-CondInsNorm-ReLU layer with k filters, and stride 2. R-k denotes a residual block that contains 3 × 3 × 3 Convolution-CondInsNorm-ReLU-Convolution-CondInsNorm layer with the same number of filters of k on both convolution layers. u-k denotes 3 × 3 × 3 Upsampling-Convolutional-InstanceNorm-ReLU layer with k filters and the scaling factor of ×2. After the last layer, we apply a tanh activation. The generator architecture is: c7s1-16, c3s2-32, c3s2-64, R-64, R-64, R-64, R-64, R-64, R-64, u-32, u-16, c7-1. The reflecting padding is used to decrease the artefacts of the output. The illustration of the generator is shown in Figure 4.9 (a).

Discriminator architectures

The input of our discriminator is a Hadamard product of the input image and a bedded domain/target class label. Wasserstein GAN [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] suggests that normalization layer should not be used in the discriminator. We use Leaky ReLU with a negative would combine neural networks with nonlocal and statistical priors [START_REF] Fablet | Joint interpolation of multisensor sea surface temperature fields using nonlocal and statistical priors[END_REF] to preserve the consistency of high-resolution textured patterns, which are missed in the observed low-resolution images.

Since the segmentation maps in Section 2.2.3 only focus on the cortex of brain, other regions of MRI images such as CSF, WM and GM can also be segmented using our proposed methods.

In addition, other supervised learning segmentation algorithms such as atlas-based methods [START_REF] Rousseau | A groupwise super-resolution approach: application to brain mri[END_REF] or CNN-based methods [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] One simple class of ANNs is a perceptron (a neuron), described as:

A.2 Activation layers

The relationship between the input and the output of a problem may be nonlinear. So, they raise the need of the layers which can model the nonlinearity. In order to extend a network to represent nonlinear functions, we can apply nonlinear activation functions such as ReLU, softmax, hyperbolic tangent activation function, etc. The ReLU f ReLU (x) is an activation function which only keeps the positive part of its input x as:

The ReLU layer is close to linear so as the gradient-based methods can easily optimize [START_REF] Goodfellow | Deep learning[END_REF]. There are several versions of ReLU layers which extend the negative parts such as: Leaky ReLU [START_REF] Maas | Rectifier nonlinearities improve neural network acoustic models[END_REF](i.e. retaining the negative part by a small fixed scaling factor), Parametric ReLU [START_REF] He | Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification[END_REF] (i.e. the scaling factor for negative part is learned). Instead of just scaling the negative part, an exponential function can be apply to this part as ELU [START_REF] Clevert | Fast and accurate deep network learning by exponential linear units (elus)[END_REF]:

where α is a scale factor. The purpose of these activation functions is to preserve the properties of linear models for optimization but also model a nonlinear transformation. Another family of nonlinear layers can be used at the end of the networks for predicting a probability, in other word presenting a probability distribution such as sigmoid, softmax or hyperbolic tangent function. The logistic sigmoid function approaches to zero or one when its input is very negative or very positive, thus, it is commonly used for logistic regression as:

In order to take advantage of sigmoid activation for multiple regression, we can use the softmax function which decomposes the arguments into K distinct linear functions as:

These above functions are often used for classification tasks. For a linear regression, the linear function is the simplest choice as:

However, data is sometimes needed to be normalized into the range of [-1, 1] because of the compatibility of different dataset and the computational cost. The hyperbolic tangent function can be used for this purpose :

A.4 Application of CNN for neural style transfer

The work in [START_REF] Gatys | Image style transfer using convolutional neural networks[END_REF] investigated an interesting application of CNNs based on [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] that is style transfer. Given a content image p, a style image s and a random image x, we would like to generate x to have the content of p with the style of s as the total loss as:

where L content (x, p) and L style (x, s) denote the content reconstruction loss and the style loss respectively, and α and β are weights. The authors demonstrated visually that higher layers lost detailed pixel information and capture the high-level content of the image. In order to perverse the content for input image x, the feature reconstruction loss function is calculated by element-wise squared error:

where F l is the feature maps of the l th layer of a pre-trained network (e.g. VGG-net [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]). The second loss of Equation A.14 brings stylistic features to the input image, described as:

where L is the number of chosen layers for style transfer, w l denotes weighting factors and G corresponds the Gram matrix (i.e. inner products of the subsets). But this method slowly finds the solution because of inference processes. [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] propose to add a independent transformation network Φ to transform an input image p to an generated image ŝ = Φ(p) which have the style of image s based on the perceptual loss:

where α Φ and β Φ are the trade-off coefficients. The first version of this method used batch normalization layers [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] to encode the mapping ŝ = Φ(p). However, this normalization applies to whole a batch of images leading to the slower optimization of networks. The work in [START_REF] Ulyanov | Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis[END_REF] proposes another type of normalization called instance normalization. The idea of this normalization layer is to calculate simply the mean and the standard deviation of the input on the sum of a single batch instead of a whole.

A.5 Generative adversarial networks

Since introduced in 2014, generative adversarial networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF] have applied for many tasks such as 3D object generation [START_REF] Wu | Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling[END_REF], super-resolution [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF] and image translation [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], MRI one-domain synthesis [START_REF] Bermudez | Learning implicit brain mri manifolds with deep learning[END_REF]. GANs consist of two networks in which one network learns how to generate candidates mapped from a latent space while other discriminates them with instances from the true data distribution. The generative possibility and the stability of GANs can be improved by utilizing convolutional neural networks (DCGAN) [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF], conditioning these two networks with class labels (cGANs) [START_REF] Mirza | Conditional generative adversarial nets[END_REF],

adding auxiliary classifier (AC-GAN) [START_REF] Odena | Conditional image synthesis with auxiliary classifier GANs[END_REF]. The generator learns a mapping from a noise z of the noise distribution P z to a image

x from the input distribution P x . Meanwhile the discriminator is trained to distinguish the generated image G(z) and the real image. We express the objective of this adversarial learning [START_REF] Goodfellow | Generative adversarial nets[END_REF] as:

where the generator G tries to minimize this object while the discriminator D tries to maximize it. In order to improve the possibility of classification, conditioned GANs (cGAN) [START_REF] Mirza | Conditional generative adversarial nets[END_REF] attempt to embed the class of images into the generator and the discriminator as:

where c is the embedded label of the real image x. Instead of feeding the discriminator with the label information, another approach is to task the discriminator predict the class of image (ACGAN). The former is now defined as :

where D c (c | x) is a probability distribution over labels computed by D. However, the use of logarithm term in the adversarial loss (known as Jensen-Shannon divergence) could be saturated and in other cases, the model tends to the collapsing mode [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF][START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Salimans | Improved techniques for training gans[END_REF]. In order to effectively avoid the mode collapsing problem, [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] (WGAN) adopt Wasserstein distance to replace the logarithm cross-entropy loss as:

where D L ≤ 1 denotes the 1-Lipschitz constraint. The authors propose to use the weightclipping method to perform the constraint. Because of the difficulty of weight clipping on the network optimization, [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] alternates this constraint by a gradient penalty.

The objective function can be rewritten by an improved version of WGAN as:

where λ gp is a trade-off and ∇ x denotes the gradient of the interpolation x between the real input and the generated input as: ǫx + (1ǫ)G(z). ǫ is a random number from a uniform distribution over an interval [0, 1]. Recently, image-to-image translation using GANs has been receiving significant attention from research community [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], Kim et al., 2017[START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017]. In the context of supervised learning, [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] investigated cGANs for paired image-to-image translation. Recent works learn this task in an unpaired learning manner [START_REF] Kim | Learning to discover cross-domain relations with generative adversarial networks[END_REF][START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF], Zhu et al., 2017]. For instance, an architecture with twoblock GANs and a connection based ℓ 1 -norm cycle-consistent loss has been investigated for translating unpaired images [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF], as demonstrated as Figure A.7. Another work similar to [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] but with ℓ 2 -norm cycle-consistent loss has also proposed in [START_REF] Kim | Learning to discover cross-domain relations with generative adversarial networks[END_REF]. A concurrent work [START_REF] Yi | Dualgan: Unsupervised dual learning for imageto-image translation[END_REF] with the same approach as cycleGAN has improved the stability of GANs but using Wasserstein GAN [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF] instead of sigmoid cross-entropy loss used in the original GANs.

A.6 Optimization of neural networks

Given a training dataset which consists of N pairs of input x i and corresponding output y i : D = {x i , y i | i = 1, 2, ...N }, network parameters are estimated by minimizing the objective function using optimization algorithms. The objective function of the MLP in Equation (A.2) with two hidden layers can be expressed as: et al., 1998]. SGD proposes to update the network parameters θ at iteration t + 1 using the negative gradient of the objective function ∇L(θ t ) at iteration t, described as:

where V t denotes the weight update, µ and α are respectively the momentum and learning rate. However, when the optimization process gets closer to a minimum, an fixed momentum causes numerical instabilities. Nesterov's accelerated gradient (NAG) [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate o (1/k2)[END_REF] proposes to calculate the gradient with added momentum, using the following update:

The gradient descent optimization with small learning rates could be lead to slow convergence.

On the other hand, high learning rates may lead to vanishing gradients [Bengio et al., 1994, Glorot and[START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]. In order to address this issue, the SGD method with an adjustable gradient clipping (SGD-GC) [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF] proposes to scale the gradients over a threshold γ to achieve an optimization with high learning rates (e.g. α = 0.1) as follows:

SGD-GC may not converge quickly because of the predefined clipping range. One family of optimization methods addresses this issue through an automatic adaption of the learning rate for each parameter as RMSProp (root-mean-square propagation) [START_REF] Tieleman | Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF] and Adam (adaptive moment estimation) [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. RMSProp method proposes to rescale the gradients to update trainable weights by the root mean square of its second moments u as:

where δ is called RMSProp decay. However, RMSProp, which does not take account of the first moment of gradients and bias corrections, may induce divergence or very large step sizes [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. Adam method uses a first-order stochastic gradient-based optimization, which relies on adaptive estimates of both the first and second moments of the gradients (m, u). The Adam method applies the following update:

where β 1 and β 2 are the first and second moment decay rates, and α is a predefined parameter.

( mj ) t and (v j ) t are called respectively the moment bias corrections of the first and second moment estimates. For further information of other optimization methods, we can refer to [START_REF] Goodfellow | Deep learning[END_REF].

A.7 Discussion

Previously, we have introduced many different architectures of CNNs and the structure of each model: from the simple perceptron to the convolutional neural networks. A perceptron may be viewed as a neuron and then a set of this element composes a network. For many image processing tasks, neural networks take advantage of the convolution operation in order to better capture the features of higher-dimensional data. Then, deeper networks (e.g. residual networks) may achieve better performance in many applications such as classification but they need many parameters to train. The densely connected networks show the potential of decreasing the depth of networks but also maintaining the good performance. On the other hand, convolutional neural networks rely on the paired training set. The study of adversarial networks is potential to solve unsupervised learning problems. In addition, the applications of these networks have been provided in order to give readers a general look. [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] and VGG-net [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] for image classification (Recreating from [Krizhevsky et al., 2012, Simonyan and[START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] Increasing image resolution through SR is a key to more accurate understanding of the anatomy. The applications of super-resolution have been shown that applying super-resolution techniques leads to more accurate segmentation maps. Sometimes, certain tissue contrasts may not be acquired during the imaging session because of time-consuming, expensive cost or lacking of devices. One possible solution is to use medical image cross-modal synthesis methods to generate the missing subject-specific scans in the desired target domain from the given source image domain. The objective of synthetic images is to improve other automatic medical image processing steps such as segmentation, super-resolution or registration. In this thesis, convolutional neural networks are applied to super-resolution and cross-modal synthesis in the context of supervised learning. In addition, an attempt to apply generative adversarial networks for unpaired cross-modal synthesis brain MRI is described. Results demonstrate the potential of deep learning methods with respect to practical medical applications.

List of Figures