Pierre-Henri Conze

Gustavo Andrade-Miranda

Vivek Kumar Singh

Vincent Jaouen

Dimitris Visvikis

Current and emerging trends in medical image segmentation with deep learning

Keywords: artificial intelligence, semantic segmentation, deep neural networks, vision Transformers, medical imaging

In recent years, the segmentation of anatomical or pathological structures using deep learning has experienced a widespread interest in medical image analysis. Remarkably successful performance has been reported in many imaging modalities and for a variety of clinical contexts to support clinicians in computer-assisted diagnosis, therapy or surgical planning purposes. However, despite the increasing amount of medical image segmentation challenges, there remains little consensus on which methodology perform best. Therefore, we examine in this paper the numerous developments and breakthroughs brought since the rise of U-Net inspired architectures. Especially, we focus on the technical challenges and emerging trends that the community is now focusing on, including conditional generative adversarial and cascaded networks, medical Transformers, contrastive learning, knowledge distillation, active learning, prior knowledge embedding, cross-modality learning, multi-structure analysis, federated learning or semi-supervised and self-supervised paradigms. We also suggest possible avenues to be further investigated in future research efforts.

I. INTRODUCTION

T HE increased volume of medical data to be interpreted by clinicians for diagnosis, therapeutic or surgical planning purposes has encouraged the development of computer-aided image analysis tools to benefit from precise, fast, repeatable and objective measurements made by computational resources. Among existing analysis tasks, medical image segmentation whose goal is to extract the boundaries of anatomical or pathological structures from medical images remains crucial. Also commonly used in computed vision [START_REF] Saporta | Multi-target adversarial frameworks for domain adaptation in semantic segmentation[END_REF], semantic segmentation is a key step for many medical imaging workflows since the information arising from the resulting voxel-wise localization can greatly help clinicians to diagnose disorders, assess disease progression, plan therapeutic interventions or monitor treatment effects. Core feature of many computeraided detection and diagnosis schemes, image segmentation is involved in the analysis of many imaging modalities including computed tomography (CT), magnetic resonance (MR), positron emission tomography (PET) or ultrasound (US).

Delineating anatomical or pathological structures from medical images is traditionally performed manually. This task is exceedingly time-consuming and requires suitable clinical expertise to get clinically-relevant contours. This is therefore This work did not involve human subjects or animals in its research. P.-H. Conze and V. Jaouen are with IMT Atlantique, LaTIM UMR 1101, Inserm, Brest, France, pierre-henri.conze@imt-atlantique.fr. V. K. Singh is with Queen's University Belfast, Belfast, United Kingdom. G. Andrade-Miranda and D. Visvikis are with Inserm, LaTIM UMR 1101, Brest, France. not applicable to large volumes of data typically produced in clinical routine or research studies. Given the potential fatigue of human experts and the wide variations in expertise, manual segmentation is prone to strong intra-and inter-expert variability [START_REF] Nelms | Variations in the contouring of organs at risk: test case from a patient with oropharyngeal cancer[END_REF]. Irregularities of the targeted structures, morphological variations or pathological deformities between patients as well as the potential lack of clearly visible boundaries with the surrounding anatomy further affect the non-agreement between operators. To ease the process, intra-subject semiautomatic techniques consisting of ascending and descending non-linear registration steps applied to manually-drawn masks can be applied to obtain volumetric results [START_REF] Zhuang | A registration-based propagation framework for automatic whole heart segmentation of cardiac MRI[END_REF][START_REF] Decaux | Semi-automatic muscle segmentation in MR images using deep registration-based label propagation[END_REF]. Although more affordable than 3D volume annotations, such propagation schemes from sparse manual delineations to remaining slices still need interactions and may require manual refinements.

Mathematical models and low-level image processing have been extensively exploited for segmentation purposes before the rise of learning techniques. In particular, model-based segmentation incorporating statistical shape models has been followed in various clinical contexts [START_REF] Heimann | Statistical shape models for 3D medical image segmentation: A review[END_REF]. These models have been further improved by exploiting prior knowledge of shape information, for instance by relying on internal shape fitting and auto-correction to guide the delineation process [START_REF] Kim | Automatic segmentation of supraspinatus from MRI by internal shape fitting and autocorrection[END_REF]. Conversely, aligning and merging manually segmented images into a specific atlas coordinate space have been developed as a reliable alternative to statistical shape models. In this context, various single-and multi-atlas methods have been proposed relying on non-linear registration [START_REF] Iglesias | Multi-atlas segmentation of biomedical images: a survey[END_REF]. Some hybrid methods relying on statistical shape models constrained with probabilistic atlases have also been investigated. Medical image segmentation has been also performed through Bayesian approaches using expectation-maximization [START_REF] Van Leemput | Automated model-based tissue classification of MR images of the brain[END_REF], possibilistic clustering [START_REF] Barra | Segmentation of fat and muscle from MR images of the thigh by a possibilistic clustering algorithm[END_REF], histogram-based thresholding followed by region growing [START_REF] Purushwalkam | Automatic segmentation of adipose tissue from thigh magnetic resonance images[END_REF], active contours [START_REF] Zhou | Active contour model based on local and global intensity information for medical image segmentation[END_REF] and more recently machine learning (ML) [START_REF] Conze | Scale-adaptive supervoxel-based random forests for liver tumor segmentation in dynamic contrast-enhanced CT scans[END_REF] techniques.

However, the previously described methodologies are not perfectly suited for high inter-subject shape variability, weak boundaries and significant differences in tissue appearance. In most cases, their robustness is not up to the inherent limitations of medical images such as noise, non-uniform contrasts or motion artifacts. Moreover, many of these methods are semiautomatic and hence require prior knowledge, associated with high computational costs. This has strongly motivated the development of deep learning (DL) techniques to exploit image characteristics (e.g. contrast variation, orientation, shape, texture patterns) in a more efficient data-driven manner.

In recent years, artificial intelligence (AI) and more particularly DL models have reached impressive performance in medical image segmentation, becoming the new state-of-theart [START_REF] Hesamian | Deep learning techniques for medical image segmentation: achievements and challenges[END_REF]. The transition between systems that use handcrafted features (i.e. ML) to systems that learn features directly from data (i.e. DL) is now acted. The most widely-used models to date are based on convolutional neural networks (CNN). Especially, the U-Net convolutional architecture [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF] has been widely adopted in the community thank to its ability to output detailed delineations using a relatively low amount of training data. Nevertheless, new paradigms recently appear based on attention mechanisms, accompanying the emerging interest devoted to pure and hybrid Transformers-based models [START_REF] Shamshad | Transformers in medical imaging: A survey[END_REF].

Despite the availability of review articles summarizing existing DL approaches in medical image segmentation [START_REF] Hesamian | Deep learning techniques for medical image segmentation: achievements and challenges[END_REF][START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF], a more exhaustive, updated and comprehensive panorama is needed to allow a wide audience (e.g. researchers, clinicians, radiologists) to benefit from the latest and emerging trends in the field. Some of these existing reviews only focus on specific aspects such as anatomy-aided techniques [START_REF] Liu | Anatomy-aided deep learning for medical image segmentation: a review[END_REF], multiorgan segmentation [START_REF] Kaur | Evolution of multiorgan segmentation techniques from traditional to deep learning in abdominal CT images: A systematic review[END_REF], 3D CNN models [START_REF] Niyas | Medical image segmentation with 3D convolutional neural networks: A survey[END_REF] or U-Net and its variants [START_REF] Siddique | U-Net and its variants for medical image segmentation: A review of theory and applications[END_REF] whereas others [START_REF] Shamshad | Transformers in medical imaging: A survey[END_REF][START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF][START_REF] Chen | Recent advances and clinical applications of deep learning in medical image analysis[END_REF] address a broader overview of medical image analysis with DL by describing various tasks including classification, detection, segmentation or registration. In addition, despite the increasing amount of medical image segmentation challenges, there remains little consensus on which methodology performs the best. In this context, this paper aims at examining the numerous developments and breakthroughs brought since the rise of U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF] inspired architectures with a novel and specific focus on the technical challenges and emerging scenarios that the community is now focusing on, including knowledge distillation, contrastive learning, medical Transformers, prior knowledge embedding, cross-modality analysis, federated, active, selfor semi-supervised learning. The recurring motivation for these new approaches lies in the difficulty to obtain large annotated datasets, as compared to other imaging tasks (e.g. classification) or fields (e.g. computer vision).

This paper starts with a description of both background (Sect.II) and clinical needs (Sect.III) before providing an in-depth overview of current trends (e.g. prior knowledge embedding) in medical image segmentation with DL (Sect.IV). By bringing the light to medical Transformers, multi-task learning, segmentation uncertainty, contrastive learning and knowledge distillation, emerging trends are then motivated and explained in Sect.V. Sect.VI focuses on emerging applications including cross-modality learning, multi-domain segmentation, semi-supervised, active and federated learning. To help readers navigate through the paper, an overview of the targeted current and emerging trends is provided in Tab.I. Sect.VII finally concludes this paper by summarizing and discussing the possible avenues to be further investigated in the future.

II. BACKGROUND

Over the last few years, CNN models have become state-ofthe-art in medical image segmentation due to their ability to learn hierarchical representations of image features in a datadriven fashion [START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF]. Before going into current and emerging trends, this section aims at explicitly formulating what is medical image segmentation using DL (Sect.II-A) and describing contains for each x x x ∈ X a map y y y of H × W × D one-hot vectors indicating the ground truth classes for all voxels. In a fully-supervised setting, a deep segmentation network φ aims at approximating a mapping function φ : x x x → φ(x x x; Θ Θ Θ φ ) = ŷ y y between intensity x x x and class labels y y y images from N training samples {x x x n , y y y n } 1≤n<N by optimizing a loss function L φ (y y y, ŷ y y) = 1 N N i=0 φ (y y y n , ŷ y y n ) with ŷ y y n = φ(x x x n ) through a stochastic optimizer. The parameters of φ, namely Θ Θ Θ φ , are optimized during the training process. A stochastic gradient descent scheme, from standard gradient descent to Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] or more elaborated optimizers, aims at finding the optimal weights Θ Θ Θ * φ such that Θ Θ Θ * φ = arg min Θ Θ Θ φ L φ (y y y, ŷ y y). In practice, the network weights are iteratively updated in the direction of the steepest descent to reach the local minimum, following:

Θ Θ Θ φ ← Θ Θ Θ φ -α∇ Θ Θ Θ φ L φ (1)
where the learning rate α is a hyper-parameter controlling the step size at each iteration. Tuning α is of paramount importance to find a good trade-off between convergence speed and stable optimization. Back-propagation deals with gradient computation, while the gradient descent algorithm, based on this gradient, aims at performing the learning procedure. φ is a per-image loss function which is usually the cross-entropy loss defined, in a multi-class scenario with C classes, as follows:

CE (y y y n , ŷ y y n ) = 1 |C ||Ω| c∈C u∈Ω -y y y n,c,u log(ŷ y y n,c,u ) (2)
where Ω is the image grid and c ∈ C a given class with C = {0, ..., C} indexing the different structures of interest as well as the background. As reviewed in [START_REF] Ma | Loss odyssey in medical image segmentation[END_REF], a wide variety of loss functions exist including distribution-based (e.g. crossentropy), region-based (e.g. Dice), compound (e.g. DiceCE) and boundary-based (e.g. Hausdorff distance) losses.

B. Seminal works

The simplest and early attempts to perform segmentation using CNN consisted in classifying each pixel individually in a patch-based manner [START_REF] Ciresan | Deep neural networks segment neuronal membranes in electron microscopy images[END_REF]. Since input patches from neighboring pixels have large overlaps, the same convolutions were computed many times. By replacing fully-connected layers with convolutional layers, fully convolutional networks (FCN) gave the opportunity to take entire images as inputs and produce likelihood maps instead of single-pixel outputs. It removed the need to select representative patches and eliminated redundant calculations due to patch overlaps. In order to avoid outputs with far lower resolution than input shapes, FCN were applied to shifted versions of input images [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]. Multiple resulting outputs were thus stitched together to get results at full resolution. Further improvements were then proposed with architectures comprising a regular FCN to extract features, followed by an up-sampling part that enables the recover the input resolution using up-convolutions [START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF]. Compared to patch-based or shift-and-stitch methods, precise localization was possible in a single pass while taking into account the full image context. This motivated the strong interest devoted to convolutional encoder-decoders among which U-Net (Sect.II-C) is the most commonly used representative.

C. U-Net

Among existing convolutional encoder-decoders architecture, most DL-based medical image segmentation models are based on U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF] and its 3D counterpart V-Net [START_REF] Milletari | V-Net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF]. U-Net and V-Net consist of symmetrical architectures comprising an encoder that gradually reduces the spatial dimension using pooling layers, a decoder progressively recovering object details and initial resolution as well as skip-connections (i.e. long-range shortcuts) which concatenate features between contracting and expanding paths to help in improving localization accuracy and convergence speed. The contracting path encoder of a standard U-Net (resp. V-Net) architecture consists of sequential layers including 3×3 (resp. 3×3×3) convolutional layers followed by batch normalization (BN) and rectified linear unit (ReLU) activations (Fig. 1). Spatial size is reduced using 2 × 2 (resp. 2 × 2 × 2) max-pooling layers. The first convolutional layer typically generates 32 or 64 channels and this number doubles after each pooling as the network deepens. The encoder finally projects each input greyscale image x x x n to a latent representation (denoted as z z z n in Fig. 1). On its turn, the decoder branch is built symmetrically with respect to the encoder, except that max-pooling layers are replaced by upsampling operations (e.g. bi/tri-linear interpolation, transpose convolution). Depending on the binary or multi-class nature of the segmentation issue at hand, a final 1×1 (resp. 1×1×1) convolutional layer with sigmoid or softmax activation achieves pixel-wise segmentation ŷ y y n = φ(x x x n ), at native resolution. V-Net-inspired models may more suffer from high computational cost and GPU memory usage than their 2D counterparts.

Numerous refinements to the U-Net encoder-decoder architecture have been proposed including, to name a few, models which embed encoders pre-trained on large non-medical imaging databases (e.g. ImageNet) to leverage low-level features typically shared between different image types [START_REF] Conze | Healthy versus pathological learning transferability in shoulder muscle MRI segmentation using deep convolutional encoder-decoders[END_REF], sequential models exploiting residual convolutions [START_REF] Milletari | V-Net: Fully convolutional neural networks for volumetric medical image segmentation[END_REF] (Fig. 1) or pyramidal atrous convolutions (instead of pooling operations) [START_REF] Zhou | AFPNet: A 3D fully convolutional neural network with atrous-convolution feature pyramid for brain tumor segmentation via MRI images[END_REF] as well as alternative attention models (Sect.IV-E) such as attention U-Net [START_REF] Oktay | Attention U-Net: Learning where to look for the pancreas[END_REF] which integrates attention gates on skip-connections to focus on salient features. As an extension to vanilla U-Net, U-Net++ [START_REF] Zhou | UNet++: Redesigning skip connections to exploit multiscale features in image segmentation[END_REF] relied on re-designed skipconnections through intermediate convolution layers as well as deep supervision (Sect.IV-D). By aggregating features of varying semantic scales at the decoder branch, nested and dense skip-connections act as a flexible feature fusion scheme.

D. Data augmentation

Deep segmentation models are most often trained with extensive on-the-fly data augmentation, towards improved generalization properties. By comprising random geometric transformations (e.g. translation, rotation, scaling, shear, flipping) and random intensity modifications (e.g. normalization, blurring, contrast adjustment), data augmentation can be seen as a clever way to artificially increase the amount of available data, with slightly modified copies of already existing images. In practice, geometric transformations are applied to both greyscale images and ground truth masks whereas intensity transformations only modify source images. Data augmentation enables to teach DL models desired invariance, covariance and robustness properties and to strongly reduce over-fitting.

More recently, it was shown that DL models could further benefit from more elaborated data augmentation techniques such as MixUp which exploits convex combinations of pairs of samples and associated labels to train neural networks [START_REF] Zhang | MixUp: Beyond empirical risk minimization[END_REF]. MixUp regularizes the neural network to favor simple linear behavior in-between training examples. Originally proposed for image classification tasks, its extension to image segmentation is straightforward and efficient, as proven in [START_REF] Panfilov | Improving robustness of deep learning based knee MRI segmentation: MixUp and adversarial domain adaptation[END_REF] where improved generalization with MixUp as a data augmentation technique is reached for knee MR segmentation purposes. This success at the input data space further inspired the use of MixUp in the latent feature space [START_REF] Verma | Manifold MixUp: Better representations by interpolating hidden states[END_REF], in a setting referred to manifold MixUp. As reviewed in [START_REF] Tajbakhsh | Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation[END_REF], synthetic augmentation based on image synthesis methods, for instance exploiting generative adversarial networks (GAN), is another powerful alternative to standard data augmentation since it samples the manifold on which the original training set resides. Although effective, especially in extreme data scarcity scenarios, synthetic augmentation is more demanding to implement. Standard and more sophisticated data augmentation systems are obviously not mutually exclusive and can be used together.

While data augmentation is typically employed during training, using it at test time recently started to get special attention. Strongly linked to the way model uncertainty can be quantified [START_REF] Wang | Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks[END_REF] (Sect.V-C), test-time data augmentation consists in performing the inference both on original and augmented versions of images, followed by a merging procedure. Gains in performance are reported in various clinical contexts such as lesion segmentation from whole-body PET-CT images [START_REF] Amiri | Improved automated lesion segmentation in whole-body FDG/PET-CT via test-time augmentation[END_REF].

III. CLINICAL NEEDS AND APPLICATIONS

An ever-increasing number of research studies have illustrated the numerous applications of medical image analysis with DL, targeting a large number of pathologies and imaging modalities [START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF][START_REF] Chen | Recent advances and clinical applications of deep learning in medical image analysis[END_REF]. On its side, medical image segmentation plays a key role in many medical imaging workflows tailored for diagnosis, disease progression assessment, surgery or therapeutic planning, follow-up, survival analysis, treatment response evaluation, dosimetry and many other applications.

Clinical needs deal, first of all, with organ delineation from anatomical CT or MR imaging given that clinical parameters (e.g. volume, shape, inner textures) can be exploited as biomarkers to diagnose or quantify disease progression, as in cardiac [START_REF] Bernard | Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?[END_REF], brain [START_REF] Salehi | Auto-context convolutional neural network (Auto-Net) for brain extraction in magnetic resonance imaging[END_REF] or prostate [START_REF] Wang | Deep attentional features for prostate segmentation in ultrasound[END_REF] disorders. Hepatic pathologies with primary or secondary liver lesions are also concerned, thus making fully-automatic liver segmentation [START_REF] Christ | Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields[END_REF] particularly useful and requested in clinical routine. Regarding pure organ volumetry, a good example is an automated assessment of the total kidney volume (TKV) from MR images in patients with polycystic kidney disease since TKV is the main image-based biomarker to follow PKD progression [START_REF] Kline | Performance of an artificial multi-observer deep neural network for fully automated segmentation of polycystic kidneys[END_REF]. Segmenting healthy organs (e.g. liver, kidneys) to obtain a measurement of volume, size or shape is also a relevant usecase in the context of transplant surgery planning [START_REF] Kavur | CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation[END_REF][START_REF] Conze | Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks[END_REF].

In other works, whole or sub-structure organ segmentation is managed as a first step toward lesion detection and delineation. The main related challenge in this context deals with class imbalance as most voxels usually belong to the non-diseased class. In particular, there are numerous research works aimed at delineating skin lesions from dermatological images [START_REF] Song | An end-to-end multi-task deep learning framework for skin lesion analysis[END_REF][START_REF] Lei | Skin lesion segmentation via generative adversarial networks with dual discriminators[END_REF], brain tumors from MR images [START_REF] Zhou | AFPNet: A 3D fully convolutional neural network with atrous-convolution feature pyramid for brain tumor segmentation via MRI images[END_REF][START_REF] Kamnitsas | Unsupervised domain adaptation in brain lesion segmentation with adversarial networks[END_REF][START_REF] Chen | 3D dilated multifiber network for real-time brain tumor segmentation in MRI[END_REF], liver lesions from CT scans [START_REF] Christ | Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields[END_REF], head and neck primary tumors, lymph nodes and organs at risk from radiotherapy computed tomography (RT-CT) or combined PET and CT images [START_REF] Jin | Deep esophageal clinical target volume delineation using encoded 3D spatial context of tumors, lymph nodes, and organs at risk[END_REF][START_REF] Ma | Combining CNN and hybrid active contours for head and neck tumor segmentation in CT and PET images[END_REF][START_REF] Lei | Automatic segmentation of organs-at-risk from head-and-neck CT using separable convolutional neural network with hard-region-weighted loss[END_REF], breast masses in mammograms [START_REF] Yan | Two-stage multi-scale breast mass segmentation for full mammogram analysis without user intervention[END_REF] or ultrasound images [START_REF] Vakanski | Attention-enriched deep learning model for breast tumor segmentation in ultrasound images[END_REF], cystic kidney tissues in MR modality [START_REF] Kline | Automatic semantic segmentation of kidney cysts in MR images of patients affected by autosomal-dominant polycystic kidney disease[END_REF] or lesions in whole-body images [START_REF] Amiri | Improved automated lesion segmentation in whole-body FDG/PET-CT via test-time augmentation[END_REF]. In oncology, PET and CT imaging held a special place for disease characterization since they contain complementary information about the metabolic or biochemical function of tissues and organs as well as the anatomy of cancer [START_REF] Oreiller | Head and neck tumor segmentation in PET/CT: the HECKTOR challenge[END_REF]. Inner-lesion tissue segmentation is also increasingly targeted as in [START_REF] Ouhmich | Liver tissue segmentation in multiphase ct scans using cascaded convolutional neural networks[END_REF] where both active and necrotic tissues are identified inside liver tumors for patients with hepatocellular carcinoma in dynamic contrastenhanced CT or in [START_REF] Kamnitsas | Ensembles of multiple models and architectures for robust brain tumour segmentation[END_REF] where low and high-grade gliomas are decomposed into several tissue types comprising necrotic and enhancing cores, non-enhancing tumor and oedema.

Another emerging application deals with automatically extracting blood vessels (e.g. retinal, brain, liver vessels) from medical images [START_REF] Keshwani | TopNet: Topology preserving metric learning for vessel tree reconstruction and labelling[END_REF][START_REF] Sadikine | Semi-overcomplete convolutional auto-encoder embedding as shape priors for deep vessel segmentation[END_REF]. Apart from class imbalance and appearance similarity with non-vascular tissues, vascular segmentation brings additional limitations: complex multi-scale geometry with decreasing diameters and contrast along treelike networks, inter-patient variability in branching patterns... Medical image segmentation is also involved in plenty of radiomics pipelines where it has been for a very long time the bottleneck in both time and automation. Thus, extracting radiomic features in an automated and high-throughput way from relevant lesion areas is requested to quantify the characteristics of medical images, comprehensively characterize objects (e.g. tumors, organs, tissues) and finally provide useful guidance for clinicians. Although initially designed to process CT and functional PET images, the radiomics approach can be applied to any imaging modality or radio-tracer [START_REF] Hatt | Machine (deep) learning methods for image processing and radiomics[END_REF]. This includes works involving automated DL-based segmentation towards patient outcome prediction such as survival analysis [START_REF] Oreiller | Head and neck tumor segmentation in PET/CT: the HECKTOR challenge[END_REF] or chemotherapy response assessment and prediction [START_REF] Islam | Deep treatment response assessment and prediction of colorectal cancer liver metastases[END_REF].

More marginal applications can also be mentioned, as for the management of musculoskeletal diseases where patientspecific information related to the degree of muscle atrophy across joints is needed to plan interventions and predict interventional outcomes. In particular, DL-based shoulder muscle segmentation from MR images [START_REF] Conze | Healthy versus pathological learning transferability in shoulder muscle MRI segmentation using deep convolutional encoder-decoders[END_REF] can be employed to analyze the shoulder strength balance, which is particularly important given that a clear relationship between muscle atrophy and strength loss [START_REF] Pons | Shoulder muscle atrophy and its relation to strength loss in obstetrical brachial plexus palsy[END_REF] has been established.

IV. CURRENT TRENDS

A. Conditional generative adversarial networks

A network design based on conditional generative adversarial networks (GAN) has been proposed as a general-purpose solution for image-to-image translation [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF]. The goal is not only to learn the mapping from input to output images but also to learn a loss function to train this mapping. This kind of strategy is obviously suitable for segmentation purposes and among the possible applications, its feasibility for medical image segmentation has been demonstrated in several recent works [START_REF] Singh | Breast tumor segmentation and shape classification in mammograms using generative adversarial and convolutional neural network[END_REF][START_REF] Conze | Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks[END_REF][START_REF] Lei | Skin lesion segmentation via generative adversarial networks with dual discriminators[END_REF][START_REF] Huo | Splenomegaly segmentation on multi-modal MRI using deep convolutional networks[END_REF]. In practice, conditional GAN architectures comprise a generator aiming at providing segmentation masks through encoding and decoding layers as well as a discriminator which assesses if a given segmentation mask is synthetic or real. The adversarial network learns to discriminate real from synthetic delineations, i.e. ground truth masks versus those arising from the generator. This enforces the generative part to create increasingly plausible segmentation masks. During the training process, the generated delineations are gradually close to the ground truth, to the point of being able to deceive the discriminator. Unlike standard post-processing schemes, such iterative refinement performed through adversarial learning [START_REF] Huo | Splenomegaly segmentation on multi-modal MRI using deep convolutional networks[END_REF] is conducted in an end-to-end manner.

As generator φ, conditional GAN pipelines may use any type of U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF] inspired model, from simple [START_REF] Singh | Breast tumor segmentation and shape classification in mammograms using generative adversarial and convolutional neural network[END_REF] to extended (using dense dilated convolution) [START_REF] Lei | Skin lesion segmentation via generative adversarial networks with dual discriminators[END_REF] and cascaded (Sect.IV-B) [START_REF] Conze | Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks[END_REF] ones. The inputs of the discriminator D are the concatenation of source images and ground truth or predicted masks to be evaluated. Defined between 0 (fake) and 1 (plausible or real), the output of D is an array where each value corresponds to the degree of segmentation likelihood for a given image crop and its associated segmentation mask. Let φ(x x x) and D(x x x, φ(x x x)) be the outputs of φ and D respectively. The loss function L φ (Θ Θ Θ φ , Θ Θ Θ D ) for the generator φ can be defined as the following combination:

L φ (Θ Θ Θ φ , Θ Θ Θ D ) = 1 N N n=1 CE (φ(x x x n ), y y y n ) + λ adv (φ(x x x n ), y y y n ) (3)
where λ is an empirically set weighting factor and Θ Θ Θ D the trainable parameters of D. The adversarial term adv (φ(x x x n ), y y y n ) equals to -log(D(φ(x x x n ), x x x n )). Minimizing CE tends to provide rough predictions whereas maximizing log D(φ(x x x n ), x x x n ) aims at improving contour delineations. Conversely, the optimizer typically fits D through crossentropy using both estimated and ground truth masks. The loss function L D (Θ Θ Θ φ , Θ Θ Θ D ) for D is therefore defined as:

L D (Θ Θ Θ φ , Θ Θ Θ D ) = 1 N N n=1 -log(D(y y y n , x x x n )) -log(1 -D(φ(x x x n ), x x x n )) (4)
The above equation maximizes the loss values for ground truth (i.e. log(D(x x x, y y y))) and minimizes loss values for generated masks (i.e. -log(1-D(x x x, φ(x x x)))). The optimization process is performed sequentially by alternating gradient descents on φ and D at each batch [START_REF] Goodfellow | Generative adversarial nets[END_REF]. To further improve the ability of conditional GAN architectures to extract the contours of the targeted anatomy or abnormalities, investigations on more robust generators than traditional U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF] is an interesting research avenue [START_REF] Conze | Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks[END_REF]. Condition GAN can also be employed as a tool combined with other constraints (Sect.IV-C) such as anatomical shape priors, as proposed in [START_REF] Boutillon | Multi-structure bone segmentation in pediatric mr images with combined regularization from shape priors and adversarial network[END_REF].

B. Cascaded networks

Managing long-range spatial context from medical images is an important feature to improve the automatic delineation process. However, increasing the network depth over and over to exploit larger receptive fields is not suitable for memory and computational reasons, especially given the volumetric nature of most medical imaging data. In addition, too many high-resolution details are discarded when the number of down-sampling operations in the encoder branch is significant. To address these challenging limitations, standard scalespace pyramid [START_REF] Adelson | Pyramid methods in image processing[END_REF] and auto-context [START_REF] Tu | Auto-context and its application to high-level vision tasks and 3D brain image segmentation[END_REF] ideas influenced the development of cascaded strategies exploiting series of convolutional encoder-decoders [START_REF] Roth | A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentation[END_REF][START_REF] Conze | Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks[END_REF]. Instead of simultaneously exploiting several pathways working at various scales [START_REF] Chen | Attention to scale: Scale-aware semantic image segmentation[END_REF], cascaded approaches consist in using a scale-space pyramid to perform segmentation at a higher resolution while also considering contextual information from lower resolutions.

By considering two convolutional encoder-decoders in cascade, the most common setup could consist in training a lowresolution model and using its weights as initialization of a high-resolution model through transfer learning and finetuning. Although this strategy can significantly speed up convergence, the ability of the high-resolution segmentation model to extract long-range contextual features remains limited. The idea of stacking (at least) two convolutional encoderdecoders to integrate multi-level information more directly came naturally [START_REF] Roth | A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentation[END_REF] and made use of auto-context [START_REF] Tu | Auto-context and its application to high-level vision tasks and 3D brain image segmentation[END_REF] such that posterior probabilities resulting from a given model can be used as features for the following one [START_REF] Salehi | Auto-context convolutional neural network (Auto-Net) for brain extraction in magnetic resonance imaging[END_REF]. The models can be trained separately [START_REF] Salehi | Auto-context convolutional neural network (Auto-Net) for brain extraction in magnetic resonance imaging[END_REF] but this prevents refining lowresolution models from the high-resolution ones during backpropagation. An end-to-end training is better suited to exploit simultaneous multi-level segmentation refinement [START_REF] Roth | A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentation[END_REF][START_REF] Conze | Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks[END_REF]. A cascade of deep modules exploiting tissue-specific geodesic distance maps as contextual information was employed in [START_REF] Wei | A cascaded nested network for 3T brain MR image segmentation guided by 7T labeling[END_REF] to gradually improve the delineation accuracy. Combined with top-down reasoning, such bottom-up strategies could better handle texture information and region discontinuities.

C. Prior knowledge embedding

Regularization plays a key role in DL since it tends to increase both robustness and generalizability of a deep model when applied to unseen data. One common strategy consists in adding a regularization term into the loss function to get more accurate and plausible results [START_REF] Nosrati | Incorporating prior knowledge in medical image segmentation: a survey[END_REF]. The regularization term R φ deals with adding some prior knowledge to the model φ and its regularization effect is achieved by adding the scaled regularizer λR φ to the loss function L φ to ensure further consistency between both predictions φ(x x x) and targets y y y. The resulting loss function can be expressed as follows: la te n t s p a c e projection Fig. 2. Integration of anatomical shape priors into a deep segmentation pipeline [START_REF] Boutillon | Multi-structure bone segmentation in pediatric mr images with combined regularization from shape priors and adversarial network[END_REF][START_REF] Oktay | Attention U-Net: Learning where to look for the pancreas[END_REF][START_REF] Boutillon | Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors[END_REF][START_REF] Ravishankar | Learning and incorporating shape models for semantic segmentation[END_REF]. Shape priors-based regularization is performed using a shape encoder arising from a convolutional auto-encoder previously optimized on ground truth segmentation masks. Refer to Sect.IV-C for further details.

L φ (Θ Θ Θ φ ) = 1 N N n=1 φ (φ(x x x n ), y y y n ) + λR φ (5)
Many different types of information can be incorporated as prior knowledge into DL frameworks such as shape constraints [START_REF] Oktay | Attention U-Net: Learning where to look for the pancreas[END_REF], topology specifications [START_REF] Keshwani | TopNet: Topology preserving metric learning for vessel tree reconstruction and labelling[END_REF], edge polarity [START_REF] Chen | DCAN: Deep contour-aware networks for object instance segmentation from histology images[END_REF] or adjacency rules between regions [START_REF] Ganaye | Removing segmentation inconsistencies with semi-supervised non-adjacency constraint[END_REF]. Nevertheless, integrating shape priors remains one of the most commonly used strategies toward anatomically meaningful predictions. In particular, the use of convolutional auto-encoder (AE) to learn anatomical shape variations from medical images has been demonstrated in multiple applications [START_REF] Boutillon | Multi-structure bone segmentation in pediatric mr images with combined regularization from shape priors and adversarial network[END_REF][START_REF] Oktay | Attention U-Net: Learning where to look for the pancreas[END_REF][START_REF] Boutillon | Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors[END_REF][START_REF] Ravishankar | Learning and incorporating shape models for semantic segmentation[END_REF]. Specifically, a convolutional AE is a deep network made of an encoder F : y y y → F (y y y; Θ Θ Θ F ) and a decoder G : F (y y y;

Θ Θ Θ F ) → G(F (y y y; Θ Θ Θ F ); Θ Θ Θ G )
where Θ Θ Θ F and Θ Θ Θ G correspond to the learnable parameters of F and G respectively. F maps the input to a low-dimensional feature space whereas G reconstructs the original input from the compact representation. To avoid the AE to copy the input, F is usually designed to be undercomplete such that the latent space is much smaller than the input dimension. By penalizing the reconstruction G • F (y y y), the cross-entropy loss function can be employed to optimize the AE following:

Θ Θ Θ * F , Θ Θ Θ * G = arg min Θ Θ Θ F ,Θ Θ Θ G 1 N N n=1 CE ((G • F )(y y y n ), y y y n ) (6)
where

Θ Θ Θ * F (resp. Θ Θ Θ * G )
are the optimal weights of the encoder (resp. decoder). After having trained the AE, its encoder component acts as a non-linear shape model and can project any predicted or ground truth segmentation masks to a shape manifold space [START_REF] Ravishankar | Learning and incorporating shape models for semantic segmentation[END_REF]. The encoder produces a feature map F (y y y) which compactly encodes the most salient characteristics of y y y.

Once the AE trained, its encoder component can be integrated into the segmentation pipeline (Fig. 2). A regularizer R φ that penalizes the deviation between predicted and ground truth segmentation masks fed as inputs of the learned shape model F is included in the global loss (Eq.5). A Euclidean distance between both latent shape representations [START_REF] Oktay | Attention U-Net: Learning where to look for the pancreas[END_REF] is usually used:

R φ (ŷ y y, y y y) = 1 N N n=1 F (y y y n ; Θ Θ Θ * F ) -F (φ(x x x n ); Θ Θ Θ * F ) 2 2 (7)
Nevertheless, a cosine distance between predicted and ground truth masks in low-dimensional space is also of interest [START_REF] Sadikine | Semi-overcomplete convolutional auto-encoder embedding as shape priors for deep vessel segmentation[END_REF].

R φ (ŷ y y, y y y) = 1 N N n=1 1 -cos(F (y y y n ; Θ Θ Θ * F ), F (φ(x x x n ); Θ Θ Θ * F )) (8)
As reported in [START_REF] Ravishankar | Learning and incorporating shape models for semantic segmentation[END_REF], shape is just one of the geometric attributes of anatomical or pathological structures one can exploit. Much more meaningful priors such as texture, topology or size can be embedded into training objectives towards stronger robustness and stability of DL segmentation networks.

D. Deep supervision

Introduced in the context of holistically-nested edge detection [START_REF] Xie | Holistically-nested edge detection[END_REF], additional convolutional operations can be applied at different levels of the decoder branch in order to exploit a deep supervision mechanism (Fig. 3) able to boost the segmentation performance [START_REF] Shamshad | Transformers in medical imaging: A survey[END_REF]. Companion objective functions are estimated at some hidden layers of the network and added to the output loss. In practice, feature maps as outputs of each intermediate decoder blocks can be up-sampled to the size of the input image using bi-or tri-linear interpolation, depending of the 2D or 3D nature of the segmentation problem. Similarly to [START_REF] Dou | A deep attentive convolutional neural network for automatic cortical plate segmentation in fetal MRI[END_REF], a convolutional operation (e.g. with 3 × 3 kernel) can be applied to create feature maps at each level of the decoder (16 in Fig. 3). These maps then go through deep supervision modules to improve the gradient flow and encourage learning more useful representations [START_REF] Dou | A deep attentive convolutional neural network for automatic cortical plate segmentation in fetal MRI[END_REF]. After having performed the concatenation of these intermediate outputs, two convolutional layers including a final 3×3 one with softmax activation finally achieve pixel-wise segmentation (multi-label in Fig. 3). In this context, the overall loss function L φ can be defined as the weighted sum of the cross-entropy losses (or any other losses) estimated at different decoder levels involving supervision:

L φ (Θ Θ Θ φ ) = M j=1 w j L j CE + w f L f CE ( 9
)
where w j and L j ce denote the weight and loss for the points of supervision at level j of the decoder, w f and L f CE the weight and loss computed at the final network output (where f stands for final). When following a VGG-13 architecture [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] as depicted in Fig. 3, M = 4 intermediate decoder levels can be considered. Following [START_REF] Dou | A deep attentive convolutional neural network for automatic cortical plate segmentation in fetal MRI[END_REF], one can use w 1 = 0.8, w 2 = 0.7, w 3 = 0.6, w 4 = 0.5 and w f = 1 where level j = 1 is closer to the network ending part than level j > 1.

However, how to balance the hyper-parameter setting among different loss components remains a matter of concern. Instead of empirically defining weights, a relative weighting can be learned from the data using homoscedastic uncertainty [START_REF] Kendall | Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[END_REF].

E. Attention mechanisms

The human visual system can concentrate and focus actively on a tiny portion of highly relevant perceptible information while disregarding other irrelevant perceivable stimuli. Attention mechanisms were introduced in DL frameworks to imitate this aspect of how the human visual system processes information. In general, they can be regarded as a dynamic selection process where the features extracted from images are adaptively weighted to pay attention to the more salient ones, i.e. the feature needed for accurately solving a specific image analysis task. Attention mechanisms, particularly in image segmentation, can suppress feature responses in irrelevant background regions, hence reducing the rate of false-positive predictions. This is particularly true for the challenging instances of small objects with high shape variability.

The attention problem is usually formulated using three vectors: query, key and value. Conceptually, we can think of key and value as a look-up table in which the query is matched to a key, and the value associated with that key is returned. In image segmentation, it is equivalent to mapping the features of the structure to segment (query) against a collection of plausible target features (keys), then presenting the bestmatched regions (values). Mathematically, let us consider that we have a query q ∈ R dq and M pairs of key

k ∈ R d k and value v ∈ R dv vectors {(k 1 , v 1 ), . . . (k M , v M )}
, where all of them can be obtained from intermediate CNN features or embedding patches of an input image x x x. The attention is computed step-by-step following three operations: alignment, weighting and contextualization. In the alignment step E(•), each query is matched against the M keys to compute a score value. Several commonly used alignment functions are further described in Tab.II where additive [START_REF] Graves | Neural turing machines[END_REF] and dot-product [START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF][START_REF] Vaswani | Attention is all you need[END_REF] functions are the most widely used. In the next step, the alignment scores are passed through a function H(•) (e.g. softmax, sigmoid) to generate the final attention weights by normalizing all the scores to a probability distribution (Eq.10). A contextualization vector (Eq.11) is then instantiated for each q as a weighted sum of the M values v i by the set of weights

Name Alignment function Additive or concatenation

E(q, k) = V V V T tanh(W W W [q; k]) [92] Content-based attention E(q, k) = cos(q, k) [95] Location-based attention E(q) = W W W q [93] General E(q, k) = qW W W k [93] Dot-product E(q, k) = q T k [93] Scaled dot-product E(q, k) = q T k √ d k [94]
A. This computation enables to define how much attention should be paid to each feature v i . Fig. 4 depicts the procedure followed to get the context vector for a particular query q.

A(q, k i ) = H(E(q, k i )) (10)
C(q, {(k 1 , v 1 ), . . . (k M , v M )}) = M i=1 A(q, k i ) × v i (11)
Four main categories of attention techniques can be found in the literature dedicated to medical image segmentation: channel attention (what to attend), spatial attention (where to attend), branch channel (which to attend) as well as hybrid (e.g. channel and spatial attention) methods.

Channel attention is based on the idea that, in deep CNN models, each channel represents a different feature map that typically denotes distinct objects. As a result, the role of channel attention is to adaptively calibrate the weight of each channel, serving as an object selector of the entities that should deserve more attention. Channel attention, particularly, squeeze-and-excitation (SE) block [START_REF] Hu | Squeeze-and-excitation networks[END_REF], has proved to be extremely effective in tasks such as head and neck primary tumor segmentation [START_REF] Iantsen | Squeeze-and-excitation normalization for automated delineation of head and neck primary tumors in combined PET and CT images[END_REF], prostate zonal segmentation [START_REF] Rundo | USE-Net: Incorporating squeeze-and-excitation blocks into U-Net for prostate zonal segmentation of multi-institutional MRI datasets[END_REF], brain structure segmentation [START_REF] Li | MSGSE-Net: Multi-scale guided squeeze-and-excitation network for subcortical brain structure segmentation[END_REF] and micro-vessel segmentation [START_REF] Shen | Selfattentional microvessel segmentation via squeeze-excitation transformer unet[END_REF]. The SE block, depicted in Fig. 5, passes an intermediate features map through a squeeze operation (i.e. global average pooling) that captures global spatial information. Then, an excitation module (alignment function) captures channel-wise relationships and outputs an attention vector using fullyconnected and non-linear layers (e.g. ReLU), followed by a sigmoid function (weighting). Lastly, each channel of the input features map is scaled by multiplying the corresponding element in the attention vector (contextualization). Others works including [START_REF] Oktay | Attention U-Net: Learning where to look for the pancreas[END_REF] extended skip-connections between both encoder and decoder branches through channel attention gates. By scaling the encoder features by importance, the network may concentrate on a specific aspect of the input to generate the segmentation. A similar approach was presented in [START_REF] Abraham | A novel focal tversky loss function with improved attention U-Net for lesion segmentation[END_REF] for lesion segmentation. This model also used the extracted feature maps from the encoder path for the computation of attention weights which are afterward merged with the upsampled feature maps in the decoder branch.

Spatial attention and channel attention have relatively similar functioning. Spatial attention consists in adaptively calibrating the weight of each part of the image. This mechanism GAP X convolution Fig. 5. Flow-chart diagram of a squeeze-and-excitation block [START_REF] Hu | Squeeze-and-excitation networks[END_REF]. GAP stands for global average pooling.

chooses where to focus attention through an adaptive spatial area selection procedure. One may integrate an attention block into the U-Net architecture to learn semantic representations that prioritize spatial regions with high saliency levels for the task of tumor segmentation [START_REF] Vakanski | Attention-enriched deep learning model for breast tumor segmentation in ultrasound images[END_REF]. In the same direction, the addition of a spatial-channel squeeze-and-excitation block can improve the performance of various convolutional architectures dedicated to medical image segmentation [START_REF] Roy | Recalibrating fully convolutional networks with spatial and channel squeeze and excitation blocks[END_REF].

Branch attention mechanisms, on its side, separate the attention problem into several sub-modules (branches), each of which focuses on a certain aspect (e.g. channel, spatial, resolution, degradation) while exchanging effective information. Then, the role of the attention mechanism is to adaptively calibrate the weight of each branch, acting as a dynamic branch selection procedure, choosing which to focus on. To segment scleral blood vessels, Yao et al. [START_REF] Yao | Claw U-Net: A U-Net variant network with deep feature concatenation for scleral blood vessel segmentation[END_REF] developed a U-Net-inspired architecture with deep feature concatenation and an attention mechanism branching into numerous attention gates. This enables the network to focus on the border segmentation of tiny blood veins. Other works such as [START_REF] Wang | Deep attentional features for prostate segmentation in ultrasound[END_REF] and [START_REF] Sinha | Multi-scale self-guided attention for medical image segmentation[END_REF] exploited the multi-scale nature of CNN models. They generated series of multi-scale attention modules at different resolutions and integrated local deep attention features with a global context. Lastly, channel and spatial attention incorporated the benefits of both attention mechanisms. This system acts as a dynamic spatial area and object selection mechanism, deciding what and where to focus attention [START_REF] Roy | Concurrent spatial and channel squeeze & excitation in fully convolutional networks[END_REF][START_REF] Fang | Spatial and channel attention modulated network for medical image segmentation[END_REF].

F. Multi-structure analysis

The multi-structure analysis aims at incorporating interstructure relations into a given DL-based segmentation model, thus leading to a more accurate representation of complex anatomies. A typical application in this direction deals with multi-organ segmentation from CT [START_REF] Roth | A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentation[END_REF][START_REF] Ma | Fast and low-GPU-memory abdomen CT organ segmentation: The FLARE challenge[END_REF][START_REF] Valindria | Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI[END_REF][START_REF] Zhou | Automatic segmentation of multiple organs on 3D CT images by using deep learning approaches[END_REF], PET [START_REF] Taghanaki | Combo loss: Handling input and output imbalance in multi-organ segmentation[END_REF] or MR [START_REF] Conze | Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks[END_REF][START_REF] Valindria | Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI[END_REF][START_REF] Taghanaki | Combo loss: Handling input and output imbalance in multi-organ segmentation[END_REF] images. Although computationally effective, global approaches which consider one single model for all structures of interest are computationally effective, extracting high-level relationship patterns between multiple structures but cannot fully exploit the local characteristics of each different component to delineate. Conversely, modeling multiple structures by a set of structure-specific models enables generating local structure-specific features but sacrifices the ability to take advantage of inter-structure relationships.

In their survey, Cerrolaza et al. reported that multi-level (e.g. nested, multi-resolution) or sequential models are robust alternatives to global or individual models as they combine the robustness and specificity of global approaches with the flexibility of structure-specific models [START_REF] Cerrolaza | Computational anatomy for multi-organ analysis in medical imaging: A review[END_REF]. Multi-level nested models decompose the data into different levels of detail according to coarse-to-fine analysis rules. Conversely, in multilevel multi-resolution models, global inter-organ relations are modeled at coarser resolutions while local organ-specific variations are extracted from higher resolutions. On its side, sequential modeling deals with the analysis of multiple structures following a pre-defined order of increasing complexity. In the same vein, it is also relevant to mention the application related to lesion segmentation for which delineating the organ of interest as a first stage before localizing its inner lesions is usually followed to promote the extraction of organ-specific features and to narrow the search area (Sect.III). As an example, such two-step sequential approach has been successfully applied in [START_REF] Christ | Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields[END_REF] for liver lesion segmentation from CT scans.

G. Learning frameworks

The success of DL in medical image segmentation not only originates from the development of novel learning paradigms but also from the network architecture design itself and the focus given to data management and optimization processes. Especially, a trend can be noticed towards the development of unified frameworks such as NiftyNet [START_REF] Gibson | NiftyNet: a deep-learning platform for medical imaging[END_REF] or nnU-Net [START_REF] Isensee | nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation[END_REF]. Since the design choices towards an optimal framework are usually dedicated to a specific segmentation task (i.e. a given tissue type for a given modality) and cannot easily be transferred to another application, pipelines that can configure their sub-components in an automated fashion are highly requested. In particular, nnU-Net [START_REF] Isensee | nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation[END_REF] which has been designed to deal with the dataset diversity found in the domain has proven its efficiency by winning many challenges. It condenses and automates the key decisions for designing a successful pipeline for any given dataset. Thus, nnU-Net has become one of the reference frameworks when targeting a new segmentation task.

In the same scope, neural architecture search (NAS) is another direction under investigation with the goal to automate the iterative network design process usually handled manually by researchers. Among the existing research works in this area, NAS-UNet [START_REF] Weng | NAS-UNet: Neural architecture search for medical image segmentation[END_REF] is based on the design of three types of primitive operation set on search space to automatically find two cell architectures (DownSC, UpSC) for semantic segmentation purposes. Promising results were reported for various imaging modalities including CT, MR and ultrasound.

V. EMERGING TRENDS

A. Medical Transformers

Transformers, as attention-based structures [START_REF] Vaswani | Attention is all you need[END_REF], have first demonstrated their tremendous force in natural language processing (NLP) [START_REF] Devlin | BERT: Pretraining of deep bidirectional transformers for language understanding[END_REF][START_REF] Radford | Improving language understanding by generative pre-training[END_REF] and have gradually gain attraction on different computer vision tasks such as image classification, detection, segmentation and video analysis. Their popularity is now also rapidly growing in medical image analysis [START_REF] Jun | Medical Transformer: Universal brain encoder for 3D MRI analysis[END_REF], especially for medical image segmentation with an exponential growth of related publications in the last year [START_REF] Shamshad | Transformers in medical imaging: A survey[END_REF][START_REF] Li | Transforming medical imaging with Transformers? a comparative review of key properties, current progresses, and future perspectives[END_REF]. The pioneering work of vision Transformers [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] was an interesting and meaningful attempt to replace convolutional backbones with convolution-free models. In contrast to CNN, vision Transformers (ViT) offer parallel processing and a complete field of view in a single layer.

ViT has a columnar structure where the 3D input volume x x x ∈ R H×W ×D×C is split into n p 3D non-overlapping patches {x x x 1 , x x x 2 , x x x 3 , . . . , x x x np } with x x x i ∈ R P ×P ×P ×C , C represents the number of modalities, (P, P, P ) is the resolution of each patch and n p = HW D/P 3 is the resulting number of patches, which is also the effective input length of the Transformer. Since Transformer layers operate over fixed size 1D set of vectors, the n p patches are flattened and mapped to a d-dimensional embedding space through a trainable linear projection matrix E E E ∈ R (P 3 •C)×d . To preserve spatial information, a 1D positional embedding E E E pos ∈ R np×d is added to each of the n p patches, and the resulting sequence of embedding is used as input to the Transformer encoder:

z z z 0 = [x x x 1 E E E; x x x 2 E E E; x x x 3 E E E; . . . ; x x x np E E E] + E E E pos (12)
The Transformers-based encoder consists of alternating L layers of multi-head self-attention (MSA) and multilayer perceptron (MLP) blocks. A layer normalization (LN) is applied before each block and a residual connection after each block. One layer of a Transformer block can be formulated as:

z z z l = MSA(LN(z z z l-1 )) + z z z l-1 z z z l = MLP(LN(z z z l )) + z z z l (13)
with l = {1, . . . , L}. The MSA block consists of h parallel self-attention (SA) heads, where each SA head attends to bring information from different representation sub-spaces at different positions through a scoring function A. To achieve this goal, an input sequence

z z z ∈ R np×d is mapped into query (Q ∈ R np×d k ), key (K ∈ R np×d k ) and value (V ∈ R np×dv ) matrices using three learnable parameters: W W W q ∈ R d×d k , W W W k ∈ R d×d k and W W W v ∈ R d×dv . Q = z z zW W W q K = z z zW W W k V = z z zW W W v (14)
Then, the attention distribution function is computed following Eq.15 and the resulting attention weights are applied to the V matrix obtaining the SA maps, as described in Eq. [START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF].

A(Q, K) = softmax Q • K T √ d k (15) SA(Q, K, V ) = A(Q, K) × V (16)
For MSA, Q, K, and V are computed once for each head using h different learnable parameters (W W W q,k,v i ), and the final attention map results from the concatenation of the h heads multiplied by a learnable aggregation matrix W W W o ∈ R hdv×d . The computational cost of single head attention with full ddimension is maintained by setting d k and d v equal to d/h.

MSA(Q, K, V ) = [head 1 ; . . . ; head h ]W W W o (17)
with

head i = SA(Q i , K i , V i ) = SA(z z zW W W q i , z z zW W W k i , z z zW W W v i ).
In the context of medical Transformers, U-Net-shaped architecture remains the preferred choice to build the Transformer segmentation models. From them and as illustrated in Fig. 6, three categories can be identified [START_REF] Andrade-Miranda | Pure versus hybrid Transformers for multi-modal brain tumor segmentation: a comparative study[END_REF]: pure Transformersbased encoder, hybrid Transformers-CNN encoder as well as full Transformers-based network.

1) Pure Transformers-based encoder: The first category exploits the global context modeling capability of Transformers to effectively encode the relationships between spatially distant voxels. A convolution-free encoder is introduced by forwarding flattened image representations to Transformers, whose outputs are then reorganized into 3D tensors followed by CNN up-sampling blocks with multi-level feature aggregation. For instance, [START_REF] Hatamizadeh | UNETR: Transformers for 3D medical image segmentation[END_REF][START_REF] Karimi | Convolution-free medical image segmentation using Transformers[END_REF] employed a 3D ViT as an encoder and connected it to the CNN decoder via skip-connections. At the bottleneck of the encoder, the feature map was reshaped and up-sampled by a factor of 2. Then, the previous Transformer layer was used as a skip connection and concatenated with the resized feature to be later up-sampled through convolution, normalization and linear activation. This process was repeated until the initial resolution was reached. However, as anatomical structures can substantially vary in scale, they cannot be properly modelled using a set of fixed sub-regions of the image. Recently, hierarchical ViT such as Swin [START_REF] Hatamizadeh | Swin UNETR: Swin Transformers for semantic segmentation of brain tumors in MR images[END_REF] or PVT [START_REF] Dong | Polyp-PVT: Polyp segmentation with pyramid vision transformers[END_REF] Transformers have been introduced to overcome these challenges by extracting features at different resolutions. This improves the performance of Transformers in dense prediction tasks while saving the linear computational complexity with respect to the image size. Hierarchical ViT architectures introduce CNN-like properties into the Transformers as they compute local attention with shifted windows, starting from small-sized patches and gradually merging neighboring patches in the subsequent layers. To reduce the design complexity of traditional hierarchical ViT, a 3D U-shape model inspired by nested hierarchical Transformers [START_REF] Zhang | Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding[END_REF] exploited the idea of global self-attention within smaller non-overlapping 3D blocks [START_REF] Yu | Characterizing renal structures with 3D block aggregate Transformers[END_REF]. Cross-block self-attention communication was achieved by hierarchically nesting these Transformers and connecting them with a specific aggregation function. Valanarasu et al. proposed in [START_REF] Valanarasu | Medical Transformer: Gated axial-attention for medical image segmentation[END_REF] a gated axial-attention model that extended previous designs by incorporating a new control mechanism in the self-attention module. Furthermore, the model operated on the whole image and patches to simultaneously learn both global and local features.

2) Hybrid CNN-Transformers encoder: This second category integrates the global context modeling ability of Transformers with the CNN inductive bias [START_REF] Chen | TransUNet: Transformers make strong encoders for medical image segmentation[END_REF][START_REF] Wang | TransBTS: Multimodal brain tumor segmentation using Transformer[END_REF][START_REF] Li | TransBTSv2: Wider instead of deeper transformer for medical image segmentation[END_REF]. CNN layers capture the multi-scale context feature maps by stacking convolution blocks. Meanwhile, Transformers capture the long-term dependencies among the features that would be potentially lost with purely convolutional models. Lastly, a CNN-based decoder gradually up-samples the Transformers output into a 4D feature map to recover the full segmentation mask (Fig. 7). Others approaches modified the traditional SA blocks using deformable Transformers [START_REF] Xie | CoTr: Efficiently bridging CNN and Transformer for 3D medical image segmentation[END_REF] or squeeze-and-expansion Transformers layers [START_REF] Li | Medical image segmentation using squeeze-and-expansion Transformers[END_REF]. In [START_REF] Chen | TransAttUNet: Multi-level attention-guided U-Net with Transformer for medical image segmentation[END_REF], Chen et al. proposed a 2D hybrid network that combines two independent self-attention blocks to model the long-range interactions and global spatial relationships. In addition, a multi-scale skip-connection scheme aggregated multiple features in the decoder at a different scale to generate more discriminative representations. PC-SwinMorph [START_REF] Liu | PC-SwinMorph: Patch representation for unsupervised medical image registration and segmentation[END_REF] performed registration-based segmentation using both CNN and patch-based contrastive strategies followed by a Swin Transformer which enforced the capture both global and local anatomical representations. Another type of hybrid Transformers-CNN encoders deal with multi-branch fusion schemes like TransFuse [START_REF] Zhang | Transfuse: Fusing Transformers and CNNs for medical image segmentation[END_REF], HybridCTrM [START_REF] Sun | HybridCTrm: Bridging CNN and Transformer for multimodal brain image segmentation[END_REF] and CrossTeaching [START_REF] Luo | Semi-supervised medical image segmentation via cross teaching between CNN and Transformer[END_REF]. Commonly, the two parallel branches, one for CNN and the other for Transformers, are fused to benefit from the two learning paradigms. The CNN branch provides the ability to focus on local information while Transformers learn long-range voxel dependencies.

3) Full Transformers network: Full Transformers architectures are built in an end-to-end Transformer-based fashion [START_REF] Zhou | nnFormer: Interleaved transformer for volumetric segmentation[END_REF][START_REF] Wu | D-former: A U-shaped dilated Transformer for 3D medical image segmentation[END_REF]. Cao et al. proposed in [START_REF] Cao | Swin-UNet: UNet-like pure Transformer for medical image segmentation[END_REF] a Swin 2D U-Net network which includes a patch expanding layer to up-sample the feature maps of the decoder. This architecture showed superior performance in capturing fine details compared to decoders based on bi-linear up-sampling. Lin et al. went a step further in [START_REF] Lin | DS-TransUNet: Dual Swin Transformer U-Net for medical image segmentation[END_REF] by first adopting a dual-scale encoder based on 2D Swin-Transformer to extract both coarse and fine-grained feature representations at different scales. They also included an interactive fusion module to effectively establish global dependencies between features of different scales through the self-attention mechanism. To better leverage multi-scale feature hierarchies, Huang et al. proposed in [START_REF] Huang | MISSFormer: An effective medical image segmentation transformer[END_REF] a 2D hierarchical encoder-decoder architecture whose main contribution was the inclusion of an enhanced Transformer context bridge to capture the correlation and local context of multi-scale features generated by the hierarchical Transformer encoder. Peiris et al. developed in [START_REF] Peiris | A volumetric Transformer for accurate 3D tumor segmentation[END_REF] VTU-Net that works directly in 3D using Swin Transformers. The Swin decoder introduces parallel cross-attention and self-attention, which creates a bridge between queries from the decoder, keys, and values from the encoder. Such parallelization enables to preserve the full global context during the decoding process, which is key towards a robust delineation of medical images.

B. Multi-task learning

Another way to improve image segmentation consists in exploiting the ability of deep models to simultaneously deal with multiple tasks. Multi-task learning aims at taking advantage of the information shared among two or more connected or auxiliary tasks while better handling each task individually [START_REF] Tajbakhsh | Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation[END_REF]. As investigated in [START_REF] Ruder | An overview of multi-task learning in deep neural networks[END_REF], multi-task learning can be seen as a form of inductive transfer where the introduced inductive bias allows to prefer some hypotheses over others (i.e. the ones that explain more than one task), towards solutions that generalize better than their individual counterparts. One of the first multi-task learning frameworks dedicated to medical image segmentation was developed in [START_REF] Moeskops | Deep learning for multi-task medical image segmentation in multiple modalities[END_REF] which investigated whether a single CNN can be trained to perform different segmentation tasks, i.e. in a multi-domain fashion (Sect. VI-B). The combined training procedure was therefore suited for identifying anatomical structures, tissue classes as well as imaging modalities at once. More globally, multi-task learning can leverage various and heterogeneous forms of annotations, from global images to finer-grained and pixel-level labels. Based on the finding that several segmentation, classification, regression or detection issues can be effectively solved at once using a single network, multi-tasking strategies have emerged through the design of a cascade of task-specific sub-networks [START_REF] Song | An end-to-end multi-task deep learning framework for skin lesion analysis[END_REF] or the development of networks with shared encoder and task-specific decoders [START_REF] Playout | A multitask learning architecture for simultaneous segmentation of bright and red lesions in fundus images[END_REF][START_REF] Murugesan | Psi-Net: Shape and boundary aware joint multi-task deep network for medical image segmentation[END_REF][START_REF] Zhang | 3D multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification[END_REF][START_REF] Chen | Multi-task attention-based semi-supervised learning for medical image segmentation[END_REF] to benefit from partial parameters sharing between tasks or sub-tasks.

In particular, the encoder-decoder architecture designed in [START_REF] Playout | A multitask learning architecture for simultaneous segmentation of bright and red lesions in fundus images[END_REF] featured a single encoding path and multiple decoding branches for concurrent segmentation tasks. The encoding module used a generic set of parameters shared by multiple tasks whereas the decoding branches were task-specific. An auxiliary cost was also added at the end of the encoding module to predict the presence or absence of lesions. Supplementary sub-tasks including contour detection and distance map estimation were incorporated in [START_REF] Murugesan | Psi-Net: Shape and boundary aware joint multi-task deep network for medical image segmentation[END_REF] to refine coarse and discontinuous segmentation predictions from convolutional models. In [START_REF] Zhang | 3D multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification[END_REF], a single multi-task network was proposed to simultaneously address gastric tumor segmentation and lymph node classification from CT scans. An attentionbased reconstruction task was integrated into the segmentation pipeline of [START_REF] Chen | Multi-task attention-based semi-supervised learning for medical image segmentation[END_REF] to leverage unlabeled medical images in a semi-supervised segmentation framework (Sect.VI-D). In [START_REF] Keshwani | TopNet: Topology preserving metric learning for vessel tree reconstruction and labelling[END_REF], Keshwani et al. improved vessel CT segmentation not only by considering a single decoding branch dedicated to vessel segmentation but also by involving two additional decoders: a centerness decoder whose task was formulated as a regression problem and a topology distance decoder aiming at enhancing the vessel connectivity which is key regarding clinical needs.

By enabling fruitful cooperation between related tasks, these multi-task approaches have been shown to outperform traditional independent or segmentation-only models. As for deep supervision (Sect.IV-D), the relative weighting between each task's loss can be tuned by hand or automatically [START_REF] Kendall | What uncertainties do we need in bayesian deep learning for computer vision?[END_REF].

C. Segmentation uncertainty

In DL-based image segmentation, the forward pass is a deterministic process that maps a voxel to a unique label. This apparent determinism however fails to take into account the various sources of uncertainty that affect a neural network pre-diction. Such awareness may be crucial to detect potential segmentation errors. There is a clear need to understand the limitations of segmentation models via the assessment of voxelwise confidence measures, which is the purpose of uncertainty quantification applied to segmentation [START_REF] Abdar | A review of uncertainty quantification in deep learning: Techniques, applications and challenges[END_REF][START_REF] Jungo | Assessing reliability and challenges of uncertainty estimations for medical image segmentation[END_REF]. Uncertainty modelling may also importantly be used to directly improve segmentation performance, as in [START_REF] Galati | From accuracy to reliability and robustness in cardiac magnetic resonance image segmentation: a review[END_REF] where the distinction between quality control and model improvement techniques is highlighted. This is typically done by averaging out voxellevel errors using multiple stochastic forward passes in models that can sample across the uncertainty space [START_REF] Wang | Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks[END_REF][START_REF] Abdar | A review of uncertainty quantification in deep learning: Techniques, applications and challenges[END_REF].

According to Bayesian terminology, uncertainty can be divided into epistemic and aleatoric uncertainty [START_REF] Kendall | What uncertainties do we need in bayesian deep learning for computer vision?[END_REF][START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF]. Epistemic (or model) uncertainty relates to the lack of accuracy in the model parameters due to insufficient training. It is a kind of uncertainty that can be reduced by providing more training time and/or data. Aleatoric uncertainty, on the other hand, relates to the inherent uncertainty of the data itself, which can further be divided into homoscedastic uncertainty (constant for all inputs) and heteroscedastic uncertainty (variable between inputs). In image segmentation, both image and label spaces are affected by aleatoric uncertainty. Example causes of homoscedastic uncertainty in radiation imaging are physical processes such as Compton scattering or positron range. Image-space heteroscedastic uncertainty can be, for instance, due to dataset shifts in multi-center studies, while labelspace heteroscedastic uncertainty may be due to heterogeneity in annotation quality [START_REF] Castro | Causality matters in medical imaging[END_REF]. Ideally, uncertainty assessment should be calibrated. Calibration means that prediction confidence c should equal its likelihood of being correct, i.e. a value of c ∈ [0, 100] should indeed translate to a model being accurate c% of the time over multiple instances, which is an open research subject in medical imaging [START_REF] Abdar | A review of uncertainty quantification in deep learning: Techniques, applications and challenges[END_REF][START_REF] Jungo | Assessing reliability and challenges of uncertainty estimations for medical image segmentation[END_REF][START_REF] Guo | On calibration of modern neural networks[END_REF]. Several techniques may be employed to produce voxel-level confidence maps for both epistemic and aleatoric uncertainty quantification of segmentation predictions. The most popular epistemic uncertainty measurement is Monte-Carlo (MC) dropout, also known as test-time dropout (TTD) [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF], where many (e.g. from 10 to 50) stochastic dropout forward passes of a model equipped with dropout weights are performed during training [START_REF] Xia | Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation[END_REF][START_REF] Nair | Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation[END_REF][START_REF] Wickstrøm | Uncertainty and interpretability in convolutional neural networks for semantic segmentation of colorectal polyps[END_REF]. The dissimilarity in the predictions, assessed through variance, mutual information or entropy [START_REF] Nair | Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation[END_REF] can then be assimilated to a voxel-wise epistemic uncertainty map. An obvious drawback of MC dropout is the requirement for dropout during training. Dropout may indeed be detrimental to segmentation performance, and a number of state-of-the-art segmentation solutions including nnU-Net [START_REF] Isensee | nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation[END_REF] do not include dropout. Alternatives to MC dropout for epistemic uncertainty quantification include performing forward passes at various training checkpoints of the optimization [START_REF] Zhao | Efficient bayesian uncertainty estimation for nnu-net[END_REF], following the empirical observation that less certain predictions are less stably predicted along training. A more computationally demanding method is deep ensembling, whereby independently trained networks are averaged together to get uncertainty maps [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF]. Albeit more demanding, deep ensembling is a common practice due to its consistency in improving segmentation results [START_REF] Isensee | nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation[END_REF]. Thus, uncertainty maps may be derived freely as a by-product of this main objective.

Data or aleatoric uncertainty, on the other hand, can be assessed through test-time data augmentation (TTA), in which multiple forward passes are performed to inputs altered through basic data augmentations (e.g. flips, rotations, scaling) [START_REF] Wang | Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks[END_REF]. The resulting outputs are then aggregated with similar methods to TTD (i.e. averaging, entropy). TTA is easier to implement than TTD as it does not require any modification to the network architecture and can readily be achieved through off-the-shelf data augmentation and segmentation frameworks. Qualitative results seem to suggest that aleatoric uncertainty estimates provide more expressive qualitative maps for medical image segmentation uncertainty assessment [START_REF] Wang | Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks[END_REF].

Regarding the improvement of performance through uncertainty sampling, using epistemic uncertainty modelling with TTD or MC dropout generally yields moderate but consistent improvement of segmentation results [START_REF] Wang | Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks[END_REF][START_REF] Zhao | Efficient bayesian uncertainty estimation for nnu-net[END_REF]. For instance, epistemic uncertainty-aware networks achieved state-of-the-art performance on the medical image segmentation decathlon challenge [START_REF] Xia | Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation[END_REF][START_REF] Antonelli | The medical segmentation decathlon[END_REF]. On the other hand, TTA seems to be more effective than TTD for improving medical image segmentation results, with performance enhanced by up to several Dice points [START_REF] Wang | Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks[END_REF]. TTA is therefore a generally recommended step if inference cost is a secondary concern. The question as to what is the optimal way to pool TTA predictions and what augmentation to select is an open research subject [START_REF] Shanmugam | Better aggregation in test-time augmentation[END_REF]. Applications of deep uncertainty modelling to PET-CT are less popular than in MR modality, with few related works in segmentation [START_REF] Ma | Combining CNN and hybrid active contours for head and neck tumor segmentation in CT and PET images[END_REF][START_REF] Hu | Coarse-to-fine adversarial networks and zonebased uncertainty analysis for NK/T-cell lymphoma segmentation in CT/PET images[END_REF]. Sudarshan et al. leverage physicsbased heteroscedastic uncertainty modelling for low-dose PET-MR image denoising [START_REF] Sudarshan | Towards lower-dose PET using physics-based uncertaintyaware multimodal learning with robustness to out-of-distribution data[END_REF]. This relative lack is arguably due to the novelty of the topic in medical imaging. Uncertainty quantification being an emerging trend, more contributions are expected in the future, especially in radiation imaging.

D. Contrastive learning

Whatever the image analysis task involving representation learning, extracting robust features means reaching distinct clusters reflecting the different classes involved. In this direction, contrastive learning tends to enforce the model to learn an efficient and disentangled feature representation by comparing the input image with comparing images (i.e. anchors). The comparison is performed between positive pairs of similar inputs (e.g. generated through data augmentation) and negative pairs of dissimilar inputs (e.g. other image samples used for training). The original contrastive loss was initially defined for classification purposes in computer vision [START_REF] Chopra | Learning a similarity metric discriminatively, with application to face verification[END_REF] and its adoption in the medical image processing community has been relatively late [START_REF] Jamaludin | Self-supervised learning for spinal MRIs[END_REF]. The underlying idea was to pull together data points from the same class while pushing apart negative samples in embedded space [START_REF] Chopra | Learning a similarity metric discriminatively, with application to face verification[END_REF], thus imposing intra-class cohesion and inter-class separation.

1) Global contrastive learning: Most existing contrastive learning methods deal with a global contrastive loss and target image classification (Fig. 8a). For image segmentation, a global contrastive loss can still be used by projecting the data through the encoder path to the latent space [START_REF] Chaitanya | Contrastive learning of global and local features for medical image segmentation with limited annotations[END_REF][START_REF] Hu | Semi-supervised contrastive learning for label-efficient medical image segmentation[END_REF]. Let us describe how a global contrastive loss can be defined in this context. We consider an input batch B = {x x x 1 , x x x 2 , . . . , x x x b } where each x x x i corresponds to a 3D medical image or one of its 2D slice.

Using data augmentation, one can transform twice each x x x i to form a pair of augmented images. These new images form a set of augmented images A = {a a a 1 , a a a 2 , . . . , a a a 2×b }. For a given image x x x i , A contains two related augmented versions, referred to a a a j and a a a k . A global contrastive loss can therefore be used:

L con = - 1 |A| j∈{1,...,2×b} log exp(z z z j .z z z k /τ ) l =j exp(z z z j .z z z l /τ ) (18)
where z z z j and z z z k are the normalized features obtained after applying a header function h(.) (e.g. multi-layer perceptron) at the output of the encoder E such that z z z j = |h(E(a a a j ))|. τ is a temperature scaling parameter. As opposed to standard approaches that operate on image classes, a dataset label information was leveraged in [START_REF] Boutillon | Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors[END_REF] to enhance intra-domain similarity and impose inter-domain margins, in a multi-task multi-domain segmentation scenario.

2) Local contrastive learning: More suited for medical image segmentation, a local contrastive learning approach can be followed by designing a local version of the contrastive loss (Eq.18) able to learn distinctive representations of local regions instead of relying on global representations. Only teaching the encoder to extract image-level disentangled features may not be sufficient since segmentation requires a class prediction for all voxels [START_REF] Hu | Semi-supervised contrastive learning for label-efficient medical image segmentation[END_REF]. Two main techniques can be followed to train the decoder at extracting distinctive local representations through contrastive learning: self-supervised (Fig. 8b) and supervised (Fig. 8c) local contrastive learning.

For the first category, Chaitanya et al. introduced in [START_REF] Chaitanya | Contrastive learning of global and local features for medical image segmentation with limited annotations[END_REF] a local extension of the contrastive loss that is helpful for perpixel segmentation as it learns distinctive representations of local regions. Thus, it compares the local features of the image to be equivalent underneath various transformations and also focuses on the dissimilarity with other regions from the same image. Validated on three MR datasets, this method lead to a substantial increment in delineation accuracy. Since generating pairs of data for the use of contrastive learning is challenging in medical image segmentation due to the potential presence of similar tissue or anatomical structure across the dataset, Zeng et al. explored in [START_REF] Zeng | Positional contrastive learning for volumetric medical image segmentation[END_REF] a novel method called positional contrastive learning. The method dealed with generating contrastive data pairs based on the position of a slice within 3D volumes. Based on the slice distance, more closest slices were referred to as positive pairs, and far slices were considered negative. This enabled the reduction of false-negative image pairs and improved the segmentation results against state-ofthe-art. Self-supervised local contrastive learning is only one of the strategies followed in self-supervised learning whose perimeter is further explained in Sect.VI-C.

Conversely, a supervised local contrastive loss (Fig. 8c) that leverages limited pixel-wise annotation to force pixels with the same label to gather around in the embedding space was proposed in [START_REF] Hu | Semi-supervised contrastive learning for label-efficient medical image segmentation[END_REF]. Some papers combine global and selfsupervised [START_REF] Chaitanya | Contrastive learning of global and local features for medical image segmentation with limited annotations[END_REF] or supervised [START_REF] Hu | Semi-supervised contrastive learning for label-efficient medical image segmentation[END_REF] local contrastive learning.

Contrastive learning was also adopted in a federated learning context (Sect.VI-E). In [START_REF] Wu | Federated contrastive learning for volumetric medical image segmentation[END_REF], Wu et al. explored contrastive learning for volumetric medical image segmentation in the presence of limited labeled data. Following a similar approach to the above-discussed ones, clients first learn a shared encoder on unlabeled data from various sites. Then, a network is finetuned on a labeled dataset. The mixed contrastive data are supplied to each medical location, enabling the use of data variousness for contrastive learning. This enables performing global structural matching to learn an encoder with suitable representations among clients.

E. Knowledge distillation

Compared to the computationally expensive scenario consisting of training many different models on the same data and then averaging their predictions, compressing the knowledge into a single model through knowledge distillation is much easier to deploy [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF]. The knowledge distillation (KD) mechanism tends to distill (i.e. transfer) information from a well-trained cumbersome teacher network to a lightweight and compact student network with the final goal of improving the performance of this student model. In a standard KD setting, the teacher model generates soft predictions which are used to supervise the student model by calculating the difference of their final layer with some measurement functions such as cross-entropy or Kullback-Leibler divergence (KL). The soft predictions are obtained after the last convolution layer by mean of a parameterized softmax function following:

q q q c = exp( z z zc τ ) exp c∈C z z z c τ ( 19
)
where the logits z z z c associated to each class c are converted into probabilities q q q c . τ is a parameter called temperature which controls the softness of the output probabilities. Since tuning τ is a difficult process and considering τ as a constant can lead to sub-optimal results, feature normalized knowledge distillation (FNKD) was proposed in [START_REF] Xu | Feature normalized knowledge distillation for image classification[END_REF] to exploit reliable soft predictions irregardless of the feature scale by rather considering the L2 norm instead of a scalar value:

q q q c = exp z z zc z z z exp c∈C z z z c z z z (20)
with . corresponding to the L2 norm. Instead of focusing on the final layer only from both teacher and student models, guiding the compact network to mimic intermediate features from the teacher network was adopted in [START_REF] Zagoruyko | Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[END_REF] by means of attention maps. Initially proposed in the context of image classification, KD mechanisms were rapidly extended for semantic segmentation purposes in computer vision [START_REF] He | Knowledge adaptation for efficient semantic segmentation[END_REF][START_REF] Liu | Structured knowledge distillation for semantic segmentation[END_REF] and medical image analysis [START_REF] Qin | Efficient medical image segmentation based on knowledge distillation[END_REF] fields. Thus, a prediction map distillation module was used in [START_REF] Qin | Efficient medical image segmentation based on knowledge distillation[END_REF] to enable the student network to learn predictive capability from the output segmentation maps provided by the teacher. Let φ s be the student model, q q q s n and q q q t n the soft probability maps respectively from the student and teacher networks. The loss function for training the student φ s with standard KD can be defined as:

L φ s = 1 N N n=1
λ KL(q q q s n ||q q q t n )+(1-λ) CE (φ s (x x x n ), y y y n ) [START_REF] Chen | Recent advances and clinical applications of deep learning in medical image analysis[END_REF] with λ ∈ [0, 1] a scalar value adjusting the contributions of both terms. Note that cross-entropy can easily replace KL for the knowledge distillation sub-loss. The standard supervised loss (cross-entropy in Eq.21) can be any dedicated loss function (e.g. DiceCE), as mentioned in Sect.II-A. Such loss function making the student mimicking the ability of the teacher to generate soft prediction maps is however not enough to really boost the performance of the student since only pixel-level information is considered. In this direction, more context and class-related information are needed. As for classification [START_REF] Zagoruyko | Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[END_REF], constraints on intermediate multi-scale features arising from both teacher and student were therefore integrated [START_REF] Qin | Efficient medical image segmentation based on knowledge distillation[END_REF] with importance maps distillation modules able to encode feature maps into a transformable form to deal with the diversity of feature sizes between teacher and student models. Other constraints were proposed in the specific context of knowledge distillation such as boundary-guided [START_REF] Wen | Towards efficient medical image segmentation via boundary-guided knowledge distillation[END_REF], region affinity [START_REF] Qin | Efficient medical image segmentation based on knowledge distillation[END_REF], class-similarity [START_REF] Tung | Similarity-preserving knowledge distillation[END_REF], anatomical knowledge [START_REF] Kats | Soft labeling by distilling anatomical knowledge for improved MS lesion segmentation[END_REF] or holistic distillation [START_REF] Liu | Structured knowledge distillation for dense prediction[END_REF] in order to align high-order relations between what both teacher and student generated.

To go further, a novel KD based framework called multiple teachers single student (MTSS) was developed as a new privacy multi-organ segmentation setting learning from multiple pre-trained single-organ segmentation models [START_REF] Zhang | Unsupervised ensemble distillation for multi-organ segmentation[END_REF]. Formulated into a special unsupervised ensemble distillation problem, multiple single-organ models served as teachers from different specialties and collaboratively teach one general student, i.e. the multi-organ segmentation model. Further, the integration of a co-training strategy and weight-averaged models unified multi-organ segmentation from few-organ datasets [START_REF] Huang | Multi-organ segmentation via co-training weight-averaged models from few-organ datasets[END_REF]. Self-distilling a Transformer-based U-Net by simultaneously learning global semantic information and local spatialdetailed features was also investigated in [START_REF] Wang | MISSU: 3D medical image segmentation via self-distilling TransUNet[END_REF].

VI. EMERGING APPLICATIONS

A. Cross-modality segmentation

Most of the approaches presented in the literature do not consider the multi-modal nature of medical imaging data, leaving aside potentially valuable cross-modal information unused. However, exploiting complementary and redundancy information across modalities can possibly improve overall segmentation performance, making better use of the scarcity of annotated medical imaging data. The research field of multi-modal image segmentation brings different technical challenges and open questions to solve [START_REF] Baltrušaitis | Multimodal machine learning: A survey and taxonomy[END_REF], including:

• Is the data available for training pairwise-aligned or comes from different patients? • How to fuse different modalities to simultaneously reduce the heterogeneity gap and enable the transfer knowledge? • How to map data from one modality to another? Considering these questions, we introduce in what follows the multi-modal works depending on the type of data available during training (paired, unpaired), the followed fusion strategy (early, mid or late) [START_REF] Zhou | A review: Deep learning for medical image segmentation using multi-modality fusion[END_REF] as well as the adopted translation approach. Fig. 9 depicts a generic categorization of the cross-modal segmentation framework whereas Fig. 10 illustrates a cross-modal pipeline when managing paired data.

1) Paired data and early fusion strategy: When multimodal paired data is available, one may carry out any of the three fusion strategies. The most straightforward is the early (also known as input-level) fusion strategy (Fig. 10a) which integrates at the input level of the deep network the different m-modalities. Therefore, the final segmentation is defined as y = φ(x 1 , x 2 , . . . , x m ) where φ represents the segmentation network. Early fusion has the advantage of its simplicity, allowing more complex segmentation strategies such as multitask, multi-view, multi-scale or GAN-based approaches. One of the first works devoted to solving multi-modal image segmentation using CNN can be found in [START_REF] Havaei | Brain tumor segmentation with deep neural networks[END_REF]. They explored two-pathway cascaded architectures using different receptive field sizes to capture both local and global context information. Qin et al. proposed in [START_REF] Qin | Autofocus layer for semantic segmentation[END_REF] an adaptive convolutional layer named autofocus to effectively change the size of the receptive field to perform multi-modal brain tumor segmentation. The autofocus layer captured the multi-scale information by parallelizing multiple convolutional layers with different dilatation rates that are later merged using a weighted soft-attention mechanism to choose the optimal scales.

The above-mentioned methods and others [START_REF] Kamnitsas | Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation[END_REF][START_REF] Pereira | Brain tumor segmentation using convolutional neural networks in MRI images[END_REF] did not make dense predictions and are therefore slow in the inference stage. To promote efficiency, encoder-decoder architecture derived from U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF] has been widely adopted. For instance, Shapey et al. used in [START_REF] Shapey | An artificial intelligence framework for automatic segmentation and volumetry of vestibular schwannomas from contrast-enhanced T1-weighted and high-resolution T2-weighted MRI[END_REF] a 2.5D U-Net to segment the vestibular schwannoma in contrastenhanced T1-weighted and high-resolution T2-weighted MR imaging. Spatial attention modules were added to each level of the decoder to deal with small target regions, giving more attention to them and penalizing voxels belonging to the background. In [START_REF] Yang | Volumetric adversarial training for ischemic stroke lesion segmentation[END_REF], the modalities were fused as multi-channel inputs and passed through an adversarial network (Sect.IV-A). The generator is a 3D residual U-Net that performs the segmentation while the discriminator distinguishes between generated segmentation and ground truth masks. An extra constraint was added via active contour modeling by measuring the dissimilarity between ground truth and prediction contours. To handle the class imbalance problem, Zhou et al. carried out in [START_REF] Zhou | One-pass multitask convolutional neural networks for efficient brain tumor segmentation[END_REF] a coarse-to-fine segmentation inspired on model cascades for brain tumor segmentation. The main difference with previous works lies in applying only a one-pass multi-task network (OM-Net) that performs three tasks that are gradually introduced in an order of increasing difficulty based on curriculum learning. The first task learns to differentiate between tumors and normal tissue until the loss curve tends to flatten. The second task is then added and split the complete tumor into intra-tumoral classes. This task continues until its loss curve displays a flattening trend. Lastly, the third task is introduced and trained simultaneously with the previous ones to precisely segment the enhancing tumor. In this way, the model parameters and the training data are transferred from an easier to a more difficult task. Unfortunately, early fusion makes it hard to discover highly non-linear relationships between the low-level features from different modalities, especially when the modalities have significantly different statistical properties.

2) Paired data and mid-fusion strategy: Mid or layer-level fusion separately processes the multi-modal data in different paths (Fig. 10b). For each modality, m, the input x m is encoded in each branch f m to learn the modality-specific representation z m . Then, each representation z m is mapped into a common latent space via a fusion operation Λ and use this as input of the decoding transformations g(Λ(z 1 , z 2 , . . . , z m )). The main goal of this strategy is to learn an optimal joint representation that emphasizes the most informative features across modalities. In mid-fusion, we can distinguish two types of multi-pathway network architectures based on the following fusion strategy: single-layer or multi-layer fusion.

Multi-modal segmentation networks based on single-layer fusion generally employ encoder-decoder architectures where each modality has its own encoder, with no interactions between them, and a single decoder. They mainly differ on the conducted fusion operation and are typically carried out via concatenation, addition, averaging or convolution. For instance, Havaei et al. used in [START_REF] Havaei | HeMIS: Heteromodal image segmentation[END_REF] modality-specific convolutional layers to later compute for each feature map the first and second moments. Then, the moments were concatenated and processed by further convolutional layers, yielding the final segmentation. For their part, Tseng et al. took in [START_REF] Tseng | Joint sequence learning and cross-modality convolution for 3D biomedical segmentation[END_REF] the encoded representation from each modality and performed a cross-modal convolution to combine the spatial information of each feature map, modeling the correlations among them. In-spired by the success of the attention mechanism (Sect.IV-E), recent fusion strategies incorporated spatial and channel-wise attention to learn more informative features among modalities [START_REF] Zhou | Fusion based on attention mechanism and context constraint for multi-modal brain tumor segmentation[END_REF][START_REF] Zhou | Latent correlation repre-sentation learning for brain tumor segmentation with missing MRI modalities[END_REF]. To name a few, Zhou et al. proposed in [START_REF] Zhou | Fusion based on attention mechanism and context constraint for multi-modal brain tumor segmentation[END_REF] a three-stage segmentation network. In the first stage, a 3D U-Net architecture got rough mask predictions. Then, binarization and erosion operations were used to obtain the context constraints for the following stage. The second stage consisted of a multi-encoder-based framework where each encoder produces a modality-specific representation that is further fused with the assistance of attention mechanisms. This process was repeated for each structure to be segmented. In the third stage, a two-encoder-based 3D U-Net segmentation network was applied to combine and refine the three single prediction results. A correlation block to discover the latent correlation between modalities was introduced in [START_REF] Zhou | Latent correlation repre-sentation learning for brain tumor segmentation with missing MRI modalities[END_REF], followed by a dual attention block that consists of a modality attention module and a spatial attention module. In this way, the network is encouraged to learn the most correlated features across modalities and more useful spatial information to boost segmentation results. Despite the great results of single-layer fusion schemes, there is no complete freedom to learn within and in-between modalities due to its single level of abstraction.

Regarding multi-layer fusion, it extends the idea of residual learning in multi-modal frameworks allowing skipconnections that by-pass spatial features between modalities [START_REF] Li | Learning cross-modal deep representations for multi-modal MR image segmentation[END_REF][START_REF] Dolz | HyperDense-Net: a hyper-densely connected CNN for multimodal image segmentation[END_REF][START_REF] Zhang | Crossmodality deep feature learning for brain tumor segmentation[END_REF]. Therefore, low-level and high-level features are fused at different levels of abstraction, increasing the learning capabilities of the network to capture complex cues across modalities. Li et al. proposed in [START_REF] Li | Learning cross-modal deep representations for multi-modal MR image segmentation[END_REF] four dilated Inception blocks consisting of three dilated convolutional layers for each modality. In this way, the receptive field of the network was expanded without losing resolution, while multi-scale features were also learned. In order to obtain the final segmentation, the features at different levels were concatenated and up-sampled. On the other hand, Dolz et al. proposed in [START_REF] Dolz | HyperDense-Net: a hyper-densely connected CNN for multimodal image segmentation[END_REF] HyperDenseNet, a 3D model where each modality has its own path. Dense connections not only occur between the layers within the same stream but also across modalities. Thus, the network can learn more powerful feature representations at all levels of abstraction. To encode more rich contextual information across modalities, Zhang et al. developed in [START_REF] Zhang | Crossmodality deep feature learning for brain tumor segmentation[END_REF] a cross-modal self-attention distillation network. The model extracted attention maps of intermediate layers to further perform layer-wise attention distillation among modalities. Significant spatial information can be distilled from an attention map of one modality and then used to ease attention learning of the other modalities. Fusing multi-modal contextual information at multi-layer stages represents the current trend. Moreover, semantic guiding across modalities by attention mechanisms can be used to bridge early feature extraction and late decision-making.

3) Paired data and late fusion: Similar to midfusion, late fusion separately processes multi-modal data (x 1 , x 2 , . . . , x m ) with the difference that the segmentation branches (φ 1 , φ 2 , . . . , φ m ) are integrated at the decision level. More precisely, during the decoding stage, all feature maps computed by the branches are mapped into a common feature space via fusion operations, (e.g. concatenation, averaging, weighted voting), followed by series of convolutional layers [START_REF] Zhang | Deep multimodal fusion for semantic image segmentation: A survey[END_REF]. The final output of late fusion can be formulated as y = Λ(φ 1 (x 1 ), φ 2 (x 2 ), . . . , φ m (x m )) where Λ is the fusion operation. Thus, common features learned by the transformation network are considered as a further refinement of decoding and prediction. Some conventional layer-level methods as [START_REF] Zhang | Modality-aware mutual learning for multi-modal medical image segmentation[END_REF] are thus categorized into late fusion strategy. Many late fusion strategies have been proposed. Most of them are based on averaging or majority voting. For averaging strategy, Kamnitsas et al. trained in [START_REF] Kamnitsas | Ensembles of multiple models and architectures for robust brain tumour segmentation[END_REF] three networks separately and then averaged the confidence of the individual networks. The final segmentation was obtained by assigning each voxel with the highest confidence. For the majority voting strategy, the final label of a voxel depends on the majority of the labels of the individual networks. The statistical properties of the different modalities are different, which makes it difficult for a single model to directly find correlations across modalities. Therefore, in a decision-level fusion scheme, the multiple segmentation networks can be trained to fully exploit multimodal features. On the other hand, Zhang et al. proposed in [START_REF] Zhang | Modality-aware mutual learning for multi-modal medical image segmentation[END_REF] a modality-aware module that fused the modalityspecific models at a high semantic level. Specifically, each modality was embedded by a different modality-specific FCN. Then, the outputs of FCN models were fused and passed to an attention module to generate a modality-specific attention map to adaptively measure the contributions of each modality. Moreover, they designed a mutual learning strategy to enable interactive knowledge transfer, where the modalities interact as teacher and student simultaneously. In the same line, Zhang et al. employed in [START_REF] Zhang | Crossmodality deep feature learning for brain tumor segmentation[END_REF] a transfer knowledge strategy across modalities that differs from previous works in the use of GAN. The authors applied cycleGAN [START_REF] Zhu | Unpaired image-toimage translation using cycle-consistent adversarial networks[END_REF] to capture the knowledge transition across modalities. Each generator represented a single-modality feature learning branch. Then, they were merged by extra convolution layers followed by an attention block to learn powerful fusion features. The intuition behind the use of GAN is that GAN models can learn the modality patterns of each modality and their content patterns.

Mid and late-fusion can achieve better performance because each modality is employed as an input of one network that can learn complex and complementary feature information compared to an input-level fusion network. However, they require more memory due to the use of multiple networks. Therefore, the trade-off between accuracy and execution time should be carefully considered. Despite the impressive advances reached in the field of multi-modal image segmentation, collecting large sets of paired images is often either prohibitively expensive or not possible. As a result, techniques that make use of unpaired datasets have attracted increasing attention in cross-modal segmentation. In cross-modal image segmentation, the feature spaces between source and target domains are identical (i.e. X S = X T ), differing only in data distribution (i.e. P (x S ) = P (x T )). Hence, the goal is to learn a segmentation function f (•) that performs well in both source and target domains by finding a transformation T (•) such that P (T (x S )) = P (T (x T )). The previous single-source single-target definition can be extended to multi-source single-target or single-source multi-target DA.

Three groups of DA techniques are used for cross-modal image segmentation: supervised domain adaption in which both labeled source and target data are available {X S , Y S , X T , Y T }, semi-supervised domain adaptation in which labeled source data in addition to some labeled target data are available, and unsupervised domain adaptation (UDA) in which both labeled source data and unlabeled target data are available {X S , Y S , X T }. Most related works in cross-modal image segmentation are based on single-source single-target UDA [START_REF] Dorent | Cross-MoDA 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation[END_REF] using reconstruction-based methods, domain-invariant feature learning [START_REF] Wilson | A survey of unsupervised deep domain adaptation[END_REF] and more largely GAN models [START_REF] Ouyang | Data efficient unsupervised domain adaptation for cross-modality image segmentation[END_REF].

The main drawbacks of existing approaches deal with their limited scalability and robustness in handling more than two domains since they rely on pairwise alignment using GAN (cycleGAN or similar [START_REF] Ouyang | Data efficient unsupervised domain adaptation for cross-modality image segmentation[END_REF]). Hence, different models should be built independently for every pair of image domains which demands high computational resources. Lastly, either paired or unpaired methods are application-dependent which limits their transferability between different clinical settings.

B. Multi-domain segmentation

A strong assumption in the way deep segmentation pipelines are usually designed and evaluated is that both training and test data arise from the same probability distribution. Their accuracy usually degrades when applied to new (i.e. unseen) data that differ from the training data [START_REF] Kamnitsas | Unsupervised domain adaptation in brain lesion segmentation with adversarial networks[END_REF]. Instead of designing pipelines specific to a given intensity domain, an emerging application consists in training a deep segmentation model over multiple intensity domains [START_REF] Boutillon | Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors[END_REF][START_REF] Valindria | Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI[END_REF][START_REF] Karani | A lifelong learning approach to brain MR segmentation across scanners and protocols[END_REF][START_REF] Dou | Unpaired multimodal segmentation via knowledge distillation[END_REF]. The underlying assumption is that exploiting the redundancy between multiple intensity domains can enable the extraction of robust domaininvariant feature representations to finally achieve better performance than domain-specific (i.e. marginal) computational models. Managing various domains can partially solve the issue of dealing with the scarcity of imaging resources [START_REF] Karani | A lifelong learning approach to brain MR segmentation across scanners and protocols[END_REF]. The improved generalization abilities of the resulting models are a further step to facilitate their integration into routine.

In practice, intensity domains can be very different in nature: multi-center, multi-scanner, multi-modal (Sect.VI-A), or multi-protocol. Reasons explaining the acquisition shift include differences in imaging systems, reconstruction settings Fig. 11. Multi-domain segmentation with shared convolutional kernels and domain-specific feature normalization, as employed in [START_REF] Boutillon | Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors[END_REF][START_REF] Dou | Unpaired multimodal segmentation via knowledge distillation[END_REF].

or acquisition protocols. It is worth mentioning that variations can even happen in a single center since both clinical practices and imaging systems may significantly evolve over time. To deal with this diversity, segmentation pipelines can integrate an adversarial network to learn domain-invariant features [START_REF] Kamnitsas | Unsupervised domain adaptation in brain lesion segmentation with adversarial networks[END_REF], exploit transfer learning and fine-tuning between domains [START_REF] Karani | A lifelong learning approach to brain MR segmentation across scanners and protocols[END_REF], share a common decoder (resp. encoder) while using domain-specific encoders (resp. decoders) [START_REF] Valindria | Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI[END_REF] or share their latent space only [START_REF] Valindria | Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI[END_REF][START_REF] Dou | Unpaired multimodal segmentation via knowledge distillation[END_REF]. In particular, the very different statistical distributions between unpaired multi-modal images making the task of learning shared representations challenging has motivated the design of a single encoder-decoder segmentation network in [START_REF] Dou | Unpaired multimodal segmentation via knowledge distillation[END_REF] through shared convolutional kernels but domain-specific feature normalization (Fig. 11). Indeed, modality-agnostic kernels can extract expressive universal representations across domains only if the features are well-calibrated upstream. Let us consider D domains {d 1 , d 2 , . . . , d D }. Let v v v j i,l,m be the m th feature map from the l th layer produced by the i th image arising from the intensity domain d j . The calibration can be performed through domainspecific batch normalization (DSBN) [START_REF] Boutillon | Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors[END_REF][START_REF] Dou | Unpaired multimodal segmentation via knowledge distillation[END_REF], following:

DSBN α j l,m ,β j l,m (v v v j i,l,m ) = α j l,m v v v j i,l,m -µ j l,m (σ j l,m ) 2 + + β j l,m (22)
where µ j l,m and σ j l,m are respectively the domain-specific mean and standard deviation computed for images from domain d j belonging to a given batch. is a scalar value that is used to reach numerical stability. DSBN layers are therefore defined by trainable domain-specific shift and scale weights {α j l,m , β j l,m } l,m set for each feature map of each layer. Rather than focusing on a given anatomical target across various intensity domains, developing a single multi-task multi-domain network can enable simultaneously segmenting multiple anatomies while leveraging shared features between various domains and datasets [START_REF] Moeskops | Deep learning for multi-task medical image segmentation in multiple modalities[END_REF]. In this direction, shared convolutional kernels and domain-specific feature normalization from [START_REF] Dou | Unpaired multimodal segmentation via knowledge distillation[END_REF] were combined in [START_REF] Boutillon | Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors[END_REF] with both contrastive (Sect.V-D) and shape regularizations (Sect.IV-C) to segment bone structures from multiple scarce pediatric datasets.

C. Self-supervised learning

The need for a large amount of annotated training data is a strong constraint given the complexity of reaching a significantly well-annotated medical imaging dataset. Generative models were therefore used to increase the number of training samples through image synthesis [START_REF] Zhang | Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network[END_REF]. As an alternative, the availability of a reduced dataset only motivated researchers to exploit the power of unlabeled images which may be easier to collect. Unsupervised learning approaches assuming that a related but unlabeled large dataset is available aim at learning transferable feature representations from unlabeled images.

In particular, self-supervised learning can consist of pretraining the model (or any of its constituents) by means of pretext tasks to finally be more able efficiently delineate the targeted structures [START_REF] Taleb | 3D self-supervised methods for medical imaging[END_REF]. With the hypothesis of good generalization ability of self-learned features, Taleb et al. investigated the effectiveness of several pretext tasks for 3D medical image segmentation purposes: rotation prediction, jigsaw puzzles, relative patch location... More complex pretext tasks such as semantic inpainting revealed their effectiveness for better solving downstream segmentation tasks [START_REF] Hu | Self-supervised pretraining with DICOM metadata in ultrasound imaging[END_REF].

Leaving aside self-supervised pre-text tasks, self-supervised contrastive learning (Sect.V-D) is another manner adopted in the medical image analysis area to learn expressive feature representations from unlabeled images [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF]. In this direction, Chaitanya et al. proposed in [START_REF] Chaitanya | Contrastive learning of global and local features for medical image segmentation with limited annotations[END_REF] a local contrastive loss able to capture local features to provide complementary information and therefore boost the segmentation accuracy.

D. Semi-supervised learning

Given the usual scarcity of many existing annotated medical dataset and apart from transfer learning [START_REF] Pan | A survey on transfer learning[END_REF] whose goal is to learn from related learning problems, researchers have also explored semi-supervised learning approaches to exploit the availability of unlabeled datasets. Among the existing semisupervised strategies [START_REF] Cheplygina | Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis[END_REF], semi-supervised consistency regularization is commonly employed through the use of a mean teacher model [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF]. In [START_REF] Yu | Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation[END_REF], a novel uncertainty-aware semisupervised learning framework was proposed and evaluated for left atrium segmentation from MR images. Teacher and student models were built in such a way that the student model learned from the teacher model by minimizing the segmentation loss on the labeled data as well as a consistency loss with respect to the targets from the teacher model on all data (i.e. labeled and unlabeled). The predicted targets from the teacher model being potentially unreliable and noisy on unlabeled data, Yu et al. designed an uncertainty-aware mean teacher framework [START_REF] Yu | Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation[END_REF], where the student model gradually learned from the meaningful and reliable targets by exploiting the uncertainty information arising from the teacher model (Sect.V-C). To better deal with noisy labels for COVID-19 pneumonia lesion segmentation, the main novelty in [START_REF] Wang | A noise-robust framework for automatic segmentation of COVID-19 pneumonia lesions from CT images[END_REF] was to propose two mechanisms: an adaptive teacher that suppresses the contribution of the student when the latter has a large training loss and an adaptive student that learns from the teacher only when the teacher outperforms the student.

As followed in [START_REF] Fan | Inf-Net: Automatic COVID-19 lung infection segmentation from ct images[END_REF], semi-supervised pseudo labeling is another strategy to deal with both annotated and unlabeled data. Thus, Fan et al. generated pseudo-labels by relying on a first training with 50 labeled images only. The newly pseudo-labeled examples were then included in the original labeled training dataset to re-train the model. This updated model was used to generate pseudo-labels for another batch of unlabeled images and so on. This process was repeated up to obtain efficient performance in COVID-19 lung infection CT segmentation. However, the created pseudo-labels usually do not have the same quality as ground truth labels, which may limit their potential for improvements from unlabeled data.

As a powerful alternative, one may adopt an auxiliary task on unlabeled data to facilitate performing image segmentation with limited labeled data. In this direction, Chen et al. proposed in [START_REF] Chen | Multi-task attention-based semi-supervised learning for medical image segmentation[END_REF] a semi-supervised image segmentation method that simultaneously optimizes both supervised segmentation and unsupervised reconstruction objectives. The reconstruction task had the particularity to exploit an attention mechanism that separated the reconstruction of image regions corresponding to different classes. Such a simple yet effective multi-task learning scheme (Sect.V-B) achieved strong improvements for brain tumor and white matter hyper-intensities segmentation.

E. Federated Learning

Collecting large medical image datasets is a difficult and time-consuming task for research needs. Accurate labeling of these images requires clinical experience and is challenging to obtain. Many imaging centers have large image datasets, but many of them are unorganized or poorly annotated in spite of their richness regarding deep model training [START_REF] Zhao | Multi-view semi-supervised 3D whole brain segmentation with a self-ensemble network[END_REF][START_REF] Ng | Federated learning: a collaborative effort to achieve better medical imaging models for individual sites that have small labelled datasets[END_REF]. In addition, medical images are usually linked to personal health information related to the patient. Data protection to prevent sharing sensitive patient data is essential when working with multiple medical institutions, in a collaborative manner.

To solve this issue, federated learning (FL) enables distributed training of DL models without really sharing data between multiple clinical institutions. Fig. 12 shows the general framework of federated learning. To work in a collaborative fashion, FL allows various clinical institutes or hospitals to work in coordination by using a central server. Each hospital keeps an individual model which focuses on the local data only. Before the training process, each institution submits a request to download the global model from a central server. The requested query is then approved by the central server and the global model weights can be downloaded. Once the training process is executed, the local client model weights are sent to the central server for updating purposes. The central server aggregates the feedback received from individual institutions and updates the global model weights based on pre-defined rules. These rules permit the model to measure the quality of the feedback obtained from the client servers.

More and more research works in medical image segmentation involve a FL scheme [START_REF] Chowdhury | A review of medical federated learning: Applications in oncology and cancer research[END_REF] More globally, FL has shown potential for improving the accuracy of medical image segmentation while protecting the privacy of individual patient data. By offering scalability, flexible training scheduling and large training datasets via multisite collaborations [START_REF] Ng | Federated learning: a collaborative effort to achieve better medical imaging models for individual sites that have small labelled datasets[END_REF], FL combines essential conditions to consider increasing its deployment in various clinical settings.

F. Active learning

Active learning (AL) is a learning technique that involves training a model on a small, initial set of labeled data and then iteratively selecting new data to be labeled and added to the training set (Fig. 13). Thus, it assists annotators in the annotation process to select the most useful samples to train a DL-based model. This is particularly useful in medical image segmentation, as manually labeling large amounts of medical images is time-consuming and costly [START_REF] Tajbakhsh | Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation[END_REF]. By using AL, the model can learn to accurately segment images with less human inputs, making the process more efficient and costeffective. Additionally, AL allows the model to focus on the most difficult and important samples, resulting in improved delineation performance. Nevertheless, choosing the best data enabling the improvement of the model learning capability remains challenging. In particular, there are multiple methods to measure informativeness which mainly involves uncertainty and representativeness criteria [START_REF] Budd | A survey on active learning and human-in-the-loop deep learning for medical image analysis[END_REF]. DL-based segmentation methods are able to measure uncertainty (Sect.V-C) to some extent. Computing for each voxel the sum of the lowest class probability is one of the simplest manners. If the prediction is uncertain, an increased number of annotated data is required to exploit richer feature information. On the contrary, representativeness deals more with choosing the samples from distinct regions of the data distribution such that the variability among the whole dataset is taken into account. In this context, a good balance between exploration and exploitation among the distribution is highly desirable.

To name a few, a cascaded 3D U-Net with CNN-correction label curation was employed in [START_REF] Kim | Active learning for accuracy enhancement of semantic segmentation with CNN-corrected label curations: Evaluation on kidney segmentation in abdominal CT[END_REF] for kidney segmentation from abdominal CT images in order to save the annotation efforts and improve the segmentation outcomes. AL was concluded to be able to reduce labeling efforts through CNNcorrected segmentation and increase training efficiency by iterative learning with limited data. Shen et al. presented in [START_REF] Shen | Labeling cost sensitive batch active learning for brain tumor segmentation[END_REF] an AL approach able to alleviate the image annotation issues towards brain tumor segmentation. The authors combined both uncertainty and representativeness information to ensure that AL selects enough informative and diverse data. Contrary to existing studies based on uncertainty or representativeness estimated at the scale of a single image, Yan et al. scored in [START_REF] Yan | Deep active learning for dual-view mammogram analysis[END_REF] dual-view mammograms according to their prediction consistency, towards better breast mass delineation. A future possible extension is to integrate multiple single and dual-view criteria to reach a unified AL system. More globally, combining the strengths of AL and human-inthe-loop computing [START_REF] Budd | A survey on active learning and human-in-the-loop deep learning for medical image analysis[END_REF] into end-to-end systems should play an increasingly important role in the upcoming years.

G. Lightweight networks

Despite high performance, applying CNN or Transformersbased networks for 3D medical image segmentation is computationally expensive. For instance, 3D convolution layers made of many filters involve a large set of parameters to train as well as a huge amount of floating point operations (FLOP) [START_REF] Ma | Fast and low-GPU-memory abdomen CT organ segmentation: The FLARE challenge[END_REF]. Medical images themselves obviously require a large computational storage cost. However, the device on which a given 3D model is deployed may have limited computational power, making the deployment process hard in clinical routine. In this context, the development of lightweight deep segmentation models with smaller model sizes, lower computational cost and inference time has recently attracted increasing attention. First attempts investigated depth-separable convolutions [START_REF] Lei | Automatic segmentation of organs-at-risk from head-and-neck CT using separable convolutional neural network with hard-region-weighted loss[END_REF] consisting in replacing 3D convolution kernel 3 × 3 × 3 with 1 × 3 × 3 intra-slice and 3 × 1 × 1 inter-slice convolutions, combinations between point-wise, group-wise and dilated convolutions [START_REF] Chen | 3D dilated multifiber network for real-time brain tumor segmentation in MRI[END_REF] or the reduction of channels [START_REF] Yaniv | V-Net light-parameter-efficient 3D convolutional neural network for prostate MRI segmentation[END_REF]. However, obtaining the same performance as computationally expensive heavy-weight models is tedious and resolving the trade-off between trainable parameters and performance remains a challenge. Towards better trade-off, Zhao et al. managed in [START_REF] Zhao | LCOV-NET: A lightweight neural network for Covid-19 pneumonia lesion segmentation from 3D CT images[END_REF] the limited feature learning ability of spatiotemporal separable convolutions via an attention-based feature calibration mechanism providing more contextual information with a larger receptive field.

The knowledge distillation mechanism fully described in Sect.V-E which tends to distill information from a welltrained cumbersome teacher network to a lightweight and compact student network [START_REF] Qin | Efficient medical image segmentation based on knowledge distillation[END_REF] could be also seen as a powerful alternative towards lightweight segmentation networks.

VII. DISCUSSION

Deep learning (DL) has proven to be a powerful tool for medical image segmentation. Its ability to automatically learn complex and hierarchical representations from data enabled to achieve a high level of robustness in segmentation tasks tackling various diseases, anatomies and imaging modalities. The availability of large datasets and open-source DL frameworks has facilitated the development and deployment of DL-based segmentation algorithms, making them more accessible to researchers and clinicians. The success of DL in medical image segmentation has been thus demonstrated in a variety of studies including whole or sub-structure organ segmentation, abnormality extraction or vascular system delineation. Especially, lesion segmentation is increasingly benefiting from the availability of combined anatomical and nuclear imaging. More globally, medical image segmentation with DL stars to have a concrete impact and to play a key role in diagnosis, surgery or therapeutic planning, follow-up, prognostic, dosimetric or radiomics applications at large.

Since the introduction of U-Net and U-shaped convolutional encoder-decoder derivatives with data augmentation and encoder pre-training, various developments and methodological breakthrough have emerged in the medical image analysis community. Among current trends, the relevancy of conditional generative adversarial networks, cascaded networks, deep supervision and attention mechanisms have been proven to enable the improvement of segmentation accuracy for both large and small anatomical or pathological structures. Regularization techniques embedding prior knowledge such as shape, topological or adjacency constraints tend to be democratized towards greater robustness and generalizability of deep segmentation models. Additionally to novel architecture designs and learning paradigms, a significantly strong focus has been recently devoted to both data management and optimization processes through the development of unified frameworks such as nnU-Net whose popularity is steadily growing. Given the substantial progress made in recent years, considering a standard U-Net as the sole baseline no longer seems relevant.

Despite these successes, there are still challenges to the use of DL for medical image segmentation. These challenges include the need for large amounts of labeled data for training, the sensitivity of DL models to noise, non-uniform contrast and artifacts in medical images, the needed incorporation of local and global context to benefit from both short-and longrange spatial dependencies, the management of small structures and weak boundaries, the robustness to inter-subject variability and various multi-center, multi-scanner, multi-modal or multi-protocol intensity domains as well as the risks for biased or non-generalizable results.

Given these challenges, the use of DL has shown great promise in line with the emergence of vision Transformers whose ability to model long-range dependencies from 3D medical images appears better than standard convolutional only architecutres. Either hybrid when used in conjunction with convolutional layers or purely Transformers-based, these approaches are still at an early stage. More works in this direction are expected, especially in the context of lowdata regimes and cross-modal analysis. The great potential of multi-task learning, constrastive learning and knowledge distillation which are likely to emerge can be also emphasize as powerful trends to follow. Multi-task learning enables to share information and exploit fruitful cooperation between connected or auxiliary tasks while contrastive learning tends to strengthen the extraction of distinctive and disentangled representations. Against computationally expensive scenarios, knowledge distillation techniques have proven great skills to distill information from a cumbersome teacher network to a lightweight student network. Uncertainty modelling is another important path to study as it may improve the learning process and provide clinicians with locally-estimated confidence information. Further research and development beyond the application of off-the-shelf DL solutions are needed to address the above-mentioned challenges and enable a wider adoption of image segmentation with DL into clinical routine.

To bridge the gap between DL paradigms and clinical needs, recent investigations have struggle with novel and concrete emerging applications. Among these applications, cross-modality segmentation has gained in popularity in order fully exploit both complementary and redundancy across modalities when managing paired or unpaired multi-modal datasets. More globally, a special attention has been paid in recent years to multi-domain segmentation strategies which are far more relevant than focusing on multiple intensity domains separately. In this direction, multi-task and multi-domain techniques with multiple anatomies as targets should deserve further investigation in the near future. Given the complexity of collecting and annotating a large amount of medical images, self-supervised, semi-supervised and active learning are subfields of clear progress. However, more research efforts are needed to maximize or avoid the time-consuming and costly manual efforts made by clinical experts. Since medical data is often sensitive and subject to strict regulations on sharing, federated learning now offers the possibility for multiple hospitals and research institutions to collaborate by training a shared model on their own local data while keeping the data private and secure. Thus, federated learning enables the use of larger and more diverse datasets, resulting in improved segmentation performance. Finally, the development of lightweight models with few memory and computational resource requirements will for sure be beneficial to ease the deployment of DL-based solutions on computationally-limited platforms.

Overall, the potential for bias in DL approaches is a common concern across medical image analysis tasks including segmentation. In this context, encouraging the collection of large and diverse datasets through collective work with various experts is highly recommended. The design of novel evaluation metrics reflecting the clinical applicability is also an area for improvement. Finally, demonstrating a better reproducibility when designing DL pipelines could increase the trust and confidence of researchers and clinicians and make them more suitable for large-scale clinical applications.

Fig. 1 .

 1 Fig. 1. Residual V-Net inspired convolutional encoder-decoder architecture for medical image segmentation purposes. Refer to Sect.II-C for further details.

Fig. 3 .

 3 Fig. 3. Convolutional encoder-decoder architecture with deep supervision. The overall loss function is the weighted sum of losses estimated at different decoder levels [25]. C is the number of classes. Refer to Sect.IV-D for further details.

Fig. 4 .

 4 Fig. 4. Flow-chart diagram of a general attention mechanism function. Refer to Sect.IV-E for further details.

Fig. 6 .

 6 Fig. 6. Typical Transformers-based model architectures for medical image segmentation: (a) hybrid Transformers-CNN encoder, (b) pure Transformers-based encoder, (c) full Transformers-based network. Refer to Sect.V-A for further details.

Fig. 7 .

 7 Fig. 7. Overview of the TransUNet architecture proposed in [29] including a schematic illustration of a Transformer layer.

Fig. 8 .

 8 Fig. 8. Self-supervised global (a), self-supervised (b) and supervised (c) local contrastive learning [164]. Refer to Sect.V-D for further details.

Fig. 9 .

 9 Fig. 9. Categorization of cross-modal segmentation frameworks. UDA stands for unsupervised domain adaptation. Refer to Sect.VI-A for further details.

FusionFig. 10 .

 10 Fig. 10. Pipeline of cross-modal frameworks for paired data. (a) Early or input-level fusion, (b) mid or layer-level fusion, (c) late or decision-level fusion.

4 )

 4 Unpaired data and domain adaptation: When only unpaired datasets are available, cross-modal segmentation is commonly managed by domain adaptation (DA) techniques. Let X m × Y m represent the joint feature space and the corresponding label space for a specific modality m. A domain can be formulated as D m = {X m , P (x m )} where X m is the feature space and P (x m ) the marginal probability distribution of the data x m . Let us assume that we have two domains (e.g. from two different modalities): a source domain D S = {X S , P (x S )} and a target domain D T = {X T , P (x T )}. The DA problem is formulated as sub-class of transfer learning where the label spaces in the source and target domains are the same Y S = Y T = Y but where the domains are different D S = D T .

Fig. 13 .

 13 Fig. 13. General framework of active learning. See Sect.VI-F for more details.

TABLE II SUMMARY

 II OF SEVERAL POPULAR ALIGNMENT FUNCTIONS USED TO COMPUTE THE MATCHING SCORE BETWEEN A QUERY AND KEYS. W W W AND V V V ARE TRAINABLE WEIGHT MATRICES, MEANWHILE d k STANDS FOR THE DIMENSION OF A VECTOR k. [.; .] STANDS FOR CONCATENATION.

  . Recently, Xu et al. introduced in[START_REF] Xu | Federated cross learning for medical image segmentation[END_REF] a new federated cross-learning segmentation approach that handles data that are not independently and identically distributed. Unlike the conventional FL methods that combine multiple individually trained local models on a server node, the proposed method named FedCross consecutively trained the global model across multiple clients in a round-robin fashion. The authors also suggested a new federated crossensemble learning technique that together trains and sets up Later, annotated images were incorporated to finetune the model. Through feature exchange in which each client exchanges the features (i.e. low-dimensional vectors) of its local data with other clients, the approach enables better local contrastive learning while avoiding raw data sharing. A global structural matching technique was developed to learn the structural similarity of encoded features with suitable representations to be shared with other remote clients.

	Client model		Client model
	Hospital I			Hospital III
	+	Central server		+
	Data	+		Data
	+	downloading	uploading	+
	Data			Data
	Client model	Global model	Client model
	Hospital II			Hospital IV
	Fig. 12. General framework of federated learning. Refer to Sect.VI-E for further details.
	various models. Wicaksana et al. [213] proposed FedMix, an
	FL strategy that employed mixed image labels specifically to
	segment anatomical region-of-interest from medical images.
	These labels incorporated substantial pixel-wise annotations,
	weak bounding boxes and image-wise class annotations. The
	authors initially created the pseudo annotations through clients
	and employed refinement under supervision to enrich pseudo-
	annotations. Later, each client contained high-quality data
	which was determined using active sample picking for local
	model updates. Based on the quantity and quality of data,
	the approach provided updates through dynamic aggregation
	techniques which allow for modification of each local client's
	weight. FedMix was validated on breast tumor segmentation
	from ultrasound and skin lesion segmentation.
	Wu et al. [38] introduced a new federated contrastive learn-
	ing (FCL) framework for 3D volumetric image segmentation
	that requires limited annotations only. The local clients first
	started with learning a shared encoder to spread unlabeled
	images.

ACKNOWLEDGMENTS

This work benefited from state aid managed by the National Research Agency under the Future Investment Program bearing the reference ANR-17-RHUS-0005 (FollowKnee project). This work was also partially funded by France Life Imaging (grant ANR-11-INBS-0006). All authors declare that they have no known conflicts of interest in terms of competing financial interests or personal relationships that could have an influence or are relevant to the work reported in this paper.