Reda Abdellah Kamraoui

Vinh-Thong Ta

Thomas Tourdias

Boris Mansencal

Pierrick José V Manjon

Pierrick Coupé

DeepLesionBrain: Towards a broader deep-learning generalization for multiple sclerosis lesion segmentation

Keywords:

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction 1.Problem Description

In recent years, the medical imaging community has witnessed a rapid increase in image processing methods based on Deep Learning (DL). These novel techniques came with remarkable performance in many tasks including Multiple Sclerosis (MS) lesion segmentation. Some automated algorithms have even reached human-level performance in controlled evaluation conditions (see [START_REF] Carass | Longitudinal multiple sclerosis lesion segmentation: resource and challenge[END_REF]). Unlike over-controlled conditions where most DL approaches have been validated, real-world data exhibit high diversity. Consequently, clinical use of MS lesion segmentation based on DL is still limited mainly because of their poor generalization on new data coming from medical sites that have not been covered during training (unseen domains). This lack of generalization of DL methods can result from several factors such as the selected solution during optimization, the diversity of the training dataset or the genericity of the extracted features.

DL is based on the assumption that training and test data are independent but come from the same distribution. This assumption is usually not respected in medical imaging data especially for Magnetic Resonance Imaging (MRI) where acquisition protocols, MRI scanner, patient populations, and software processing may vary depending on the clinical center or the cohort. As a result of these differences of data distribution (covariate shift), a decrease in performance is observed between the training data (source domain) and the test data derived from different distributions (target domain). This is known as the domain shift.

An intuitive method to reduce this problem is to train on a wider and more heterogeneous dataset (as shown by [START_REF] Mårtensson | The reliability of a deep learning model in clinical out-of-distribution mri data: a multicohort study[END_REF]). However, this requires a large dataset annotated by experts which are rarely available and tedious to produce. Some deal with this phenomenon by applying extensive data augmentation (such as [START_REF] Zhang | Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation[END_REF]). Others use few available labeled images from the target domain to reduce the covariate shift, such as few-shot and one-shot learning strategies (see [START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Valverde | One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks[END_REF]).

Besides, DL requires the tuning of a large number of parameters relative to the number of training data samples. Thus, it usually ends up converging to one of the many possible local minima as opposed to the theoretical best parameter configuration which leads to the global minimum. Consequently, the generalization ability of the model depends on the selected solution. The selection of the best generalizing local minimum is still an open question. On one hand, some works have proposed to select it using the local characteristics (e.g., flatness) of the loss function (see [START_REF] Keskar | On large-batch training for deep learning: Generalization gap and sharp minima[END_REF][START_REF] Wu | Towards understanding generalization of deep learning: Perspective of loss landscapes[END_REF] ). On the other hand, an alternative strategy consists in combining several local minima to improve the generalization capability of the method. This can be done by averaging several local minima of one model (e.g., snapshot ensemble [START_REF] Huang | Snapshot ensembles: Train 1, get m for free[END_REF]) or by combining outputs of different models trained independently (e.g., classical ensemble [START_REF] Zhang | Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices[END_REF] and spatially distributed networks [START_REF] Coupé | AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation[END_REF][START_REF] Huo | 3D whole brain segmentation using spatially localized atlas network tiles[END_REF]).

Unlike classical methods that use hand-crafted features, Convolutional Neural Networks (CNNs) automatically extract the most suitable set of features for a particular task. Although this strategy is very efficient to extract relevant features for a particular source domain, this set of features may not generalize well for the target domain. Some works proposed to learn invariant features that coexist across different source domains [START_REF] Motiian | Unified deep supervised domain adaptation and generalization[END_REF][START_REF] Muandet | Domain generalization via invariant feature representation[END_REF][START_REF] Yang | Multi-view discriminant transfer learning[END_REF]. They tried to apply a regularization to learn an abstract representation of the specific computer vision task (i.e., just like humans understand high-level concepts). Indeed, the extraction of generalizing features lies between the freedom of the optimization process to find the optimal combination from data and the constraints used for minimizing domain bias.

The successful deployment of DL based methods for MS lesion segmentation requires generalization capabilities that can guarantee high performance for unseen domains. First, such methods should ensure the convergence of the DL model to generalizing minima. Second, the training process should anticipate the reduction of the covariate shift. Moreover, the method should be enforced to learn MS lesion generalizing features from the source domain, to effectively delineate lesions despite the target domain distribution. Finally, this solution should not require additional annotation in case of processing unseen domains.

Related Works

Recently, many works have been proposed for MS lesion segmentation using CNNs. First, [START_REF] Brosch | Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation[END_REF] proposed a deep 3D encoder-decoder network, with joint training of the encoder and the decoder. The authors used shortcut connections between the two interconnected pathways for integrating high and low-level features. This pioneering work demonstrated the high potential of deep learning for MS lesion segmentation. [START_REF] Valverde | Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach[END_REF] proposed a cascade of two patch-wise 3D CNNs, composed of a first sensitive network to reveal possible lesion candidates followed by a second network to reduce misclassified voxels. This cascade allows refined segmentation but it uses a small receptive field that prevents capturing the global context. Later, the authors [START_REF] Valverde | One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks[END_REF] improved their method by proposing a one-shot domain adaptation model which uses transfer learning and partial fine-tuning. However, this domain adaptation needs a labeled example from the new domain. Moreover, such strategies lead to different versions of the method after each adaptation, this results in discrepancies in the segmentation. [START_REF] Hashemi | Asymmetric loss functions and deep densely-connected networks highly-imbalanced medical image segmentation: Application to multiple sclerosis lesion detection[END_REF] considered the problem of data imbalance (i.e., the under-sampling of the lesion class) by using an asymmetric similarity loss function based on Tversky index to train a 3D CNN that performed better than Dice or cross-entropy measures. This result suggests that further work should be done on choosing an adequate loss function. Although the proposed loss can be tuned for the optimal trade-off between precision and recall in a particular domain, the generalization to unseen domains has to be proven. [START_REF] Zhang | Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices[END_REF] used a fully convolutional densely connected network for MS lesion segmentation. They stacked adjacent 2D slices of different modalities with a channel-wise concatenation, before forwarding this stack through a 2D CNN. The final segmentation is based on a majority vote along different orientations. While this method showed competitive performance on a well-controlled challenge, the stacking using only the two directly adjacent slices gives a weak insight into the 3D nature of the data. Moreover, 2D features may not be considered as generalizing features when processing 3D volumes and can result in the limited generalization of the method. [START_REF] Aslani | Multi-branch convolutional neural network for multiple sclerosis lesion segmentation[END_REF] proposed an end-to-end encoder-decoder 2D network with multiple downsampling branches, one for each input modality, and a decoder part where features from the different modalities are put together at multiple scales. This separation in encoder branches enables the model to encode information efficiently from each modality, before combining them in a later stage. However, this 2D approach does not combine features based on axial, coronal, and sagittal orientations that may greatly reduce its generalization on 3D images. [START_REF] Feng | A self-adaptive network for multiple sclerosis lesion segmentation from multi-contrast mri with various imaging sequences[END_REF] considered MRI modality unavailability during segmentation by introducing sequence dropout. This is an important point since the availability of all the modalities is not always ensured between datasets that can greatly reduce the generalization capacity of a method. This framework randomly drops specific MRI sequences during training, with the intent to learn the intrinsic information of each sequence. This technique showed it can produce acceptable segmentation even in the absence of one or two modalities. Nonetheless, it is less efficient than other state-of-the-art methods when all modalities are available (will be detailed in section 3.3). [START_REF] Aslani | Scanner invariant multiple sclerosis lesion segmentation from MRI[END_REF] tackled the problem of generalization to new domains by integrating a regularization network to the traditional encoder-decoder network. The regularizer penalizes the network when the latter learns features that allow the prediction of MRI scanning sites. However, [START_REF] Li | Domain generalization with adversarial feature learning[END_REF] have argued that such strategies suffer from overfitting, the obtained representation could well generalize for all the source domains but poorly for the unknown target domains.

All the cited MS methods ( [START_REF] Brosch | Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation[END_REF][START_REF] Valverde | Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach[END_REF][START_REF] Feng | A self-adaptive network for multiple sclerosis lesion segmentation from multi-contrast mri with various imaging sequences[END_REF][START_REF] Hashemi | Asymmetric loss functions and deep densely-connected networks highly-imbalanced medical image segmentation: Application to multiple sclerosis lesion detection[END_REF][START_REF] Zhang | Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices[END_REF][START_REF] Aslani | Multi-branch convolutional neural network for multiple sclerosis lesion segmentation[END_REF][START_REF] Feng | A self-adaptive network for multiple sclerosis lesion segmentation from multi-contrast mri with various imaging sequences[END_REF]) focused on obtaining accurate segmentation within a same domain evaluation. However, the use of out-of-domain datasets is essential to ensure a good evaluation of the generalization capabilities of a method. This question is right now a hot topic (see [START_REF] Mårtensson | The reliability of a deep learning model in clinical out-of-distribution mri data: a multicohort study[END_REF], and [START_REF] Bron | Cross-cohort generalizability of deep and conventional machine learning for mri-based diagnosis and prediction of alzheimer's disease[END_REF]) and an important recommendation from the clinical world (see [START_REF] Omoumi | To buy or not to buy-evaluating commercial ai solutions in radiology (the eclair guidelines)[END_REF]). Experiments using training and testing images derived from the same domain are known to be biased [START_REF] Omoumi | To buy or not to buy-evaluating commercial ai solutions in radiology (the eclair guidelines)[END_REF] and do not ensure generalization. Therefore, a model used in clinical conditions should produce accurate segmentation for new domain images without the need of retraining with expert segmentation on the new domain.

Proposals

In this paper, we propose DeepLesionBrain (DLB), a novel method for MS lesion segmentation robust to domain shift, validated on out-of-domain testing (cross-dataset testing).

First, we use a large group of compact 3D CNNs spatially distributed over the brain with overlapping receptive fields between regions.

By associating a distinct network with each region of the brain, the spatially distributed networks (see [START_REF] Coupé | AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation[END_REF][START_REF] Huo | 3D whole brain segmentation using spatially localized atlas network tiles[END_REF]) strategy simplifies the MS lesion segmentation from a single complex task on the whole brain to multiple simpler sub-tasks on each region. Moreover, the overlapping regions ensure consistent and stable consensus.

Applied to brain segmentation, this strategy demonstrated good generalization on unseen domains (e.g., child's brain or patients with Alzheimer's disease) when trained on healthy adult brains [START_REF] Coupé | AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation[END_REF].

Second, to extract more relevant features for MS that may lead to better generalization, we focus on feature learning strategy. We consider that a generalizing model should learn two types of features: first global and generic features, and second local and specific features. Therefore, we propose Hierarchical Specialization Learning (HSL) to efficiently extract those features in two-steps. In the first step, a single network (the generic network) is trained on all brain regions. In the second step, each network of the spatially distributed networks is initialized with the generic network weights and specialized for a specific region of the brain. Third, DLB is trained with a novel Image Quality Data Augmentation (IQDA) method, which mimics real-world data diversity by adding realistic alterations to the training images. As shown in the works of [START_REF] Zhang | How does mixup help with robustness and generalization?[END_REF], such a type of regularization technique aims to reduce covariate shift. IQDA proposes specific augmentations that constrain task learning to be independent from source data acquisition resolution, data contrast, or data quality. Consequently, the proposed augmentation strategy enables domain shift robustness.

Finally, we propose a method using only two modalities (T1w and FLAIR) to ensure its compatibility with a large number of datasets. Most of the methods ( [START_REF] Feng | A self-adaptive network for multiple sclerosis lesion segmentation from multi-contrast mri with various imaging sequences[END_REF][START_REF] Brosch | Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation[END_REF][START_REF] Valverde | Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach[END_REF][START_REF] Zhang | Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices[END_REF]) optimize their segmentation using T1w, FLAIR, PD, and T2 modalities. However, in clinical conditions, not all these sequences are always available. Therefore we focused our work on developing a robust approach using only two modalities. DLB is based on a spatially distributed networks strategy, proposed by [START_REF] Coupé | AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation[END_REF] and [START_REF] Huo | 3D whole brain segmentation using spatially localized atlas network tiles[END_REF]. Such a strategy uses a group of compact networks, where each network is specialized in a particular region of the brain, and processes a sub-volume of the global volume. The receptive fields of the neighboring networks overlap with one another, and the final segmentation of the whole volume is obtained with a majority vote of the local predictions. Employing our spatially distributed compact networks is equivalent to a big network with more filters and a higher receptive field (see Fig. 1). This particular configuration with overlapping receptive fields aims to produce a more robust segmentation compared to an individual network. As shown in the work of [START_REF] Huo | 3D whole brain segmentation using spatially localized atlas network tiles[END_REF] that compared between 8 networks configuration with no-overlapping and 27 networks with overlapping, having an average prediction over overlapping regions led to significantly better performance on the evaluation data-sets including out-of-domain (on infants) data (i.e. better generalization). Similarly, [START_REF] Coupé | AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation[END_REF] confirmed that larger overlapping led to better performance until the limit of 125 networks where performance peaked and stayed stable for 125, 216, and 343 networks. Additionally, we assume that averaging the prediction of a group of networks that have been trained separately on different sub-volumes is less likely to collapse than having a single network trained on the same set of data.

Hierarchical Specialization Learning (HSL)

To improve generalization for the task of MS lesion segmentation, we take inspiration from MS lesion features and propose a better learning strategy. MS lesions features can be grouped into two categories: First, some lesion characteristics are considered generic and shared among lesion types. Such features are independent of lesion localization. They have a common and inherent significance at the global scale of the brain volume, we will refer to them in this paper as "generic global features". Second, other relevant features for MS lesions depend on brain structure and some distinct regions (see [START_REF] Filippi | Assessment of lesions on magnetic resonance imaging in multiple sclerosis: practical guidelines[END_REF]). In this work, we refer to these features as "specialized local features".

On the one hand, training each specialized network on a specific sub-region of the brain (see Fig. 1 right) would prevent the efficient learning of "generic global features", since each specialized member of our group would be trained on a particular region of the whole brain. On the other hand, using a single 3D CNN to learn "specialized local features" over the whole brain volume would require a large model which may not fit into memory and a large dataset to train it.

To overcome this limitation, we propose a novel Hierarchical Specialization Learning (HSL). Fig. 1 shows our two-step learning process. First, the "generic network" is trained with data samples from all over brain regions to learn general knowledge about lesions by extracting "generic global features". Second, each network in the spatially distributed strategy is specialized over a specific sub-volume of the brain.

The generic network is used as an initialization for each network of our spatially distributed networks, by transferring the generic network weights to each individual specialized network. The knowledge gained from this transfer learning transmits the ability to extract "generic global features", while the specialized network training will specialize them in extracting local "specialized local features".

In our ablation study, we will show that this hierarchical specialization learning of the specialized networks performs better than training a single network over the whole brain, or training the specialized networks without HSL.

Image Quality Data Augmentation (IQDA)

The quality of the MRI greatly varies between datasets. In fact, the quality of the images depends on several factors such as signal-to-noise ratio, contrast-to-noise ratio, resolution, or slice thickness. To address this issue, we propose a data augmentation strategy that considers image quality disparity. During training, we simulate "on the fly" altered versions of 3D patches. We randomly introduce at each iteration either blur, edge enhancement, or axial subsampling distortion (2D FLAIR are usually acquired along the axial direction). For the blur, a gaussian kernel is used with a randomly selected standard deviation ranging between [0.5, 1.75]. For edge enhancement, we use unsharp masking with the inverse of the blur filter. For axial subsampling distortion, we simulate acquisition artifacts that can result from the varying slice thickness. We use a uniform filter (a.k.a mean filter) on the axial direction with a size of [1 × 1 × sz] where sz ∈ 2, 3, 4. Ground truth is kept the same as the original version. This process reduces the domain bias when learning to extract relevant features caused by data variability.

Selection of the Required Modalities

To use a trained model for MS lesion segmentation with optimal performance, it usually requires the use of the same set of modalities that have been chosen during training. DLB proposes a method that needs only T1w and FLAIR sequences to be compatible with all benchmark MS datasets and most already available MS patients data.

Our method is built with the purpose to generalize on unseen datasets, thus it uses the minimum necessary modalities. Indeed, increasing the number of sequences requires a longer scan acquisition time. Besides, it needs more complex processing which may be prone to error, such as multimodal image registration. Furthermore, the use of more sequences during the training on a dataset may reduce the generalization to other image domains.

In addition to the wide use of T1w and FLAIR for MS diagnosis, the choice of these modalities has also been motivated by the fact that FLAIR is the most relevant sequence for revealing most MS lesions (see [START_REF] Narayana | Are multi-contrast magnetic resonance images necessary for segmenting multiple sclerosis brains? a large cohort study based on deep learning[END_REF]), while T1w can provide complementary structural information needed for accurate segmentation. The number of input channels (NC) depends on the modality number (i.e., NC= 2, for using T1w and FLAIR ). Each block is composed of group normalization (GN), Convolution (Conv) and Rectified Linear Unit (ReLU) activation.

Implementation Details

The network architecture used in our spatially distributed strategy is based on 3D U-Net composed of a downsampling part and an upsampling one, linked with one another by skip connections at the multiple scales. This 3D CNN architecture, shown in Figure 2, has been used for all the networks in our approach. Dropout with 0.5 rate is used after max-pooling layers to prevent the overfitting of our model to the training data. Due to GPU memory constraints, we trained with a batch size of 1, and so we used Group Normalization (GN) [START_REF] Wu | Group normalization[END_REF] with 8 groups before each convolution.

We have chosen Rectified Linear Units (ReLu) to introduce non-linearity after convolution layers. DLB is optimized with Adam [START_REF] Kingma | A method for stochastic optimization[END_REF] using a learning rate of 0.0001 and a momentum of 0.9. The experiments have been performed with Keras 2.2.4 [START_REF] Chollet | [END_REF] and Tensorflow 1.12.0 [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF] on NVIDIA Titan Xp 12 GB GPU.

Method Description

To obtain sub-volumes for each image, we first divide our whole MRI registered into the MNI space into multiple overlapping 3D patches. We perform a cropping operation over the whole image using a sliding window of the size (P x, P y, P z), and a stride of (Sx, Sy, Sz). We take Sx < P x, Sy < P y, Sz < P z to ensure the overlapping.

Sub-volumes from different images, that represent the same receptive field into the MNI space (the same sub-volume region of the whole volume), are grouped together. They are used for training a network specialized for that particular region.

In this work, we explored many combinations of sub-volume sizes and numbers. We chose a configuration with 125 sub-volumes by taking experimentally P x = P y = P z = 96, Sx = Sy = 76, and Sz = 67 as a good trade-off between the overall performance and computation resources.

Symmetrical Training

To limit redundant training and to use the most possible data for a particular brain region, we choose to train specialized networks only on one hemisphere. By flipping (mirroring) the sub-volumes of the second hemisphere, it is possible to train a single specialized network for both sides. Thus, we can use twice the amount of data for each region while reducing the number of networks to train to nearly a half (due to sub-volume overlapping, the plane of symmetry cuts through the median networks which cover equivalent symmetrical regions from both hemispheres). Consequently, unlike previous works with spatially distributed networks (i.e., [START_REF] Coupé | AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation[END_REF]), instead of using 125 networks we use only 75 specialized networks. We experimentally verified that using 125 networks or only 75 trained with twice the number of patches, produced similar segmentation accuracy.

Loss Function

MS lesion segmentation task suffers from class imbalance since lesion volume is considerably low compared to healthy volume. Thus, we use a smooth version of the Generalized Jaccard Loss (GJL), which considers this issue [START_REF] Manjon | DeepHIPS: A novel deep learning based hippocampus subfield segmentation method[END_REF].

GJL = 1 - σ + N c c=1 w c N i=1 p ci t ci σ + N c c=1 w c N i=1 (p ci + t ci ) - N i=1 p ci t ci (1)
Where w c = 1/(1 + N i=1 t ci ), σ is the smoothness factor, N is the number of voxels, N c is the number of classes, p ci and t ci are respectively the predicted probability and the ground truth probability of the voxel i for the class c.

During inference, we combine the overlapping predictions in a straightforward majority vote technique. The class of each voxel (either lesion or healthy tissue) is chosen based on the most predicted class among the networks which cover that voxel.

Datasets

To assess the robustness of a model, it is crucial to test its ability to generalize on unseen domains. Therefore, DLB has been trained and validated using different datasets to assess its domain generalization ability (see 3.2 ). These datasets exhibit high heterogeneity in terms of resolution, data processing, acquisition sites, delineation protocols, and they also cover a large variety of clinical scenarios.

ISBI Longitudinal Multiple Sclerosis Lesion

The ISBI dataset [START_REF] Carass | Longitudinal multiple sclerosis lesion segmentation: resource and challenge[END_REF] consists of five subjects for training, fourteen subjects for testing, with a mean of 4.4 time-points per subject (21 images for training and 61 images for testing). Two human expert raters delineated MS lesions, from the four available modalities acquired on 3.0 Tesla MRI scanner: 3D MPRAGE T 1 -weighted (T1w) of 0.82 × 0.82 × 1.17 mm 3 voxel size, 2D T 2 -weighted (T2), 2D T 2 -weighted fluid attenuated inversion recovery (FLAIR), and 2D Proton Density weighted (PD), of 0.82 × 0.82 × 2.2 mm 3 voxel size each.

For the training, we used the ISBI training dataset with available annotations from the two experts. For test and evaluation, we segmented the test data with no available expert annotation, and submitted our results to the ISBI challenge website 1 . The ISBI pipeline already included preprocessing. Each first time-point T1w was inhomogeneitycorrected using N4 [START_REF] Tustison | N4itk: improved N3 bias correction[END_REF], skull-stripped [START_REF] Carass | A joint registration and segmentation approach to skull stripping[END_REF], dura stripped [START_REF] Shiee | Reconstruction of the human cerebral cortex robust to white matter lesions: method and validation[END_REF], followed by a second N4 inhomogeneity correction, and rigid registration to a 1 mm 3 isotropic MNI template. Then, this image was used as a target for the remaining T1w time-points and all modalities for the same subject. These images were N4 corrected and then rigidly registered to the T1w in the MNI space. The skull and dura-stripped mask from the target T1w was applied, which were then N4 corrected again. We added an intensity normalization step using kernel density estimation for all images.

MICCAI2016 MS Challenge Dataset

The MSSEG'16 training dataset [START_REF] Commowick | Msseg challenge proceedings: Multiple sclerosis lesions segmentation challenge using a data management and processing infrastructure[END_REF] contains 15 patients from 3 different clinical sites. Five modalities are available for each patient: 3D FLAIR, 3D T1w, 3D T1w GADO, 2D PD, and 2D T2. The images were acquired on 1.5T and 3T MRI scanners with multiple resolutions: 3D FLAIR modalities ranging from 1 × 0.5 × 0.5 to 1.25 × 1.04 × 1.04 mm 3 , and 3D T1w sequences between 0.85 × 0.74 × 0.74 and 1.08 × 1.08 × 0.9 mm 3 .

Seven human experts have manually segmented the multiple sclerosis lesions. Each patient modalities have been preprocessed with the same pipeline. First, each sequence was denoised using the non local means algorithm [START_REF] Coupé | An optimized blockwise nonlocal means denoising filter for 3-d magnetic resonance images[END_REF]. Second, a rigid registration of each modality on the FLAIR was performed [START_REF] Commowick | Block-matching strategies for rigid registration of multimodal medical images[END_REF]. Then, skull stripping of T1w was performed using the volBrain platform [START_REF] Manjón | volbrain: an online mri brain volumetry system[END_REF], the same mask is applied to other modalities. Finally, bias field correction was applied using the N4 algorithm [START_REF] Tustison | N4itk: improved N3 bias correction[END_REF]. In addition to these steps that have been performed on the available images, each modality was registered to the MNI space for our experiments. Similarly, to the ISBI images, we used kernel density estimation for the normalization step.

In-house Dataset

For further evaluation of our approach, we used an In-house 3D MRI dataset, with 3D T1w and 3D FLAIR modalities [START_REF] Coupé | Lesionbrain: an online tool for white matter lesion segmentation[END_REF]. This dataset contains 43 subjects diagnosed with MS. The images were acquired with different scanners and multiple resolutions (0.6 × 0.6 × 0.65 mm 3 , 0.5 × 0.5 × 0.9 mm 3 , and 1 × 1 × 1 mm 3 ).

The dataset lesion masks have been obtained by human experts manual delineation. The images were pre-processed using the lesionBrain pipeline from the volBrain platform [START_REF] Manjón | volbrain: an online mri brain volumetry system[END_REF]. First, it included denoising of each modality [START_REF] Coupé | An optimized blockwise nonlocal means denoising filter for 3-d magnetic resonance images[END_REF]. Second, an affine registration to MNI space was performed on the T1w , then the FLAIR was registered to the transformed T1w. Skull stripping and bias correction have been performed on the modalities, followed by a second denoising. Finally, the intensities have been normalized.

Datasets Summary

Table 1 summarizes the main differences between the 3 datasets. We focused specifically on the resolution of FLAIR due to its known relevance in MS lesion segmentation. To summarize, ISBI train-set contains multiple time points of only five subjects, acquired in a single clinical site with two human expert segmentations. Except for 3D MPRAGE T1w, the other three modalities are in 2D. MSSEG'16 dataset is a multi-site database comprising 15 patients, with seven human segmentations. This dataset contains 5 available modalities with 3D FLAIR. Finally, the In-house dataset is the Figure 3 shows examples from the three presented datasets, each image represents a sagittal section of the FLAIR modality in the MNI space after intensity normalization. The two images on the left are examples from the ISBI dataset. We notice blurring effects which makes it hard to distinguish precisely brain structures. This blur comes from 2D low-resolution acquisitions. In the middle, the two examples come from the MSSEG'16 dataset. These 3D FLAIR are noticeably of higher resolution than the other images. Therefore, lesion boundaries are more easily delineated and main structures are apparent. The final two images on the right are from our In-house 3D dataset. The 3D resolution enables the differentiation of white matter, gray matter, and shows the lesions clearly. In terms of FLAIR images, we notice that both MSSEG'16 and In-house dataset (3D FLAIR) propose better visual quality than ISBI dataset (2D FLAIR).

Validation Framework

Evaluation Metrics

The assessment of a segmentation method is usually measured by a similarity metric between the predicted segmentation and the human expert ground truth.

First, we use several complementary metrics to assess segmentation performance. Namely, we use the Dice similarity coefficient, the Positive Predictive Value (PPV or the precision), true positive rate (TPR, known as recall or Sensitivity), and Pearson's correlation coefficient (CORR).

P P V = T P T P + F P , T P R = T P T P + F N , (2)
Dice = 2 × T P (T P + F N ) + (T P + F P ) , (3)
where TP, FN, FP represent respectively true positives, false negatives, and false positives.

Second, recent works (i.e., [START_REF] Commowick | Objective evaluation of multiple sclerosis lesion segmentation using a data management and processing infrastructure[END_REF]) question the relevance of classic metrics (Dice) compared to detection metrics, which are used for MS diagnostic and clinical evaluation of the patient evolution. Thus, in addition to the voxel-wise metrics, we also use lesion-wise metrics that focus on the lesion count. Such as Lesion False Positive Rate (LFPR) and Lesion True Positive Rate (LTPR).

LT P R = LT P LT P + LF N , LF P R = LF P LT P + LT N , (4)
where LTP, LFN, LFP represent respectively lesion true positives, lesion false negatives, and lesion false positives.

Moreover, [START_REF] García-Lorenzo | Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging[END_REF] pointed out that even though Dice is commonly used and simplifies method comparison, multiple complementary metrics are needed to provide a better understanding of the performance. Recently, international challenges took into consideration several metrics ( [START_REF] Carass | Longitudinal multiple sclerosis lesion segmentation: resource and challenge[END_REF] and [START_REF] Commowick | Objective evaluation of multiple sclerosis lesion segmentation using a data management and processing infrastructure[END_REF]). Consequently, we decided to evaluate our methods using Hybrid score proposed by [START_REF] Carass | Longitudinal multiple sclerosis lesion segmentation: resource and challenge[END_REF]. This metric combines voxel-wise segmentation, lesion-wise detection, and volumetric metrics. It is defined as:

Hybrid = Dice 8 + P P V 8 + (1 -LF P R) 4 + LT P R 4 + CORR 4 (5)
Finally, we also use the ISBI Submission Score for the evaluation of ISBI test-set segmentations. [START_REF] Carass | Longitudinal multiple sclerosis lesion segmentation: resource and challenge[END_REF] defined it as the average of the hybrid scores of all image examples with the different human raters and with inter-rater variability taken into consideration. This score is computed after submitting the segmentation to ISBI's challenge website 2 . Obtaining an ISBI score of 90 or higher with a segmentation technique indicates that this method is similar to the human raters.

Reference Methods

During experiments, our method was compared to three publicly available state-of-the-art approaches. We performed training and validation for all three compared methods, in the same conditions regarding datasets and preprocessing. The reference methods are nicMSlesion by [START_REF] Valverde | One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks[END_REF], DeepMedic by [START_REF] Kamnitsas | Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation[END_REF], and 2.5D Tiramisu by [START_REF] Zhang | Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices[END_REF]. These methods have been selected for the availability of the authors source code and the relevance of their contributions in the MS segmentation community.

nicMSlesion: This method is based on a cascade of two 3D patch-wise CNNs. The first one is trained to be sensitive to reveal lesion candidates. The second one is trained to reduce the misclassified voxels from the first network. Training is performed on 11 × 11 × 11 patches randomly augmented with flipping and rotations. Therefore, nicMSlesion involves classical data augmentation. In the first network, the negative class is under-sampled to the same number of existing lesion voxels. It is composed of patches extracted from all of the available lesion voxels and a random selection of normal-appearing tissue voxels. Afterwards, an evaluation of the first CNN model is computed by performing inferences on the same train-set and identifying negative voxels that have been misclassified as lesions (False Positives). Finally, the second model is trained using a balanced set composed of all the lesion voxels and a random selection from the identified False Positives in the previous step.

DeepMedic: This method is based on an 11-layers deep dual pathway 3D CNN designed for brain lesion segmentation. to incorporate both local and larger contextual information, the dual pathway architecture processes the input images at multiple scales simultaneously. To overcome the computational burden, the authors use a hybrid dense training scheme processing adjacent image patches into one pass through the network. To refine the network segmentation and remove false positives, a 3D fully connected conditional random field is used. The training includes data augmentation with sagittal reflections.

2.5D Tiramisu:

This method is based on a fully convolutional densely connected network. The model uses stacked slices from all three anatomical planes to achieve a 2.5D based method. Individual slices from a given orientation provide global context along the plane and the stack of adjacent slices adds local context. The training also includes flipping and rotations of the 2D patches for data augmentation. Therefore, 2.5D Tiramisu involves classical data augmentation. For each stack of 2D 128 × 128 slices composed of a center slice and its 2 adjacent slices, the model produces the segmentation of the center slice. Then, the inference results along the different orientations are combined via a majority vote to output the final segmentation. For both these methods, we use the implementations provided publicly by the authors (see3 and4 ).

Statistical Test

To assert the advantage of a technique obtaining the highest average score, we conducted a two-sided Wilcoxon test (i.e., paired statistical test) over the lists of hybrid scores measured at image level (for the consistency of the segmentation section we took the lists of dice indices between the two segmentations). The significance of the test is established for a p-value below 0.05. In the following tables, * indicates a significantly better average score when compared with the rest of the other approaches.

Results

Ablation Study

To demonstrate the impact of each proposed contribution on domain generalization, we measured separately their effects on different metrics. To show both the effect on accuracy improvement and the domain shift robustness, we propose an out-of-domain and in-domain ablation study. First, we trained each method configuration on ISBI challenge train-set, then we validated on both ISBI test-set (see Table 2) and MSSEG'16 (see Table 3). To ensure a fair comparison, each configuration is trained until convergence. Specifically, we used an early stopping criterion of 50 epochs (i.e., the training stops if the loss function does not improve on the validation set during 50 epochs) with a maximum number of 500 epochs. We verified that none of the configurations reached this maximum number.

Table 2: Ablation study results with different variants of our approach trained on ISBI challenge train-set and tested on ISBI test-set. DeepLesionBrain (DLB) refers to using our spatially distributed specialized networks, each network in charge of segmenting a sub-volume. The generic network represents the variant of DLB with a single network (without the spatially distributed strategy). Hierarchical Specialized Learning (HSL) indicates that we initialized the "specialized networks" with the "generic Network". To evaluate the performance of the proposed Data Augmentation, we compared variants with IQDA (previously defined in 2.1.2) and without IQDA. For each metric, the bold values indicate the best result. In the hybrid score column, * indicates a significantly better score than the other approaches using the two-sided Wilcoxon test. Table 2 shows the effect of each contribution to segmentation accuracy, when trained on ISBI challenge train-set and tested on ISBI test-set. First, the best performing combination is DLB with HSL and IQDA, it obtained an ISBI Score of 92.849. Second, both the versions of DLB without IQDA and DLB without HSL are less accurate. They obtained respectively ISBI scores of 92.383 and 91.713. The later comparison shows the impact of HSL on the accuracy of segmentations. Moreover, the generic network is less accurate than our spatially distributed approach used in DLB. The variant of generic Network with IQDA obtained a score of 92.491, whereas the variant without IQDA obtained a hybrid score of 92.425. Finally, we compare HSL with other weight initialization strategies. Specifically, HSL is compared with the neighbor transfer learning from AssemblyNet proposed by [START_REF] Coupé | AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation[END_REF] and models genesis proposed by [START_REF] Zhou | Models genesis[END_REF]. Although both variants obtained a better score compared to DLB without HSL, both initialization strategies gave a lower score than DLB with HSL and IQDA. Table 3 shows the effect of each contribution to domain shift robustness, when trained on ISBI challenge train-set and tested on MSSEG'16. First, the most robust combination is DLB with HSL and IQDA, it obtained a hybrid score of 0.684. Second, both the variants of DLB without IQDA and DLB without HSL are less accurate. They obtained hybrid scores of 0.673 and 0.648 respectively. Moreover, the generic network is less robust than our spatially distributed approach with DLB. The variant of the generic network with IQDA obtained a score of 0.672, whereas the variant without IQDA obtained a hybrid score of only 0.626. The later comparison shows the impact of IQDA on robustness even without the spatially distributed networks. Finally, HSL is compared with Assembynet [START_REF] Coupé | AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation[END_REF] and model genesis [START_REF] Zhou | Models genesis[END_REF] initialization strategies. The variants with model genesis and AssemblyNet initialization methods obtained respectively hybrid scores of 0.623 and 0.6103.

Cross-dataset Testing

In this section, we assess the cross-dataset robustness and generalization ability of our proposed approach. We chose to compare our method with three state-of-the-art approaches: nicMSlesion [START_REF] Valverde | One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks[END_REF], DeepMedic [START_REF] Kamnitsas | Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation[END_REF], and 2.5D Tiramisu [START_REF] Zhang | Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices[END_REF]. During the proposed validation, all the methods have been trained on exactly the same dataset (i.e., same preprocessing, same number of modalities, etc.) to ensure a fair comparison of method performance. Although reference methods have been originally proposed with a specific number of modalities (i.e., Tiramisu 2.5D and nicMSlesion were tested with 4 and 3 modalities respectively), their implementation is independent of the number of modalities since all modalities are concatenated and fed to the CNN. Besides, their official open-source implementations support the usage of only T1w and FLAIR sequences. Thus, in this evaluation, all methods are trained using only these two modalities. The following cross-dataset testing (cross-domain testing) consists in training each technique on one dataset at each time. Afterward, the obtained models are evaluated on the other datasets which contain unseen domains. We verified the average inference time per image for each method on the same machine and the same preprocessed images: 57.353s for nicMSlesion, 17.547s for DeepMedic, 47.471s for 2.5D Tiramisu, and 38.014s for DLB (this time does not include image preprocessing). Unlike using a single network to segment patches coming from the entire image, DLB uses multiple networks, one network for each sub-volume. These networks are loaded one by one to enable the use of a common GPU hardware solution (e.g., NVIDIA Titan Xp with 12 GB in our setup). Even though DLB requires sequential loading of multiple networks on GPU, the inference time over the whole image is similar to using a single network since network weights loading time is negligible compared to patch segmentation time. The ISBI score is returned by the challenge website only for ISBI test-set evaluation, and thus this metric is not available (NA) for testing on other datasets. When validating the methods on MSSEG'16, we report that 2.5D Tiramisu obtained slightly better results (not significantly) than DLB, in terms of hybrid score whereas nicMSlesion and DeepMedic performed relatively worse with 0.537 and 0.51 respectively. On our in-house dataset, DLB performed significantly better with a hybrid score of 0.696 while 2.5D Tiramisu, DeepMedic, and nicMSlesion obtained respectively 0.654, 0.523, and 0.419. We can notice that nicMSlesion offers poor cross-domain performance on 3D FLAIR when trained with a 2D FLAIR dataset. Table 5 shows the results of segmentation when training the different approaches on the MSSEG'16 dataset comprising 3D T1w and 3D FLAIR modalities. First, we notice that our approach obtained significantly better hybrid scores for both the ISBI test and the In-house datasets. Second, when validating on ISBI, the obtained submission score is 89.043 for DLB (the closest to human performance), 87.344 for DeepMedic, 87.173 for nicMSlesion, and 86.686 for 2.5D Tiramisu (the farthest from human performance). In the same conditions, 2.5D Tiramisu obtained the average Dice of 0.165 which indicates a failure of the method and thus a lack of generalization in this scenario (when trained on high-quality 3D FLAIR and tested on low-quality 2D FLAIR). Finally, for the In-house dataset, DLB produced significantly better segmentation than other methods. DLB obtained a hybrid score of 0.697 while nicMSlesion obtained 0.669, 2.5D Tiramisu obtained 0.664, and DeepMedic obtained the lowest score of 0.597.

Trained on ISBI

Trained on MSSEG'16

Trained on In-house

Table 6: Results of the different approaches trained on In-house dataset, with T1w and FLAIR modalities. For each metric, the bold values indicate the best result. In the hybrid score column, * indicates a significantly better score than the three other approaches, using the two-sided Wilcoxon test. Red values indicate hybrid scores lower than 0.5 or Dice index below 0.25. Table 6 shows the results of segmentation when training on our In-house dataset with 3D FLAIR. First, the obtained results when testing on MSSEG'16 indicates a close segmentation accuracy for DLB and 2.5D Tiramisu in terms of hybrid score (0.745 and 0.741) and slightly lower performance from nicMSlesion and DeepMedic (0.7 and 0.717). Second, we notice that our approach obtained a significantly higher hybrid score when validating on the ISBI testing dataset, with a submission score of 89.843 compared to 2.5 Tiramisu, DeepMedic, and nicMSlesion with 89.289, 86.810, and 84.512 respectively. In this scenario, nicMSlesion obtained the worst score with a Dice of 0.131 indicating a failure of the method.

Cross-dataset Testing Summary

First, it is noteworthy that when our approach obtained a better score, the superiority was statistically significant. On the contrary, when one of the other approaches obtained a higher score, the advantage was not significant using the Wilcoxon test.

Second, it should be pointed out that in all the considered cross-domain cases, DLB did not degenerate not even once while maintaining high scores. We reported for nicMSlesion trained on ISBI and validated on the In-house dataset a hybrid score of 0.419. We also recall the low performance of 2.5D Tiramisu trained on MSSEG'16 and tested on ISBI (0.462 hybrid score). This shows the cross-domain robustness of the proposed strategy. '16). We notice that DLB obtains the highest hybrid score and Dice index by a large margin compared to 2.5D tiramisu, DeepMedic, and nicMSlesion. Table 7: Summary of the cross-dataset experiment. The table represent the average of cross-dataset experiment results (see Table 4, Table 5, and Table 6) based on the number of images for each dataset. For each metric, the bold values indicate the best result. In the hybrid score column, * indicates a significantly better score than the three other approaches using the Wilcoxon test.

Strategy

Hybrid Second, Table 9 shows the current top-performing methods on the ISBI challenge website. 2.5D Tiramisu [START_REF] Zhang | Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices[END_REF] is the best-ranked method with the current highest ISBI Score of 93.21, followed in second place by nnUnet [START_REF] Isensee | NNU-Net: Breaking the spell on successful medical image segmentation[END_REF] with 93.09. Both approaches rely on 4 modalities (T1w, FLAIR, T2, PD). Our approach comes in third place using only 2 modalities, and obtained the ISBI submission score of 92.85. Although DLB uses a lower number of modalities, it obtained better results than IMAGINE [START_REF] Hashemi | Asymmetric loss functions and deep densely-connected networks highly-imbalanced medical image segmentation: Application to multiple sclerosis lesion detection[END_REF], Self-adaptive network [START_REF] Feng | A self-adaptive network for multiple sclerosis lesion segmentation from multi-contrast mri with various imaging sequences[END_REF], and Multi-branch [START_REF] Aslani | Multi-branch convolutional neural network for multiple sclerosis lesion segmentation[END_REF] that obtained respectively the scores of 92.49, 92.41, and 92.12. Approach Modalities CNN type Submission Score 2.5D Tiramisu [START_REF] Zhang | Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices[END_REF] T1w, FLAIR, T2, PD 2D 93.21 nnUnet [START_REF] Isensee | NNU-Net: Breaking the spell on successful medical image segmentation[END_REF] T1w, FLAIR, T2, PD 2D and 3D 93.09 DLB [ours] T1w, FLAIR 3D 92.85 IMAGINE [START_REF] Hashemi | Asymmetric loss functions and deep densely-connected networks highly-imbalanced medical image segmentation: Application to multiple sclerosis lesion detection[END_REF] T1w, FLAIR, T2, PD 3D 92.49 Self-adaptive network [START_REF] Feng | A self-adaptive network for multiple sclerosis lesion segmentation from multi-contrast mri with various imaging sequences[END_REF] T1w, FLAIR, T2, PD 3D 92.41 Multi-branch [START_REF] Aslani | Multi-branch convolutional neural network for multiple sclerosis lesion segmentation[END_REF] T1w, FLAIR, T2 2D 92.12

The high accuracy of the results was expected as both the training and testing sets share the same domain (same acquisition conditions, and same scanner...). By tuning and adapting a method to this specific domain conditions, we can obtain artificially higher performance (e.g., DLB with 4 modalities obtained a score of 92.92, and a 2D version of DLB obtained 93.14 5 ). However, in our opinion, results reported in the same domain experiment do not truly reflect methods performances. For instance, the best performing method of this section (2.5D Tiramisu) failed when trained on different datasets (obtained submission scores of 89.043 and 89.289 in Table 5 andTable 6). The limitation of such a validation strategy is one of the main messages of our paper. Hence, we consider that cross-dataset evaluation with diverse images from different domains is a better alternative for method assessment.

Cross-dataset Segmentation Consistency

Finally, a usually under-investigated method property is its cross-dataset segmentation consistency. To assess the consistency of our model segmentation, we decided to compare the segmentation produced by each approach on the same data when the model is trained on different datasets. We compute the Dice between the different segmentations of a method as a similarity index to quantify the prediction consistency. Table 10 shows the segmentation consistency for each approach in our cross-dataset setting.

First, we analyzed the segmentations on In-house when the models are trained respectively on ISBI train-set and MSSEG'16. In this case, DLB obtained the best score of 0.647, followed by 2.5D Tiramisu and DeepMedic with 0.6261 and 0.602 respectively. Lastly, nicMSlesion obtained a score of 0.217. Second, we analyzed the segmentations on MSSEG'16 when the models are trained respectively on ISBI train-set and In-house. In this case, we obtained close consistency scores for 2.5D Tiramisu and DLB with Dice scores around 0.72 while DeepMedic and nicMSlesion are less consistent with 0.537 and 0.514 respectively. Finally, we analyzed the segmentations on ISBI test-set when comparing the models trained on ISBI train-set, the models trained on In-house, and the models trained on MSSEG'16. For all settings, DLB was significantly more consistent than both other methods with a Dice ranging from 0.63 to 0.649. 2.5D Tiramisu segmentation consistency index varies from 0.217 to 0.485. DeepMedic consistency index fluctuates from 0.49 to 0.602. nicMSlesion is the least consistent with scores ranging from 0.177 to 0.512.

During our cross-dataset consistency experiment, DLB was the only method capable of ensuring segmentation consistency independent of the training dataset. Both other methods failed several times as indicated with red color in Table 10. 4 represents an image from the In-house dataset and the segmentation of the different methods when trained on the ISBI challenge and MSSEG'16 datasets. First, both nicMSlesion and 2.5D Tiramisu fail to segment the majority of lesions when trained on ISBI challenge dataset. This exhibits the limitation of the robustness of these methods to domain shift, especially for 2.5D Tiramisu currently considered as the state-of-the-art approach on the ISBI challenge. Second, DLB detects almost all the lesions in the same conditions. Third, although DeepMedic also detects most of the lesions, it is more prone to false positives compared to the other methods. Finally, when choosing MSSEG'16 as a training dataset, DLB produces the most similar segmentation to expert annotation. Figure 5 represents an image from MSSEG'16 dataset and the segmentation of the different methods when trained on ISBI challenge and In-house datasets. First, when trained on ISBI dataset, the segmentations of 2.5D Tiramisu, DeepMedic, and DLB are more accurate than nicMSlesion segmentation, although all techniques missed a large portion of the central lesion (False Negative) located around the midsagittal plane. These common voxels misclassification can result from the subjectivity of raters between training and testing datasets. Second, when trained on In-house, DLB delineates successfully most of the lesions. Especially in the case of small lesions, DLB missed only one lesion, whereas both nicMSlesion and 2.5 D Tiramissu missed four lesions, and DeepMedic misses two lesions. trained on MSSEG'16. Likewise, 2.5D Tiramisu produced better segmentation when trained on In-house than on MSSEG'16. Although DeepMedic is consistent for this case, the produced segmentation was less precise and prone to false positives compared to the other methods.

Discussion and Conclusion

Discussion

Deep learning-based segmentation models can be prone to generalization failure due to domain shift between training and testing data. Such a domain shift may be caused by hardware and preprocessing diversity, the difference in acquisition protocol or annotation protocol, that results in a difference between the distributions of training and testing datasets. Besides, we also have to acknowledge the subjectivity of raters in training datasets. Indeed, the disagreement between expert segmentations, both in the same dataset and across different datasets, can make it difficult to train a generalizing model. Our experiments showed the limited generalization capability of state-of-the-art approaches, whereas DLB was able to adapt across different domains. Our study emphasizes the importance of cross-dataset validation, particularly when considering the clinical application of machine learning.

DLB uses a group of several separately trained networks, each network is specialized in a particular sub-volume of the brain. In the ablation study (see Tables 2 and3), our spatially distributed networks strategy showed better generalization and higher accuracy than using a single model.

In our work, we considered both specialized local features, and generic global features of MS lesions. The hierarchical specialization learning proposes an alternative to network cascades (i.e., [START_REF] Valverde | Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach[END_REF]). Instead of using cascades that are prone to error propagation, we suggested a logical hierarchy during learning based on data selection and transfer learning. The ablation study (see Tables 2 and3) exhibits the contribution of HSL to accuracy and domain generalization compared to DLB without HSL.

In this paper, the proposed method was validated using an out-of-domain cross-dataset evaluation. This strategy ensures that the performance obtained is not biased by the training dataset domain information. Indeed, the use of testing and training images from the same domain is questionable and does not reflect the generalization ability. The community should start considering this issue for both the validation and the comparison of proposed methods. Automated MS lesion segmentation should be able to render the most accurate segmentation with the minimum number of modalities, to be efficiently adopted in clinical conditions and to limit inter-modality dependence. Many experts agree that FLAIR is the most important modality for MS lesion delineation. Moreover, T1w modality can provide complementary information for better white-matter, gray-matter, and cerebrospinal-fluid distinction. FLAIR and are the most available modalities for MS patients and in all MS benchmark datasets. Our method achieved a competitive performance using these two modalities even on unseen domains.

In this paper, we proposed a novel data augmentation technique to reduce domain shift introduced by the variability of image resolution and quality. IQDA simulates different acquisition conditions to reduce covariate shift. Our ablation study (see Tables 2 and3) showed IQDA as a solid contribution to segmentation accuracy and cross-domain generalization. Indeed, while other methods (nicMSlesion, DeepMedic, and 2.5D Tiramisu) involve usual data augmentation (rotation and flipping), such simple strategies failed to ensure good generalization on unseen datasets.

Both domain generalization and adaptation are concerned with reducing dataset bias. The difference between these strategies is that for domain adaptation, some unlabeled data or even a few labeled data from the target domain are exploited to capture properties of the target domain for model adaptation. However, in domain generalization, no samples of any kind are used from the target domain. Domain generalization has been proposed to address the problem of unavailability of target domain samples by leveraging the labeled data to learn a universal representation to generalize for any target domain and without any prior insight from that domain. In this work, we emphasize on testing the domain generalization of our approach with cross-dataset evaluation. Unlike domain adaptation such as one-shot domain adaptation (i.e., [START_REF] Valverde | One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks[END_REF]), DLB does not need expert segmentation from the target domain. Our testing conditions draw a clear distinction between training data containing source domains and testing data containing unseen target domains.

In section 3.2, we reported that the best performances of DLB have been obtained when using high-resolution 3D FLAIR datasets and multi-rater consensus ground truth for training. The resulting model can render more accurate segmentations for both 2D and 3D image resolution data, even across unseen domains. This observation led us to believe that to efficiently train 3D CNN-based models for domain generalization, it may be desirable to optimize the model using high-resolution training data.

With current available hardware, it is unfeasible to exploit 3D CNNs with equivalent depth and kernel size as state-of-the-art 2D CNNs. Consequently, many neuroimaging automated pipelines are still using 2D CNNs despite processing 3D data. Our results suggest that using multiple compact networks can approximate a larger and more stable model since the sum of features extracted by the group of specialized networks and the features of a hypothetical big network may be equivalent in terms of relevant information for MS lesion segmentation. In our work, we have chosen to break down the complexity of the task spatially, based on the sub-volume division of the whole brain volume. One other advantage of this distribution is the ability to train networks in parallel since network weights and images of each region are independent. It is possible to use several GPUs for parallel training.

Our full pipeline including the preprocessing, MS lesion segmentation with DLB, and an easy-to-read report is available on our repository6 .

Conclusion

DeepLesionBrain is a deep learning framework for MS lesion segmentation designed for domain generalization. First, we use a spatially distributed strategy of multiple compact 3D CNNs with large overlapping receptive fields, to produce consensus-based segmentation robust to domain shift. Our method is trained using hierarchical specialization learning to efficiently incorporate both generic and specialized features. Second, we propose a novel image quality data augmentation to increase training data variability in a realistic way. Finally, we use only T1w and FLAIR modalities to propose a method compatible with a large number of datasets.

The ablation study showed the impact of each contribution on segmentation accuracy and domain generalization. The out-of-domain cross-dataset testing is suggested as an alternative for method evaluation in areas that are sensitive to domain bias (i.e., medical imaging). Our validation showed the generalization ability of our method and its robustness to domain shift. We also proved experimentally that DLB produces consistent segmentations compared to other state-of-the-art approaches regardless of the training data domain.

Figure 1 :

 1 Figure 1: The two-steps training process of DeepLesionBrain (see Sect. 2.1.1 for more details)

Figure 2 :

 2 Figure 2: Illustration of the considered U-Net architecture. The number of input channels (NC) depends on the modality number (i.e., NC= 2, for using T1w and FLAIR ). Each block is composed of group normalization (GN), Convolution (Conv) and Rectified Linear Unit (ReLU) activation.

Figure 3 :

 3 Figure 3: FLAIR examples from the considered three datasets in the MNI space and after intensity normalization. From left to right, the two images are from ISBI, MSSEG'16, our in-house data, respectively.

Figure 6 :

 6 Figure 6: Part A (left) axial sections of multi-modal MRI (T1w and FLAIR) from ISBI dataset, and its respective raters segmentations for MS lesion segmentation. Part B (right) cross dataset segmentation of the image section shown in Part A. The first and second rows illustrate the segmentations of methods when trained respectively on MSSEG'16, and In-house datasets. First, second, third, and fourth columns represent respectively the segmentations of nicMSlesion, DeepMedic, 2.5D Tiramisu, and DeepLesionBrain.

Table 1 :

 1 Description of datasets used in this work.

		FLAIR resolution	Site	# Subjects	# Raters	Modalities
	ISBI train-set	2D	Mono 5, multiple time-points	2	T1w, FLAIR, T2, PD
	MSSEG'16	3D	Multi	15	7	T1w, T1w GADO, FLAIR, PD, T2
	In-house	3D	Multi	43	1	T1w, FLAIR

Table 3 :

 3 Ablation study results with different variants of our approach trained on ISBI challenge train-set and tested on MSSEG'16 (see caption of Table2for details).

	Method	Hybrid Score Dice	PPV	TPR LFPR LTPR
	DLB with HSL and IQDA	0.684*	0.639 0.768 0.608 0.319 0.700
	DLB with HSL and without IQDA	0.673	0.669 0.728 0.671 0.416 0.725
	DLB without HSL and with IQDA	0.648	0.562 0.806 0.489 0.320 0.629
	DLB with models genesis init. and IQDA	0.623	0.593 0.737 0.576 0.436 0.665
	DLB with AssemblyNet init. and IQDA	0.610	0.541 0.708 0.537 0.466 0.705
	The generic Network with IQDA	0.672	0.665 0.721 0.673 0.413 0.727
	The generic network without IQDA	0.626	0.625 0.763 0.588 0.449 0.611

Table 4 :

 4 Results of the different approaches trained on the ISBI training dataset, with T1w and FLAIR modalities. For each metric, the bold values indicate the best result. In the hybrid score column, * indicates a significantly better score than the three other approaches using the two-sided Wilcoxon test. Red values indicate hybrid scores lower than 0.5 or Dice index below 0.25.

	Trained on ISBI	Approach	Hybrid Score Dice	PPV	TPR LFPR LTPR CORR Submission Score
		nicMSlesion	0.537	0.442 0.614 0.423 0.504 0.629	0.495	NA
	MSSEG'16	DeepMedic 2.5D Tiramisu	0.510 0.711	0.476 0.542 0.560 0.829 0.850 0.664 0.741 0.658 0.284 0.695	0.509 0.730	NA NA
		DLB	0.684	0.639 0.768 0.608 0.319 0.700	0.650	NA
		nicMSlesion	0.419	0.204 0.727 0.129 0.309 0.361	0.158	NA
	In-house dataset	DeepMedic 2.5D Tiramisu	0.523 0.654	0.536 0.633 0.499 0.805 0.765 0.545 0.871 0.410 0.204 0.476	0.549 0.635	NA NA
		DLB	0.696*	0.675 0.850 0.564 0.342 0.644	0.718	NA
	Table 4 shows the results of segmentation when training the different approaches using T1w and FLAIR modalities,
	on the ISBI training dataset (2D resolution FLAIR).

Table 5 :

 5 Results of the different approaches trained on the MSSEG'16 dataset, with T1w and FLAIR modalities. For each metric, the bold values indicate the best result. In hybrid score column, * indicates a significantly better score than the three other approaches using the two-sided Wilcoxon test. Red values indicate hybrid scores lower than 0.5 or Dice index below 0.25.

	Trained on MSSEG'16	Approach	Hybrid Score Dice	PPV	TPR LFPR LTPR CORR Submission Score
		nicMSlesion	0.555	0.398 0.717 0.292 0.368 0.206	0.822	87,173
	ISBI test-set	DeepMedic 2.5D Tiramisu	0.547 0.462	0.378 0.801 0.265 0.416 0.298 0.165 0.937 0.096 0.075 0.160	0.717 0.212	87.344 86,686
		DLB	0.618*	0.535 0.697 0.471 0.353 0.373	0.835	89.043
		nicMSlesion	0.669	0.686 0.689 0.705 0.467 0.717	0.737	NA
	In-house dataset	DeepMedic 2.5D Tiramisu	0.597 0.664	0.645 0.647 0.670 0.721 0.811 0.706 0.766 0.694 0.432 0.801	0.650 0.552	NA NA
		DLB	0.697*	0.746 0.681 0.847 0.478 0.754	0.799	NA

Table 7

 7 sums up cross-dataset experiments results. This table presents the average score estimated over all the images obtained during the three experiments presented in Table 4, Table 5, and Table 6 (61 images for ISBI test-set, 43 images for In-house, 15 images for MSSEG

  Despite the previously mentioned limitations of in-domain validation, we also provide experiments using the same domain as complementary results. First, Table8shows the results of DLB, nicMSlesion, DeepMedic, and 2.5D Tiramisu on ISBI test-set after being trained on ISBI train-set (same domain), with T1w and FLAIR modalities. The three approaches give close results with submission scores of 92.923 for 2.5D Tiramisu, 92.849 for DLB, and 92.161 for nicMSlesion. DeepMedic comes last with a submission score of 90.866.

		Score Dice	PPV	TPR LFPR LTPR CORR
	nicMSlesion	0.526	0.365 0.695 0.308 0.362 0.338	0.595
	DeepMedic	0.554	0.483 0.725 0.429 0.556 0.520	0.649
	2.5 D Tiramisu	0.608	0.443 0.870 0.368 0.179 0.402	0.537
	DLB	0.663*	0.601 0.775 0.550 0.299 0.484	0.780
	3.3 Same Domain Validation

Table 8 :

 8 Results of the different approaches trained on the ISBI training dataset and tested on ISBI test-set, with T1w and FLAIR modalities. For each metric, the bold values indicate the best result. two-stepin the hybrid score column, * indicates a significantly better score than the three other approaches using the two-sided Wilcoxon test.

	Approach	Hybrid Score Dice	PPV	TPR LFPR LTPR CORR Submission Score
	nicMSlesion	0.724	0.639 0.853 0.541 0.144 0.432	0.863	92.161
	DeepMedic	0.649	0.643 0.827 0.557 0.408 0.530	0.873	90.866
	2.5D Tiramisu	0.750	0.672 0.865 0.592 0.150 0.513	0.868	92.923
	DLB	0.748	0.646 0.888 0.545 0.131 0.486	0.868	92.849

Table 9 :

 9 State-of-the-art published results for the ISBI challenge.

Table 10 :

 10 The consistency of the segmentations for each approach in the cross-dataset setting. The consistency index represents the test-set average of Dice values, each Dice is computed between two segmentations produced by the same method when trained on two different train-sets. Higher values indicate better consistency in the segmentations. The bold values indicate the best result and red values indicate consistency lower than 0.5. * indicates a significantly better segmentation consistency score than the three other approaches, using the two-sided Wilcoxon test.

	Test-set		In-house	MSSEG'16		ISBI Test-set
	Train-sets	Dataset 1 vs. Dataset 2	ISBI Train-set ISBI Train-set ISBI Train-set MSSEG'16 In-house MSSEG'16	In-house MSSEG'16	ISBI Train-set In-house
	The consistency of	nicMSlesion	0.217	0.514	0.512	0.250	0.177
	the model predictions	DeepMedic	0.602	0.537	0.490	0.602	0.496
	when trained on	2.5D Tiramisu	0.615	0.726	0.217	0.485	0.460
	Dataset1 vs. Dataset2	DLB	0.647	0.719	0.630*	0.637*	0.649*
	Figure

https://smart-stats-tools.org/lesion-challenge-upload-results

https://smart-stats-tools.org/lesion-challenge

https://github.com/sergivalverde/nicMSlesions

https://github.com/MedICL-VU/LesionSeg

https://smart-stats-tools.org/lesion-challenge

https://github.com/volBrain/DeeplesionBrain

Acknowledgements

This work benefited from the support of the project DeepvolBrain of the French National Research Agency (ANR-18-CE45-0013). This study was achieved within the context of the Laboratory of Excellence TRAIL ANR-10-LABX-57 for the BigDataBrain project. Moreover, we thank the Investments for the future Program IdEx Bordeaux (ANR-10-IDEX-03-02, HL-MRI Project), Cluster of excellence CPU and the CNRS/INSERM for the DeepMultiBrain This study has also been supported by the DPI2017-87743-R grant from the Spanish Ministerio de Economia, Industria Competitividad. The authors gratefully acknowledge the support of NVIDIA Corporation with their donation of the TITAN Xp GPU used in this research.


Antoine Lavault

Axel Roebel
email: roebel@ircam.fr

Matthieu Voiry

StyleWaveGAN: Style-based synthesis of drum sounds using generative adversarial networks for higher audio quality

Keywords: Percussive Sound Synthesis, Generative Models, Creative Interfaces

In this paper we introduce StyleWaveGAN, a stylebased drum sound generator that is a variation of StyleGAN, a state-of-the-art image generator. By conditioning StyleWaveGAN on the type of drum, we are able to synthesize waveforms faster than real-time on a GPU directly in CD quality up to a duration of 1.5s while retaining some control over the generation. We also introduce an alternative to the progressive growing of GANs and experimented on the effect of dataset balancing for generative tasks. The experiments are carried out on an augmented subset of a publicly available dataset comprised of different drums and cymbals. We evaluate against two recent drum generators, WaveGAN and NeuroDrum, demonstrating significantly improved generation quality using two quality measures: first the Frechet Audio Distance and second a perceptual test.

I. INTRODUCTION

Drum machines are musical devices creating percussion sounds using analog or digital signal processing [START_REF] Reid | Practical Snare Drum Synthesis[END_REF], [START_REF]Practical Cymbal Synthesis[END_REF]. The characteristic sound of this synthesis process contributed to their use in the '80s and their appreciation nowadays. However, these drum machines did not provide an extensive set of controls over the generation.

Following the success of deep learning, several generative processes for percussive sounds have been proposed in the recent years, and two approaches retained our attention. [START_REF] Drysdale | Adversarial Synthesis of Drum Sounds[END_REF] used a generative adversarial network (GAN) for waveform generation with a conditioning on the type of drum, generating 0.3s at 44100Hz. There is also [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF], where a GAN was trained to generate STFT of drum sounds, allowing them to generate 1s at 16kHz. Both of them used the progressive growing of GANs [START_REF] Karras | Progressive Growing of GANs for Improved Quality, Stability, and Variation[END_REF].

In this paper, we build upon the same idea of conditional synthesis using discrete and continuous controls, with time-domain generation like [START_REF] Drysdale | Adversarial Synthesis of Drum Sounds[END_REF] with a style-based approach (SGAN) [START_REF] Karras | A Style-Based Generator Architecture for Generative Adversarial Networks[END_REF], [START_REF] Karras | Analyzing and Improving the Image Quality of StyleGAN[END_REF]. We conduct our experiments on an augmented version of the ENST-Drums [START_REF] Gillet | ENST-Drums: An extensive audio-visual database for drum signals processing[END_REF] dataset, containing kick, snare, toms and hi-hats and comprising about 120k samples amounting to 100 hours of recordings. To evaluate the quality of the model on this dataset, we are using the Fréchet Audio Distance (FAD) [START_REF] Kilgour | Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms[END_REF], in an attempt to obtain a reference-free automatic evaluation of the generated samples. We also performed perceptual tests on the generated samples to measure how the generated samples are perceived by human listeners.

All in all, our goal is to create an algorithm for drum sound synthesis suitable for professional music production. In other words, we expect good output quality, real-time generation and relevant controls. We will especially compare to a few networks whose performances are summarized in table I. WaveGAN [START_REF] Dessein | Regularized optimal transport and the rot mover's distance[END_REF] is the first use of a generative adversarial network (GAN) for temporal generation of audio and Neu-roDrum [START_REF] Ramires | Neural Percussive Synthesis Parameterised by High-Level Timbral Features[END_REF] was the first to introduce perceptual features as part of the control for drum synthesis. DrumGAN [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF] uses a GAN with perceptual features as part of the control for drum synthesis and finally [START_REF] Drysdale | Adversarial Synthesis of Drum Sounds[END_REF] uses a GAN for drum sound synthesis at higher sample rate but with shorter duration than the previously mentioned networks.

Reference

Sample Rate Duration WaveGAN [START_REF] Dessein | Regularized optimal transport and the rot mover's distance[END_REF] 16kHz 1.1s NeuroDrum [START_REF] Ramires | Neural Percussive Synthesis Parameterised by High-Level Timbral Features[END_REF] 16kHz 1s DrumGAN [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF] 16kHz 1.1s Drysdale et al. [START_REF] Drysdale | Adversarial Synthesis of Drum Sounds[END_REF] 44.1kHz 0.4s Ours 44.1kHz 1.5s

II. MODEL A. Generative Adversarial Networks and StyleGAN

Generative Adversarial Networks (GAN) are a family of training procedures in which a generative model (the generator) competes against a discriminative adversary (the discriminator) that learns to distinguish whether a sample is real or fake [START_REF] Goodfellow | Generative Adversarial Networks[END_REF].

Instead of using a vanilla GAN, we are using an evolution called StyleGAN [START_REF] Karras | A Style-Based Generator Architecture for Generative Adversarial Networks[END_REF], [START_REF] Karras | Analyzing and Improving the Image Quality of StyleGAN[END_REF]. StyleGAN attempts to mitigate the entangled representation when using noise as latent and input of the generator. The key idea here is to use a style encoding, a vector which is obtained through a mapping network and is then used to control (through an affine transform) every layer of a synthesis network.

B. Proposed architecture

Since StyleGAN was originally used for high-quality image generation, we have to modify it for direct waveform generation. In particular, we transform 2D convolution (3 × 3) into 1D causal convolutions (1 × 9) [START_REF] Van Den Oord | WaveNet: A Generative Model for Raw Audio[END_REF], the upsampling is done with an averaging filter before each convolution block in the synthesis network, the mapping networks has 4 layers instead of 8 and the loss function is WGAN-LP [START_REF] Petzka | On the regularization of Wasserstein GANs[END_REF] (see figure 1).

We use the same number of filters, with respect to the depth, as StyleGAN2 [START_REF] Karras | Analyzing and Improving the Image Quality of StyleGAN[END_REF]. Just like StyleGAN2 , the synthesis network uses input/output skips and the discriminator is a residual network. In this work we follow [START_REF] Drysdale | Adversarial Synthesis of Drum Sounds[END_REF], [START_REF] Ramires | Neural Percussive Synthesis Parameterised by High-Level Timbral Features[END_REF] using a temporal signal representation. Informal perceptual evaluations performed in the initial phase of this study supported our idea that the temporal representation produces better audio quality than spectral representation : we suppose it is because of the high amount of noise and the importance of the transient in the drum sounds.

C. Noise addition layers

We modified the noise addition layers of StyleGAN to make them style-dependant. We also add noise shaping (with a linear fade out) to avoid noisy tails. Having controlled noise addition is useful since some classes need more noise than other to get a good quality synthesis.

D. Output envelopes

One of the main complaints during informal perceptual tests for StyleWaveGAN was the generated sounds have an audible noisy tail which makes them easily identifiable. To avoid this pitfall, we added envelopes after the output of the network.

These envelopes where generated using the training dataset, one per type of drum. For each sample of one given type, the final envelope is the filtered mean of the analytical part of the Hilbert transform of these normalized samples. A small fade out is applied to avoid audible clicks at the end of the generated sounds.

E. Controlling the network

To control the network, we use 5 labels describing the type of drum (kick, snare, toms, closed hi-hat and open hihat). The labels are fed into an embedding layer which is then concatenated to the latent z (c.f figure 1) and fed to the mapping network. These labels are concatenated after the mapping network too. This is done to allow the user to control the synthesis network without the necessity of a mapping network.

In our experiments, we are using 5 labels. These labels are added to the network with a one-hot vector. We expect to have a better disentanglement between the class label during the style encoding by using this method.

F. AutoFade

While Progressive Growing of GANs [START_REF] Karras | Progressive Growing of GANs for Improved Quality, Stability, and Variation[END_REF] is used in recent papers [START_REF] Drysdale | Adversarial Synthesis of Drum Sounds[END_REF], [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF], we have to note it was dropped by its creators [START_REF] Karras | Analyzing and Improving the Image Quality of StyleGAN[END_REF]. As a compromise, we introduce AutoFade. It is a ResNet architecture with a convolution path and a bypass where a learned parameter is used to fade more or less of one path. Rather than fixing a value like ResNet, we let the network choose the best value as part of the training process, without the need of training it block by block. If x and y represents the two different branches, we have:

sin(α)x + cos(α)y (1)
α is independent of x or y. It makes this structure an intermediate between ResNet and Highway Networks. By using trigonometric function in equation (1), we guarantee the conservation of the standard deviation, if both inputs have equal variance. Informal listening tests showed that AutoFade was of great use in the discriminator, but did not bring any improvement when used in the generator. The Autofade feature will therefore be evaluated in the following sections, only as part of the discriminator.

III. EXPERIMENTAL SETUP

A. Dataset

We are using a subset of ENST-Drums [START_REF] Gillet | ENST-Drums: An extensive audio-visual database for drum signals processing[END_REF], comprised of 350 samples of close miking of kicks, snares, toms and hihat. Since 350 elements is too low for a data-driven approach, we used an augmentation method similar to [START_REF] Jacques | Data Augmentation for Drum Transcription with Convolutional Neural Networks[END_REF]. We used SuperVP 1 to process the original dataset. The modifications applied to the sounds consist of a gain applied to transient/attack components [START_REF] Röbel | A new approach to transient processing in the phase vocoder[END_REF], noise components as well as independent transposition of the signal source and the spectral envelope.

The set of parameters is shown in table II. The limits have been obtained by means of subjective evaluation of the modified sounds aiming to avoid transformations that can be perceived as unnatural by a human listener. Examples are available in the supplementary material which can be found at https://alavault.github.io/stylewavegan eusipco/.

As a supplementary metric, the Fréchet Audio Distance between the original dataset and the augmented one is 0.62.

B. Training procedure

The training procedure is the same as StyleGAN 2 [START_REF] Karras | Analyzing and Improving the Image Quality of StyleGAN[END_REF], except that we trained the network on 2M samples. With a batch size of 10, it totals to 200k iterations.

C. Imbalanced dataset

Balancing datasets is common in classification tasks but to our knowledge, has never been done for generation tasks. As shown in table III, our augmented dataset is quite unbalanced, so to obtain a balanced dataset, we use a sampler which takes elements from sub-datasets (one per label) at random according to a uniform distribution. We call it "equal-proportion sampling", even if it is a form of oversampling method.

D. Baseline

In our evaluation, we will use NeuroDrum [START_REF] Ramires | Neural Percussive Synthesis Parameterised by High-Level Timbral Features[END_REF] and Wave-GAN [START_REF] Dessein | Regularized optimal transport and the rot mover's distance[END_REF] as baselines when comparing using an objective measure, and we will use demo samples provided by the author of DrumGAN [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF] as a baseline for a perceptual test. These demo samples have a sample rate of 16kHz.

Because NeuroDrum [START_REF] Ramires | Neural Percussive Synthesis Parameterised by High-Level Timbral Features[END_REF] works with 16kHz sample rate we adapted our model to use this sample rate for this comparison. We also compared with WaveGAN [START_REF] Dessein | Regularized optimal transport and the rot mover's distance[END_REF], a model with a similar training method, using our dataset with 44.1kHz. Here we configured both networks to generate 0.3s (@44.1kHz).

Unfortunately, DrumGAN is not reproducible because of missing source code or/and missing or unknown meta parameter. We also don't have any insights on the contents of the private dataset used for training DrumGAN, even if it sounds like synthetic drums were used. DrumGAN results are provided as a comparison to one of the most recent neural drum synthesizer.

The lack of available source code and meta parameters for [START_REF] Drysdale | Adversarial Synthesis of Drum Sounds[END_REF] makes it unreproducible as well. However, we know the dataset they built for this task is comprised of synthetic samples since they aimed at drum sound synthesis for electronic music production.

E. Evaluation 1) Reference-free evaluation:

We chose to use the Fréchet Audio Distance (FAD) [START_REF] Kilgour | Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms[END_REF], a reference-free evaluation metric for audio generation algorithms using a VGGish model trained on AudioSet. We compare the embedding of the augmented database to the embedding obtained from 64k samples generated by the evaluated network. In terms of computational cost, we achieve a generation rate of 52drum sounds/s on one 1080GTX with the network in full resolution (1.5s@44.1kHz).

Network

FAD Baseline [START_REF] Ramires | Neural Percussive Synthesis Parameterised by High-Level Timbral Features[END_REF] 25.35 StyleWaveGAN@16kHz 11.48 2) perceptual testing: In our perceptual evaluation framework, we are evaluating the quality of generation among 4 sets of sounds : the original dataset, the sounds of the augmented dataset that are generated with the most extreme setting of the transformation parameters in table II, sounds from StyleWaveGAN at full resolution(1.5s at 44.1kHz) and sounds from DrumGAN (courtesy of Javier Nistal). The comparison of these 4 sets is motivated as follows: first, results obtained for the most extreme examples of the augmented dataset will provide a lower bound for our model, and second the evaluation of DrumGAN synthesis on their private dataset establishes a baseline which is proven to be better than NeuroDrum [START_REF] Ramires | Neural Percussive Synthesis Parameterised by High-Level Timbral Features[END_REF] in terms of FAD and claiming to be comparable to [START_REF] Dessein | Regularized optimal transport and the rot mover's distance[END_REF] in terms of perceived quality.

Instead of NeuroDrum or WaveGAN, we could use Drum-GAN [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF] or Drysdale et al. [START_REF] Drysdale | Adversarial Synthesis of Drum Sounds[END_REF] as our perceptual test baseline. Both are unreproducible due to lack of available source code and metaparamaters. Drysdale et al. uses drum type conditioning when DrumGAN only has perceptual feature conditioning. However, we were able to get samples with drum type conditioning for DrumGAN through personal communication with the main author of [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF]. Their performance differ, since Drum-GAN can generate samples of 1.1s at 16kHZ, while Drysdale et al. can generate samples of 0.4s at 44.1kHz. We will see in the following discussion that the test participants indicate that the decay time is important to evaluate the realness of drum sounds, which gives an advantage to DrumGAN with its longer samples even if the sample rate is lower. Also, both of these solutions do not provide much insights about their training datasets. We know that Drysdale et al. focus on sample-based electronic music (EM), as described in their article. Samples used in EM are inherently synthetic and are built to sound different from real drums. Since our perceptual testing aims to evaluate how close the synthesis sounds like a real drum, having synthetic samples in the set will always be evaluated as worse.

Given the limitations discussed above, we selected Drum-GAN as our baseline in our perceptual test. Its role is to represent one of the models of the state of the art.

The mean opinion score (MOS) is calculated as the average of the score given by the test participants. Part of the samples generated with StyleWaveGAN and used for the test are available in the supplementary material here.

Due to its low-risk nature, this evaluation didn't need an ethic approval from the host organization.

IV. EXPERIMENTAL RESULTS

A. Impact of our contributions

The first result we have is that we improved in terms of FAD (tables IV and V) when comparing to NeuroDrum and WaveGAN. We can also see from table V that using AutoFade in the discriminator helped at getting a better generation.

The results with dataset balancing are mitigated. It improved the supervised generation, as seen on table VI. However, without the label conditioning, using it didn't bring any decrease in the FAD : since it makes the training and evaluation dataset different (in proportions), the learned distribution differs, impacting negatively the FAD.

Envelopes on the output are the greatest contribution to the quality of generation in terms of FAD, almost halving it when comparing to StyleWaveGAN without the output envelope.

B. Results of perceptual testing

9 people took part on the test. The total Mean Opinion Score (MOS), with their confidence interval at 95%, is shown in table VIII as well as more detailed per-class results. Even if the number of participant is low, most of them (5 out 9) are audio professionals. Each of them were presented with 24 samples to evaluate along a scale going from "1-Poor" to "5-Real drum". Data was randomly picked among the original and augmented dataset as well as samples generated by StyleWaveGAN with a fixed label.

The score of the augmented samples is slightly lower than the real samples. This indicates that the extreme cases of our augmentation strategy are a bit too extreme. Here less extreme augmentation parameters with more intermediate values should be selected for future work. The main problem that can be found against the augmented samples comes from the pitch changes made by the augmentation process. The pitch change affect negatively the attack of the sound, making them sounding less natural than the real data. While the change is minor, it is sufficiently present to be perceived and graded worse than a real sample. Note however that these extreme parameter combinations remain rather rare in the full set of augmented sounds.

Comparing StyleWaveGAN to DrumGAN we can conclude that StyleWaveGAN trained on augmented data produces results that are perceived either similarly close (kick) or significantly closer (snare, cymbals) to real drums than DrumGAN trained on a drum dataset obtained from sources that are not further detailed in [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF]. We conclude that the StyleWaveGAN model trained on augmented data achieves state of the art performance for drum synthesis.

We now discuss the StyleWaveGAN results in details. Given StyleWaveGAN was trained on the full dataset of augmented samples, a perfect model should produce results in between the test results of the real data and the extreme examples of the augmented data. We note that StyleWaveGAN achieves this performance only for the cymbals. Snare and kick synthesis remain less natural. A discussion with the participants of the perceptual tests reveals the following problems: for the kick drum sounds, the SWG model does not produce the characteristics long tail of the resonances and is also missing some energy in the frequency band below 100Hz. For snare drum synthesis the main problem appears to be the fact that SWG creates hybrids of sounds generated with sticks, mallets and brushes. Concatenating for example an attack of a snare sound obtained with a stick with a decay of a snare sound obtained with a brush creates fair sounding but unrealistic samples. These problems with kick and snare sounds indicate that the current implementation of the discriminator is not sufficient and further investigation will be required to improve the discriminator loss such that it avoids these perceptual problems.

V. CONCLUSION AND FUTURE WORK

In this paper, we presented a new method for drum synthesis using StyleWaveGAN, an adaptation of a state of the art image generator. The proposed method has explicit controls on drum type to give some basic controllability.

We have shown the proposed style-based synthesis achieves a significantly reduced FAD compared to recent DNN based drum synthesis methods [START_REF] Dessein | Regularized optimal transport and the rot mover's distance[END_REF], [START_REF] Ramires | Neural Percussive Synthesis Parameterised by High-Level Timbral Features[END_REF]. perceptual tests also show that our network performs quite well in terms of perceived quality when comparing to processed and real data as well as DrumGAN [START_REF] Nistal | DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks[END_REF]. To the best of our knowledge the proposed DNN is the first achieving drum synthesis with 44.1kHz sample rate (for sounds with a duration of 1.5s) with an inference speed more than 50 times faster than real-time on a consumer GPU: in 1 second we can generate 50 sounds 1.5 s long at described sample rate.

In terms of future work we will continue to work on the sound quality and additional controls for velocity as well as high-level control using perceptually relevant audio descriptors.

Fig. 1 .

 1 Fig. 1. StyleWaveGAN

Fig. 2 .

 2 Fig. 2. Generated envelopes from the training dataset

TABLE I COMPARISON

 I OF STATE OF THE ART NEURAL DRUM SYNTHESIZERS

  SuperVP is available free of charge in form of a Max/MSP object at https://forum.ircam.fr/projects/detail/supervp-for-max/

	Element	Proportion
	Kick	3%
	Snare	18%
	Toms	45%
	Closed hi-hat	10%
	Open hi-hat	22%
	TABLE III
	DATASET POPULATION
	1

TABLE VIII MOS

 VIII ON DIFFERENT SETS OF SOUNDS, GLOBAL AND PER-LABEL (1 IS LOWEST, 5 IS HIGHEST)