Florian Allender

Rémi Allègre

Cédric Wemmert

Jean-Michel Dischler

Data augmentation based on spatial deformations for histopathology: An evaluation in the context of glomeruli segmentation

Keywords: Histopathological images, glomeruli segmentation, data augmentation, random spatial deformations

come

Introduction

In the field of digital pathology, deep learning techniques for whole slide image (WSI) analysis have attracted considerable attention in recent years [START_REF] Litjens | Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis[END_REF][START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF][START_REF] Komura | Machine learning methods for histopathological image analysis[END_REF][START_REF] Dimitriou | Deep learning for whole slide image analysis: An overview[END_REF][START_REF] Srinidhi | Deep neural network models for computational histopathology: A survey[END_REF][START_REF] Wemmert | Deep learning for histopathological image analysis[END_REF]. Deep models based on Convolutional Neural Networks (CNN) have the ability to automatically identify microscopic structures, which brings significant support to diagnosis. However, these models are based on learning patterns in images and require a large amount of annotated data to be generalizable. This requirement is a challenge, especially for application of deep models to diseases where few samples are available. This is the case for renal pathology on which we focus on [START_REF] Barisoni | Digital pathology and computational image analysis in nephropathology[END_REF]. Data augmentation [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF] tackles this limitation by expanding the variety of training datasets in an artificial manner, which may involve various image manipulations such as spatial and colorimetric transformations, noise injection, image mixing, or synthetic image generation using generative models. While data augmentation has proven to improve the performance of deep models for various applications, augmentations are domain-dependent and require data-specific design [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF]. Automatically finding optimal augmentation policies has been investigated recently for classification, detection or segmentation tasks, using meta-learning schemes or adversarial training (see e.g. AutoAugment [START_REF] Cubuk | AutoAugment: Learning augmentation strategies from data[END_REF][START_REF] Zhang | Adversarial AutoAugment[END_REF]). These methods still require to select appropriate models for transformations.

In this paper we consider the supervised segmentation of glomerular structures in WSI patches of renal histopathological slides. These complex structures are responsible for blood filtration and production of urine. Renal histopathology is crucial in the study of kidney diseases, and especially transplant rejection, which occurs with an incidence of 7.9% in the first year [START_REF] Naik | Renal transplantation rejection[END_REF]. Glomeruli are ball-shaped structures that incorporate multiple substructures: membranes, capillaries, mesangial and endothelial cells, podocytes. The appearance of glomeruli can thus exhibit great variability in WSIs (Fig. 1). Besides sectioning, this variability also stems from stainings, patients, procedures in laboratories, and/or microscope and imaging devices, making the robust detection and segmentation of glomeruli challenging. While there has been recent advances regarding invariance to staining variability [START_REF] Lampert | Strategies for training stain invariant CNNS[END_REF][START_REF] Liu | Towards staining independent segmentation of glomerulus from histopathological images of kidney[END_REF][START_REF] Vasiljević | Self adversarial attack as an augmentation method for immunohistochemical stainings[END_REF], we focus here on the variability in shapes, i.e. geometric variability, which has not yet been addressed thoroughly in the Medical Imaging literature. We rely on a standard U-Net model [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] for segmentation and evaluate the impact of augmenting training data with random spatial deformations. We define a random spatial deformation as a vector field resulting from the filtering of a white noise, i.e. the realization of a stochastic process with uniform or Gaussian distribution. Random spatial deformations can be implemented in many different ways, and a vast variety of models has been proposed in the literature, differing on how the vector field is generated.

Considering a selection of state-of-the-art models for random spatial deformations, we address the following questions: (1) Among the considered deformation models, which are best suitable for data augmentation, with the objective of segmenting glomeruli in WSI patches using U-Net? (2) Which range of parameter values are adequate? (3) How can we improve deformation models to achieve better segmentation results? In this context, our contributions are the following:

• We propose a comparative study of a selection of state-of-the-art random spatial deformation models, including Random Displacement Fields and deformations based on control points. • We provide appropriate range of parameter values for each of the considered random spatial deformation models. • We propose an original method based on the detection of cell nuclei centers for the automatic positioning of scattered control points, that simplifies the control of random deformations and reduces the computational complexity of finding best parameter values. • We provide a complete framework for identifying which deformation models and range of parameter values are the best choices for a given application.

In the following section, we review related work (Section 2). Next, we provide a description of the spatial deformation models considered in this paper (Section 3). Our evaluation protocol is then developed (Section 4), before we present and discuss our experimental results (Section 5 and 6). Complementary results discussed in the paper are provided as a Supplemental material.

Related work

Glomeruli segmentation. We consider the semantic segmentation of glomeruli from tissue (i.e. background) in patches extracted from WSIs [START_REF] Bel | Automatic segmentation of histopathological slides of renal tissue using deep learning[END_REF][START_REF] Liu | Towards staining independent segmentation of glomerulus from histopathological images of kidney[END_REF][START_REF] Bueno | Glomerulosclerosis identification in whole slide images using semantic segmentation[END_REF][START_REF] Merveille | An automatic framework for fusing information from differently stained consecutive digital whole slide images: A case study in renal histology[END_REF][START_REF] Davis | Deep learning segmentation of glomeruli on kidney donor frozen sections[END_REF][START_REF] Salvi | Automated assessment of glomerulosclerosis and tubular atrophy using deep learning[END_REF], which consists in solving a two-class problem at pixel level for each input patch. We suppose that each patch contains either a single glomerulus surrounded by tissue, or partial glomeruli, or tissue only. The U-Net model [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] is the state-of-the-art CNN architecture for segmentation in Biomedical Imaging [START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF], especially for glomeruli segmentation in WSI patches [START_REF] Bel | Automatic segmentation of histopathological slides of renal tissue using deep learning[END_REF][START_REF] Merveille | An automatic framework for fusing information from differently stained consecutive digital whole slide images: A case study in renal histology[END_REF][START_REF] Davis | Deep learning segmentation of glomeruli on kidney donor frozen sections[END_REF]. Some variants like U-Net++ [START_REF] Zhou | UNet++: A nested U-Net architecture for medical image segmentation[END_REF][START_REF] Zhou | UNet++: Redesigning skip connections to exploit multiscale features in image segmentation[END_REF] or SE U-Net [START_REF] Roy | Concurrent spatial and channel 'squeeze & excitation' in fully convolutional networks[END_REF] result in slight improvements, as reported in [START_REF] Liu | Towards staining independent segmentation of glomerulus from histopathological images of kidney[END_REF]. Other architectures like Seg-Net [START_REF] Bueno | Glomerulosclerosis identification in whole slide images using semantic segmentation[END_REF] or U-Net with a ResNet34 backbone [START_REF] Salvi | Automated assessment of glomerulosclerosis and tubular atrophy using deep learning[END_REF], pre-trained on the ImageNet dataset, have been proposed. Both works report improved segmentation results w.r.t. U-Net, Salvi et al. [START_REF] Salvi | Automated assessment of glomerulosclerosis and tubular atrophy using deep learning[END_REF] adding a post-processing stage based on cell nuclei and lumen detection to refine the segmentation results. These works only consider flipping, rotation and scaling as geometric transformations. In this paper, we choose to rely on the basic, well established and still widely used standard U-Net model, in order to study and understand the benefits of random spatial deformations as an augmentation. Note also that we do not use a pre-trained model. [START_REF] Bel | Automatic segmentation of histopathological slides of renal tissue using deep learning[END_REF]

[18] [19] [20] [24] [26] [27] [29] [38] [31] [36] Rotations ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Mirroring ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Shift/crops ✓ ✓ ✓ ✓ Scaling ✓ ✓ ✓ ✓ ✓ ✓ Shearing ✓ ✓ Elastic deformations ✓ ✓ ✓ ✓ ✓ ✓ ✓ Blur ✓ ✓ ✓ ✓ ✓ ✓ Additive noise ✓ ✓ ✓ Cutout ✓ Color shifting ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Table 1: Augmentations used in a selection of papers from the digital histopathology field.

Glomeruli segmentation in patches of WSIs is different from direct segmentation of WSIs, as achieved e.g. in [START_REF] Hermsen | Deep Learning-Based Histopathologic Assessment of Kidney Tissue[END_REF][START_REF] Gadermayr | Cnn cascades for segmenting sparse objects in gigapixel whole slide images[END_REF][START_REF] Altini | Semantic segmentation framework for glomeruli detection and classification in kidney histological sections[END_REF][START_REF] Kuklyte | Evaluation of the Use of Single-and Multi-Magnification Convolutional Neural Networks for the Determination and Quantitation of Lesions in Nonclinical Pathology Studies[END_REF], where modified U-Net architectures are used to account for high-resolution input images. Also note that segmentation of glomeruli in WSI patches is different from glomeruli detection, which is achieved with CNN architectures conceived for object detection like Faster RCNN or YOLOv3 in the most recent works [START_REF] Temerinac-Ott | Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities[END_REF][START_REF] Gallego | Glomerulus classification and detection based on convolutional neural networks[END_REF][START_REF] Liu | Towards staining independent segmentation of glomerulus from histopathological images of kidney[END_REF][START_REF] Kannan | Segmentation of glomeruli within trichrome images using deep learning[END_REF][START_REF] Heckenauer | Real-time detection of glomeruli in renal pathology[END_REF].

Data augmentation for histopathology. Data augmentation has been introduced in deep learning to prevent overfitting issues that arise from small and imbalanced datasets. A complete review is beyond the scope of this paper, the reader can refer to [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF] for a survey. There exist two main approaches: 1) The transformation approach consists in applying individual (or compositions of) transformations to training datasets, or in some cases, to test datasets; 2) The generative approach consists in generating new data by means of a trained generative model, e.g. a Generative Adversarial Network (GAN). We focus here on augmentations of the first category, applied to training datasets. Geometric transformations like affine transformations and spatial deformations have become usual for almost all applications, besides color shifting, addition of noise, or image mixing or blending, that are part of libraries like Augmentor [START_REF] Bloice | Biomedical image augmentation using Augmentor[END_REF]. In the field of digital histopathology, common augmentations include geometric transformations and specific transformations related to staining variations [START_REF] Van Eycke | Strategies to reduce the expert supervision required for deep learning-based segmentation of histopathological images[END_REF][START_REF] Xiao | A new color augmentation method for deep learning segmentation of histological images[END_REF][START_REF] Mpinda Ataky | Data augmentation for histopathological images based on gaussian-laplacian pyramid blending[END_REF][START_REF] Faryna | Tailoring automated data augmentation to H&E-stained histopathology[END_REF]. Those types of augmentations have been especially used for glomeruli detection and segmentation [START_REF] Bel | Automatic segmentation of histopathological slides of renal tissue using deep learning[END_REF][START_REF] Altini | Semantic segmentation framework for glomeruli detection and classification in kidney histological sections[END_REF][START_REF] Hermsen | Deep Learning-Based Histopathologic Assessment of Kidney Tissue[END_REF][START_REF] Heckenauer | Real-time detection of glomeruli in renal pathology[END_REF][START_REF] Merveille | An automatic framework for fusing information from differently stained consecutive digital whole slide images: A case study in renal histology[END_REF][START_REF] Davis | Deep learning segmentation of glomeruli on kidney donor frozen sections[END_REF][START_REF] Faryna | Tailoring automated data augmentation to H&E-stained histopathology[END_REF][START_REF] Salvi | Automated assessment of glomerulosclerosis and tubular atrophy using deep learning[END_REF]. See Table 2 for a summary of the augmentation methods used in these papers. One can note that rotations and mirroring are systematically used, and that staining transformations and elastic deformations are among the most commonly used augmentation methods. Elastic deformations refer to random spatial deformations based on control points on a regular grid, as typically proposed by Ronneberger et al. [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. The benefits of staining transformations [START_REF] Tellez | Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology[END_REF][START_REF] Lampert | Strategies for training stain invariant CNNS[END_REF][START_REF] Vasiljević | Self adversarial attack as an augmentation method for immunohistochemical stainings[END_REF][START_REF] Faryna | Tailoring automated data augmentation to H&E-stained histopathology[END_REF] and affine transformations [START_REF] Faryna | Tailoring automated data augmentation to H&E-stained histopathology[END_REF] have been quantified. The impact of using different augmentations for the segmentation of WSIs of rat organs is evaluated in [START_REF] Kuklyte | Evaluation of the Use of Single-and Multi-Magnification Convolutional Neural Networks for the Determination and Quantitation of Lesions in Nonclinical Pathology Studies[END_REF], including affine transformations and elastic deformations, but only with a binary approach (with/without). In this paper, we propose to fill the gap regarding random spatial deformations and the adjustment of the parameter values for the case of glomeruli segmentation in WSI patches.

✓ ✓ ✓ ✓ ✓ Mirroring ✓ ✓ ✓ ✓ ✓ Shift/crops ✓ ✓ Scaling ✓ ✓ ✓ ✓ Elastic deformations ✓ ✓ ✓ ✓ Blur ✓ ✓ ✓ ✓ Additive noise ✓ ✓ Cutout ✓ Color shifting ✓ ✓ ✓ ✓
Table 2: Augmentations used in a selection of papers from the digital histopathology field.

Automatic data augmentation. Some recent works attempt to find optimal data augmentation policies automatically, through meta-learning schemes [START_REF] Cubuk | AutoAugment: Learning augmentation strategies from data[END_REF][START_REF] Lim | Fast AutoAugment[END_REF][START_REF] Cubuk | Randaugment: Practical automated data augmentation with a reduced search space[END_REF], adversarial training [START_REF] Zhang | Adversarial AutoAugment[END_REF] or a combination for better performance [START_REF] Tang | Onlineaugment: Online data augmentation with less domain knowledge[END_REF]. [START_REF] Faryna | Tailoring automated data augmentation to H&E-stained histopathology[END_REF] apply the Ran-dAugment methodology [START_REF] Cubuk | Randaugment: Practical automated data augmentation with a reduced search space[END_REF] in the context of a WSI patch classification task for breast cancer detection. In this paper, we evaluate the impact of data augmentation by considering a single deformation model at a time, i.e. we do not mix deformed images obtained with different models. Therefore we use a simple meta-learning scheme that performs parameter sampling inspired by RandAugment [START_REF] Cubuk | Randaugment: Practical automated data augmentation with a reduced search space[END_REF], in order to find the parameter values that achieve the best segmentation performance.

Spatial deformation models. The most basic spatial deformation model consists of linear transformations, that include affine and projective transformations. While linear transformations are ubiquitous in data augmentation policies [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF], often only consisting of flips and rotations, they enable limited variations in image content. Non-linear models have been developed to increase the variety and locality of spatial deformations, some of them describing the deformations in terms of a physical process, and known in the literature as elastic deformations, non-rigid deformations, or domain warping (see [START_REF] Nealen | Physically based deformable models in computer graphics[END_REF] for a survey). In the Medical Imaging community, non-linear deformations have been widely employed for image registration [START_REF] Wang | A review of deformation models in medical image registration[END_REF]. They attracted attention more recently for the purpose of improving the training of deep learning models. Chlap et al. [START_REF] Chlap | A review of medical image data augmentation techniques for deep learning applications[END_REF] surveyed deformation models in the context of data augmentation for CT and MRI images. In the field of digital histopathology, correspondences cannot be established between images, which makes image registration or statistical shape models inadequate to produce variations of training images. The alternative is to generate random deformations, i.e. vector fields resulting from the realization of a stochastic process with controlled statistics.

Previous work in the field of digital histopathology [START_REF] Bel | Automatic segmentation of histopathological slides of renal tissue using deep learning[END_REF][START_REF] Altini | Semantic segmentation framework for glomeruli detection and classification in kidney histological sections[END_REF][START_REF] Hermsen | Deep Learning-Based Histopathologic Assessment of Kidney Tissue[END_REF][START_REF] Heckenauer | Real-time detection of glomeruli in renal pathology[END_REF][START_REF] Merveille | An automatic framework for fusing information from differently stained consecutive digital whole slide images: A case study in renal histology[END_REF][START_REF] Davis | Deep learning segmentation of glomeruli on kidney donor frozen sections[END_REF] report the use of two random spatial deformation models involved in data augmentation for the segmentation of WSIs patches: Random Displacement Fields as proposed in [START_REF] Simard | Best practices for convolutional neural networks applied to visual document analysis[END_REF], and grid-based deformations, as suggested in the original U-Net paper [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. Two choices have to be done: 1) the deformation model, and 2) the parameter values. In this paper, we review these models (Section 3), and we evaluate their impact when used for data augmentation in the context for glomeruli segmentation (Section 4). We note that the influence of the parameter values chosen in previous work is not evaluated, and that these values result in deformed images that are visually close to the available training ones. The search for optimal parameter values for the data augmentation task has not yet been investigated. In the present paper, we explore the appropriate values for each of the considered deformation models.

Random spatial deformations

Background and notations

A 2D spatial deformation is a continuous function D : R 2 → R 2 that maps input coordinates x = [x, y] to transformed coordinates D x (x), D y (y) , possibly not bijective. Applying deformations to images involves a sampling (i.e. continuous-to-discrete) problem solving. Transformed coordinates may not correspond to pixel locations, and may not cover the whole spatial image domain. Therefore, the transformation D is usually defined in an inverse fashion (a.k.a. reverse warping), and computed from the pixel grid of the output image to the spatial domain of the input image. In what follows, we will consider square images of size N 2 . The pixel grid of the output (deformed) image I will be denoted as

P = {p i } N 2 i=1
, where p i = [x i , y i ] are the pixel coordinates. The pixel grid of the associated input image I ′ will be denoted as

P ′ = {p ′ i } N 2 i=1 .
In our context, we can write:

D(P) = P + - → D (1)
where

- → D = { ⃗ d i } N 2
i=1 is a set of displacement vectors, or flow field. Following this approach, pixel values in I are obtained by interpolating pixel values in I ′ , considering the pixel locations that are the closest from the deformed coordinates D(P) in the spatial domain of I ′ , i.e.: I = S(D(P),

I ′ ) = S(P + - → D, I ′ ) ( 2
)
where S is a continuous interpolation function. Common choices for S are bilinear interpolation, and higher-order interpolation such as bicubic or spline interpolation. We consider that this choice is not critical for our study, and rely either on bilinear, or bicubic interpolation. Note that some locations in D(P) may fall outside the bounds of the input pixel grid. We solve this issue by extending the input image with symmetric padding along the edges of the input image.

Characterization of random deformations

Definition. A random spatial deformation model is defined by a specific method to generate a flow field -→ D from a set of random values, so that -→ D is a multivariate Gaussian random field where the displacement vectors on sampling (i.e. pixel) locations are spatially correlated. Those spatial correlations may arise from filtering (e.g. for Random Displacement Fields), or from the distance to control points and associated interpolation function (e.g. for grid-based deformations). Geometric properties. Random deformations may preserve the shapes and textures contained in an image only at some degree, making the result more or less "realistic", or "distorted". While previous work consider augmenting data with visually realistic deformed images, our goal is to compare the deformation models and determine the optimal amount of distortion for the training of the segmentation model. For this purpose, we propose to characterize random deformations from a geometric point of view.

The ability of random deformations to preserve the shapes and textures contained in an image is directly related to the preservation of distances and angles. One special case of distortion is foldover, that occurs when the orientation between pairs of pixel locations is flipped. The occurrence of foldover can be seen as an upper limit of distortion, since the induced topological inconsistencies may cause profound changes in the local appearance of the deformed images.

The Another feature of flow fields involved in shape and texture preservation is the local scaling of angles. A deformation that preserves angles is said to be conformal, and its Jacobian matrix is a rotation matrix multiplied by a scalar. We evaluate the closeness of a deformation to a conformal deformation by computing the dot product of the normalized components of the Jacobian matrix, i.e. S(p)

= D x (p) ∥D x (p)∥ • D y (p)
∥D y (p)∥ . The value of S(p) ranges between -1 (maximum angle expansion) and 1 (maximum angle shrinking). The closest to 0, the highest the angle preservation.

For the selected deformation models and varying parameter values, we examine the distribution of the previous metrics, calculated on 1k randomly generated flow fields in the following form:

• Foldover rate, computed as:

|{p i ∈ P s.t. J(p i ) ≤ 0} N 2 i=1 |/N 2 ; • Preservation of distance, computed as the histogram of {J(p i ), p i ∈ P};
• Preservation of angles, computed as the histogram of {S(p i ), p i ∈ P}. Fig. 3 shows some plots of these distributions for the Random Displacement Fields model. Additional plots are available in the Supplemental material.

Deformation models

In this section, we review Random Displacement Fields and deformations based on control points for generating random deformations. We also introduce a novel approach based on unstructured control points located at detected cell nuclei centers.

Random Displacement Fields

Random Displacement Fields (RDF) have been proposed by Simard et al. [START_REF] Simard | Best practices for convolutional neural networks applied to visual document analysis[END_REF] to simulate smooth deformations. First, a displacement field

- → V = {⃗ v i } N 2 i=1 is generated, where v i = [v x , v y ] is such that v x = rand(-1, +1
) and v y = rand(-1, +1). The function rand(-1, +1) returns uniformly distributed pseudo-random numbers between -1 and +1. Then, -→ V is convolved with a Gaussian filter G σ of standard deviation σ, where σ acts as an "elasticity" coefficient (small values result in a completely random field, while large values tend to produce rigid deformations). The RDF is finally multiplied by a scaling factor α that controls the amplitude of the deformation. The flow field is therefore obtained as:

- → D = α G σ ( - → V) (3)
The only work related to glomeruli segmentation that mentions the usage of RDF for data augmentation is the work by Merveille et al. [START_REF] Merveille | An automatic framework for fusing information from differently stained consecutive digital whole slide images: A case study in renal histology[END_REF], where σ = 10 and α = 100, for patches of size N = 508. This choice is not discussed by the authors. The sampling resolution w.r.t. tissue size is the same as in our experiments (0.506 µm/pixel), making it possible to deform our patches of size N = 256 with the same impact. These values produce results that are visually close to the original images, as shown in Fig. 4.

The curve plot of Fig. 3(a), first row, illustrates the emergence of foldover for the cases σ = 10, that occurs starting from α ≈ 150 (see the previous Section for interpretation). The foldover rate remains low (less than 1%) for values of 150 < α < 200, and only affects 10% of the deformation fields. Distances remain well preserved, while the scale factor of angles approximately ranges between -0.5 and 0.5. These elements question about the benefit of larger values of α w.r.t. σ, that may contribute favorably to data augmentation, and thus are to be considered for a study of the optimal parameter values. Fig. 4 shows deformation results for σ = 10 and σ = 40, with varying α. This figure, besides Fig. 3 shows that deformations with low distortion can be produced with larger values of σ, and appropriate values of α, such as α = 2400 for σ = 40. This can be explained by the band-limiting effect of the Gaussian filter, that increases with σ, and requires larger values of α to produce distortions. Controlling the amount of distortion is a difficult task with this model, since there is no intuitive idea of the amplitude of the deformations induced by a value of α for a given value of σ. Fig. 5 shows that, for a given value of σ, a linear relation exists between α and the displacement distance computed as the norm of the displacement vectors, with a slope that is however not proportional to σ. The following section focuses on deformations based on control points, that remedy to this shortcoming.

Deformations based on control points

Deformations based on control points are highly common, and a number of interpolation methods have been adapted to this task. Given a set of source control points

C ′ = {c ′ k } K k=1 , target locations C = {c k } K k=1
are computed with random displacements. As suggested by the authors of U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], we consider here that the local magnitude of the displacements follows a Normal distribution with zero mean and standard deviation σ, i.e.:

C = C ′ + σ - → V (4)
where

- → V = {⃗ v k } K k=1 is such that v k ∼ N(0, 1). The flow field - →
D is computed by interpolating the displacement vectors between the target control points C, i.e.

-→

D = T (P, - - → V) ( 5
)
where T is an interpolation function. If topological relations between control points are available, e.g. in the form of a regular grid, common choices are bilinear, or bicubic interpolation as suggested by the authors of the U-Net model [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. For a higher degree of smoothness and unstructured control points respectively, we consider spline and Moving Least Squares (MLS) interpolation, that we detail below. In this approach, the source control points C ′ and the scaling σ are the usercontrolled parameters. We detail the available options to generate control points later on.

Spline interpolation. Polyharmonic splines can be applied to structured or unstructured control points. The interpolating function has the form:

f (p) = K k=1 w k ϕ(∥p -c k ∥) + a ⊤ p 1 ( 6
)
where the first term is a weighted sum of polyharmonic Radial Basis Function (RBF) terms, with centers C, and the second term is a polynomial linear term. In this paper we consider the two following choices for ϕ:

• ϕ(r) = r 2 ln(r), which corresponds to the well-known Thin Plate Spline (TPS) [START_REF] Bookstein | Principal warps: thin-plate splines and the decomposition of deformations[END_REF];

• ϕ(r) = r 3 , which is the cubic spline. The weights {w k } K k=1 and the coefficients of the polynomial a are estimated such that the interpolating function exactly fits the value of the function at the control points C, which is achieved by solving a linear system.

Moving Least Squares deformations. Moving Least Squares (MLS) deformations have been introduced by Schaefer et al. in [START_REF] Schaefer | Image deformation using moving least squares[END_REF]. Given the set of source control points C ′ and target points C, MLS deformation finds a transformation function D that satisfies three conditions: where M is an affine transformation matrix, and weights w k are defined by w k = 1 ∥c k -p∥ 4 . The points c and c′ are the weighted centroids of the target and source control points, respectively. We focus here on the as-rigid-as-possible version of MLS deformations (MLS-ARAP), where the constraint M ⊤ M = I is introduced (i.e. the matrix M contains no scaling). Further mathematical details are provided in Appendix A.

K k=1 w k ∥(c k -c)M -(c ′ k -c′ )∥ 2 , (7)
The goal of MLS-ARAP is to produce a realistic result by preserving as much as possible the shapes and textures in the input image when moving the control points. For this purpose, control points should be placed on rigid parts, whereas other areas are considered as flexible.

Control points on a regular grid. Control points on a regular grid are easily generated by computing a tessellation of the input image. This method can thus be used to deform any type of image content in a random fashion, and we call this kind of method Grid-Based Deformation (GBD). The spacing between grid points determines how much the displacement vectors at neighbor pixel locations are correlated, and hence fixes the maximum amount of distortion of the flow field. The value of the scaling factor σ determines the amount of distortion.

The grid points spacing is generally set by choosing the grid size for a given image size, which we will denote by n 2 = K. The corresponding distance in pixels between neighbor control points is thus d = N/(n -1). For the same value of d, a large scaling factor σ will yield a higher amount of distortion than a small scaling factor (see Supplemental material for corresponding plots of foldover rate, preservation of distances and preservation of angles).

The maximum displacement in pixels along one axis for a given value of σ obey the three-sigma rule, i.e. about 68% of the displacements will be less than σ pixels, while 28% will be comprised in [σ, 2σ], and 4% will fall in [2σ, 3σ], which offers a more intuitive control than Random Displacement Fields.

Fig. 6 shows examples of deformations based on control points on a regular grid using a cubic spline (GBD order-3, or GBD-3) interpolation, including the parameter values proposed in the U-Net paper [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], scaled to our patch size, which gives n = 3 and σ = 5. The parameter values proposed in the U-Net paper result in very moderate distortions, which leads us to consider stronger deformations. The other examples use n = 5 with σ ranging from 5 to 20, yielding increasing distortions.

Fig. 3 and4 of the Supplemental material show the behavior of random grid-based deformations with TPS (order-2) and cubic (order-3) interpolation, respectively, with varying values of n and σ, for an image size N = 256. Almost no difference is observed between TPS and cubic interpolation. It can be seen that foldover starts to occur with increasing probability among random deformation fields when σ ≈ 10 for n = 3 (d ≈ 85), and σ > 5 for n = 5 (d ≈ 51) and n = 10 (d ≈ 26). A simple way to drastically limit foldover would be to consider d/2 as an upper bound for σ. However, this approach excludes some foldover-free deformations that may contribute to the data augmentation. For example, approximately 30% of the deformations with n = 3 and σ = 50 are foldover-free (Fig. 3 and4 of the Supplemental material). In our study on the performance of data augmentation for the glomeruli segmentation task, we examine the impact of foldover-free deformations vs. deformations containing foldover.

Unstructured control points. Applying deformations based on unstructured control points requires to place source control points C ′ at appropriate locations in the input image. Instead of positioning them at random positions, we propose to perform a content-aware choice, and utilize detected cell nuclei centers. We describe the method in the following subsection.

Cell nuclei-based deformations

In this section, we describe our novel deformation method based on detected cell nuclei centers and MLS-ARAP deformations. The motivation is that the appearance of cell nuclei centers is almost invariant across WSI patches. That is, they are assumed to be rigid parts in the content of the patches, and our goal is to preserve their shape through a MLS-ARAP interpolation. We proceed in three steps, given a WSI patch to be deformed: 1. We detect cell nuclei centers in the patch, that become the source control points C ′ (this step has to be done only once for a given patch). 2. We compute displacement vectors at source control points to obtain the target control points C.

We interpolate the displacement vectors

{ --→ c ′ i c i } K i=1
between source and target control points to get output pixel locations P. We use the method described in [START_REF] Mahmood | Deep adversarial training for multi-organ nuclei segmentation in histopathology images[END_REF] for cell nuclei detection in WSI patches, although any other method could be used. Since cell nuclei are irregularly spaced, moving control points independently is prone to foldover. Instead, we obtain the displacement vectors of control points by sampling a multivariate Normal distribution N(0, Σ), where Σ is a covariance matrix of size K × K that takes account of the distances between detected cell nuclei centers. More precisely, we define the entries of Σ as follows:

Σ i, j = σ exp       - ∥c ′ i c ′ j ∥ √ 2N       (8)
for i, j ∈ [1, K], where σ represents a scaling factor that controls the maximum amplitude of the deformations. Since cell nuclei are spread over the whole image in each patch, we use the size of the image diagonal √ 2N to normalize the distances. The usercontrolled parameters therefore only consist in the scaling coefficient σ. We call this method CNB-MVN-MLS for short (Cell Nuclei Based, Multivariate Normal, MLS-ARAP). Having a single parameter is advantageous over the other considered methods for the purpose of parameter estimation. A result obtained with this method is shown in Fig. 7, where K = 310, and σ = 12. In our patches, the number of detected nuclei cell centers in a single WSI patch varies between 250 and 320. Additional results are presented in Fig. 8. It can be seen that the obtained deformations are richer than deformations produced with RDF or grid-based deformations: they incorporate low and high frequencies, which is a consequence of both the spatial correlation of control points and the rigidity constraint. Distortions have a behavior that is similar to the other examined deformation models (see Supplemental material Fig. 5 for foldover and distance/angle preservation plots).

Evaluation protocol and implementation

In this section, we present our training protocol, followed by the details of our implementation.

Evaluation protocol

Glomeruli segmentation is formulated as a two classes problem where each pixel of a WSI receives a label glomeruli, or a label tissue. Below we detail the dataset used to train the U-Net segmentation model, as well as our baseline and approach to assess the impact of random spatial deformations as an augmentation technique.

Dataset. The input to the U-Net are patches sampled from WSIs, such that each patch may either contain a complete glomerulus, or up to three partial glomeruli, or no glomeruli (i.e., tissue). We have at our disposal several WSIs from different patients, acquired with the same sampling resolution (0.506 µm/pixel), and distributed between training, validation and test, from which we extract 512 × 512 patches with and without glomeruli. The composition of this dataset is summarized in Table 3. We select randomly 40 patches from the training base, 20 patches from the validation base and 100 patches from the test base, half with glomeruli and the other half without glomeruli (Table 3). We purposely choose a small number of examples for this experiment in order to make it easier to highlight the differences in the results. From each patch, we randomly extract 5 crops of size 256 × 256. Those new patches are the ones presented to the U-Net. The test set remains the same for all subsequent experiments.

Baseline. Our baseline is composed of all the 90°rotations and flips we can apply to an image. Since the sampling resolution is the same for all patches, we did not include global scaling in our baseline. The observed objects are at the same scale in all images, even if their sizes can differ. These local variations in size are simulated in a more faithful way by spatial deformations.

Application of deformations. To assess the impact of using random spatial deformations as augmentation techniques on segmentation results using a U-Net, we propose a protocol that aims to provide a fair comparison of the different deformation models.

To do so, we have to make sure that the parameter values used for each model are, as far as possible, close to optimal parameter values. We use a meta-learning scheme akin to RandAugment [START_REF] Cubuk | Randaugment: Practical automated data augmentation with a reduced search space[END_REF]. For each model we consider the (one or two) parameter(s) described in section 3, for which we hand-select a few values. Those values are chosen to cover different cases, with varying distortion amounts. However, due to the computational cost of meta-learning (a single training takes approximately two hours on an NVidia P100), we do not search for the exact set of optimal parameter values, but rather aim to find a good range of parameter values, and highlight the influence of the characteristics of the deformations. We performed 5 training for each set of parameter values in a grid-search manner, and averaged the results. The selected parameter values for the chosen deformation models are summarized below.

• RDF (σ, α) -σ = {5, 10, 20} -α = {100, 200, 400, 800} • GBD order-2 (n, σ) -n = {3, 5, 10} -σ = {5, 10, 20, 50} • GBD order-3 (n, σ) -n = {3, 5, 10} -σ = {5, 10, 20, 50} • CNB-MVN-MLS (σ) -σ = {5, 10, 15, 30, 100}
The augmentations are computed offline before training, which means that the training time is the same for all methods. For a given deformation method and a given set of parameter values, we apply 10 deformations to each training image. Each of those 10 deformations is randomly sampled, meaning that we sample 2000 flow fields for each set of parameter values and each deformation model. To the training and validation sets, we then apply our baseline, thus multiplying the training dataset size by 7 (14,000 images in total for the training set).

Since we have a low number of repetitions (5 trainings per experiment), we use the Wilcoxon-Mann-Whitney statistical test [START_REF] Wilcoxon | Individual comparisons by ranking methods[END_REF][START_REF] Mann | On a test of whether one of two random variables is stochastically larger than the other[END_REF] to give an indication on whether or not we can consider two experiments as equivalent w.r.t their average Dice scores (see Appendix B for p-values).

Implementation

We implemented the U-Net in Python using TensorFlow 2, as described in the original paper [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] with only one convolution layer per stage, using transposed convolutions for upsampling, and cross entropy loss, which is a standard choice for stable training (see e.g. [START_REF] Taghanaki | Deep semantic segmentation of natural and medical images: a review[END_REF] for a review of loss functions). The contracting path is composed of 6 convolutional layers and the expanding path of 5 convolutional layers, with 64 filters in the first layer. The number of filters doubles at each stage, with a maximum of 512. We use the Adam optimizer [START_REF] Kingma | A method for stochastic optimization[END_REF] with a 4.10 -6 learning rate.

The deformation models described in Section 3 were also implemented in Python. For Random Displacement Fields and Grid-Based Deformations, we use the Tensor-Flow Addons Image module, that runs on the GPU. We implemented the deformation models as CPU code for CNB-MVN-MLS deformations, although they could be implemented on the GPU either. The time required to sample a flow field and deform a 256×256 image for CNB-MVN-MLS deformations with our CPU codes is about 1.5 s, whereas RDF and GDB take 60 ms per image. Similar performance (60 ms per image) would be expected for for CNB-MVN-MLS deformations running on the GPU.

We make our code publicly available through a GitHub repository2 . We provide the complete framework for evaluating the impact of the deformation models that we considered as augmentations for any 2D image segmentation task based on a U-Net.

Experiments and results

We first analyze the segmentation performance w.r.t. augmentation with spatial deformations as described in the protocol presented in the previous section. Then we study the influence of geometric distortions on the obtained performance results.

Segmentation results

We ran each training for 30 epochs. When using a larger number of epochs, we have verified experimentally that the validation loss was increasing for each training. We kept the trained segmentation model with the lowest validation loss. At test time, we evaluated the model on four metrics: Dice score (or F1-score), precision, recall and specificity. All are averaged over 5 repetitions. We recall the formulas of those metrics: For simplicity, we will focus only on the average Dice score, but the results for all metrics can be found in the Supplementary material. We present the average Dice score for all deformation models in Table 4. We highlight in red the best score, and in blue the scores that are equivalent to the best one w.r.t. the Wilcoxon test. Note that there can be a difference of up to 0.06 in average Dice score for two equivalent experiments in the sense of the Wilcoxon test, meaning we cannot consider the sets of parameter values corresponding to these experiments as interchangeable. However, the Wilcoxon test gives us an interval where we can safely assume the best parameter values are w.r.t. the chosen value ranges, thus potentially helping us refine the grid search. In particular, in order to provide the best bounds possible, we tried to have non-equivalent parameter values (i.e. values not highlighted) at the extremities of each line with the best average Dice score. This is the reason why we conducted some additional experiments w.r.t. the ranges of values defined in the protocol. We also give the average Dice score when only the baseline (i.e. no deformation) is used for data augmentation: 0.6231 ± 0.0401. It should be noticed that there may be multiple local maxima in the landscape of the average Dice scores. The chosen ranges of parameter values determine some maxima that are not guaranteed to be absolute.

When focusing on a single deformation model at a time, the impact of the parameter values clearly appears. For example, in the case of the RDF method, some parameter values are detrimental, such as σ = 20 and α = 50, while others can boost the average Dice score from 0.62 to more than 0.75. All the selected parameter values for the GBD-3 method improve the results, the values corresponding to the highest average Dice score being n = 3 and σ = 20.

We summarize below the best average Dice scores for each deformation model and corresponding parameter values, obtained from the sample values described in Section 4. The best average Dice scores are obtained with the GBD with order-3 interpolation method, but we note that aside from the RDF method, all results are close. Considering the variance and the small number of repetitions, we can state that all methods (aside RDF) are equivalent in performance. This is corroborated by the Wilcoxon test, that we also applied to the best average dice scores obtained with the different deformation models. Note that this does not mean that the RDF method is necessary inferior: we could not find parameter values that provide the same performance as the other methods, which indicates that they either do not exist or are more difficult to find. This corroborates the observation made in Section 3 that the RDF model is more difficult to control than the other deformation models. We also note that the change of the interpolation method from order-2 to order-3 for GBD does not provide a noticeable change in the average Dice score (0.005 gap).

The CNB-MVN-MLS model perfoms on par with the other models, but with a single parameter to control, which significantly reduces the computational complexity of finding a best parameter value.

This leads us to the conclusion that changing methods is not the most decisive factor and that an optimization of the parameter values is crucial to obtain the best segmentation results. Since the maximum reported average Dice scores are close from one deformation model to the other, we can also state that the obtained values are close to a global maximum with good confidence. The observed similar performance of the different deformation models is consistent with the fact that they all rely on the sampling of a filtered white noise, as detailed in 3. A significant advantage of the CNB-MVN-MLS model lies in its control through a single parameter.

For more completeness, we made a complementary experiment mixing deformations with several parameter values. We first examined picking the parameter values of the GBD order-3 method at random. We fixed n = 3 and uniformly sampled σ among the values 10, 20 and 50, for each of the 2000 transformations we apply in the protocol. We obtain an average Dice score of 0.8116 ± 0.0287, the same as for (n = 3, σ = 10). Thus mixing the parameter values does not seem to provide additional diversity and seems to hold the performance to the lowest score among the scores corresponding to single parameter values.

Impact of distortions

When looking at the deformed images, we observe that deformations with moderate distorsions, which are often used to keep the examples "realistic", perform lesser than deformations involving stronger distortions that may look unrealistic. As presented in Section 3, deformations with a high amount of distortions implies a lesser preservation of distances and angles in the result images. Varying the angles have important consequences on the shapes, which leads to better generalizability (or robustness) of the segmentation model. As one could expect, deformations provoking foldover result in a degradation of the segmentation performance as the foldover rate increases. However, deformations containing moderate foldover can still yield better performance than the baseline. As an example, GBD-3 deformations with n = 3 and σ = 5 induce very limited distortion, and lead to +0.15 in average Dice score in comparison with the baseline. With the same model, deformations with n = 3 and σ = 50 yield a higher amount of distortion, with 1/3 of deformations containing foldover (see Fig. 3 of the Supplemental material), and lead to a gain of +0.23. Deformation with n = 10 and σ = 50 provoke a high foldover rate (more than 40%), yielding a +0.08 amelioration in average Dice score.

Our interpretation is that foldover creates new patterns that are not found in the original images, thus introducing variety that may be beneficial for training the U-Net model. However, if those new patterns are dominant, the original appearance of the image (especially the topological structure) may be completely lost, yielding to poor contribution to training, or even degrading the segmentation performance.

Limitations

Since they are based on the RandAugment strategy [START_REF] Cubuk | Randaugment: Practical automated data augmentation with a reduced search space[END_REF], our proposed protocol are computationally expensive. We had to perform 420 trainings, which amounts to approximately 840 GPU hours on an NVidia Tesla P100 SXM2 with 16 GB RAM. While our parameter sampling choices and statistical tests yield only bounds on the best parameter values, estimating tighter bounds or exact best parameter values for each method would require additional computational resources with this approach. In particular, more repetitions for each set of parameter values could provide a better separation between close parameter values, or conversely show that they are equivalent. We provide our complete evaluation framework to the end of both enabling parameter values refinement, depending on the available computing power, and evaluating data augmentation based on spatial deformation for other datasets or applications.

It is worth mentioning that our conclusions cannot be generalized to other dataset sizes or to other dataset types, i.e. other images. As we know from [START_REF] Cubuk | Randaugment: Practical automated data augmentation with a reduced search space[END_REF], the size of the training dataset has an impact on the optimization process of a deep model hyperparameters. In order to exhibit a difference between the performance gain enabled by the deformation models and their parameter values, we used only 20 images with glomeruli. It helped us show the impact of deformations as an augmentation technique with a potential gain of more than 0.20 in average Dice score, but at the cost of a relatively high variance across trainings. Thus the low number of images is another factor, in conjunction with the small number of repetitions, that contributes to the large ranges obtained for the estimation of best parameter values.

Regarding other dataset types, the impact of the deformation models may be different, since changing the images changes the shapes and textures. We did not conducted experiments with images other than renal WSI patches, but the framework we provide allows for investigations on any image type and dataset size.

Conclusion and future work

In this paper, we evaluated the impact of random spatial deformations used as augmentation in the case of renal histopathology patches segmentation using a U-Net

Figure 1 :

 1 Figure 1: Examples of glomeruli in patches of WSIs, with variable shapes (Periodic acid-Schiff staining).

Figure 2 :

 2 Figure 2: Two examples of deformations applied to a WSI patch containing a single glomerulus, with plots of the corresponding values of the determinant of the Jacobian matrix J. The first row shows a deformation with moderate distortions, whereas the second row shows a deformation with foldover, which is reflected by negative values of J. (a): Deformed image; (b): Deformation applied to a checkerboard; (c): HSV colorcoded flow field; (d): Plot of J.

  amount of local distortion induced by a flow field can be characterized by the Jacobian matrix of the deformation function D at every point p i ∈ P, denoted by ∇D(p i ). The determinant of the Jacobian matrix, i.e. J(p i ) = det(∇D(p i )), encodes the local scale factor by which the distances are shrunk (|J(p)| ∈ [0, 1)) or expanded (|J(p)| > 1), and its sign reveals whether the deformation produces foldover (J(p) ≤ 0) or not (J(p) > 0). The distances are preserved when |J(p)| = 1. The less the displacement vectors at neighbor pixel locations are correlated, the more foldover is thus likely to occur locally. Fig 2 shows plots of J for two examples of deformations obtained with the Random Displacement Fields model. The case of the second line exhibits foldover: J is negative at some locations of the flow field, representing about 20% of the pixel locations.

Figure 3 :

 3 Figure 3: Characterization of the flow fields generated with Random Displacement Fields, based on 1k randomly generated flow fields, for σ = 10 (first column) and σ = 40 (second column) and increasing values of α. (a): Foldover rate (the central curve represents the median of the distribution, while surrounding curves correspond to percentiles); (b): Preservation of distances (symmetric log 2 scale is used for bin edges); (c): Preservation of angles. Plots for random flow fields generated with the other models considered in this paper are provided in the Supplemental material.

σ = 10 ,Figure 4 :Figure 5 :

 1045 Figure 4: Examples of RDF deformations applied to a WSI patch containing a single glomerulus, for varying values of σ and α. Note that the directions of displacement vectors are identical in the four examples. (a): Deformed patch; (b): Deformation applied to a checkerboard; (c): HSV color-coded flow field.

( 1 )

 1 Interpolation: the control points C map to C ′ , i.e. D(C) = C ′ ; (2) Smoothness: D should produce smooth deformations; (3) Identity: if C ′ = C, then D should be the identity function, i.e. D(C) = C. For any input pixel location p to be transformed, D(p) is constructed using Moving Least Squares to minimize

n = 3 , σ = 5 n = 5 , σ = 5 n = 5 , σ = 10 n = 5 ,Figure 6 :

 3555556 Figure 6: Examples of grid-based deformations with order-3 interpolation applied to a WSI patch containing a single glomerulus, for varying values of n and σ. Note that the directions of displacement vectors are identical in the three examples where n = 5. (a): Deformed patch; (b): Deformation applied to a checkerboard; (c): HSV color-coded flow field.

Figure 7 :

 7 Figure 7: Example of CNB-MVN-MLS deformation applied to a WSI patch containing a single glomerulus, for K = 310 and σ = 12; (a): Input WSI patch; (b): Detected cell nuclei; (c): Cell nuclei displacements; (d): Deformed image; (e): Deformation applied to a checkerboard; (f): HSV color-coded flow field.

Figure 8 :

 8 Figure 8: Additional examples of CNB-MVN-MLS deformations applied to a WSI patch containing a single glomerulus, for K = 310 and two values of σ; (a): Cell nuclei displacements; (b): Deformed image; (c): Deformation applied to a checkerboard; (d): HSV color-coded flow field.

•

  Dice score = 2 × Precision×Recall Precision+Recall • Precision = |True positives| |True positives|+|False positives| • Recall = |True positives| |True positives|+|False negatives| • Specificity = |True negatives| |True negatives|+|False positives|

1 :

 1 • Baseline: 0.6231 ± 0.0401 • RDF (σ = 10, α = 400): 0.7892 ± 0.0370 • GBD order-2 (n = 5, σ = 20): 0.8541 ± 0.0125 • GBD order-3 (n = 3, σ = 20): 0.8593 ± 0.0162 • CNB-MVN-MLS (σ = 15): 0.8525 ± 0.0182

  Bel et al, 2018 Merveille et al, 2021 Davis et al, 2021 Salvi et al, 2021 Altini et al, 2020 Rotations

Table 3 :

 3 Number of patches used in each dataset. The number in parentheses is the total number of patches available in the original dataset.

Table 4 :

 4 U-Net glomeruli segmentation performance considering the average Dice score for the parameter values of the deformation models considered in the paper.

	σ	5	α	100 0.7586 ± 0.0233	200 0.7275 ± 0.0387	400 0.5342 ± 0.0399	800 0.4807 ± 0.0448	n	σ	5			10	20	50
		10		0.7250 ± 0.0523	0.7748 ± 0.0184	0.7892 ± 0.0370	0.7787 ± 0.0175	3		0.7884 ± 0.0224	0.8176 ± 0.0383	0.8434 ± 0.0181	0.8502 ± 0.0158
		20		0.6653 ± 0.0519	0.7367 ± 0.0481	0.7729 ± 0.0292	0.7859 ± 0.0256	5		0.8014 ± 0.0116	0.8320 ± 0.0243	0.8541 ± 0.0125	0.7888 ± 0.0351
	σ		α	2400	5000			10	0.8028 ± 0.0202	0.8402 ± 0.0317	0.7648 ± 0.0321	0.7171 ± 0.0473
		40		0.7554 ± 0.0292	0.6899 ± 0.0223
								(b) Average Dice score w.r.t. to the GBD-2 parameter val-
	(a) Average Dice score w.r.t. to the RDF parameter values	ues
	n	3	σ	5 0.7688 ± 0.0135	10 0.8177 ± 0.0184	20 0.8593 ± 0.0162	50 0.8510 ± 0.0139	σ	5 0.8212 ± 0.0116	10 0.8501 ± 0.0121	15 0.8525 ± 0.0182	30 0.8337 ± 0.0161	100 0.70417 ± 0.0298
		5		0.7931 ± 0.0207	0.8407 ± 0.0344	0.8114 ± 0.0319	0.7381 ± 0.0542
		10		0.8113 ± 0.0311	0.8448 ± 0.0165	0.7506 ± 0.0328	0.7011 ± 0.0197
								(d) Average Dice score w.r.t. to the CNB-MVN-MLS pa-
	(c) Average Dice score w.r.t. to the GBD-3 parameter values	rameter values

Corresponding author.

https://github.com/ASTex-ICube/spatial_deformations_aug

model. We focused our study on spatial deformations based on a multivariate Gaussian random field. We performed a comparative study of state-of-the-art deformation models based on this approach used in the field of digital histopathology, considering geometric aspects. We proposed an original content-aware random deformation model named CNB-MVN-MLS based on the detection and displacement of cell nuclei centers that involves a single parameter.

We compared the performance offered by these deformation models when used for augmentation in the considered segmentation task. We used a meta learning scheme to optimize their parameter values and find appropriate bounds. We found out that the different models show similar performance (measured in average Dice score) and that using the good range of parameter values can boost the average Dice score by up to 0.23 on our application. We showed that deformations with relatively strong distortions yield the best performance increase for segmentation, while previous work only report the use of deformations with low distortions. Our CNB-MVN-MLS model performs on par with the other considered deformation models but with only one parameter to control, improving the performance of best parameter estimation.

In future work, we would like to develop an online protocol, and make a comparison with the proposed offline protocol. By using more images and/or trainings, we could also lower the variance across training and thus provide better bounds for the optimal parameters. We also plan to extend the evaluation to alternative pre-trained network architectures for segmentation like Seg-Net [START_REF] Bueno | Glomerulosclerosis identification in whole slide images using semantic segmentation[END_REF] or U-Net with a ResNet34 backbone [START_REF] Salvi | Automated assessment of glomerulosclerosis and tubular atrophy using deep learning[END_REF].

Following the direction of work of our content-aware CNB-MVN-MLS random deformation model, we would like to further develop models with an improved correlation between the displacement vectors and the content of images. Methods based on physical simulation [START_REF] Nealen | Physically based deformable models in computer graphics[END_REF] take account of material and mechanical properties, but at a greater computational cost for the generation of deformation fields, which would require to identify a trade-off.

Acknowledgments

The WSI patches used in our experiments are courtesy of the ERACoSysMed project "SysMIFTA", co-funded by EU H2020 and the national funding agencies German Ministry of Education and Research (BMBF) project management PTJ (FKZ: 031L-0085A), and Agence National de la Recherche (ANR), project number ANR-15-CMED-0004.

The authors would like to acknowledge the High Performance Computing Center ROMEO hosted by the University of Reims Champagne-Ardenne for access to computing resources.

Declaration of competing interest

The authors declare that they have no conflict of interest.

Appendix A. As-rigid-as-possible MLS deformations

Referring to [START_REF] Schaefer | Image deformation using moving least squares[END_REF]

where

where ⊥ is an operator on 2D vectors such that [x, y] ⊥ = [-y, x].

Appendix B. Wilcoxon-Mann-Whitney statistical test

Considering the Dice scores of two experiments as independent observations from two distributions X and Y, we test the null hypothesis H0, i.e. the distributions of both populations are equal. If we note X 1 , ..., X n and Y 1 , ..., Y m the samples drawn from each distribution, we can define the U-statistic as

With n = m = 5, we can reject the null hypothesis when U ≤ 2 or U ≥ 23 according to tables in [START_REF] Wilcoxon | Individual comparisons by ranking methods[END_REF]. We give below the associated p-values for every comparison we made.

Appendix B.1. Comparison between deformation models

We present in Table 5 ((a) and (b)) of the Supplemental material the comparison between the considered deformations models, under two different hypothesis and using the best set of parameters for each model. Table 5(a) gives the p-values when testing H0: the baseline Dice score is higher (one-sided test), (b) gives the p-values when testing H0: the two considered distributions are equal.

Appendix B.2. Comparison between parameter values for each deformation model

Table 5 ((b) to (f)) of the Supplemental material presents the comparison between the parameter values for the considered deformation models individually, under the hypothesis that two distributions are equal (two-sided test). A p-value greater then 0.05 means that the two sets of parameter values yield significantly close results.


Florian Allender Icube

Rémi Allègre
email: remi.allegre@unistra.fr

Cédric Wemmert
email: wemmert@unistra.fr

Jean-Michel Dischler
email: dischler@unistra.fr

Conditional image synthesis for improved segmentation of glomeruli in renal histopathological images 1 st

Keywords: Digital Histopathology, Glomeruli segmentation, Data augmentation, Conditional image synthesis

In a context of limited data availability, we consider the supervised segmentation of glomerular structures in patches of renal histopathological whole slide images. These structures are complex, include multiple substructures, and exhibit great variability in their shape, making their robust segmentation challenging. In this context, using appropriate data augmentation techniques is crucial to obtain more robust results. We investigate data augmentation based on random spatial deformations and conditional image synthesis for the training of a U-Net model. We rely on a SPADE model to perform the synthesis, using label maps built from the real patches available for training as input. Synthesis from ground truth masks only results in noisy patches, where substructures are absent, whereas additional structure information yield more realistic patches. We show that the best improvements of the segmentation performances are obtained by mixing real patches with synthetic patches generated from ground truth masks only, which yields an increase of up to 0.76 of average dice score w.r.t. augmentation based on spatial deformations only. We conclude that, using conditional image synthesis, patches synthesized with no additional structure information better contribute to the robustness of glomeruli segmentation than patches synthesized with structure information extracted from available real patches.

I. INTRODUCTION

In the field of digital histopathology, Deep Learning aims at bringing support to pathologists for faster and more accurate diagnoses [START_REF] Litjens | Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis[END_REF], [START_REF] Wemmert | Deep Learning for Biomedical Data Analysis: Techniques, Approaches, and Applications, chapter Deep Learning for Histopathological Image Analysis[END_REF]. Deep models based on Convolutional Neural Networks (CNN), like e.g. U-Net for segmentation [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF], have the ability to learn patterns in images, which necessitates large amounts of annotated data to prevent overfitting. This requirement is a challenge, because manually segmenting Whole Slide Images (WSIs) requires expert knowledge and is extremely time-consuming and costly.

To circumvent the limitations related to small or imbalanced datasets, a standard approach in Deep Learning is to perform Data augmentation [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF]. It consists in expanding the variety of training data in an artificial manner, which may involve image manipulations (spatial and colorimetric transformations, noise injection, image mixing) or synthetic image generation. The goal of synthetic image generation is to increase the diversity of training data by producing novel realistic-looking images. State-of-the-art approaches rely on Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF], highly popular for their ability to learn data distributions. While the image manipulation approach raises the questions of the appropriate set of transformations and parameter tuning (see e.g. [START_REF] Faryna | Tailoring automated data augmentation to h&e-stained histopathology[END_REF]), synthetic image generation raises the issue of the whole synthesis pipeline, and is itself concerned by the size and balance of the available data [START_REF] Shamsolmoali | Image synthesis with adversarial networks: A comprehensive survey and case studies[END_REF]- [START_REF] Tseng | Regularizing generative adversarial networks under limited data[END_REF]. GAN-based image synthesis requires a large amount of data for training [START_REF] Shamsolmoali | Image synthesis with adversarial networks: A comprehensive survey and case studies[END_REF]. To cope with limited data, augmentations based on image manipulations [START_REF] Karras | Training generative adversarial networks with limited data[END_REF] or regularization techniques [START_REF] Tseng | Regularizing generative adversarial networks under limited data[END_REF] have been proposed.

In the field of digital histopathology, GANs have been successfully used for stain transfer or stain-to-stain translation [START_REF] Vasiljević | Towards histopathological stain invariance by unsupervised domain augmentation using generative adversarial networks[END_REF], and also for the generation of tissue patches [START_REF] Tschuchnig | Generative adversarial networks in digital pathology: A survey on trends and future potential[END_REF]. Previous works have addressed the synthesis of tissue patches to improve the training of classifiers in the context of cancer diagnosis [START_REF] Wei | Generative Image Translation for Data Augmentation in Colorectal Histopathology Images[END_REF]- [START_REF] Quiros | Pathologygan: Learning deep representations of cancer tissue[END_REF], or improve cell nuclei segmentation [START_REF] Mahmood | Deep adversarial training for multi-organ nuclei segmentation in histopathology images[END_REF], with demonstrated benefits.

Renal histopathology is crucial in the study of kidney diseases, and especially transplant rejection, that occurs with an incidence of 7.9% in the first year [START_REF] Naik | Renal transplantation rejection[END_REF]. In a context of limited data, we consider the supervised segmentation of glomerular structures in patches of renal WSIs. These complex ballshaped structures are responsible for blood filtration and con-tain multiple substructures: membranes, capillaries, mesangial and endothelial cells, podocytes. The appearance of glomeruli thus exhibits great variability (Fig. 1), making their robust segmentation challenging. The state-of-the-art approaches rely on deep segmentation models such as U-Net [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF], [START_REF] Bel | Automatic segmentation of histopathological slides of renal tissue using deep learning[END_REF]- [START_REF] Davis | Deep learning segmentation of glomeruli on kidney donor frozen sections[END_REF] or SegNet [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF], [START_REF] Altini | Semantic segmentation framework for glomeruli detection and classification in kidney histological sections[END_REF], possibly with a pre-trained backbone [START_REF] Bueno | Glomerulosclerosis identification in whole slide images using semantic segmentation[END_REF], [START_REF] Salvi | Automated assessment of glomerulosclerosis and tubular atrophy using deep learning[END_REF].

For improved generalizability, standard augmentation techniques include geometric transformations as well as transformations related to staining variations [START_REF] Vasiljević | Towards histopathological stain invariance by unsupervised domain augmentation using generative adversarial networks[END_REF], [START_REF] Merveille | An automatic framework for fusing information from differently stained consecutive digital whole slide images: A case study in renal histology[END_REF], [START_REF] Li | Deep learning segmentation of glomeruli on kidney donor frozen sections[END_REF]. Murali et al. [START_REF] Murali | Generative modeling for renal microanatomy[END_REF] explored the usage of Deep Convolutional GANs [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF] to generate glomeruli images, accompanied by a visual evaluation by experts. However, no evaluation in the context of glomeruli segmentation has been performed. The complexity of the glomerular structures makes their synthesis a difficult task and classic methods either fail to reproduce fine details or suffer from mode collapse.

In this paper, we study the mixing of random spatial deformations and GAN-based conditional image synthesis as a data augmentation technique in the context of glomeruli segmentation in patches of renal WSI using a standard U-Net. We use random spatial deformations to inject geometric diversity in the training dataset, and conditional image synthesis to introduce texture variations. We describe our method and validation protocols in Section II and discuss the results in Section III.

II. BASELINE AND PROPOSED METHOD

The segmentation task consists in separating glomerular structures from surrounding tissue at pixel level in input patches. Our baseline for training the U-Net model consists of real glomeruli patches with ground truth binary segmentation masks (see Fig. 1 for examples), such that each patch contains either exactly one complete glomerulus, and possibly some parts of other glomeruli. The training, validation and test datasets are respectively composed of 660, 400 and 431 patches of size 256×256 The validation and test datasets will remain the same for all experiments. The training dataset is augmented live during training with rotations and flips with a probability of 0.5. We propose to augment the training dataset offline with random spatial deformations and synthetic patches, and evaluate the impact on segmentation.

A. Spatial deformations

For random spatial deformations, we use grid-based random deformations as suggested in the original U-Net paper [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF], i.e. a 3×3 regular grid of control points, random displacements following a Gaussian distribution at control points, and cubic spline interpolation of displacements between control points. For our patches, we set the standard deviation of the displacements to 20, which amounts to a local magnitude of up to 60 pixels. We use symmetric padding along the edges of the input images for deformed locations that fall outside their bounds.

B. Patch synthesis

Architecture: We consider an image-to-image translation approach -label map to image in our case, which is a Fig. 2. Architecture of SPADE [START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF]. Top-left: The Spatially-Adaptive Normalization layer; Top-right: The SPADE residual block; Bottom row: The SPADE generator (images taken from [START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF]). type of conditional synthesis, and employ the state-of-the-art GAN-based SPADE model [START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF]. The SPADE design draws its name from the "Spatially-Adaptive Normalization". While the commonly used normalization layers tend to wash away useful information from the inputs label maps [START_REF] Wang | High-resolution image synthesis and semantic manipulation with conditional gans[END_REF], SPADE redesigns the generator in such a way that the normalization of the input is learned through convolutional layers (Fig. 2 top-left). The label maps are injected into SPADE "residual blocks" (Fig. 2 top-right) inserted between upsampling layers, which allows to skip the contracting path used in previous image-to-image translation architectures. The generator can then take a random vector as input, which enables multi-modal synthesis and reduces mode collapse (Fig. 2

bottom row).

Training and synthesis: We train the SPADE network with adaptive discriminator augmentation, as introduced in StyleGAN2-ADA [START_REF] Karras | Training generative adversarial networks with limited data[END_REF]. The probability to use augmentations evolves w.r.t. to the performance of the discriminator, preventing it to overfit and thus allowing for better results in the case of small datasets. Our set of augmentations includes rotations, flips and spatial deformations applied in the way described in Section II-A. These transformations have the most significant impact according to the authors [START_REF] Karras | Training generative adversarial networks with limited data[END_REF]. The model is trained on 660 glomeruli patches, different from the training set of the segmentation model.

We first train the model using ground truth binary segmentation masks as label maps. To generate new glomeruli patches, we apply random spatial deformations (using the technique described in Section II-A) to the segmentation training set masks and use the results as inputs to SPADE. The generated images lack visual fidelity, with absent substructures, as highlighted by Fig. 3. Lumina of vessels are lost, as well as relevant placement of cell nuclei. This led us to consider the construction of enriched label maps.

Label maps enrichment: We propose to enrich glomeruli ground truth binary masks with additional structure information extracted from glomeruli images to improve the guidance of the SPADE generator. No tissue feature nor internal components of glomeruli reflecting substructures can reliably be segmented, since no corresponding ground truth is available. We therefore chose to separate the non-textured areas (i.e. lumina of vessels, urinary space, slide outer space) from textured areas (glomeruli and surrounding tissue) in order to get structure information reliably. To do so, we assume that non-textured areas are not affected by stainings, nor important artifacts, and correspond to the color of the microscope light, which is close to white [START_REF] Korzynska | The influence of the microscope lamp filament colour temperature on the process of digital images of histological slides acquisition standardization[END_REF]. We convert glomeruli patches in HSV color space and then perform thresholding on the S component to obtain binary masks for textured areas. Fig. 4 shows examples of binary texture masks for some real glomeruli patches. For our datasets, we set the threshold to the absolute value of 0.1. Next, we combine these texture masks with corresponding glomeruli masks to produce 4-class label maps that we call "structure label maps". Training and synthesis: We performed a new training of the SPADE model using structured label maps obtained as described above. We generate synthetic glomeruli images by passing structure label maps extracted from the baseline training dataset and spatially deformed with the method described in Section II-A, as inputs to the generator. Fig. 5 shows some artificial glomeruli patches synthesized with our method. The enriched label maps enforce spatial constraints resulting in generated glomeruli visually closer to real ones. However, some fine texture details, like cell nuclei, do not appear as sharp as in real patches.

C. Validation protocol

Our objective is to evaluate the impact of training a U-Net segmentation network [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] by mixing augmentations based on random spatial deformations and synthetic images obtained via SPADE. As described earlier, we apply spatial deformations to label maps provided as inputs to the SPADE generator, and we evaluate our synthetic data against deformed real images. We therefore created 5 datasets offline:

• "Real": our baseline dataset, with 10 random deformations applied to each images (6600 images). • "SPADE-GTMask": SPADE-generated dataset, with 10 times randomly deformed ground truth binary masks from the baseline as input (6600 images). • "SPADE-Structure": SPADE-generated dataset, with 10 times randomly deformed structure label maps from the baseline glomeruli patches as input (6600 images).

• "Real + SPADE-GTMask": baseline dataset, with 5 random deformations applied to each images (3300 images) + SPADE-generated dataset, with 5 times deformed ground truth binary masks from the baseline as input (3300 images).

• "Real + SPADE-Structure": baseline dataset, with 5 random deformations applied to each images (3300 images) + SPADE-generated dataset, with 5 times deformed structure label maps from the baseline glomeruli patches as input (3300 images). We train a U-Net network on each dataset 5 times and average the results. A schematic view of our method workflow can be found in Fig. 6.

III. RESULTS AND DISCUSSION

The glomeruli segmentation performance results of the U-Net model are summarized in Table I for the training datasets presented in Section II-C. The best average dice score is obtained with the dataset that mixes real and synthetic images generated with SPADE and the ground truth masks as label maps. This score is 0.76 above the score obtained with the Real dataset, and 0.85 above the score obtained with SPADE and structure label maps. Our interpretation is that the patches of SPADE-Structure remain too close from the real patches of the training dataset, despite the texture variations. The noisy aspect of the patches of SPADE-GTMask better contributes to make the U-Net model more robust. We note that when training on synthetic images alone, the SPADE-Structure dataset obtains a score 2.47 above SPADE-GTMask, which supports our claim that the SPADE-Structure patches are more realistic than the SPADE-GTMask patches. It is also worth mentioning that the lower standard deviations on the dice scores for the datasets mixing real and synthetic patches support improved segmentation robustness.

These results show that the visual fidelity of the synthetic images is not necessarily crucial when a segmentation model is trained with a mix of real and synthetic data. The small obtained performance improvement also shows that the chosen generative approach has a limited ability to expand the diversity of the training dataset. The amount of beneficial randomness in texture and geometry, and the question of how to enable the emergence of novelty, deserve more investigations. Future lines of research will include the exploration of alternative ways to further increase the diversity of the training dataset. This could be achieved by enabling and controlling structure variations w.r.t. the baseline training set, rather than relying on already seen structures. This would require to devise a generative model with appropriate design to synthesize new structure label maps to feed the conditional patch synthesis model. Alternatives to the U-Net model, such as SegNet, or pre-trained versions of such models [START_REF] Bueno | Glomerulosclerosis identification in whole slide images using semantic segmentation[END_REF], [START_REF] Salvi | Automated assessment of glomerulosclerosis and tubular atrophy using deep learning[END_REF], will be considered. Another line of research would be to explore unconditional patch synthesis, using e.g. StyleGAN2-ADA [START_REF] Karras | Training generative adversarial networks with limited data[END_REF], that we did not consider in this study, and perform comparisons with conditional synthesis.

Fig. 1 .

 1 Fig. 1. Top row: Examples of glomeruli in patches of WSIs, with variable shapes (Periodic acid-Schiff staining); Bottom row: glomeruli ground truth binary masks.

Fig. 3 .

 3 Fig. 3. Results obtained with the SPADE model trained with ground truth binary masks only. Top row: Input glomeruli binary masks; Bottom row: Synthesis results. Many details are lost and the images are noisy.

Fig. 4 .

 4 Fig. 4. Binary masks separating textured and non-textured areas for the glomeruli patches presented in Fig. 1.

Fig. 5 .

 5 Fig. 5. Results obtained with the SPADE model trained with 4-class "structure" label maps. Top row: Input structure label maps; Bottom row: Synthesis results. The synthesized images are visually closer to real images, although the finest texture details are not reproduced.

Fig. 6 .

 6 Fig. 6. Workflow process. Step 1: A SPADE model is trained with pairs of glomeruli patches and label maps. Label maps are either ground-truth binary masks (a) or structure label maps obtained by thresholding the S component of the glomeruli patches in HSV color space (b). Step 2: The U-Net segmentation model is trained. First, random spatial deformations are applied to the real training dataset. The resulting images are fed to the SPADE models trained in Step 1, thus creating the datasets SPADE-GTMask (c) and SPADE-Structure (d), that we use to train the U-Net segmentation model.

TABLE I AVERAGE

 I DICE, PRECISION AND RECALL W.R.T. TRAINING DATASET FOR GLOMERULI SEGMENTATION.

	Training dataset	Dice	Precision	Recall
	Real	93.31 ± 0.46	95.64 ± 0.42	91.10 ± 0.95
	SPADE-GTMask	85.86 ± 1.67	96.77 ± 0.39	77.20 ± 2.84
	SPADE-Structure	88.33 ± 1.46	93.70 ± 2.16	82.41 ± 5.40
	Real + SPADE-GTMask	94.07 ± 0.17	94.85 ± 0.85	93.30 ± 0.79
	Real + SPADE-Structure	93.22 ± 0.14	94.86 ± 0.44	91.64 ± 0.45

IV. ACKNOWLEDGEMENTS

The glomeruli patches used in our experiments are courtesy of the ERACoSysMed project "SysMIFTA", co-funded by EU H2020 and the national funding agencies German Ministry of Education and Research (BMBF) project management PTJ (FKZ: 031L-0085A), and Agence National de la Recherche (ANR), project number ANR-15-CMED-0004.

The authors acknowledge the High Performance Computing Center ROMEO hosted by the University of Reims Champagne-Ardenne for access to computing resources.