Tristan Lazard
email: tristan.lazard@mines-paristech.fr

Marvin Lerousseau
email: marvin.lerousseau@mines-paristech.fr

Etienne Decencière
email: etienne.decenciere@mines-paristech.fr

Thomas Walter Cbio

Giga-SSL: Self-Supervised Learning for Gigapixel Images

Whole slide images (WSI) are microscopy images of stained tissue slides routinely prepared for diagnosis and treatment selection in medical practice. WSI are very large (gigapixel size) and complex (made of up to millions of cells). The current state-of-the-art (SoTA) approach to classify WSI subdivides them into tiles, encodes them by pretrained networks and applies Multiple Instance Learning (MIL) to train for specific downstream tasks. However, annotated datasets are often small, typically a few hundred to a few thousand WSI, which may cause overfitting and underperforming models. Conversely, the number of unannotated WSI is ever increasing, with datasets of tens of thousands (soon to be millions) of images available. While it has been previously proposed to use these unannotated data to identify suitable tile representations by self-supervised learning (SSL), downstream classification tasks still require full supervision because parts of the MIL architecture is not trained during tile level SSL pre-training. Here, we propose a strategy of slide level SSL to leverage the large number of WSI without annotations to infer powerful slide representations. Applying our method to The Cancer-Genome Atlas, one of the most widely used data resources in cancer research (16 TB image data), we are able to downsize the dataset to 23 MB without any loss in predictive power: we show that a linear classifier trained on top of these embeddings maintains or improves previous SoTA performances on various benchmark WSI classification tasks. Finally, we observe that training a classifier on these representations with tiny datasets (e.g. 50 slides) improved performances over SoTA by an average of +6.3 AUC points over all downstream tasks.

Introduction

Whole slide images (WSI) are microscopy images of stained tissue sections. They are enormous (billions of pixels) and complex, often containing millions of individual cells, their environments, and the overall tissue structure. They are routinely used in cancer treatment centers for diagnosis, patient stratification, and treatment selection. Computational pathology is the field concerned with the automatic analysis of WSI. The most clinically impactful task in computational pathology is to make predictions directly from the WSI, such as predicting cancer subtype, survival of the patient, or response to treatment. The major challenges in building predictive models operating on WSI are:

• Prohibitive memory requirements (typically 15GB uncompressed per WSI);

• Signal/noise: The high amount of biological material, not necessarily related to the output variable, is making models: (i) fail to identify the region of interests; (ii) prone to overfitting.

• Technical complexity: WSI are technically demanding to deal with given their large size, which presents a considerable barrier for multi-modal analyses of genomic and pathology data.

Today, the leading methods for WSI classification rely on Multiple Instance Learning (MIL): WSI are tessellated into small images, called tiles, which are encoded by an embedder. Tile embedders are usually pre-trained, either on natural images or -more recently and with great effect -by self-supervised learning (SSL). WSI are then seen as bags of tiles, and the slide representation is obtained by combining the tile embeddings, which are then used as input for the slide classification network. The agglomeration strategy comes in different flavors and usually relies on tile selection or weighted averaging of tile embeddings [9, [START_REF] Ilse | Attention-based Deep Multiple Instance Learning[END_REF][START_REF] Li | Dual-stream Maximum Selfattention Multi-instance Learning[END_REF][START_REF] Lu | Data-efficient and weakly supervised computational pathology on whole-slide images[END_REF][START_REF] Rymarczyk | Kernel Self-Attention in Deep Multiple Instance Learning[END_REF]. The slide classification network is usually trained from scratch on the specific classification task.

While these methods successfully predict a large variety of output variables, such as grade, cancer subtype, gene signatures, mutations or response to treatment [START_REF] Campanella | Clinical-grade computational pathology using weakly supervised deep learning on whole slide images[END_REF][START_REF] Coudray | Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning[END_REF][START_REF] Echle | Deep learning for the detection of microsatellite instability from histology images in colorectal cancer: A systematic literature review[END_REF][START_REF] Nikolas Kather | Pancancer image-based detection of clinically actionable genetic alterations[END_REF][START_REF] Lazard | Deep Learning identifies new morphological patterns of Homologous Recombination Deficiency in luminal breast cancers from whole slide images[END_REF][START_REF] Naylor | Neural network for the prediction of treatment response in Triple Negative Breast Cancer[END_REF][START_REF] Qu | Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning[END_REF], the performances remain highly dependent on the size of the training dataset [START_REF] Campanella | Clinical-grade computational pathology using weakly supervised deep learning on whole slide images[END_REF]. Indeed, MIL performance reaches saturation when using thousands of slides with associated ground truth for training [START_REF] Campanella | Clinical-grade computational pathology using weakly supervised deep learning on whole slide images[END_REF]. This might be realistic for the most frequent cancer types and routinely acquired output variables, but in most real-world projects only a few tens or hundreds of WSI with corresponding ground truth are available. However, with the digitalization of many pathology facilities, there is an increasing access to WSI without ground truth which are digitalized in clinical routine. Following the SSL paradigm that has been successfully applied at the tile level [7,[START_REF] Dehaene | Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology[END_REF][START_REF] Lazard | Deep Learning identifies new morphological patterns of Homologous Recombination Deficiency in luminal breast cancers from whole slide images[END_REF][START_REF] Courtiol | Deep learning-based classification of mesothelioma improves prediction of patient outcome[END_REF], there is a challenging opportunity to make use of these unannotated data at the slide level to derive meaningful slide representations. These would be particularly useful for small cohorts and non-standard output variables, such as prognosis for rare cancer types or prediction of treatment response in clinical trials.

However, learning representations at the WSI level is difficult since WSI cannot be manipulated as one image object due to their size, impeding the straightforward use of selfsupervised learning frameworks developed on natural images. The community needs to innovate to translate SSL at the WSI level regarding the design of pertinent augmentations. For instance, the crop augmentation plays a central role for learning good representations with SSL on natural images [START_REF] Chen | A Simple Framework for Contrastive Learning of Visual Representations[END_REF][START_REF] Misra | Self-Supervised Learning of Pretext-Invariant Representations[END_REF]. However, randomly cropping one memoryfittable image from a WSI can lead to a complete loss of the cells and tissues that determine its ground-truth, due to the inherent heterogeneity of tissues. Further developments should also be done on the architecture of a SSL framework for WSI representations, as was done in the only paper tackling SSL at the WSI level [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF].

Here, we propose Giga-SSL, a strategy to perform SSL for gigapixel images. Designed for pathology data, our method is capable of leveraging large datasets, such as The Cancer Genome Atlas (TCGA) [START_REF] Weinstein | The Cancer Genome Atlas Pan-Cancer Analysis Project[END_REF], to learn representations at the WSI level without using any ground truth databut only whole slide images. Our main contributions are:

• Giga-SSL, an efficient self-supervised learning framework for gigapixel images.

• Extensive experiments show that a linear classifier that uses these embeddings outperforms the current stateof-the-art performance on several clinically impactful classification tasks. The gains are especially significant for small datasets.

• We publicly release the WSI embeddings of the whole TCGA dataset, compressing it by a factor of almost 1 million from 16Tb to 23Mb, and thus making this large image datasets amenable for future research.

We expect that this method will have an important impact in the field of computational pathology in two ways:

(1) Our method specifically boosts performance for small datasets, which are very common in practice. We therefore address a major bottleneck in computational pathology. [START_REF] Caron | Emerging Properties in Self-Supervised Vision Transformers[END_REF] We can make image data accessible to a larger community of researchers in cancer bioinformatics, in order to investigate the complex relationships between genetic, transcriptomic and phenotypic data. To facilitate reproducibility and the broad use of Giga-SSL, the complete source code of this work as well as the full TCGA-FFPE encodings are available at https://github.com/trislaz/gigassl.

Background

Multiple instance learning for gigapixel images

In the MIL paradigm, objects (called bags) comprise other objects (called instances). For gigapixel images, the bag is a gigapixel image, and its instances are subimages (also called tiles or patches) extracted throughout the gigapixel image. While traditional MIL assumes independent and identically distributed (i.i.d.) instances within each bag [START_REF] Ilse | Attention-based Deep Multiple Instance Learning[END_REF], this assumption is relaxed for gigapixel images because instances are extracted from the same image, and are therefore not independent. Given a gigapixel image X made of n x instances (x 1 , . . . , x nx ), MIL is implemented as a combination of three modules: (i) an instance embedder e θ1 (•), (ii) a pooling operator p θ2 (•) and (iii) a classifier c θ3 (•) such that a decision ŷ is obtained with ŷ = c θ3 p θ2 {e θ1 (x 1 ), . . . , e θ1 (x n )} .

Most MIL architectures differ in the design of the pooling operator p θ2 . There are two families of operators: (i) those that consider instances as i.i.d. and (ii) those that exploit the relationship between instances of a bag. Architectures that consider instances as i.i.d. are either parameterless (e.g. using the operators average, maximum, a concatenation of both [START_REF] Lerousseau | Multimodal brain tumor classification[END_REF], or a noisy-OR function [START_REF] Srinivas | A Generalization of the Noisy-Or Model[END_REF]), or trainable, such as an attention-based neural network [START_REF] Ilse | Attention-based Deep Multiple Instance Learning[END_REF]. While these architectures obtain good performances, instances of gigapixel images are dependent and contain information that can be leveraged to produce accurate predictions. Modern MIL architecture for gigapixel images have been designed to exploit the spatial relationship of instances. For instance, transformer-based MIL approaches [START_REF] Shao | TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification[END_REF] extend the attention mechanism of Ilse et al. [START_REF] Ilse | Attention-based Deep Multiple Instance Learning[END_REF] by incorporating the positions of instances for decision prediction. Of particular interest in this work, the SparseConvMIL [START_REF] Lerousseau | SparseConvMIL: Sparse Convolutional Context-Aware Multiple Instance Learning for Whole Slide Image Classification[END_REF] architecture leverages spatial information by building a sparse map from both the instance embeddings and their sampled locations. This map is further processed by a sparse-input convolutional neural network that outputs a latent vector to be further classified by a generic classifier.

Self-supervised learning for gigapixel images

Self-supervised learning have been investigated in computational pathology at the tile level, i.e. for patches extracted from whole slide images [7,[START_REF] Dehaene | Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology[END_REF][START_REF] Lazard | Deep Learning identifies new morphological patterns of Homologous Recombination Deficiency in luminal breast cancers from whole slide images[END_REF][START_REF] Courtiol | Deep learning-based classification of mesothelioma improves prediction of patient outcome[END_REF]. The findings suggest that SSL indeed improved the performance on WSI classification tasks by using the SSL pre-trained tile level model as a frozen tile encoder. Because patches extracted from WSI are of size similar to datasets of natural images, the majority of the work successfully used off-theshelf frameworks developed on natural images such as Sim-CLR [START_REF] Chen | A Simple Framework for Contrastive Learning of Visual Representations[END_REF] or MoCo [START_REF] He | Momentum Contrast for Unsupervised Visual Representation Learning[END_REF].

To the best of our knowledge, only one prior work has proposed a self-supervised learning framework for learning representations directly at the WSI level [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF]. To do so, the authors design a new architecture made of 3 hierarchically stacked visual transformers [12] which is trained on unlabelled WSI with the DINO framework [START_REF] Caron | Emerging Properties in Self-Supervised Vision Transformers[END_REF], notably by enforcing consistency between two perturbed views of the same object. As stated by the authors [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF], their approach cannot be trained end-to-end due to memory issues and needs to be trained in stages, starting from the visual transformer at higher magnification. on top of time-consuming SSL pre-training, a drawback is the need to retrain all transformers at lower magnifications when modifying one visual transformer. A major bottleneck of this approach is the necessity to retrain the last transformer from scratch, implying that (i) the whole system does not benefit fully of SSL pretraining, and that (ii) linear embeddings cannot be extracted for new slides and used as input vectors for downstream tasks [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF]. Conversely, we designed an efficient method for learning WSI representations that obtained state-of-the-art performance with a linear classifier without the need to finetune any part of our system.

Methods

Algorithmic design

Notations and algorithmic background Giga-SSL training comprises 6 sequential steps to extract WSI representations which we details here and which is illustrated in Figure Fig. 1. Lets us consider a WSI X. Giga-SSL uses an extension of the SparseConvMIL architecture for WSI classification [START_REF] Lerousseau | SparseConvMIL: Sparse Convolutional Context-Aware Multiple Instance Learning for Whole Slide Image Classification[END_REF] by considering a ResNet network f θ (e.g. ResNet18) [START_REF] He | Deep Residual Learning for Image Recognition[END_REF], which is cut at the beginning of the fourth residual block into two sequential parts:

1. the first part, acting as the tile embedder e θ1 , is made of all layers of f θ up to the first layer of the fourth block, 2. the second part, acting as the pooling function p θ2 , is made of all layers after and including the fourth block of f θ , such that for any image i, the ResNet embedding is:

f θ (i) = p θ2 e θ1 (i) ∈ R 512 .
Step Step 4: Augmentations of the WSI at the slide-level While WSI are difficult to manipulate due to their huge size, a sparse map can be augmented with geometric transformations, enabling our framework to perform slide-level transformations in real-time. S 1 and S 2 are randomly flipped, rotated, and scaled with a factor uniformly sampled in [0.5, 2] independently for the x and y axis.

Step 5: Embedding of the sparse maps into two augmented WSI representations To compute representations, we apply p θ2 on both augmented sparse maps S 1 and S 2 . It should be noted that p θ2 is not a conventional CNN model but has been converted into a submanifold convolutional network [START_REF] Graham | Submanifold Sparse Convolutional Networks[END_REF] with the same architecture such that it can process sparse data. At this stage, the two augmented views of the input WSI X (augmented at the tile-level and at the slide-level) are vector representations of the WSI. Step 6: Loss optimization As is done in SimCLR, augmented views are finally fed to a projector, giving two augmented projections with which the loss will be computed. We train the weights of the pooling function p θ2 by optimizing the contrastive loss NT-XENT loss [START_REF] Chen | A Simple Framework for Contrastive Learning of Visual Representations[END_REF]. Given a minibatch B of augmented WSI (X i 1 , X i 2 ) i∈B , we set the loss function for a positive pair of WSI as

ℓ i = -log exp sim X i 1 , X i 2 /τ x∈B 1 {x =X i 1 } exp sim X i 1 , x /τ (1)
where τ is the temperature parameter and 1 {.} the indicator function. The final loss is computed as the average of these terms across all views.

Design choices

Selection of the underlying CNN architecture and loss function Giga-SSL does not theoretically rely on a ResNet architecture. There are many choices of good architectures that could be used for the comprising tile encoder and pooling function, including two parts of different architectures. However, the pooling function must be implemented such that it can handle sparse data since it processes the augmented sparse maps (see Step 5 Sec. 3.1).

Freezing the tile encoder A key computational bottleneck of this strategy is the online computation of tile embeddings for a batch of B WSI, each composed of T tiles.GPU memory limitations put constraints on B and N t , which effectively limits the number of total tiles per batch that can be used. Besides, it has been shown in SSL for natural images that a large batch size is required to yield representations with good downstream classification performances [START_REF] Chen | A Simple Framework for Contrastive Learning of Visual Representations[END_REF][START_REF] Chen | Intriguing Properties of Contrastive Losses[END_REF][6]. A strategy for overcoming these issues is to freeze the tile encoder e θ1 and pre-compute the embeddings of randomly sampled and augmented tiles for each WSI, i.e. essentially bypassing steps 1 and 2 of Sec. 3.1. For encoding a WSI, this is implemented by: (i) sampling 50 tile-level augmentation functions (both color and geometric augmentations) (t k ) k 50 , (ii) for each k, randomly subsampling 256 tiles from the WSI and augment them with t k , and (iii) concurrently and independently forwarding each augmented tile into e θ1 and storing them. This process leads to N *50*256 tile embeddings where N is the total number of WSI of the Giga-SSL training dataset.

Giga-SSL is then trained, starting from step 3 Sec. 3.1 by performing the following to sample a view of a WSI: (i) sample one of the 50 tile-level augmentations, (ii) sample a subset T of the 256 embeddings obtained from this augmentation, (iii) build the sparse map, and (iv) carry on from step 4 of Sec. 3.1.

Experimental validation 4.1. Step 1: self-supervised pre-training

Self-supervised pre-training of Giga-SSL is done using The Cancer Genome Atlas (TCGA) [START_REF] Weinstein | The Cancer Genome Atlas Pan-Cancer Analysis Project[END_REF], a public dataset that comprises 11754 whole slide images containing tissue from virtually all types of solid cancers. This dataset is the result of an international data-collecting effort and therefore features a high variety of participant centers (190). Such slides are crucial for patient care since they are the basis of diagnosis and treatment selection. On average, images have a width of 93000 pixels and a height 67500 pixels, for an average of 6.5 billion pixels per image. Fully compressed, TCGA weighs more than 16 Terabytes, i.e. 3 orders of magnitude more than ImageNet [START_REF] Deng | ImageNet: A large-scale hierarchical im-age database[END_REF]. We tesselated non-overlapping square patches of size 256 pixels from all diagnostic slides of the TCGA at 10x magnification.

e θ1 pre-training We choose to pre-train e θ1 using MoCo [START_REF] He | Momentum Contrast for Unsupervised Visual Representation Learning[END_REF]. We trained a full ResNet18 on a subset of 6 million of these tiles extracted from a random set of 3000 slides from the TCGA for 200 epochs. e θ1 is then extracted from this network as described in Sec. 3.1. More details about this pre-training are available in the supplementaries.

Giga-SSL pretraining: we trained Giga-SSL on the full TCGA dataset, with frozen augmented embeddings extracted with the previously described pre-trained tile embedder (see Sec. 3.2), with Adam [START_REF] Diederik | Adam: A Method for Stochastic Optimization[END_REF] for 1000 epochs.

Step 2: learning from linear embeddings

Training design For Giga-SSL, similarly to the works on natural images [START_REF] Caron | Emerging Properties in Self-Supervised Vision Transformers[END_REF][START_REF] Chen | A Simple Framework for Contrastive Learning of Visual Representations[END_REF][START_REF] He | Momentum Contrast for Unsupervised Visual Representation Learning[END_REF], we measured the quality of the learned representations by performing linear probing either with all the labels available for a given task or by artificially reducing the number of labels to simulate a semi-supervised setting. To do so, one representation was extracted for each WSI after SSL pretraining. These representations were then used as input data to train a logistic regression for each considered downstream task.

Datasets This protocol was applied to six diagnostic WSI classification tasks highly pertinent for clinical practice:

• 3 tasks performed by Chen et al. [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF] aiming at automating the routine diagnosis of Non-Small Scell Lung Cancer (NSCLC), Breast Cancer (BRCA), and Kidney Cancer (RCC);

• 3 tasks aiming at inferring molecular properties from tissue slides towards faster, cheaper and more accessible molecular testing for cancer therapy selection.

For each of these 6 tasks, Tab. 1 reports the number of training WSI of the corresponding dataset, and their class distribution. All the datasets for these tasks are subsets of the TCGA [START_REF] Weinstein | The Cancer Genome Atlas Pan-Cancer Analysis Project[END_REF]. Results were computed on 10 bootstrapped splits of the data for each experiment, as was done in Chen et al. [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF], and we also used their train/test splits to ensure fairness of performance comparisons. Default settings The number T of tiles sampled per slide to 5. For a slide X, we bootstrap R = 50 views without tile augmentation (i.e. differing only in the sampled tiles), compute their embedding {W r } 1,...,50 and consider the WSI representation as the elementwise average of the {W r } 1,...,50 . Average embeddings are normalized using a standard scale, while the Giga-SSL embeddings are normalized using the L2 unit. More details about training parameters are given in the supplementaries.

Task

Results

Classification results on benchmarked tasks Table 2 synthesizes the results on all tasks for 5 models i.e. average, an attention-based MIL [START_REF] Ilse | Attention-based Deep Multiple Instance Learning[END_REF] on top of a ResNet18 pretrained with MoCo , DeepSMILE [START_REF] Schirris | DeepSMILE: Self-supervised heterogeneity-aware multiple instance learning for DNA damage response defect classification directly from H&E whole-slide images[END_REF] and HIPT [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF].

Results from HIPT and DeepSMILE are taken from their respective articles, and constitute the SoTA on the task on which they are cited. Our proposed approach, Giga-SSL, outperforms the state-of-the-art on two out of three tasks benchmarked in ( [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF]) when using 100% of the available training labels NSCLC and BRCA subtyping. For BRCA subtyping, the AUC is increased by 3 points. Our proposed approach also achieves superior performances for all the other remaining tasks (mHRD, tHRD and BRCA molecular profiling). However, the power of the proposed approach seems to be in the low data regime. This is evident by the results obtained by using only 25% of the available labels. In this semi-supervised regime, the proposed approach obtained the best results on all tasks. While this finding may be expected when comparing Giga-SSL to methods without pretraining, Giga-SSL obtained superior results compared to the other SSL-based approach HIPT. For example, there is a gain of 6.9 AUC points for BRCA subtyping.

Compared to attention-based MIL and HIPT, the proposed approach (Giga-SSL) provides an overall gain in per-Method Giga-SSL (proposed) AverageMIL DeepMIL [START_REF] Ilse | Attention-based Deep Multiple Instance Learning[END_REF] HIPT [START_REF] Chen | Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning[END_REF] DeepSMILE [START_REF] Schirris | DeepSMILE: Self-supervised heterogeneity-aware multiple instance learning for DNA damage response defect classification directly from H&E whole-slide images[END_REF] Linear formance while working in a linear regime. This is in contrast to HIPT and attention-based methods, which require fine-tuning and learning from scratch, respectively. Consequently, the downstream training pipeline for Giga-SSL is extremely efficient in comparison to the other two approaches. For instance, training for BRCA subtyping with 100% of the training data on 10 bootstrapped splits took 1.25 CPU-seconds for the proposed approach versus 150 GPU-minutes for attention-based MIL. This is a difference of 7200 times in favor of Giga-SSL -while also obtaining superior performances.

Tiny datasets

In practice, pathological datasets can be tiny for the prediction of treatment response. For instance, phase II clinical trials typically involve 50 patients. Training a model to identify responding and non-responding patients is therefore challenging due to the low number of available labels.

We measured the performance of Giga-SSL in such a context by artificially reducing the size of all 6 datasets to 250, 100 and 50 samples. We compare Giga-SSL to the DeepAttnMIL model, which performances are on par with all other benchmarked algorithms (see Tab. 2).

Figure 2 shows that the performance gap between the proposed approach and the standard WSI classification method strengthens as the number of samples decreases. The average improvement over all tasks brought by Giga-SSL features is of 5.1 AUC points when using 100 WSI and up to 6.3 AUC points when using only 50 WSI.

Ablation study and sensitivity analyses

In this section, we aim to understand the impact of some of Giga-SSL design choices over the predictive power of the learned representations. All subsequent experiments were conducted with the same conditions (including hyperparameters, epochs, and training dataset) as in the previous experiments, unless otherwise stated.

Sharing tile augmentations within views improves performance Table 3 reports the performance of Giga-SSL when removing one component at a time, i.e. (i) with a tile embedder pre-trained on ImageNet rather than pre-trained with MoCo on histopathological data (Giga-SSL im ), (ii) without slide-level augmentation during the WSI-level SSL pretraining; (iii) without shared augmentations across all tiles of a view, i.e. each tile is transformed by a randomly and independently sampled augmentation. On the contrary, the slide-level augmentation does not seem to be extremely important for the SSL task, as removing it has a small to no impact on performances. However, applying independent transformations to each tile (not shared) degrades substantially the performances with an average decrease of 1.9 AUC points using 100% of the data down to 2.8 AUC points when using only 50 WSI, over the classification tasks. When ablating the shared transformations from a Giga-SSL model trained with tile features pretrained with ImageNet, the drop of performances compared to a Giga-SSL im is even more important: 2.1 AUC points with 100% of the data, 3.2 AUC points with 50 WSI.

Using shared augmentation thus allows the learning of useful features in abundant and scarce data regimes. We hypothesize key features linked to the slide preparation and shared by all the tiles on the slide are still available for shortcut learning if the tile-level augmentations are not shared. It seems that these shortcut features may be more present in ImageNet than in MoCo. Highlighting such features and finding even more stringent ways to hide them when learning Giga-SSL would improve even more its performances.

The fewer tiles, the better Figure 3.A presents the performances of 4 Giga-SSL models trained with different numbers of sampled tiles per view. The fewer tiles we sample, the better the resulting WSI representations. This behaviour strengthens when the downstream problem has a smaller training set and is comparable among all the downstream classification tasks. Interestingly, we can observe the opposite effect when using a DeepMIL model to classify a WSI: the fewer tiles used at training time, the worse the per-formances [START_REF] Lerousseau | Multimodal brain tumor classification[END_REF]. A very small number T of sampled tiles per view when training Giga-SSL can be seen as an aggressive augmentation. It has been reported ( [START_REF] Chen | A Simple Framework for Contrastive Learning of Visual Representations[END_REF]) that SSL benefits from stronger augmentations more than classification tasks, and Tian et al. ( [33]) have shown that there is an optimal strength of augmentation for each downstream task. This optimum results from a trade-off between keeping enough information to solve the downstream task and minimizing irrelevant features.

As sampling 5 tiles per WSI is enough to learn useful information to solve all the proposed downstream tasks, we can deduce that the signal relative to these problems is distributed among most of the tiles of the WSI. It would be interesting to test the performances of Giga-SSL on a classification task for which we know that the signal is highly concentrated on a few instances.

Ensembling representations brings improvement

We show in supplementaries that a Giga-SSL model with a SparseConvMIL aggregation module must use the same number of tiles per WSI at inference and training. We therefore decided to bootstrap R views of a WSI at inference time before averaging the Giga-SSL embeddings of these R views. Figure Fig. 3.B investigates the effect of R on the downstream performances of the Giga-SSL representations. It shows that without this ensembling strategy, Giga-SSL loses up to 4 AUC points on NSCLC subtyping. The gain in performance saturates around R = 50. Generalization Giga-SSL has been trained on the full TCGA dataset, and downstream classification dataset also comes from the TCGA. In order to investigate the extent to which Giga-SSL could transfer to other datasets, we extracted from the TCGA all slides coming from the 41 centers that contributed to the NSCLC dataset, leading to an in- 4. Linear classification performances on NSCLC subtyping of embeddings trained on either the full TCGA or a subset of the TCGA independent from the downstream task dataset. when trained on a set of WSI totally independent from the downstream task set. This suggests that Giga-SSL would generalize well on a different dataset.

Attention-deep-MIL unlearns when trained with SSL Instead of using a sparse-CNN as a tiles features aggregator, one could choose any other MIL model. We trained a Giga-SSL model with a DeepMIL aggregation module and evaluated its downstream linear performances on the NSCLC dataset. Figure 3.C shows that the performances of such a model decrease while the SSL training is in progress. Although the DeepMIL shows very good classification performances Tab. 2 when trained from scratch, this architecture seems not suitable for Giga-SSL pretraining. We suspect that the DeepMIL architecture has too easily access to shortcuts features to learn the WSI identity. Understanding what causes its collapse may highlight key pitfall for Giga-SSL training and therefore allow to improve it. For all of the latter points, we report in the supplementaries a similar behaviour on the other downstream classification tasks.

Conclusion

Limitations While Giga-SSL has been shown to generalize well outside of its training data distribution, the tileembedder is not pre-trained on a dataset that is entirely independent from the downstream tasks datasets. It would be interesting to conduct the same experiment as Sec. 5 but excluding the WSI from the tile-embedder pre-training dataset too. In addition, a drawback of working with frozen embeddings of WSI is that it removes any possibility of building explainable models.

Finally, we have explored self-supervised learning for whole slide images with a versatile design based on specific data augmentation tailored for the multiple instance learning framework. Our proposed approach achieved or beat state-of-the-art performance over a wide range of clinically impactful tasks in both high and low data regimes. In particular, for small datasets (e.g. 50 slides), our approach achieved a performance improvement of 6.3 AUC points on average compared to competing methods. Ablation studies and sensitivity analyses highlighted the key components of our approach -including tile encoder pretraining and how to apply augmentations to tiles -to better understand the pitfalls of self-supervised whole slide image representation learning.

The public release of the learned representations for all diagnostic slides of The Cancer Genome Atlas in a man-ageable size has the potential to decipher new knowledge about cancer and to develop new tools for diagnosis assistance and treatment response prediction towards improved patient survival.

Figure 1 .

 1 Figure1. Overview of the Giga-SSL method. First, random augmentations of a WSI are used to create two different views X1 and X2 of the same WSI. Next, T tiles randomly extracted from each view are embedded using a tile-embedder network e θ 1 , resulting in T embeddings in R F . These embeddings and their associated tile coordinates are fed into a sparse-input CNN model p θ 2 , producing two WSI representations W1 and W2. A contrastive loss is applied on a minibatch of several whole slide images in order to update both e θ 1 and p θ 2 .

Figure 2 .

 2 Figure 2. Difference between the average AUC performances of Giga-SSL and DeepMIL (in %) as a function of the training set size. The red line represents equal performance. Above the red line, the advantage is given to Giga-SSL

Figure 3 .

 3 Figure 3. Experiments on key parameters of Giga-SSL. Each point is a 10-fold cross-validated AUC performance of a logistic regression fed with Giga-SSL features. The classification task is NSCLC subtyping for the three experiments. A. Effect of the number of sampled tiles T per WSI during training. B. Effect of the number R of bootstrapped non-augmented views of WSI to feed Giga-SSL at inference time (see supplementaries). C. Evolution of the performances of a Giga-SSL with a SparseConvMIL (blue line, normal situation) or an attention-MIL network (orange line) as an aggregator.

  1: Augmentation of the WSI at the tile-level Two augmentation functions t 1 and t 2 are sampled from an image augmentation domain A made of color augmentations (color jitter, grayscale) and geometric augmentations (flips, rotations, scaling, blurring). First, T tiles are subsampled from X for each augmentation function t 1 and t 2 , yielding two sets of patches {X 1 } and {X 2 }. The coordinates of the top-left pixel of the tiles are stored for further processing.

Finally t 1 is applied to all patches of {X 1 }, yielding a set of augmented patches denoted as t 1 {X 1 } , and similarly a set t 2 {X 2 } for the second set patches {X 2 }.

Step 2: Embedding of tiles Each tile of both t 1 {X 1 } and t 2 {X 2 } are concurrently and independently forwarded through the tile embedder network e θ1 . Each image is thus converted into a feature map which is averaged across all pixels, yielding a tile embedding of size F (256 for ResNet18) for each tile of t 1 {X 1 } and t 2 {X 2 }

Step 3: Building of the sparse maps Following the framework of SparseConvMIL

[START_REF] Lerousseau | SparseConvMIL: Sparse Convolutional Context-Aware Multiple Instance Learning for Whole Slide Image Classification[END_REF]

, a sparse map S 1 is built by assigning each produced embedding of t 1 {X 1 } at the location where each of its original tiles was sampled in Step 1 Sec. 3.1 but downsampled by a factor d = 224. Similarly, a sparse map S 2 is built from the embeddings t 2 {X 2 } .

Table 1 .

 1 Total number of samples and number of samples per class for all of the 6 benchmarked tasks in this paper.

		# samples # labels per class
	BRCA subtyping	1041	831 -210
	Kidney subtyping	924	510 -294 -120
	NSCLC subtyping	1033	528 -505
	BRCA Molecular	595	129 -466
	BRCA mHRD	912	447 -465
	BRCA tHRD	634	318 -316

Table 2 .

 2 Benchmark study reporting the 10-fold cross-validated AUC performances of a logistic regression trained with Giga-SSL WSI representations or AverageMIL WSI representations, and retrained from scratch for other benchmarked approaches. For each task, we evaluate the methods with two data budgets with either 100% or 25% of the available training data.

			✓	✓	✗	✗	✗
	Task	% data
	NSCLC subtyping	100 25	0.952 ± 0.020 0.939 ± 0.017	0.913 ± 0.023 0.948 ± 0.017 0.952 ± 0.021 0.885 ± 0.036 0.922 ± 0.034 0.923 ± 0.020	--
	BRCA subtyping	100 25	0.905 ± 0.032 0.890 ± 0.058	0.859 ± 0.038 0.874 ± 0.050 0.874 ± 0.060 0.822 ± 0.072 0.860 ± 0.042 0.821 ± 0.069	--
	RCC subtyping	100 25	0.982 ± 0.007 0.975 ± 0.012	0.973 ± 0.011 0.986 ± 0.008 0.980 ± 0.013 0.959 ± 0.015 0.970 ± 0.016 0.974 ± 0.012	--
	BRCA molecular	100 25	0.938 ± 0.035 0.853 ± 0.075	0.920 ± 0.037 0.924 ± 0.042 0.799 ± 0.068 0.810 ± 0.093	--	--
	BRCA mHRD	100 25	0.756 ± 0.028 0.743 ± 0.039	0.706 ± 0.030 0.736 ± 0.047 0.643 ± 0.050 0.660 ± 0.046	--	0.727 ± 0.010 -
	BRCA tHRD	100 25	0.855 ± 0.023 0.781 ± 0.050	0.799 ± 0.034 0.836 ± 0.052 0.698 ± 0.078 0.721 ± 0.075	--	0.838 ± 0.012 -


Mélanie Lubrano
email: melanie.lubrano@minesparistech.psl.eu

Yaëlle Bellahsen-Harrar
email: yaelle.bellahsen-harrar@aphp.fr

Rutger Fick
email: rfick@tribun.health

Cécile Badoual
email: cecile.badoual@aphp.fr

Thomas Walter
email: thomas.walter@minesparis.psl.eu

Lubrano Fick

Badoual Walter

Simple and Efficient Confidence Score for Grading Whole Slide Images

Keywords: Uncertainty estimation, confidence score, grading, multiple instance learning, whole slide images

Grading precancerous lesions on whole slide images is a challenging task: the continuous space of morphological phenotypes makes clear-cut decisions between different grades often difficult, leading to low inter-and intra-rater agreements. More and more Artificial Intelligence (AI) algorithms are developed to help pathologists perform and standardize their diagnosis. However, those models can render their prediction without consideration of the ambiguity of the classes and can fail without notice which prevent their wider acceptance in a clinical context. In this paper, we propose a new score to measure the confidence of AI models in grading tasks. Our confidence score is specifically adapted to ordinal output variables, is versatile and does not require extra training or additional inferences nor particular architecture changes. Comparison to other popular techniques such as Monte Carlo Dropout and deep ensembles shows that our method provides state-of-the art results, while being simpler, more versatile and less computationally intensive. The score is also easily interpretable and consistent with real life hesitations of pathologists. We show that the score is capable of accurately identifying mispredicted slides and that accuracy for high confidence decisions is significantly higher than for low-confidence decisions (gap in AUC of 17.1% on the test set). We believe that the proposed confidence score could be leveraged by pathologists directly in their workflow and assist them on difficult tasks such as grading precancerous lesions.

Introduction

Cancer diagnosis usually implies the assessment of biological tissue samples by pathologists. For some cancer types, samples need to be graded according to histological patterns to assess the severity of the disease. This grading task is crucial since it affects the choices in patient's care and follow-up. For head and neck cancers, precancerous lesions (or dysplasia) are graded according to the extent of abnormality detected in the tissue [START_REF] Gale | Evaluation of a new grading system for laryngeal squamous intraepithelial lesions-a proposed unified classification[END_REF]. However, grading morphological patterns according to their degree of abnormality is a difficult task: WHO nomenclatures try to impose sharp boundaries on a continuous spectrum of lesions, which often leads to poor inter-and intra-grader-reproducibility [START_REF] Gale | Laryngeal dysplasia: persisting dilemmas, disagreements and unsolved problems-a short review[END_REF][START_REF] Slot | Laryngeal precursor lesions: Interrater and intrarater reliability of histopathological assessment[END_REF].

In the past years, Artificial Intelligence (AI) has been successfully applied to pathology: to detect metastasis [START_REF] Babak | Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer[END_REF], to determine cancer subtype [START_REF] Coudray | Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning[END_REF] or to estimate patient prognosis [START_REF] Courtiol | Deep learning-based classification of mesothelioma improves prediction of patient outcome[END_REF]. Several studies even showed the positive benefit that AI tools can have when combined with manual examinations [START_REF] Kiani | Impact of a deep learning assistant on the histopathologic classification of liver cancer[END_REF][START_REF] Ba | Assessment of deep learning assistance for the pathological diagnosis of gastric cancer[END_REF]. However, these algorithms are not yet widely accepted for use in clinical practice as standalone diagnostic tools ( [START_REF] Van Der Laak | Deep learning in histopathology: the path to the clinic[END_REF][START_REF] Yogesh K Dwivedi | Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy[END_REF] due to concerns about their instability on external cohorts, sensitivity to domain shifts, and lack of interpretability. AI models also deliver their predictions regardless of histologic ambiguity, without providing information about the confidence of the decision, even if the sample is difficult to grade for the pathologist himself. On such ambiguous cases, a pathologist may ask for a second opinion or additional analysis, while AI would not flag the uncertainty.

Measuring Neural Network's (NN) confidence could bring additional information on these difficult cases. Confidence (or equivalently 'uncertainty') of AI models have been explored in the past years [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF][START_REF] Osband | Risk versus uncertainty in deep learning: Bayes, bootstrap and the dangers of dropout[END_REF], and applied to medical context. Still, very little work focused on histopathology applications: its first use was to boost model performances for segmentation tasks [START_REF] Nair | Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation[END_REF][START_REF] Camarasa | Quantitative comparison of monte-carlo dropout uncertainty measures for multi-class segmentation[END_REF], for active learning [START_REF] Scandalea | Deep active learning for axon-myelin segmentation on histology data[END_REF] or detection of Out Of Distribution (OOD) samples [START_REF] Linmans | Efficient Out-of-Distribution Detection in Digital Pathology Using Multi-Head Convolutional Neural Networks[END_REF].

Benchmark of the most popular confidence measures were recently conducted: Poceviciute et al. ( 2022) compared their capacity to identify misclassified patches, and [START_REF] Thagaard | Can you trust predictive uncertainty under real dataset shifts in digital pathology[END_REF] to detect OOD samples at inference. [START_REF] James M Dolezal | Uncertainty-Informed Deep Learning Models Enable High-Confidence Predictions for Digital Histopathology[END_REF] proposed an uncertainty measure robust to domain shift and tested it on cancer subtyping tasks (binary classification of LUAD vs LUSC1 ). However, none of these methods were designed for multiclass problems or grading tasks (ordinal output) and all focused on binary classification of whole slides. In addition, these studies used tile-based methods requiring either local annotation or tile level information (if the disease is uniformly spread on the slide for instance). This paradigm is not suitable for grading tasks because multiple grades of lesions can co-exist on the same slide.

Most of all, AI models for WSI classification are difficult to train due to the large size of pathology datasets. Yet, the most common methods benchmarked in these studies are computationally intensive (relying on generation of a distribution of predictions) and thus not suitable for such datasets containing thousands of slides and millions of tiles.

In this work we propose a measure of uncertainty designed for grading tasks, that does not require extra training or inferences nor particular architecture changes. This uncertainty measure is easily interpretable as it quantifies how much a model hesitates between two classes, and thus particularly suitable for grading tasks. We compared our grade sensitive confidence score with popular ones and showed that our method manages to accurately separate high confidence from low confidence slides and matches the behavior of pathologists. We believe that measuring the confidence of AI models can provide useful information to pathologists, helping them to standardize their diagnoses and increase their trust in AI.

Related Work

Multiple methods have been proposed to quantify models' confidence. The simplest one consists in considering the softmax output of the model as a direct measure of confidence. This was argued against in [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF] who proposed the Monte Carlo (MC) dropout method as an approximation of a variational Bayes estimator. In MC dropout, multiple forward inferences are generated while keeping the dropout active. The uncertainty is quantified by the variability between the MC samples and measured by the standard deviation of this distribution. Recently, deep ensembles have proven to be better estimates of uncertainty [START_REF] James M Dolezal | Uncertainty-Informed Deep Learning Models Enable High-Confidence Predictions for Digital Histopathology[END_REF][START_REF] Poceviciute | Generalisation effects of predictive uncertainty estimation in deep learning for digital pathology[END_REF]: multiple networks are trained from different random initializations. Again the uncertainty measure is derived from the distribution of predictions generated. Other techniques have been explored such as Test Time Augmentation (TTA) which consist in applying random transformation to an input image, generating a distribution of predictions. However, the size of histological datasets make such techniques hardly feasible in practice in the field of computational pathology.

Methods

Weakly-supervised WSI classification

Grading of head and neck precancerous lesions

Grading WSI is a difficult task: often, the borders between each grade are blurry and related histologic patterns overlap. Additionally, several grades of lesions can co-exist on the same WSI. In this study we focused on grading head and neck precancerous and cancerous lesions with respect to their gravity. Such lesions are graded by assessing several morphological and cytological abnormalities of the tissue according to the WHO recommendations [START_REF] El-Naggar | WHO classification of head and neck tumours[END_REF]. Head and neck epithelium can be either normal/benign (0), or present a low grade lesion (1), a high grade lesion (2) or have infiltrative carcinoma (3). (See Appendix A.1 for examples).

Dataset

To assess the benefit of our confidence measure in the context of a grading task we used an in-house dataset of head and neck precancerous and cancerous tissue samples obtained from the HEGP pathology department. The dataset was composed of 2121 Hematoxylin and Eosin (H&E) slides. Each slide was labeled with the most severe lesion it contained (grade 0 to 3). Considering the inherent ambiguity of these histologic patterns, a gold standard test set of 128 slides was blindly reviewed by 2 pathologists and consensus diagnosis was found. An extensive description of the dataset can be found in [START_REF] Lubrano | Diagnosis with Confidence: Deep Learning for Reliable Classification of Squamous Lesions of the Upper Aerodigestive Tract[END_REF].

Attention MIL Architecture

We used an architecture derived from the Attention MIL model proposed by [START_REF] Ilse | Attention-based deep multiple instance learning[END_REF]. Each WSI is cut into small tiles and fed to a NN. A global label associated with the WSI is used to train the model. An attention mechanism allows to detect the most relevant tiles for the classification. Details of the architecture can be found in Appendix A.2.

Cost sensitive training

To leverage the ordinal nature of the class we used the cost aware classification loss introduced in [START_REF] Chung | Cost-aware pre-training for multiclass cost-sensitive deep learning[END_REF]: the Smooth One Sided Regression (SOSR) loss. The output of the network is a class specific risk Ĉ rather than a posterior probability. No softmax is used. The predicted class corresponds to the class minimizing this risk. The loss is defined as follows:

L SOSR = i j ln(1 + exp(2 i,j • (ĉ i -C i,j ))) (1)
With 2 i,j = -1 i =j + 1 i=j , ĉi the i-th coordinate of the network output and C the error table. The use of ordinal risk predictions is a crucial requirement for pathologists as it penalizes errors with high medical impact. In contrast, if we used a normal classification setting with cross-entropy, severe misclassifications would be likely to be more frequent. In this paper, we defined the error table as the one proposed by the TissueNet Challenge [START_REF] Loménie | Can ai predict epithelial lesion categories via automated analysis of cervical biopsies: The tissuenet challenge[END_REF]. It can be found in the Appendix A.3.

Confidence measures

3.2.1. Our method: grade sensitive confidence score Our confidence score was derived from the risk estimation vector outputted by the last layer of the network. The softmax of the inverted risk (-risk vector) was computed, turning cost estimation into probabilities. This makes the score easier to interpret and to compare, while not changing the ordering of values. The confidence score was defined as the difference between the two highest risk probabilities: if the probabilities were close, the network was hesitating between two classes, if they were far, the network was considered more confident (see Algorithm 1).

Algorithm 1: Grade sensitive confidence score (ours)

Input: Y ∈ R n , with n the number of classes Output: u, the confidence score Y ← sof tmax(-Y ); Y ← sort(Y, ascending = F alse); U ← Y 0 -Y 1 ;
We benchmarked our method against the most popular ones in medical image analysis (MC Dropout and Deep Ensembles). As a baseline we considered the raw output vector of the NN as a confidence measure.

MC Dropout

Our model was trained with dropout layers with rate r. At inference, M forward passes with dropout layers enabled were performed as described in [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF]. The confidence was derived from the standard deviation of the generated Monte-Carlo samples according to the equation 2. The standard deviation was inverted and normalized between 0 and 1 to be compared with the grade sensitive confidence measure.

U unormalized = M m=0 (Y m -Ȳ ) M -1 (2)
With Y ∈ R n the output vector, n the number of classes, M the number of MC samples.

Deep ensembles

D networks were initialized with different seeds and trained independently from each other. The confidence score was computed from the standard deviation of the D predictions made from the different networks similarly as Equation (2). Again, for comparison purposes, the standard deviation is inverted and normalized between 0 and 1.

Implementation and training details

The dataset was split in 5 training and validation sets in a stratified way to perform a 5folds cross validation. The folds were used for training and hyper-parameter optimization. The gold standard test was only used for evaluation purposes. WSI were cut in tiles of 224x224 pixels without overlap at resolution of 10X (1µm/pixel) and background tiles were removed, resulting in a total of 3.9 million of tiles (from 2121 slides). DenseNet121 was used to extract features from the tiles. This convolutional part of the network was frozen and initialized with pre-trained weights obtained from self-supervised learning. The selfsupervised architecture relied on SimCLR method [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] and is described in Appendix A.4. Dropout rates were set to 0.5. Deep Ensembles were obtained by training 5 models (D = 5) with different seeds for the 5 data folds. 50 MC samples were generated (M = 50) for the MC dropout method. Predictions were made on the gold standard test set by averaging the output vectors of the D inferences (for deep ensembles) or the M inferences (for MC dropout) and then averaged on the 5 folds. Models were trained for 100 epochs with early stopping on the validation sets. RMSProp optimizer was used with a momentum of 0.5 and a learning rate of 1e-4. At each training step, all the tiles of the randomly sampled slide were used. The implementation was done with Tensorflow (TF 2.8).

Experiments and results

Capacity to detect difficult slides

To compare the confidence measures capacity to detect the most ambiguous slides, we evaluated the classification performances on the gold standard test set for a range of confidence thresholds. As the threshold increased, more slides were considered uncertain by the model and removed from the test set. Metrics were then computed on the remaining slides. The baseline consisted in the raw risk estimates obtained when training with the SOSR loss. It was compared with the deep ensembles, the MC dropout, and our proposed grade sensitive measure. Comparison was also made with the evolution of the AUC when slides were picked randomly (Figure 1). We observed that the grade sensitive measure, raw risk and deep ensembles have similar behaviors: they consistently identify more difficult slides on which the model has more chance to make a misprediction. On the contrary, MC dropout fails to identify the most uncertain samples: the AUC is flat when removing the first slides (the most uncertain according to the MC Dropout measure), similar to random removal of slides. .

Classification performances at a particular confidence level -Screening use case

In the setting of a screening use-case (e.g. to use AI help only if the prediction is reliable), a confidence threshold has to be set. In real life cases the threshold would be set on the validation sets, based on clinical constraints: to reach a Negative Predictive Value (NPV) on the abnormal (dysplasia + carcinoma) class > 0.95 for instance. Here, to compare the metrics on the same amount of slides and ensure a fair comparison between sets and methods, we defined the threshold as the median of the confidence level. The distribution of confidence scores across classes revealed that confidence levels were always high for the classes 0 and 3, that are indeed the most simple classes to detect for pathologists. On the other hand, confidence levels are often lower on intermediary classes (1 and 2) which are more difficult to grade and the criteria for each class are difficult to assess (Appendix B.3).

In Table 1, we observe that the grade sensitive measure leads to the larger gaps between the low and high confidence subsets of slides.

Correlation with histologic ambiguity

We compared the level of confidence with the results of the dual blind review of the gold standard test set. Confidences were measured for slides on which pathologist agreed in first place and slides on which pathologist disagreed during the blind review. Confidence scores for deep ensembles and grade sensitive methods were significantly correlated with the review: higher confidence were associated with slides easier for pathologists (the one they agreed on) and lower confidence on more difficult slides (the one they disagreed on). Figure 2 suggests that the confidence level of the model correlates with the risk of disagreement between reviewers. We note that the high variability of confidence scores are expected. Indeed, if the grading of a slide was so difficult that the assignment was random, there would still be a 25% chance that the reviewers agree. If in this case the slide was given a low confidence by the score, it would lead to a low confidence score for a slide the reviewers agreed on.

As the confidence scores are measuring different aspects of the prediction process (distance measure, variance measure, risk measure), they are expected to behave differently and vary on different ranges, but this has no negative impact on the quality of the score.

Discussion and Conclusion

We benchmarked our grade sensitive confidence score with popular uncertainty measures for the difficult task of grading precancerous lesions on histopathological samples. We showed that the grade sensitive score, as well as deep ensembles scores allowed us to accurately detect poor confidence samples while MC dropout measure was actually not performing well in the grading setting (Fig. 1). This observation corresponds to what is found in the literature [START_REF] Thagaard | Can you trust predictive uncertainty under real dataset shifts in digital pathology[END_REF][START_REF] Poceviciute | Generalisation effects of predictive uncertainty estimation in deep learning for digital pathology[END_REF]. Additionally, our method led to larger gains in classification performance between low and high confidence scores (Table 1), demonstrating that it is the most appropriate method for clinical practice among the techniques tested.

Our grade sensitive method is computationally efficient and is suitable to be used in weakly supervised workflows as it does not requires any annotations. In contrast to deep ensembles that require the training of an extra 4 models to assess the confidence, multiplying by such the training and prediction time. Given that WSI classification typically relies on large datasets and requires long training times, this is an important advantage of our method. Surprisingly, the raw risk estimates were a good assessment of the confidence, managing to stratify the test set between high and low confidence subsets almost as well as deep ensembles. This goes against accepted theory that softmax output of a network is not a good uncertainty estimate [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF][START_REF] Pearce | Understanding softmax confidence and uncertainty[END_REF]. We believe this contradictory behavior can be caused by the grading setting. Indeed, in other studies, uncertainty estimates were always computed for binary tasks. In such setting, the softmax layer inevitably tends to force extreme decisions, the subtleties needed to stratify the data according to their confidence are erased by this excessive behavior. On the other hand, in the context of grading, softmax can have more difficulties forcing a class to a probability of 1, leading to a better distribution of probabilities that can thus be used more efficiently to measure a model's confidence.

We identified that the most uncertain slides according to our measure were mostly low grade (1) and high grade lesions (2), ie. intermediate classes. This observation corresponds to what pathologists experiment in practice. Indeed, the grading system of head and neck precancerous lesions has always been controversial [START_REF] Gale | Laryngeal dysplasia: persisting dilemmas, disagreements and unsolved problems-a short review[END_REF][START_REF] Slot | Laryngeal precursor lesions: Interrater and intrarater reliability of histopathological assessment[END_REF], and no less than 4 grading nomenclature have been proposed in the last decades. This illustrates the inherent difficulty of the grading task, where borders between classes are subject to noise and pathologists' own calibration.

Finally, we showed that grade sensitive, raw risks and deep ensembles measures were significantly correlated with pathologists slides agreements (Figure 2). This interesting finding demonstrates the portability of such measures to clinical practice. Indeed, the confidence is able to detect slides on which pathologists will also struggle, and probably disagree. This can be of great help for detecting more difficult slides and either filter them out or directly ask for a second opinion.

In conclusion, we proposed new approach for evaluating the reliability of AI models for WSI grading. This approach is easy to understand, cost-effective, and takes into account the inherent uncertainty of histological samples. We believe this contribution is an interesting step toward more reliable AI model to be used in clinical practice, especially for subjective tasks such as grading precancerous lesions.

A.3. Cost Matrix

Misclassification errors do not lead to equally serious consequences. Accordingly, a panel of pathologists established a grading of each of these errors i.e they attributed to each pair of possible outcome (i, j) ∈ {0, 1, 2, 3} 2 a severity score 0 C i,j 1 (Table 3).

A.4. Self-supervised Implementation Details

Our Self-supervised model relied on SimCLR architecture. It consisted of a feature extractor (DenseNet121) and a projection head (3 dense layers). The model was trained on 3.5 million tiles of 336x336 pixels from the dataset, using a batch size of 864, a temperature of 0.1, and a learning rate of 10e-4. The feature extractor was initialized with ImageNet pretrained weights and trained for 5 epochs with the feature extractor frozen. After this, the model was trained for a total of 300 hours (143 epochs) using augmentations such as cropping and resizing the tiles to 224x224 pixels, random rotations of 90 degrees, flipping, custom stain augmentation, and color jittering to adjust brightness, hue, contrast, and saturation. can have on the patient's care, cure and follow up. For instance, according to experts, diagnosing a benign case (0) instead of a low grade lesion (1) has less impact than diagnosing a high grade (2). Without any prior knowledge or experts to set error costs, one could use more standard matrices with linear or quadratic weights for instance. The following results compare performances obtained from training with the custom costs matrix and a linear cost matrix as defined below.

Linear cost matrix:

Figure 1 :

 1 Figure 1: Overall AUC evolution on the Gold Standard Test set as we remove the most uncertain slides. We measured the overall AUC (average on the 4 classes) on the same number of slides, after removing the most uncertain slides of the test set. The faster the AUC increases, the better the selected slides are. Per class performances can be visualized in Appendix B.4. Evolution with respect to the confidence score threshold is available in Appendix B.2

Figure 2 :

 2 Figure 2: Confidence score distributions compared to pathologist agreements

B. 2 .

 2 Figure 5: Evolution of the Overall AUC and the number of slides removed with respect to the confidence score threshold. Dashed line are link to the right axis (% slides remaining in the test set), et full lines are linked to the left axis (Overall AUC). This representation allows us to verify that not all the slides are removed immediately when increasing the threshold.

Figure 6 :Figure 7 :

 67 Figure 6: Confusion Matrices on high and low confidence (threshold=median) based on grade sensitive measure (left) and distribution of confidence scores for each class (right)

Figure 8 :

 8 Figure 8: ROC AUC on the Gold Standard Test set after dichotomized between High Confidence slides and low confidence slides (threshold = median, 10 000 bootstraps)

  Per class classification performances (avg +/-std on the 5 folds) on the gold standard test set Normal (0) Low Grade (1) High Grade (2) Carcinoma (3) AUC Custom Matrix Overall AUC= 0.875 +/-0.005 0.911 +/-0.0078 0.78 +/-0

Figure 10 :

 10 Figure 10: Similar to Figure 1. Overall AUC evolution on the Gold Standard Test set as we remove the most uncertain slides. Comparison with trainings done using a linear cost matrix in yellow.

Table 1 :

 1 AUC for the stratified gold standard test set in high confidence and low confidence slides. Predictions on the test set are obtained by averaging the predicted probabilities from the 5 trainings on the different folds (ensembling) and bootstraping was conducted to compute final metrics. Other metrics and confidence intervals are available in Appendix B.5

	threshold= median	AUC (Average on 4 classes)	NPV Carcinoma (3)		NPV Abnormal (1+2+3)
	1000 bootstraps	Low Confidence	High Confidence	Gap	Low Confidence	High Confidence	Gap	Low Confidence	High Confidence	Gap
	MC-Dropout	0.842	0.884	4,2%	0.950	0.936	-1,4%	0.664	0.733	6,90%
	Raw risk (SOSR)	0,797	0,934	13,70%	0,920	1.0	8,0%	0,621	0,742	12,10%
	Deep Ensembling	0.790	0.928	13,8%	0.938	1.0	6,2%	0.540	0.809	26,90%
	Grade Sensitive (Ours)	0.770	0.941	17,1%	0.920	1.0	8,0%	0.496	0.867	37,10%

Table 2 :

 2 Whole Slide Classification -Summary Table

	Layers	Type
	Feature Extractor	DenseNet121 [1024]
		Dimensionality Reduction
		FC [128] + tanh
		Dropout
		Tile Scoring
		FC [128] + softmax
	Attention-MIL	Dropout Classification
		FC [200] + relu
		Dropout
		FC [100] + relu
		Dropout
		FC [4]

Table 3 :

 3 Weighted Accuracy Error Table -Error table to ponderate misclassification according to their gap with the ground truth

	Ground Truth Benign (pred) Low-grade (pred) High-grade (pred) Carcinoma (pred)
	Benign	0.0	0.1	0.7	1.0
	Low-grade	0.1	0.0	0.3	0.7
	High-grade	0.7	0.3	0.0	0.3
	Carcinoma	1.0	0.7	0.3	0.0

Table 4 :

 4 Average Confidence values per slides for each method

	Classes	Grade Sensitive (ours) Deep Ensembles MC Dropout Raw Risk
	Benign (0)	0.74	0.79	0.47	0.25
	Low Grade (1) 0.42	0.66	0.45	0.08
	High Grade (2) 0.66	0.63	0.47	0.13
	Carcinoma (3)	0.89	0.82	0.50	0.46

LUAD = Lung Adenocarcinoma, LUSC = Lung Squamous Cell Carcinoma

Acknowledgments

ML was supported by a CIFRE PhD fellowship founded by TRIBUN HEALTH and ANRT. This work was supported by the French government under management of ANR as part of the "Investissements d'avenir" program, (ANR-19-P3IA-0001,PRAIRIE 3IA Institute).

Appendix A. Methods

B.6. Analysis on most hesitating slides, comparison between ordinal loss and cross-entropy

B.7. Comparison with different cost matrices

The cost matrix defined in the TissueNet Challenge [START_REF] Loménie | Can ai predict epithelial lesion categories via automated analysis of cervical biopsies: The tissuenet challenge[END_REF] was set by an expert consensus within a scientific council. It reflects the medical impact a diagnosis error


Amandine Bibard

Davide Martinetti

Giraud Aymeric

Albert Picado De Puig

Karine Chalvet-Monfray

Thibaud Porphyre

Risk assessment of BTV incursion in Europe from Sardinia by Culicoides spp. wind dispersal

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Risk assessment of BTV incursion in Europe from

Sardinia by Culicoides spp. wind dispersal Amandine Bibard Boehringer Ingelheim France amandine.bibard@boehringer-ingelheim.com Sardinia island (Italy) experiences regular orbiviruses incursions like serotype 3 of BlueTongue Virus (BTV) in 2018, or Epizootic Hemorrhagic Disease Virus (EHDV) in 2022. Once the disease is established in Sardinia, what is the risk of introduction for the other regions in Europe?

Long-distance dispersal of the Culicoides vectors through the wind is a major disease introduction pathway but remains rarely considered in quantitative risk assessment models. However, combination of epidemiological and meteorological models can provide useful insights on risk estimations. A methodological framework and its first application for Sardinia as the infection source, are hereby presented. It expands to the west as the vector flight duration increases but very rarely to the north.

BTV infection of vector population in west southern

 Risk of new introduction remains mostly limited to Sicily, continental Italy and Corsica.

Annual risk for continental France (southwest) and Spain (Baleares islands) was only evidenced considering 72h of flight and competence of both vector types.

 If onlyC. imicola contributes to the disease transmission, there is no significant risk of further introduction to neighbouring countries.

 Mediteranean islands are exposed almost the entire period whereas continental areas are exposed during only few weeks, at most 3 months.

Next steps:

• Finetune estimations of vectors abundance data

• Perform sensitivity analysis

• Make the model generic for any combination of source and destination

References: [1] https:/ /doi.org/10.5334/ohd.33 ; [2] 10.1046/j.1439-0450.2003.00632.x; [3] 10.1038/sdata.2018.227; [4] https://doi.org/10.1038/s41597-020-00587-y; [5] 𝑰𝑰𝑰𝑰𝑰𝑰 𝒗𝒗 , 𝒊𝒊 ~ Disease prevalence in hosts, vector abundance [1] , weekly presence probability [2] , livestock abundance [3] , vector competence [4] and other vector specific parameters Probability to be airborne from i to j

Probability to be still alive after dispersion

Disease can be established at destination only if 𝑅𝑅 0 > 1 Save my poster!



  Sardinia : 𝐼𝐼𝐼𝐼𝑓𝑓 𝑣𝑣 𝑖𝑖 First local establishment at destination : 𝑅𝑅 0 𝑗𝑗 Probability of long-distance dispersal : 𝑃𝑃 𝐿𝐿𝐷𝐷𝐷𝐷𝐷𝐷 𝑗𝑗 1. Simulate HYSPLIT® aerial trajectories from source area (i) to destination cells (j) : • daily from Week 11 (mid March) to Week 46 (mid November), named « study period » • at sunrise & sunset 2. Filter trajectories according to survival conditions of the Culicoides: o Max duration of aerial transport : scenarios for 24, 48 and 72 hours o Trajectory below planetary boundary layer o Temperature >10°C 3. Calculate frequency of trajectories reaching j as a proxy for the probability to be dispersed from i to destination j by the wind 4. Calculate probability of long distance dispersal considering ecology & biology of the Culicoides spp. × × 𝑹𝑹 𝟎𝟎 = Basic Reproduction Number 𝑅𝑅 0 = Average number of secondary infections in destination produced by one exogenous infected vector Prevalence arbitrary set in source: 50 cattle cases <=> 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑐𝑐 = 0.66% 200 small ruminant cases <=> 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑠𝑠 = 0.06% Infection in i Long distance dispersal to destination j by wind Scenario 2: only C. imicola contributes to virus transmission Introduction Risk Introduction risk in j is a combination of the 3 previous steps Risk = Number of local infections in destination j per unit of time Yearly risk of introduction at destination Scenario 1: both C. imicola & Obsoletus complex contribute to virus transmission Vector infection starts end of April and ends mid of November C. imicola: [0,5 -1,9] infected midges / week C. obsoletus/scoticus : [45 -2500] infected midges / week Mean of weekly probabilities for long distance dispersal from source area i (+) according to 3 scenarios of maximal duration of aerial transport (1, 2 or 3 days) from Turner et al. [5] consider : • different hosts populations and vectors populations • potential cross infections between vector populations Probability to be uplifted by the wind Endemic situation Number of weeks at risk for 3 neighbouring countries 48h-dispersionMean of weekly secondary infections at destination according to vector contribution to transmission The vector dispersion halo from Sardinia mostly points to the southeastern direction.

  1 ; Martinetti, Davide 2 ; Giraud, Aymeric 2 ; Picado De Puig, Albert 1 ; Chalvet-Monfray, Karine 3 ; Porphyre, Thibaud 4 1 Boehringer Ingelheim Animal Health France, Global Innovation, Transboundary and Emerging Diseases, Site de Lyon Porte des Alpes, Saint Priest, France 2 INRAE UMR 0546 Biostatistiques et Processus Spatiaux, Avignon, France 3 Université Clermont Auvergne, INRAE, VetAgro Sup, UMR EPIA, Saint-Genès-Champanelle, France 4 Université de Lyon, Université Lyon 1, CNRS, VetAgro Sup, Laboratoire de Biométrie et Biologie Evolutive, Villeurbanne, France 24h-flight 24h-dispersion