Antoine De Mathelin
email: antoine.de-mathelin-de-papigny@michelin.com

Francois Deheeger
email: francois.deheeger@michelin.com

Mathilde Mougeot
email: mathilde.mougeot@ens-paris-saclay.fr

Nicolas Vayatis
email: nicolas.vayatis@ens-paris-saclay.fr

Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification

Keywords: Deep Ensemble, Uncertainty, Out-of-distribution, Antiregularization

We consider the problem of uncertainty quantification in high dimensional regression and classification for which deep ensemble have proven to be promising methods. Recent observations have shown that deep ensemble often return overconfident estimates outside the training domain, which is a major limitation because shifted distributions are often encountered in real-life scenarios. The principal challenge for this problem is to solve the trade-off between increasing the diversity of the ensemble outputs and making accurate in-distribution predictions. In this work, we show that an ensemble of networks with large weights fitting the training data are likely to meet these two objectives. We derive a simple and practical approach to produce such ensembles, based on an original anti-regularization term penalizing small weights and a control process of the weight increase which maintains the in-distribution loss under an acceptable threshold. The developed approach does not require any out-of-distribution training data neither any trade-off hyper-parameter calibration. We derive a theoretical framework for this approach and show that the proposed optimization can be seen as a "water-filling" problem. Several experiments in both regression and classification settings highlight that Deep Anti-Regularized Ensembles (DARE) significantly improve uncertainty quantification outside the training domain in comparison to recent deep ensembles and out-of-distribution detection methods. All the conducted experiments are reproducible and the source code is available at https://github.com/antoinedemathelin/DARE.

Introduction

With the adoption of deep learning models in a variety of real-life applications such as autonomous vehicles [START_REF] Choi | Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving[END_REF][START_REF] Feng | Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection[END_REF], or industrial product certification [START_REF] Mamalet | White Paper Machine Learning in Certified Systems[END_REF], providing uncertainty quantification for their predictions becomes critical. Indeed, various adaptations of classical uncertainty quantification methods to deep learning predictions have been recently introduced as Bayesian neural networks [START_REF] Mackay | Bayesian methods for adaptive models[END_REF][START_REF] Neal | Bayesian learning for neural networks[END_REF], MC-dropout [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF], quantile regression [START_REF] Romano | Conformalized quantile regression[END_REF] and deep ensembles [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF][START_REF] Wen | Batchensemble: an alternative approach to efficient ensemble and lifelong learning[END_REF][START_REF] Wenzel | Hyperparameter ensembles for robustness and uncertainty quantification[END_REF]. These methods appear to be quite efficient in predicting the uncertainty in the training domain (the domain defined by the training set), called in-distribution or aleatoric uncertainty [START_REF] Abdar | A review of uncertainty quantification in deep learning: Techniques, applications and challenges[END_REF]. However, when dealing with data outside the training distribution, i.e. out-of-distribution data (OOD), the uncertainty estimation often appears to be overconfident [START_REF] D'angelo | Uncertainty-based out-of-distribution detection requires suitable function space priors[END_REF][START_REF] Liu | The peril of popular deep learning uncertainty estimation methods[END_REF][START_REF] Ovadia | Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift[END_REF]. This is a critical issue, because deep models are often deployed on shifted distributions [START_REF] De Mathelin | Handling distribution shift in tire design[END_REF][START_REF] Saenko | Adapting visual category models to new domains[END_REF][START_REF] Xu | Self-ensembling attention networks: Addressing domain shift for semantic segmentation[END_REF]; overconfidence on an uncontrolled domain can lead to dramatic consequences in autonomous cars or to poor industrial choices in product design.

One problem to be solved is to increase the output diversity of the ensemble in regions where no data are available during training. This is a very challenging task as neural network outputs are difficult to control outside of the training data regions. In this perspective, contrastive methods make use of real [START_REF] Pagliardini | Agree to disagree: Diversity through disagreement for better transferability[END_REF][START_REF] Tifrea | Semi-supervised novelty detection using ensembles with regularized disagreement[END_REF] or synthetic [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF][START_REF] Mehrtens | Improving robustness and calibration in ensembles with diversity regularization[END_REF][START_REF] Segonne | Robust uncertainty estimates with out-of-distribution pseudo-inputs training[END_REF] auxiliary OOD data to constrain the network output out-of-distribution. However, these approaches can not guarantee prediction diversity for unseen OOD data as the auxiliary sample may not be representative of real OODs encountered by the deployed ensemble. Another set of methods assumes that the diversity of the ensemble outputs is linked to the diversity of the networks' architectures [START_REF] Zaidi | Neural ensemble search for uncertainty estimation and dataset shift[END_REF], hyper-parameters [START_REF] Wenzel | Hyperparameter ensembles for robustness and uncertainty quantification[END_REF], internal representations [START_REF] Ramé | Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation[END_REF][START_REF] Sinha | Dibs: Diversity inducing information bottleneck in model ensembles[END_REF] or weights [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF][START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF]. The main difficulty encountered when using these approaches is to solve the trade-off between increasing the ensemble diversity and providing accurate prediction in-distribution. The current approach to deal with this issue consists in setting a trade-off parameter with hold-out validation [START_REF] Jain | Active image segmentation propagation[END_REF][START_REF] Liu | Ensemble learning via negative correlation[END_REF][START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF] which is time consuming and often penalizes the diversity.

Considering these difficulties, a question that naturally arises is how to ensure important output diversity for any unknown data region while maintaining accurate in-distribution predictions? In this work, we tackle this question with the following reasoning: an ensemble of large weights networks essentially produces outputs with large variance for all data points. Furthermore, to make accurate prediction on the training distribution, the output variance for training data needs to be reduced, which requires that some of the network's weights are also reduced. However, to prevent the output variance from being reduced anywhere other than the training domain, the network weights should then be kept as large as possible. Following this reasoning, we seek an ensemble providing accurate prediction in-distribution and keeping the weights as large as possible.

To meet these two objectives, deviating from traditional recommendations for deep learning training, we propose an anti-regularization process that consists in penalizing small weights during training optimization. To find the right trade-off between increasing the weights and providing accurate prediction in-distribution, we introduce a control process that activates or deactivates the weight increase after each batch update if the training loss is respectively under or above a threshold. Thus, an increase of the weights induces an increase of the prediction variance while the control on the loss enforces accurate in-distribution predictions. Synthetic experiments on toy datasets confirm the efficiency of our proposed approach (cf Figure 1). We observe that the uncertainty estimates of our Deep Anti-Regularized Ensembles (DARE) increase for any data point deviating from the training domain, whereas, for the vanilla deep ensemble, the uncertainty estimates remain low for some OOD regions. The contributions of the present work are the following:

-A novel and simple anti-regularization strategy is proposed to increase deep ensemble diversity. -An original control process addresses the trade-off issue between in-distribution accuracy and reliable OOD uncertainty estimates. -We provide theoretical arguments to understand DARE as a "water-filling" optimization problem where a bounded global amount of variance is dispatched among the network weights. -A new experimental setup for uncertainty quantification with shifted distribution is developed for regression. Experiments are also conducted for out-of-distribution detection following the setup of [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF].

2 Deep Anti-Regularized Ensemble

Notations

We consider the supervised learning scenario where X ⊂ R p and Y ⊂ R q are respectively the input and output space, with p, q ∈ N. The learner has access to a training sample, S = {(x 1 , y 1 ), ..., (x n , y n )} ∈ X × Y of size n ∈ N. We consider a set H of neural networks h θ ∈ H where θ ∈ R d refers to the network weights. We consider a loss function l : Y × Y → R + and define the average error of any

h θ ∈ H on S, L S (h θ ) = 1 n (xi,yi)∈S l(h θ (x i ), y i ).

Optimization formulation

The main assumption of this present work is that accurate in-distribution prediction and output diversity for OOD can be obtained by increasing the networks' weights as much as possible while maintaining the training loss under a threshold. We further assume that the variance is increased when the weights of each network is increased. Based on these assumptions, we propose the simple following optimization for every member h θ ∈ H of the ensemble:

min θ L S (h θ ) -λ R(θ) (1)
with R : R d → R + a monotone function growing with ||θ|| 2 . The parameter λ ∈ {0, 1} is a binary variable which controls the trade-off between the indistribution loss and the regularization. At each batch computation, λ is updated as follows:

λ = 1 if L S (h θ ) ≤ τ 0 if L S (h θ ) > τ (2)
with τ ∈ R a predefined threshold which defines the acceptable level of performance targeted by the learner.

-The first term of the optimization objective in Eq. ( 1): L S (h θ ) is the loss in-distribution. This term induces the network to fit the training data which implies smaller in-distribution prediction variances. -The second term -λR(θ) acts as an "anti-regularization" term which induces an increase of the network weights. This implies a larger variance of the ensemble weights, and therefore a larger prediction variance especially for data "far" from the training distribution on which the network's predictions are not conditioned.

The underlying idea of the proposed optimization is that, to fulfill both objectives: reducing the loss in-distribution and increasing the weights, large weights will appear more likely in front of neurons which are never or weakly activated by the training data. Therefore, if an out-of-distribution data point activates one of these neurons, large values are propagated through the networks which induces larger prediction variances. We show in Section 4, that this claim is supported by theoretical analysis and empirical observations. The control process is necessary to temper the weight increase, because a large increase of the weights induces an unstable network with reduced accuracy on training data. To be sure to fulfill a performance threshold τ , the weight increase is stopped (λ = 0) until the loss in-distribution comes back under the threshold. Therefore, the resulting ensemble is composed of networks fitting the training data with weights as large as possible.

Implementation

Parallel optimization. Each network of the ensemble is trained independently of the others with the objective of Eq. (1). This approach allows parallel training of the ensemble. It is theoretically possible that each network ends up reaching the same optimum resulting in no ensemble diversity. However, we observe that this degenerate case never occurs in our experiments due to the random process of the optimization and aleatoric weights initialization. Regularization function. We propose the following choice of regularization function:

R(θ) = 1 d d k=1 log(θ 2 k ) (3)
With θ = (θ 1 , ..., θ k ) the network weights. The use of the logarithmic function is motivated by the "water-filling" interpretation of DARE (cf. Section 4). Control threshold. The control threshold τ should be chosen by the learner based on his targeted error level in-distribution. Smaller τ leads to smaller increase of the weights. For τ = -∞, DARE is equivalent to a vanilla deep ensemble. A selection heuristic for τ is proposed and discussed in Section 6.

Optimization algorithm Each network is trained through batch gradient descent. The optimization algorithm is described in Algorithm 1.

Algorithm 1 Anti-Regularized Network

Theoretical Analysis

The purpose of the following theoretical analysis section is to support the fundamental claim that : "large weights do not activate on in-distribution samples while mostly activate for OOD, leading to larger output variance". We first provide theoretical insights on a simple case of linear networks which is then extended to multi-layer fully-connected networks. In a third part, an in-depth analysis of the layer activations is conducted on a syntetic dataset to illustrate this leveraged assumption.

Theoretical insights on linear networks

In the following, we propose theoretical insights for anti-regularized ensemble of linear networks (equivalent to linear regression). This theoretical result helps to understand the underlying dynamic of DARE. In particular, we show both following results which hold under appropriate assumptions:

-The weights' variance of the DARE networks is inversely proportional to the variance of the input features (Theorem 1). -As a consequence of the previous results, the DARE prediction variance is very sensitive to small deviations of low variance input features (Corollary 1).

To formalize and demonstrate these results we consider the regression problem with scalar outputs (q = 1) where X ∈ R n×p and y ∈ R n are respectively the input and output data of the training set S. We consider the ensemble of linear networks H = {x → x T θ; θ ∈ R p } such that, any h θ ∈ H is composed of two layers: the input and output layers of respective dimension p and 1. The loss function is the squared error such that for any h θ ∈ H we have nL S (h θ ) = ||Xθ -y|| 2 2 . To simplify the calculation without loosing in generality, we assume that the problem is linear, i.e. there exists θ * ∈ R p such that L S (h θ * ) = 0. We denote

s 2 = (s 2 1 , ..., s 2 p ) ∈ R * + p the diagonal of the matrix 1 n X T X.
We now consider an anti-regularized ensemble H τ produced by Algorithm 1. To characterize this ensemble, we make the following assumptions for any h θ ∈ H τ :

θ ∼ Θ σ 2 ; E[θ] = θ * , Cov(θ) = diag(σ 2 ) (4)
E [L S (h θ )] ≤ δ τ (5)
Where δ > 0 and diag(σ 2 ) is the diagonal matrix of values σ 2 ∈ R p + verifying:

σ 2 = argmax σ 2 =(σ 2 1 ,...,σ 2 p ) p k=1 log θ * k 2 + σ 2 k (6)
As presented in Assumption (4), the anti-regularized ensemble distribution is described by the random variable θ centered in θ * with variance σ 2 . Assumption (5) implies that P (L S (h θ ) ≥ τ ) ≤ δ, by Markov inequality, which models the fact that the loss of each member of DARE is maintained above a threshold τ thanks to the control process on λ (cf Section 2.2). Definition (6) approximates the impact of the anti-regularization term -R(θ) in the DARE optimization formulation with an upper bound of max σ E[R(θ)]. The weights are increased as much as possible while the loss stays under the threshold.

Our first theoretical result shows that the weight variance of the antiregularized ensemble is solution of a "water-filling" optimization problem [START_REF] Boyd | Convex optimization[END_REF], and is proportional to 1/s 2 , i.e. the inverse of the input features variance.

Theorem 1. There exist a constant C > 0 such that for any k ∈ [|1, p|], the variance of the k th weight component is expressed as follows:

σ 2 k = max C δ τ s 2 k -θ * k 2 , 0 (7)
Sketch of Proof. A detailed proof is reported in Appendix. The proof consists in first noticing that E[L S (h θ )] = p k=1 s 2 k σ 2 k .
By combining this result with Assumptions 5 and 6, we show that σ 2 is solution of the above water filling problem: maximize

σ 2 ∈R p + p k=1 log(σ 2 k + θ * k 2 ) subject to p k=1 s 2 k σ 2 k ≤ δ τ (8)
As the log function is strictly concave, and the constraints form a compact set on R p , the problem (8) has a unique solution which is given by ( 7).

The "water-filling" interpretation of the DARE optimization ( 8) is very insightful: δ τ is the "global variance capacity" that can be dispatched to the network weights. As, it grows as a function of τ , the more the learner accept a large error in-distribution, the more the global variance capacity increases. We stress that each weight component has a different "variance cost" equal to s 2 k : for high feature variance s 2 k , the increase of the corresponding weight variance σ 2 k penalizes more the training loss. Thus, large weights appear more likely in front of low variance features. Notice also that, when

θ * k 2 is high, C δ τ s 2 k -θ * k 2
is more likely to be negative, leading to σ k = 0 (cf Eq. ( 7)). Besides, θ * k 2 is generally higher for higher s 2 k as it corresponds to more informative feature, enhancing the effect σ k = 0 for large s 2 k . We see the importance of choosing a strictly concave function like the logarithm (cf Section 3), if instead of log, we choose the identity function for instance, then the solution of (8) degenerates to σ 2 = 0, ..., 0, δ τ s 2 p with s 2 p the lowest feature variance. In this case, all the weight variance is assigned to one component, which reduces the likelihood to detect a deviation of a potential OOD input on another low variance feature.

From Theorem 1, we now derive the expression of the DARE prediction variance for any data x ∈ R p : Corollary 1. Let H τ be the large weights ensemble defined by Assumptions 4, 5, 6 and x ∈ R p an input data, the variance of prediction for h θ ∈ H τ is:

Var θ∼Θ σ 2 [h θ (x)] = p k=1 x 2 k max C δ τ s 2 k -θ * k 2 , 0 (9)
We deduce from Corollary 1 that the prediction variance for x is large when the components x 2 k are large for features with low variance (s 2 k 1). Thus, the predicted uncertainty of DARE is correlated with deviations in directions in which the training input data has small variance.

Extending the results to multi-layer fully connected networks

Transposed to the hidden layers of a multi-layer neural network, Theorem 1 means that higher network weights likely appear in front of neurons which are weakly activated by the training data. To generalize Corollary 1 to the multi-layer case, we introduce an anti-regularized ensemble H τ of fully-connected deep neural networks with L hidden layers and network weights distributed according to Θ σ 2 in R d . We denote p > 0, the number of neurons of the layer ∈ [0, L + 1]. We denote g : R → R the activation function applied at each hidden layer. For every ∈ [|0, L|], we denote φ ( ) (X) ∈ R n×p the representation of the th layer, with = 0 the input layer. For any j ∈ [|1, p +1 |], we denote θ ( ,j) ∈ R p the network weights between the layer and the j th component of the layer + 1.

We assume that, for any random variable Z in R there exists α > 0 such that Var[g(Z)] > αVar[Z]. This condition is true for the leaky-relu activation, and also true for other classic activations as relu, softplus and sigmoid if Z is centered. We define φ * ( ) (X) = E[φ ( ) (X)] and denote s 2 ( ) the diagonal of the matrix 1 n φ * ( ) (X) T φ * ( ) (X). We finally assume that, there is δ > 0 such that, for any ∈ [|0, L|] and any j ∈ [|1, p +1 |], φ ( ) (X) and θ ( ,j) verify Assumptions (4), ( 5), [START_REF] D'angelo | Uncertainty-based out-of-distribution detection requires suitable function space priors[END_REF]

with θ ≡ θ ( ,j) , X ≡ φ * ( ) (X), y ≡ φ * ( ) (X)θ * ( ,j) and θ * ≡ θ * ( ,j) ≡ E[θ ( ,j) ]
Theorem 2. There exists a constant γ > 0 such that, for any h θ ∈ H τ and x ∈ R p , the prediction variance verifies:

Var θ∼Θ σ 2 [h θ (x)] ≥ γ L =0 φ * ( ) (x) 2 , σ2 ( ) With, σ2 ( ) = p +1 j=1 max C ( ,j) s 2 ( ) -θ * ( ,j) 2 , 0 ; C ( ,j) ∈ R * +
Theorem 2 states that the prediction variance for any x ∈ X is a growing function of φ * ( ) (x) 2 , 1/s 2 ( ) , which means that the uncertainty predicted by DARE is high if one hidden neuron weakly activated by the training data is strongly activated by x. Figure 2 shows an illustration of the expected behavior of DARE for one OOD data x ood . The underlying assumption is that the distance D between x ood and the learning domain is expressed on one hidden feature with low variance for the training data.

Verification on a synthetic dataset

We consider the "two-moons" binary classification dataset and use a three hiddenlayer network with 100 neurons each and ReLU activations as base network. We use a linear activation for the output layer and the mean squared error as loss function. As we consider a classification task, uncertainty can be obtained with a single network through the formula: uncertainty is the sum of the individual network members uncertainty plus the prediction variance between members. Figure 3 presents the predicted uncertainty heat-map for one DARE network as well as the internal layer representations. We observe that the OOD component values grow from one layer to the next. A correspondence between features with low variance for the training data and large weights can be clearly observed. In the last hidden layer (layer 3), the OOD components are large in direction of low training variance (components 80 to 100) to which correspond large weights. This observation explains the large uncertainty score for the OOD example.

min h θ (x) 2 , (1 -h θ (x)) 2 . The DARE

Related Works

Increasing ensemble diversity has been an enduring paradigm since the early days of the ensemble learning research. At first, diversity was seen as a key feature for improving the generalization ability of the ensembles [START_REF] Brown | Managing diversity in regression ensembles[END_REF][START_REF] Kuncheva | Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy[END_REF][START_REF] Liu | Ensemble learning via negative correlation[END_REF][START_REF] Zhang | Constraint projections for ensemble learning[END_REF]. Then, with the growing interest in uncertainty quantification, the primary objective of ensemble diversity becomes to produce good uncertainty estimates. In this perspective, a first category of methods propose to increase diversity by using diverse architectures or training conditions among an ensemble of deep neural networks [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF][START_REF] Wen | Batchensemble: an alternative approach to efficient ensemble and lifelong learning[END_REF][START_REF] Wenzel | Hyperparameter ensembles for robustness and uncertainty quantification[END_REF][START_REF] Zaidi | Neural ensemble search for uncertainty estimation and dataset shift[END_REF]. The underlying idea is that the diversity of architecture or local minima reached by the different networks induces a diversity of predictions. Another category of methods proposes to explicitly impose a diversity constraint in the loss function of the networks. The loss function is then written L + λP where L is the loss for the task (e.g. mean squared error or negative log-likelihood (NLL)), P is a penalty term which decreases with the diversity of the ensemble and λ is the trade-off parameter between the two terms. Three kinds of penalization are distinguished in the literature. The first kind makes use of training data to compute the penalty term. It includes the Negative Correlation method (NegCorr) [START_REF] Shui | Diversity regularization in deep ensembles[END_REF][START_REF] Zhang | The diversified ensemble neural network[END_REF] which applies the penalization from [START_REF] Liu | Ensemble learning via negative correlation[END_REF] to deep ensembles to enforce a negative correlation between the errors of the networks on the training set. Similarly, [START_REF] Ross | Ensembles of locally independent prediction models[END_REF] imposes an orthogonality constraint between the gradients of the ensemble members on training data. Penalizing the similarity between hidden representations of the networks has also been proposed by [START_REF] Ramé | Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation[END_REF][START_REF] Sinha | Dibs: Diversity inducing information bottleneck in model ensembles[END_REF] using adversarial training. The second kind of penalization refers to contrastive methods that enforces diversity on potential OOD instances rather than training data. This avoids the issue of being over-conditioned by the training domain that can be encountered by previous methods. In this category, several methods suppose that an unlabeled sample containing OOD is available, [START_REF] Pagliardini | Agree to disagree: Diversity through disagreement for better transferability[END_REF][START_REF] Tifrea | Semi-supervised novelty detection using ensembles with regularized disagreement[END_REF]. Others avoid this restrictive assumption and simulate potential OOD with random uniform data [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF][START_REF] Mehrtens | Improving robustness and calibration in ensembles with diversity regularization[END_REF] or instances localized around the training data [START_REF] Segonne | Robust uncertainty estimates with out-of-distribution pseudo-inputs training[END_REF]. In the last approach, considered by Anchored-Networks (AnchorNet) [START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF] and Repulsive Deep Ensemble (RDE) [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF], the penalization P is a function of the network's parameters which forces the networks to reach local minima spaced from each other in parameters space. Our proposed DARE approach relates particularly to these two methods. Our assumption is that imposing weights diversity has more chance to obtain a global output diversity rather than imposing it on specific data regions as done by the two previous kind of penalization. Anchored-Networks appears to be an efficient tool, for instance, in the detection of corrupted data [START_REF] Ulmer | Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data[END_REF], however, it is very hard to set the right anchors and trade-off parameter [START_REF] Scalia | Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction[END_REF]. Large initial variance can lead to large weight variance but may not converge to accurate model in-distribution. The DARE approach is more practical as it starts to increase the weights after reaching an acceptable loss threshold which ensures to fit the training data.

Experiments

The experiments are conducted on both regression and classification datasets. In the majority of previous works, OOD uncertainty quantification is studied in the perspective of OOD detection in the classification setting where examples from other classes / datasets are considered as OOD [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF][START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF][START_REF] Liu | A simple approach to improve single-model deep uncertainty via distance-awareness[END_REF][START_REF] Van Amersfoort | Uncertainty estimation using a single deep deterministic neural network[END_REF]. For regression, few attempts of uncertainty quantification on shifted datasets have been conducted: [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF] separates male and female faces for age prediction dataset and [START_REF] Foong | in-between'uncertainty in bayesian neural networks[END_REF][START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF][START_REF] Segonne | Robust uncertainty estimates with out-of-distribution pseudo-inputs training[END_REF] propose OOD version of several UCI regression datasets [START_REF] Dua | UCI machine learning repository[END_REF]. In this work, we propose a novel regression setup for uncertainty estimations on shifted domains based on the CityCam dataset [START_REF] Zhang | Understanding traffic density from large-scale web camera data[END_REF] which has been used in several domain adaptation regression experiments [START_REF] De Mathelin | Adversarial weighting for domain adaptation in regression[END_REF][START_REF] Zhao | Adversarial multiple source domain adaptation[END_REF]. Our setup models real-life domain shift scenarios where uncertainty quantification is challenged and offers a clear visual understanding of the domain shifts (cf Figure 4). For the classification experiments, we consider the OOD detection setup developed in [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF]. The source code of the experiments is available on GitHub3 .

Regression experiments on CityCam

We consider the CityCam dataset [START_REF] Zhang | Understanding traffic density from large-scale web camera data[END_REF]. This dataset is composed of images recorded from different traffic cameras. The task consists in counting the number of vehicles present on the image, which is useful, for instance, to control the traffic in the city. To get relevant features for the task, we use the features of the last layer of a ResNet50 [START_REF] He | Deep residual learning for image recognition[END_REF] pretrained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. We propose three different kinds of domain shift: 1. Camera-shift: This experiment uses the images from 10 cameras in the CityCam dataset. For each round, 5 cameras are randomly selected as indistribution while the 5 remaining cameras are considered as out-of-distribution.

2. Bigbus-shift: Images marked as "big-bus" referring to the fact that a bus appears and masks a significant part of the image [START_REF] Zhang | Understanding traffic density from large-scale web camera data[END_REF] are used to create the OOD dataset.

3. Weather-shift: Blurry images caused by water drops landed on the camera are used as OOD dataset.

These three experiments model real-life uncertainty quantification problems as the generalization of uncertainty estimates to unseen domains (camera-shift), the robustness to changes in data acquisition (weather-shift) and the detection of rare abnormal event (bigbus-shift). Further details on these experimental setups are provided in Appendix.

We consider the following competitors: Deep-Ensemble (DE) [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF], Neg-Corr [START_REF] Shui | Diversity regularization in deep ensembles[END_REF], AnchorNet [START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF], MOD [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF] and RDE [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF]. All are deep ensemble methods which focus on increasing the diversity among members. AnchorNet, NegCorr and MOD introduce a penalty term in the loss function multiplied by a trade-off parameter λ. The trade-off λ and the anchor initialization parameter σ for AnchorNet are selected through hold-out validation, as suggested in [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF][START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF]. The kernel bandwidth for RDE is adaptive and chosen with the median heuristic as suggested in the corresponding work [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF]. The parameter τ of DARE is chosen equal to 1 + δ times the DE validation loss. The underlying heuristic is to maintain the in-distribution performance level close to that of the vanilla DE. However, a small penalty δ > 0 should be accepted to allow the weight increase. In these experiments, we intuitively select δ = 0.25 without further fine-tuning. An ablation study of the impact of the δ parameter on the DARE performances is presented in Appendix.

The experiments are performed with ensemble of 5 fully-connected networks with 3 hidden layers of 100 neurons each and ReLU activations. The Adam optimization algorithm is used with learning rate 0.001 [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. The batch size is chosen equal to 128. The experiments are repeated 5 times to compute standard deviations for the scores. We use the NLL defines in [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF] as loss function.

Choosing the right number of optimization epochs to perform is not trivial. Previous works have used a fixed number of epochs [START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF] or early stopping [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF]. For DARE, a natural checkpoint strategy is to save the network if the validation loss of the current epoch is below the threshold τ . This process ensures a targeted level of in-distribution performance and allows the weight increase during several epochs. For the competitors, we observed that restoring the weights of the best validation loss epoch provides the best results.

To assess the ensemble quality for the regression experiments, we consider the NLL and expected calibration error (ECE) computed on the out-of-distribution dataset as used in the works [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF], [START_REF] Mehrtens | Improving robustness and calibration in ensembles with diversity regularization[END_REF]. The results are averaged over 5 runs and reported in Table 1 2: in the camera-shift experiment for instance, the in-distribution NLL of DARE is increased of only 0.18 compare to the deep ensemble method, while the NLL out-of-distribution is decreased of 2.4 (cf Table 1). The impact of the λ control in the DARE optimization is presented in Figure 5, we observe that the setting λ = 0 (equivalent to DE) leads to a small weight increase whereas the setting λ = 1 leads to a loss explosion. The control setting λ ∈ {0, 1} allows the weight increase while maintaining the loss above τ .

Classification Experiments

We consider the experimental setup defines in [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF] for OOD detection on Fashion-MNIST and CIFAR10. The MNIST dataset is used as OOD dataset for Fashion-MNIST and the SVHN dataset for CIFAR10. We extend the experiments by adding CIFAR10 as OOD for Fashion-MNIST and CIFAR100 as OOD for CI-FAR10. Thus, for both experiments, OOD detection is performed on both "Near-OOD" and "Far-OOD" datasets [START_REF] Liu | A simple approach to improve single-model deep uncertainty via distance-awareness[END_REF].

We first observed that a direct application of DARE to this setup leads to negative results. The first issue comes from the softmax activation at the end layer, which cancels the effect of increasing the weights. Indeed, the softmax activation inverses the correlation between large outputs and high uncertainty, resulting in over-confidence for OOD data. A second issue comes from the use of convolutional layers in the CIFAR10 experiment. These layers have fewer parameters than fully-connected ones, which reduces the possibility of increasing the weights in front of weakly activated neurons.

To overcome these difficulties, considering that DARE is efficient for fullyconnected networks and linear end-activation, we remove the softmax activation at the end-layer of the DARE networks and use the mean squared error. Uncertainty scores are then obtained by computing the distance between the predicted logits and the one-hot-encoded vector of the predicted class. For the CIFAR10 experiment, we use a pretrained ResNet32 and replace the end layer by deep ensemble of fully-connected networks. This setup is more practical, as only one ResNet32 is required, which reduces the need of computational resources [START_REF] Lee | Why m heads are better than one: Training a diverse ensemble of deep networks[END_REF].

The obtained results are reported in Table 3 for DARE and the competitors. To fully evaluate the impact of the DARE optimization, we add the results obtained with a Deep Ensemble trained with the mean squared error (DE (MSE)) which is equivalent to a DARE with λ = 0. We train 5 networks in each ensemble and repeat the experiments 5 times. The AUROC metric is used, computed with the the uncertainty score defined previously for DARE and DE (MSE) and the entropy for the other methods. We observe that DARE globally improves the OOD detection. For instance, in Fashion-MNSIT, we observe an improvement of 8 points on CIFAR10 and 34 points on MNIST, with a lost of only 2 points of in-distribution accuracy.

A Proof of Theorem 1

We first remind the assumptions and notations used for the Theorem: Notations:

-X ∈ R n×p and y ∈ R n are respectively the input and output data which compose the training set S.

-H = {x → x T θ; θ ∈ R p } is the ensemble of linear networks. -L S (h θ ) = 1 n ||Xθ -y|| 2 2 is the mean squared error. -s 2 = (s 2 1 , ..., s 2 p ) ∈ R * + p is the diagonal of the matrix 1 n X T X.
-Θ σ 2 is the weights distribution of the anti-regularized ensemble H τ ⊂ H, with σ 2 the weights variance.

Assumptions:

∃ θ * ∈ R p ; Xθ * = y (10)
For any h θ ∈ H τ :

θ ∼ Θ σ 2 ; E[θ] = θ * , Cov(θ) = diag(σ 2 ) (11) E [L S (h θ )] ≤ δ τ (12)
Where δ > 0 and diag(σ 2 ) is the diagonal matrix of values σ 2 ∈ R p + verifying:

σ 2 = argmax σ 2 =(σ 2 1 ,...,σ 2 p ) p k=1 log θ * k 2 + σ 2 k ( 13
)
Theorem A.1. There exist a constant C > 0 such that for any k ∈ [|1, p|], the variance of the k th weight component is expressed as follows:

σ 2 k = max C δ τ s 2 k -θ * k 2 , 0 (14)
Proof. Let's introduce the variable z ∼ N (0, diag(σ 2 )), verifying: θ = θ * + z.

From Assumption 12, we derive that:

L S (h θ ) = 1 n ||Xθ -y|| 2 2 = 1 n ||X(θ * + z) -y|| 2 2 = 1 n ||Xθ * -y + Xz|| 2 2 = 1 n ||Xz|| 2 2 (by definition of θ * ) = 1 n n i=1 p k=1 p j=1 X ik X ij z k z j (15)
Thus, we have:

E[L S (h θ )] = 1 n n i=1 p k=1 p j=1 X ik X ij E[z k z j ] = 1 n n i=1 p k=1 X 2 ik σ 2 k (by definition of z) = p k=1 1 n n i=1 X 2 ik σ 2 k = p k=1 s 2 k σ 2 k ( 16
)
Combining this results with Assumption 13, we show that σ 2 verifies: maximize

σ 2 ∈R p + p k=1 log(σ 2 k + θ * k 2 ) subject to p k=1 s 2 k σ 2 k ≤ δ τ (17)
This expression is a "water-filling" problem (cf [START_REF] Boyd | Convex optimization[END_REF] Example 5.2) with weighted constraint. The inequality constraint p k=1 s 2 k σ 2 k ≤ δ τ can be written as an equality constraint, as any increase of σ 2 k induces an increase of the objective function.

To solve this problem, we introduce the Lagrange multipliers µ ∈ R p + for the constraints σ 2 ≥ 0 and the multiplier α ∈ R for the constraint p k=1 s 2 k σ 2 k = δ τ . By considering the Lagrangian as a function of σ 2 , the KKT conditions are then written:

-1 θ * k 2 + σ 2 k + αs 2 k -µ k = 0 ∀ k ∈ [|1, p|] (18)
σ 2 k ≥ 0, µ k ≥ 0, µ k σ 2 k = 0 ∀k ∈ [|1, p|] and p k=1 s 2 k σ 2 k = δ τ (19)
Leading to:

σ 2 k -1 θ * k 2 + σ 2 k + αs 2 k = 0 ∀ k ∈ [|1, p|] (20)
σ 2 k ≥ 0, αs 2 k ≥ 1 θ * k 2 + σ 2 k , ∀k ∈ [|1, p|] and p k=1 s 2 k σ 2 k = δ τ (21)
Then, for any k ∈ [|1, p|], we have:

σ 2 k = 0 or σ 2 = 1 αs 2 k -θ * k 2 (22)
As σ 2 k ≥ 0, we deduce that:

σ 2 k = max 1 αs 2 k -θ * k 2 , 0 (23)
With α verifying:

p k=1 s 2 k max 1 αs 2 k -θ * k 2 , 0 = δ τ p k=1 max 1 αδ τ - s 2 k θ * k 2 δ τ , 0 = 1 (24)
By defining C = 1 αδ τ , we have:

p k=1 max C - s 2 k θ * k 2 δ τ , 0 = 1 (25)
Which imposes C > 0.

We conclude then, that there exists C > 0 such that, for any k ∈ [|1, p|]:

σ 2 k = max Cδ τ s 2 k -θ * k 2 , 0 (26)

B Proof of Corollary 1

Corollary B.1. Let H τ be the large weights ensemble defined by Assumptions 11, 12, 13 and x ∈ R p an input data, the variance of prediction for h θ ∈ H τ is:

Var θ∼Θ σ 2 [h θ (x)] = p k=1 x 2 k max C δ τ s 2 k -θ * k 2 , 0 (27)
Proof. Let's consider x ∈ R p and h θ ∈ H τ :

Var

θ∼Θ σ 2 [h θ (x)] = Var θ∼Θ σ 2 p k=1 x k θ k = p k=1 x 2 k σ 2 k (by Assumption 11) (28)
C Proof of Theorem 2

Notations:

-X ∈ R n×p and y ∈ R n are respectively the input and output data which compose the training set S.

D Synthetic Experiments

We consider the two synthetic experiments : Two moons classification and 1D regression :

-Two Moons Classification : 200 random data points are generated from the two moons generator4 to form the training set. We consider the three uncertainty quantification methods: Deep Ensemble (NLL), Deep Ensemble (MSE) and Deep Anti-Regularized Ensemble (DARE). The first method implements a softmax activation function at the end layer and use the multiclassication NLL as loss function. The two others use a linear activation at the end layer and the mean squared error as loss function. A fully-connected network of three layers with 100 neurons each and ReLU activations is used as base network. Each ensemble is composed of 20 networks. The Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] optimization is used with learning rate 0.001, batch size 32 and 500 epochs.

For DARE, the threshold is set to τ = 0.001. To compute the uncertainty scores, the entropy metric [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF] is used for Deep Ensemble (NLL) while Deep Ensemble (MSE) and DARE use the uncertainty metric for classification with MSE loss function defined in Appendix F. We report the uncertainty map produced by each method in Figure 6. We also report the results for the OOD detection task in Figure 7. To produce this last results, we sample 50 random points from the two moons generator which acts as validation data. We compute the uncertainty scores on the validation set and use the 95% percentiles as threshold for OOD detection. Any data points below this threshold are considered as in-distribution data (in white) and the others as OOD (in dark blue).

-1D Regression : We reproduce the synthetic univariate Regression experiment from [START_REF] Jain | Active image segmentation propagation[END_REF]. We consider the three methods: Deep Ensemble (NLL), Deep Ensemble (MSE) DARE. We use the gaussian NLL loss for regression defined in [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF] for Deep Ensemble (NLL) and DARE. The mean squared error is used as loss function for Deep Ensemble (MSE). The threshold for DARE is set to τ = 0.1. For any data x ∈ R, each method, return two values µ x and σ x (cf [START_REF] Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF]). We report in Figure 8 the confidence intervals [µ x -2σ x , µ x + 2σ x ] (in light blue) for any x in the input range.

E CityCam Experiments

We consider the three following experiments:

-Weather-shift: in this experiment, we select the images of the three cameras n°164, 166 and 572 and use the images recorded during February the 23 th . During this particular day, the weather significantly change between the beginning and the end of the day. Thus, to evaluate the robustness of the method to a change in the weather condition, we split the dataset in two subsets: we consider the images recorded before 2 pm as in-distribution and the others as out-of-distribution. we observe a shift in the weather condition around 4 pm which produces a visual shift in the images. After 4 pm, the rain starts to fall and progressively damages the visual rendering of the images due to the reverberation. This is particularly the case for camera n°164, as the water drops have landed on the camera and blur the images. Each dataset is composed of around 2500 images. -Camera-shift: this experiment uses the images from 10 cameras in the CityCam dataset. For each round, 5 cameras are randomly selected as in-distribution while the 5 remaining cameras are considered as out-ofdistribution. In average, each set is composed of around 20000 images. -Bigbus-shift: The CityCam dataset contains images marked as "big-bus" referring to the fact that a bus appears and masks an significant part of the image [START_REF] Zhang | Understanding traffic density from large-scale web camera data[END_REF]. We then select the 5 cameras for which some images are marked as "big-bus" and use this marker to split the dataset between in-distribution and out-of-distribution samples. The in-distribution set is composed of around 17000 images while the other set contains around 1000 images.

F OOD Detection Experiments

For the OOD detection experiments, we follow the setup of [START_REF] D'angelo | Repulsive deep ensembles are bayesian[END_REF]. For the Fashion-MNIST experiemnt, a three layers fully-connected network is use as base network, with ReLU activations. We use the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with learning rate 0.001 and a batch size of 128. The maximal number of epochs is set to 50. For the CIFAR10 experiment, the same base network is used on top of a pretrained ResNet32. We consider the same optimization parameters.

For DARE and Deep Ensemble (MSE), the uncertainty score is computed as follows:

OOD-score(x) = 1 M M k=1 ||h k (x) -y k || 2 2 + 1 M M k=1 h k (x) - 1 M M i=1 h i (x) 2 2 (35)
With M the number of network in the ensemble, h k the network members and y k the one-hot-encoded vector of the predicted class multiplied by the number of classes.

Fig. 1 :

 1 Fig. 1: Synthetic datasets uncertainty estimation. White points represent the training data. For each experiment, the ensemble are composed of 20 neural networks. For classification, darker areas correspond to higher predicted uncertainty. For regression, the confidence intervals for ±2σ are represented in light blue. The full experiment description is reported in Appendix.

Fig. 2 : 1 1 2 2 1 .

 2121 Fig. 2: Schematic description of DARE behavior: an OOD data x ood is distant from D > 0 from the training domain. In a hidden layer of the network h θ , the training domain is composed of a strong component φ ( ) (X) 1 of variance s 2 1

Fig. 3 :

 3 Fig. 3: Internal analysis of a large weights network The uncertainty produced by one member of an anti-regularized ensemble is presented on the left. On the right, the two figures on top present the expression of the training distribution in the three hidden layers (in blue) compared to the representation of one OOD example (in red). The components are sorted in descending order of training variance. The two bottom figures present the average weight in front of each component, i.e. the mean weights that multiply the layer components to produce the next layer representation.

Fig. 4 :

 4 Fig. 4: CityCam experimental setups. The top blue images correspond to in-distribution examples and bottom orange images to OOD examples.

Fig. 5 :

 5 Fig. 5: Training behaviors for different λ settings. Evolution of the weights, the training loss and the average trade-off parameter over the epochs ((a) ≡ DE and (b) ≡ DARE).

Fig. 6 :Fig. 7 :Fig. 8 :

 678 Fig. 6: Two-moons uncertainty estimation Darker areas correspond to higher predicted uncertainty.

Fig. 9 :

 9 Fig.9: Ablation study for the δ parameter. The DARE NLL is reported for different values of δ with τ = 1 + δ. We globally observe a decrease of the OOD NLL and a sligth increase of the in-distribution NLL when δ increases.

Table 1 :

 1 . We observe that DARE outperforms other uncertainty quantification methods in both term of NLL and ECE for the bigbus-shift and camera-shift experiments. For the weather-shift experiment, DARE provides Out-of-distribution NLL and ECE for CityCam

		Camera-shift	Bigbus-shift	Weather-shift
	Methods	NLL	ECE	NLL	ECE	NLL	ECE
	DE	6.04 (0.40) 0.28 (0.02) 3.66 (0.17) 0.28 (0.01) 2.25 (0.09) 0.12 (0.02)
	AnchoreNet	6.03 (0.24) 0.28 (0.02) 3.86 (0.16) 0.29 (0.01) 2.25 (0.12) 0.12 (0.03)
	MOD	6.24 (0.89) 0.29 (0.02) 3.66 (0.21) 0.29 (0.01) 2.32 (0.17) 0.15 (0.03)
	NegCorr	5.97 (0.54) 0.29 (0.03) 4.02 (0.27) 0.29 (0.01) 2.30 (0.10) 0.14 (0.02)
	RDE	5.05 (0.25) 0.27 (0.02) 3.67 (0.11) 0.28 (0.00) 2.22 (0.14) 0.13 (0.04)
	DARE	3.64 (0.34) 0.23 (0.03) 2.99 (0.13) 0.25 (0.01) 2.33 (0.13) 0.07 (0.04)
		DE	AnchorNet	MOD	NegCorr	RDE	DARE
	Cameras	1.03 (0.08) 1.00 (0.07) 1.04 (0.06) 1.04 (0.06) 1.06 (0.06) 1.18 (0.09)
	Bigbus	0.89 (0.03) 0.87 (0.02) 0.91 (0.04) 0.89 (0.06) 0.91 (0.03) 1.06 (0.04)
	Weather	1.25 (0.04) 1.24 (0.05) 1.26 (0.05) 1.26 (0.06) 1.28 (0.05) 1.27 (0.06)

Table 2 :

 2 In-distribution NLL for CityCam the best ECE score. Moreover, these gains in uncertainty estimation come with little penalization of the performance in-distribution as reported in Table

Table 3 :

 3 OOD detection results. AUROC scores and ID accuracy are reported7 Limitations and PerspectivesFor now, the efficiency of DARE is limited to fully-connected neural network with piece-wise linear activation. We have seen, however, that DARE can benefit to a final fully-connected network head placed on top of deep features obtained with convolutions. Thanks to the practical aspect of DARE, the method can be combined to other deep ensemble or OOD detection methods. One can use a specific training process and then apply DARE afterwards to increase diversity. Future work will consider a Bayesian version of DARE by adding gaussian noise with increasing variance to the weights of pretrained networks.

https://github.com/antoinedemathelin/DARE

https://scikit-learn.org/stable/modules/generated/sklearn.datasets. make_moons.html

-H τ is an anti-regularized ensemble of fully-connected deep neural networks with L hidden layers and network weights distributed according to Θ σ 2 in R d . -For any ∈ [|0, L + 1|], p is the number of neurons of the th layer.

g : R → R is the activation function applied at each hidden layer.

-For every ∈ [|0, L|], we denote φ ( ) (X) ∈ R n×p the representation of the th layer, with = 0 the input layer.

-For any j ∈ [|1, p +1 |], we denote θ ( ,j) ∈ R p the network weights between the layer and the j th component of the layer + 1.

-

Assumptions:

For any random variable Z in R there exists α > 0 such that:

There is δ > 0 such that, for any ∈ [|0, L|] and any

φ ( ) (X) and θ ( ,j) verify Assumptions 11, 12, 13 with

Theorem C.1. There exists a constant γ > 0 such that, for any h θ ∈ H τ and x ∈ R p , the prediction variance verifies:

Considering Assumption 30, we apply Theorem A.1 to φ ( ) (x) j for any ∈ [|0, L|] and any j ∈ [|1, p +1 |], such that:

With C ( ,j) ∈ R * + . Notice that the max function is applied element-wise.

For any ∈ [|0, L -1|] and any j ∈ [|1, p +1 |], the following inequalities are verified:

Var φ ( +1) (x) j = Var g φ ( ) (x)θ ( ,j)

≥ αVar φ ( ) (x)θ ( ,j) (by Assumption 29)

With γ ( ,j) ≥ min

The fourth inequality comes from the variance decomposition of the product of two independent variables:

The last inequality in [START_REF] Romano | Conformalized quantile regression[END_REF] provides an expression of recurrence for Var φ ( ) (x) j . We notice that, Var φ (0) (x) j = Var [x j ] = 0 for any j ∈ [|1, p 0 |]. For the last layer, we have:

γ ( ,j) and γ = min[1, γ, γα L ] > 0. We then deduce, by considering the recurence expression, that:

With, σ2


Antoine De Mathelin
email: antoine.de-mathelin-de-papigny@michelin.com

Mounir Atiq
email: mounir.atiq@ens-paris-saclay.fr

Guillaume Richard
email: guillaume.richard@ens-paris-saclay.fr

Alejandro De La Concha
email: alejandro.delaconchaduarte@ens-paris-saclay.fr

Mouad Yachouti
email: mouad.yachouti@ens-paris-saclay.fr

François Deheeger
email: francois.deheeger@michelin.com

Mathilde Mougeot
email: mathilde.mougeot@ens-paris-saclay.fr

Nicolas Vayatis
email: nicolas.vayatis@ens-paris-saclay.fr

ADAPT : Awesome Domain Adaptation Python Toolbox

Keywords: Domain Adaptation, Transfer learning, Deep networks, Importance weighting, Fine tuning, Machine learning, Python

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction and Motivation

Transfer learning and domain adaptation (DA) aim to correct the shifts that exist between the training distribution of a machine learning model (referred as source) and the target distribution on which the model is deployed. This research field has known an important development over the past decades. The presence of "domain shifts" between source and target distributions is the typical framework encountered in real-life applications which makes domain adaptation algorithms particularly usefull for numerous use-cases. Domain adaptation techniques are often needed in scenarios where labels are easily available on a source domain but are expensive on the target domain. For example, it may be used to leverage information from a synthetic dataset to build a classifier for real data [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF] such as the adaptation of GTA images for autonomous car segmentation [START_REF] Saito | Maximum classifier discrepancy for unsupervised domain adaptation[END_REF]. Furthermore, domain adaptation is of great interest to correct bias from the training samples such as sample bias [START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF] when one population in the training set is over-represented with respect to the overall population. DA methods are also used to correct shifts in the input features, caused by sensor or technological drifts [START_REF] Courty | Optimal transport for domain adaptation[END_REF]. Finally domain adaptation is useful for specifying a pre-trained model on a sub-task with few labels, as segmentation of medical images [START_REF] Ravishankar | Understanding the mechanisms of deep transfer learning for medical images[END_REF].

Nowadays, domain adaptation has become an essential tool to handle domain shifts in real applications. Many DA methods have been developed in recent years and a large number of DA implementations are spread on the web. Paradoxically, the large number of DA variants makes their accessibility to users more difficult. Indeed, finding which methods will best fit a given domain adaptation problem is a difficult task. In practice, the user would like to try different methods and select the most promising one. However, most DA algorithms available on the web are not implemented under the same basis, some of them rely on PyTorch (Paszke et al., 2019), others use scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] or TensorFlow [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF]. Moreover, most open source implementations have for primary objective to reproduce the experimental results of a particular publication and an extra effort is then required to apply them on other data. Facing these difficulties, we propose ADAPT 1 , a new open-source Python library compatible with scikit-learn and TensorFlow. This library aims at facilitating the access to transfer learning and domain adaptation methods to a wide public including industrial practitioners. Inspired by the scikit-learn library, which manages to make machine learning accessible to everyone, ADAPT offers DA methods that can be easily used in the same format as each object implements the fit, predict and score functions as any scikit-learn estimator. In a model deployment perspective, the ADAPT objects are compatible with the convenient features of scikit-learn as cloning and gridsearch. DA specific metrics are provided to allow an unsupervised selection of hyper-parameters. Finally, as we consider that the accessibility of a DA method essentially relies on the user understanding, a very detailed documentation is available online with many real and synthetic examples. A user guide is provided to allow newcomers to find the right DA method for their problem based on practical considerations (see the ADAPT flowchart 2 ). ADAPT offers the possibility of developing its own transfer method easily by subclassing existing classes. The library works on Linux, Mac and Windows for the four latest Python versions 3.6 to 3.9. ADAPT is nowadays appreciated by various users both from the academic and industrial world, with more than 1k downloads by month.

Existing transfer libraries

The emergence of domain adaptation algorithms first started around 2006. Progressively, many algorithms have been developed and open source implementations have been released. At some point, it became necessary to group several algorithms together to allow their comparison by the machine learning community. Some compilations of algorithms have then been developed as libTLDA (Wouter, 2015) or DA-Toolbox [START_REF] Yan | Domain adaptation toolbox[END_REF]. These libraries implement "classic" DA methods, i.e which do not use deep learning, as KMM [START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF] or TCA [START_REF] Sinno Jialin Pan | Domain adaptation via transfer component analysis[END_REF]. These first attempts to group DA methods offered the opportunity to quickly test several methods on a same basis. However the proposed libraries were lacking of documentation and modularity. After that, many deep DA methods have been developed and some libraries have then proposed to group several variants under the same repository such as salad [START_REF] Schneider | Salad: A toolbox for semi-supervised adaptive learning across domains[END_REF]. Since then, many repositories for deep learning have been released, the most notable one being TLlib (Junguang Jiang, 2020) which makes the great work of regrouping more than 40 deep DA algorithms. It should be noted that all these repositories propose PyTorch implementations and are designed in a benchmark purpose, i.e. they mainly focus on comparing the results of each variant on well known datasets. Using these repositories on the user dataset often requires an extra effort. In 2021, ADAPT has been released, with the purpose to open the DA algorithms to newcomers and, in particular, industrial players. For this purpose, we group the main "classic" and "deep" DA methods into a same library and provide a very detailed documentation and robust code, tested through unit tests. The library is available on PyPI, offering more than 30 algorithms implemented in a user friendly scikit-learn style (cf Table 1).

Organization

The ADAPT library is divided into three modules : feature-based, instance-based and parameter-based corresponding to the three main DA strategies. A list of all implemented methods is presented in Table 2 (Appendix A). As mentioned in the DA survey [START_REF] Weiss | A survey of transfer learning[END_REF], some DA methods are based on the use of a small number of labeled target data and are referred as supervised domain adaptation methods (SDA), others use only unlabeled target data along with the sources (UDA). Some methods perform the adaptation and the learning of the task in one stage, others in two.

Feature-Based Methods

The purpose of feature-based DA methods (Figure 1) is to learn a new representation of the input features in which both source and target distribution match. This DA strategy is mostly used for unsupervised DA. Feature-based methods generally consider the assumption that the domain shift is due to an unknown transformation of the input space caused for instance by sensor drifts or any changes in the acquisition conditions [START_REF] Courty | Optimal transport for domain adaptation[END_REF][START_REF] Ganin | Domain-adversarial training of neural networks[END_REF].

Instance-Based

The goal of instance-based methods (Figure 2) is to perform a reweighting of source instances in order to correct the difference between source and target distributions. This kind of methods are mostly used in sample bias scenario and assume that source and target distribution share the same support in the input space [START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF][START_REF] Sugiyama | Direct importance estimation with model selection and its application to covariate shift adaptation[END_REF].

Parameter-Based

Parameter-based methods (Figure 3), also called "source-free DA", aim to adapt the parameters of a pre-trained source model to a few target observations. These DA methods are mostly used in computer vision where deep model trained on huge data sets are fine-tuned on a smaller data set of images for a specific task [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF].

Installation and Usage

ADAPT provides several widely used domain adaptation methods using different approaches. The provided methods allow to cope with the main DA settings encountered in real applications as Supervised DA and Unsupervised DA which respectively refer to the cases (a) Applying CORAL (Sun et al., 2016) under Unsupervised DA.

(b) Applying TrAdaBoost [START_REF] Dai | Boosting for transfer learning[END_REF] under Supervised DA.

(c) Applying FineTuning [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF] under Source-free DA.

ADAPT Guidelines

The API is written in pure Python using scikit-learn, SciPy, NumPy, TensorFlow and cvxopt. The principal features of the API are given below:

• Documentation: Each method is documented following the standards of scikit-learn. Algorithms explanations are provided along with a full description of the parameters.

For each proposed method, illustrative examples are given on both synthetic datasets and real DA problems to offer visual understanding of the methods and empirical comparisons on known DA issues.

• Code Quality : checkers are used in all implemented objects to ensure that arguments defined by the user are valid and throw comprehensive warnings and exceptions to help the user. The code is tested with an high coverage and illustrative examples visually show that the methods are behaving as expected.

• Developer: ADAPT is released under a BSD2 License on GitHub. Anyone can contribute to the project by reporting issues and/or making pull requests. A Developer Guide is given to help DA researcher to include their works. Continuous integration is implemented to check the code compliance with unit tests.

Conclusion and Future work

Since its release, ADAPT has already been used for several research and industrial problems as fall detection [START_REF] Minvielle | Transfer learning on decision tree with class imbalance[END_REF], tire design (Mathelin et al., 2021) and even for cosmology applications [START_REF] Gilda | Unsupervised domain adaptation for constraining star formation histories[END_REF]. Future work will focus on adding more diversified algorithms to handle multisource and semi-supervised domain adaptation.

Figure 1 :

 1 Figure 1: Feature-based strategy illustration.

Figure 2 :

 2 Figure 2: Instance-based strategy illustration

Figure 3 :

 3 Figure 3: Parameter-based strategy illustration

Figure 4 :

 4 Figure 4: Examples of ADAPT usage in three different settings. X s , y s are referring to the source data and X t , y t to the target data.

Appendix A : list of implemented methods [START_REF] Uguroglu | Feature selection for transfer learning[END_REF] UDA 2-stages SA [START_REF] Fernando | Unsupervised visual domain adaptation using subspace alignment[END_REF] UDA 2-stages Feature CORAL (Sun et al., 2016) UDA 2-stages DeepCORAL (Sun and Saenko, 2016) UDA 1-stage DANN [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF] UDA 1-stage ADDA [START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF] UDA 2-stage WDGRL [START_REF] Shen | Wasserstein distance guided representation learning for domain adaptation[END_REF] UDA 1-stage CCSA [START_REF] Motiian | Unified deep supervised domain adaptation and generalization[END_REF] SDA 1-stage CDAN (Long et al., 2018) UDA 1-stage MCD [START_REF] Saito | Maximum classifier discrepancy for unsupervised domain adaptation[END_REF] UDA 1-stage MDD [START_REF] Zhang | Bridging theory and algorithm for domain adaptation[END_REF] UDA 1-stage KMM [START_REF] Huang | Correcting sample selection bias by unlabeled data[END_REF]) UDA 2-stages KLIEP [START_REF] Sugiyama | Direct importance estimation with model selection and its application to covariate shift adaptation[END_REF] UDA 2-stages TrAdaBoost [START_REF] Dai | Boosting for transfer learning[END_REF] SDA 1-stage IWC [START_REF] Bickel | Discriminative learning for differing training and test distributions[END_REF] UDA 2-stages LDM [START_REF] Mansour | Domain adaptation: Learning bounds and algorithms[END_REF] UDA 2-stages ULSIF [START_REF] Kanamori | A least-squares approach to direct importance estimation[END_REF] Regular Transfer NN [START_REF] Chelba | Adaptation of maximum entropy capitalizer: Little data can help a lot[END_REF]) SDA 1-stage Fine-Tuning [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF] SDA 1-stage SER-STRUT [START_REF] Segev | Learn on source, refine on target: A model transfer learning framework with random forests[END_REF] SDA 1-stage SER*-STRUT* [START_REF] Minvielle | Transfer learning on decision tree with class imbalance[END_REF] SDA 1-stage