Antoine De Mathelin

Antoine De-Mathelin-De

Franc ¸ois Deheeger
email: francois.deheeger@michelin.com

Mathilde Mougeot
email: mathilde.mougeot@

Nicolas Vayatis
email: nicolas.vayatis@

Maximum Weight Entropy

Keywords: Epistemic Uncertainty, Out-of-distribution detection, Deep Ensemble, Bayesian Neural Networks, Maximum Entropy 1

. Considering that this issue is mainly related to a lack of weight diversity, we claim that standard methods sample in "over-restricted" regions of the weight space due to the use of "over-regularization" processes, such as weight decay and zero-mean centered Gaussian priors. We propose to solve the problem by adopting the maximum entropy principle for the weight distribution, with the underlying idea to maximize the weight diversity. Under this paradigm, the epistemic uncertainty is described by the weight distribution of maximal entropy that produces neural networks "consistent" with the training observations. Considering stochastic neural networks, a practical optimization is derived to build such a distribution, defined as a trade-off between the average empirical risk and the weight distribution entropy. We develop a novel weight parameterization for the stochastic model, based on the singular value decomposition of the neural network's hidden representations, which enables a large increase of the weight entropy for a small empirical risk penalization. We provide both theoretical and numerical results to assess the efficiency of the approach. In particular, the proposed algorithm appears in the top three best methods in all configurations of an extensive out-of-distribution detection benchmark including more than thirty competitors.

Introduction

In many practical deep learning scenarios, neural network models are deployed on unknown data distributions that can significantly differ from the training distribution. For instance, when building deep learning models of object detection for autonomous cars, the training dataset cannot cover any potential situation that the model can encounter, in terms of weather conditions, geography or camera obstructions for examples. In this context, the learner aims at providing confidence guarantees on the model prediction for any data belonging to the whole input space. This task is related to uncertainty quantification and out-of-distribution (OOD) detection for deep learning [START_REF] Abdar | A review of uncertainty quantification in deep learning: Techniques, applications and challenges[END_REF][START_REF] Shen | Towards out-of-distribution generalization: A survey[END_REF]. In this research area, the general framework is depicted by an input and output spaces X , Y, a training set S containing several paired observations (x, y) ∈ X × Y, drawn independently of the training distribution p(x, y), and a hypothesis set H of neural networks of specified architecture mapping X to Y. The primary goal is to find the hypothesis h * in H with the best predictive power on X . To provide an approximation of h * , the learner typically considers a hypothesis ĥ with low empirical risk on S, computed through empirical risk minimization algorithms. In the epistemic uncertainty quantification framework [START_REF] Kendall | What uncertainties do we need in bayesian deep learning for computer vision[END_REF][START_REF] Hüllermeier | Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods[END_REF], the learner aims at estimating, for any input x ∈ X , the potential discrepancy between the predicted value ĥ(x) and the best possible prediction h * (x). When dealing with neural network hypotheses, the set H is typically very large and many different hypotheses may provide low empirical risk on the training set S. Informally, this collection of consistent hypotheses form a subset H S ⊂ H which provides probable candidates for the best hypotheses h * . Prediction uncertainty for a novel input observation x ∈ X is then described by the prediction diversity of the consistent hypotheses: {h(x); h ∈ H S } [START_REF] Hüllermeier | Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods[END_REF].

In the case of universal approximators such as neural networks, epistemic uncertainty is related to the distance between a new test instance and previous training examples. Indeed, for an input instance x ∈ X far from the support of the training data, there are likely two consistent hypotheses h, h ′ ∈ H S that produce very different outputs for x. More precisely, if H is the set of k-Lipschitz functions, the error on x between any consistent hypothesis h ∈ H S and the best model is bounded by a value proportional to the distance between x and the training inputs [START_REF] John Sullivan | Optimal uncertainty quantification for legacy data observations of lipschitz functions[END_REF][START_REF] Malherbe | Global optimization of lipschitz functions[END_REF][START_REF] De Mathelin | Discrepancybased active learning for domain adaptation[END_REF]. Therefore, a proxy of the epistemic uncertainty can be estimated by computing the distance to the support of the training set. Methods developed under this paradigm are referred to as distance-based uncertainty quantifiers, which includes, for instance, derivative of Gaussian processes [START_REF] Edward | Gaussian processes in machine learning[END_REF], Deterministic Uncertainty Quantification (DUQ) [START_REF] Van Amersfoort | Uncertainty estimation using a single deep deterministic neural network[END_REF], Mahalanobis distance (Lee et al., 2018b) or Deep Nearest Neighbors (Sun et al., 2022). The main challenge faced by distance-based uncertainty approaches is to find a relevant notion of distance to use [START_REF] Liu | A simple approach to improve single-model deep uncertainty via distance-awareness[END_REF]. For high-dimensional machine learning problems, using the Euclidean distance in the input space X is generally irrelevant and one looks for geometric distances computed in encoded spaces. For instance, [START_REF] Liu | A simple approach to improve single-model deep uncertainty via distance-awareness[END_REF] and [START_REF] Cao | Deep hybrid models for out-of-distribution detection[END_REF] develop distance preserving networks using spectral normalization. Finally, computing the distance to the training distribution support can also be performed by density estimation techniques, such as auto-encoders or GANs, which have been used for OOD detection [START_REF] Zhou | Rethinking reconstruction autoencoder-based out-of-distribution detection[END_REF][START_REF] Ryu | Out-of-domain detection based on generative adversarial network[END_REF]. The distance to the training set is then computed through the reconstruction error of the decoder or by the predicted likelihood of the discriminator.

The main alternative to distance-based approach consists in directly looking for a set of hypotheses that are coherent with the observations and to use the diversity of their predictions as uncertainties. It essentially includes ensemble and Bayesian methods [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF]Mackay, 1992). The ongoing challenge of this approach is to produce diversity in the ensemble of networks, i.e. to avoid sampling similar hypotheses. It has been observed, indeed, that most of the main baselines lead to a lack of prediction diversity, in particular outside the training support, i.e. for out-of-distribution data [START_REF] Ovadia | Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift[END_REF][START_REF] Liu | The peril of popular deep learning uncertainty estimation methods[END_REF][START_REF] Henning | Are bayesian neural networks intrinsically good at out-of-distribution detection?[END_REF]. Facing this issue, several attempts propose to increase the prediction diversity by adding a penalizing term to the loss. For instance, negative correlation methods penalize the correlation between the outputs of the ensemble members on the training data [START_REF] Liu | Ensemble learning via negative correlation[END_REF][START_REF] Shui | Diversity regularization in deep ensembles[END_REF][START_REF] Zhang | The diversified ensemble neural network[END_REF]. Related methods, referred to as contrastive approaches, penalize small output variances on synthetic OOD data produced by sampling uniformly in the input space [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF][START_REF] Alexander Mehrtens | Improving robustness and calibration in ensembles with diversity regularization[END_REF] or in the neighborhood of the training instances [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF][START_REF] Segonne | Robust uncertainty estimates with out-of-distribution pseudo-inputs training[END_REF]. The drawback of these methods is the lack of generalization to any OOD data that the model can encounter [START_REF] Cao | Deep hybrid models for out-of-distribution detection[END_REF]. Alternative approaches consist in penalizing the similarity between the ensemble members in the parameter space [START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF][START_REF] Francesco | Repulsive deep ensembles are bayesian[END_REF], with the underlying assumption that an ensemble of neural networks with weights distant from each other produce diversified outputs. Under this paradigm, a recent method, called Deep Anti-Regularized Ensemble (DARE), proposes an anti-regularizing process which penalizes small weights in the network while maintaining the training loss under an acceptable threshold [START_REF] De Mathelin | Deep antiregularized ensembles provide reliable out-of-distribution uncertainty quantification[END_REF]. The authors advocate that this technique provides a sample of hypotheses at the edge of the set of consistent hypotheses, resulting in increased prediction diversity, especially for OOD data. Building on this previous work, we claim that the key feature for producing accurate uncertainty quantification for any data point x ∈ X is to sample in the whole space of consistent hypotheses. Indeed, we argue that standard Bayesian and ensemble methods often provide over-confident predictions for OOD data because the hypotheses they produce are sampled in restricted regions of the consistent hypothesis space due to over-regularization processes and hyper-parameters selection based on hold-out validation.

Considering stochastic neural networks with parameterized weight distribution [START_REF] Valentin Jospin | Hands-on bayesian neural networks-a tutorial for deep learning users[END_REF], we cast the problem as a trade-off between sampling in low empirical risk regions and increasing the weight diversity. We consider the entropy as a measure of weight diversity, and show that the optimization boils down to solving a maximum entropy problem [START_REF] Edwin | Prior probabilities[END_REF], where we aim at selecting the weight distribution of maximal entropy under the constraint that the training loss is acceptable. We derive a practical optimization formulation to solve this problem, called Maximum Weight Entropy (MaxWEnt), and show that it can be tackled with stochastic variational inference [START_REF] Matthew D Hoffman | Stochastic variational inference[END_REF] using the reparameterization trick [START_REF] Diederik | Auto-encoding variational bayes[END_REF]. The proposed optimization consists in penalizing the training loss with a term imposing the increase of the weight distribution entropy. We provide a theoretical framework to understand the dynamics of this approach and show that the spread of the weight distribution is inversely proportional to the neuron activation amplitude for the training data, which extends the theoretical analysis of DARE to stochastic neural networks. The entropic penalization of MaxWEnt can then be interpreted as an anti-regularization, enforcing the weight distribution to cover the whole set of consistent weights. Numerical experiments conducted on several regression and classification datasets demonstrate the strong benefits of this approach in OOD detection compared to state-of-the-art methods dedicated to this task.

Figure 1 presents the comparison of MaxWEnt with the main baselines Deep Ensemble [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF] and MC-Dropout [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF] on a classification and a regression synthetic datasets. We observe that Deep Ensemble and MC-Dropout produce overconfident estimation outside the training support due to a lack of hypothesis diversity. In the classification experiment, for instance, the hypotheses produced by both methods are restricted to half-space separators. There is no prediction uncertainty in the upper left and lower right areas of the input space, despite the lack of training data in these regions (cf. top Figures 1.a and 1.b). In contrast, MaxWEnt provides a clear discrimination between the in-distribution and out-of-distribution domains in terms of prediction uncertainty. In Figure 1.c, the uncertainties produced by MaxWEnt are reported when no regularity assumption is made on the labeling function. In this case, we observe that the uncertainty quickly increases when leaving the training support, which truly represents the epistemic uncertainty in the absence of prior knowledge about the labeling function. Figure 1.d reports the MaxWEnt uncertainty estimation when considering Lipschitz constraints. These results can be obtained with a small modification of the previous MaxWEnt model in the form of weight clipping. The full description of these synthetic experiments is reported in Section 7.1. Below: 1D-regression [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF]. For classification, uncertainty estimates, in shades of blue, are computed with the average of prediction entropy over multiple predictions (darker areas correspond to higher uncertainty). For regression, the ground-truth is represented in black and the predicted confidence intervals of length 4σ w (x) in light blue, with σ w (x) computed as the standard deviation over multiple predictions.

Setup and Objective

Notations

We consider the supervised learning framework provided with the input space X of finite dimension b ∈ N, and the output space Y. We denote by p * (y|x) the "ground truth" conditional law defined over Y for any x ∈ X . Furthermore, we distinguish the in-distribution and out-of-distribution domains by considering that only a subset D X ⊂ X can be sampled. The subset D X is called "training domain" and any data from the complementary X \ D X is considered as "out-of-distribution". We assume that the learner has access to the training set S = {(x 1 , y 1 ), ..., (x n , y n )} ∈ D X × Y of size n ∈ N where the training instances (x i , y i ) are supposed independently identically distributed (iid) according to the joint distribution p(x, y) defined over D X ×Y and verifying p(y|x) = p * (y|x) ∀ x ∈ D X . We consider a continuous loss function ℓ : Y × Y → R + and define the optimal predictor f * : X → Y as follows:

f * (x) = argmin y ′ ∈Y y∈Y ℓ(y ′ , y) dp * (y|x). (1)
We denote H the set of neural networks of a specified architecture, mapping X to Y. The set H is assumed to be "large". We denote W ⊂ R d (d ∈ N) the set of weights corresponding to the hypotheses in H. For any h ∈ H, we define the empirical risk as follows:

L S (h) = 1 n (x,y)∈S ℓ(h(x), y), (2)
denoted indifferently L S (w), when considering the weights w ∈ W associated to the hypothesis h ∈ H, also referred as h w . Finally, we consider a metric over the space of functions mapping X to Y, denoted |||., .|||, and define the best hypothesis h * as follows:

h * = argmin h∈H |||h, f * |||. (3)
2.2 The epistemic uncertainty is described by the set of consistent hypotheses

In this work, we distinguish the following four sources of uncertainty:

1. Aleatoric uncertainty: the intrinsic random noise of the data, i.e. p * (y|x). This uncertainty cannot be reduced, even with an infinite number of observations (e.g. outcome of a coin flip).

2. Model uncertainty: the discrepancy between f * and h * . The model uncertainty is related to the choice of hypothesis set H. It can be reduced by increasing the size of H or by acquiring prior knowledge about f * (e.g. Lipschitz constraint).

3. Statistical uncertainty: the partial knowledge about p(x, y) given by the finite number of data S. This uncertainty, also referred as approximation uncertainty [START_REF] Hüllermeier | Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods[END_REF] or data variability (Huang et al., 2021b), is linked to the discrepancy between h * and its estimation. It can be reduced by the acquisition of novel observations drawn according to p(x, y) or by prior knowledge about the intrinsic random noise (e.g. Gaussian homoscedastic noise of known variance).

4. Out-of-distribution uncertainty: the absence of observation over the out-of-distribution domain X \ D X . This uncertainty can remain large even with an infinite number of training observations. Indeed, for complex hypotheses as neural networks, different hypotheses can match h * (x) on D X but produce different outputs on X \ D X .

The first three sources of uncertainty are described in details in [START_REF] Hüllermeier | Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods[END_REF], sources (2) and (3) are referred to as epistemic uncertainty, and are related to the lack of knowledge about f * . Source ( 4) is an additional distinction of the epistemic uncertainty, similar to the setup introduced in [START_REF] Liu | A simple approach to improve single-model deep uncertainty via distance-awareness[END_REF]. This distinction is useful to understand the out-of-distribution detection task. In the following, we focus our uncertainty estimation on the epistemic uncertainty (sources (2-4)), moreover, considering the denseness property of neural networks, we assume that f * is close to H, i.e. h * ≃ f * , and then neglect the model uncertainty. Our work then focus on the two last sources, which are related to the indetermination of the best hypothesis h * . The goal is then to model this epistemic uncertainty for any x ∈ X through a distribution in the label space Y. Because of lack of complete knowledge, the learner cannot perfectly determine the best hypothesis h * and then the best predictions h * (x). If no data is available, the prediction uncertainty for x ∈ X is given by the distribution of the predicted values h(x) for all hypotheses h ∈ H. When acquiring more observations, the learner can discriminate between relevant and irrelevant candidates for h * , i.e. between "consistent" and "inconsistent" hypotheses with respect to the observations S (assuming that a notion of "consistency" can be formally defined). By denoting H S the set of consistent hypotheses, the epistemic uncertainty for the prediction of the model for x is then given by the distribution of predictions h(x) with h ∼ H S .

The notion of consistency depends on the underlying assumption that the learner considers about the data sample S. A strong assumption is the "no noise" framework, where the learner assumes that the best hypothesis necessarily verifies h * (x) = y for any (x, y) ∈ S. In this case, the set of consistent hypotheses is the set: H S = {h ∈ H; h(x) = y} [START_REF] Tom | Version spaces: A candidate elimination approach to rule learning[END_REF]. In general, the learner assumes a moderated noise level. Then, the notion of consistency is related to the empirical error L S (h), such that consistent hypotheses provide "low" empirical error on S. For instance, if the learner is only interested in deploying models with greater accuracy than τ = 0.99, then the set of consistent hypotheses is defined as H S = {h ∈ H; L S (h) ≤ 1 -τ } (assuming that ℓ is the 0-1 loss). In the Bayesian setting, a noise model, p(y|x, h) is generally assumed (e.g. Gaussian noise of unknown mean and variance), then a gradual notion of consistency is obtained through the likelihood of the hypothesis h ∈ H given the sample S, i.e. p(h|S) (D'Angelo and Fortuin, 2021).

The main limitation of epistemic uncertainty estimation for deep learning

Based on the previous considerations, the epistemic uncertainty estimation is then considered accurate when the learner is able to determine the whole set of consistent hypothesis H S (or to determine the likelihood of any hypotheses in the Bayesian framework). However, as H is an infinite set, computing the empirical risk for any hypothesis from H to determine which hypothesis belong to H S is impossible. Moreover, with deep neural network hypotheses, determining the subspace H S is generally intractable, because of the non-linear relationship between the neural network parameters and the empirical error.

To overcome this issue, common practice consists in using empirical risk minimization algorithms to produce a sample or a distribution of consistent hypotheses. To avoid sampling always the same empirical risk minimizer, deep ensemble methods use random initialization and random batch order with early stopping [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF], while Bayesian neural networks algorithms learn a weight distribution [START_REF] Kendall | What uncertainties do we need in bayesian deep learning for computer vision[END_REF]. Although such approaches foster hypothesis diversity, they cannot guarantee to produce a representative sample of the whole set of consistent hypotheses. Moreover, common practices in deep learning training induce important biases which narrow the sampling in a restricted region of the consistent hypotheses' subspace. For instance, the use of weight decay (ℓ 2 penalization) and random weights initialization of relatively small variance (e.g. equal to the inverse of the number of neurons in the layer [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF])) drive the sample in low weight regions. Consistent hypotheses with high weights are then excluded, even though they can explain the observations as well, but in a different way, which would contribute to increase the potential prediction diversity. Similarly, in the Bayesian framework, it has been recently observed that the most commonly used prior, i.e. the Gaussian centered prior, is "unintentionally informative" (Wenzel et al., 2020a). Finally, the evaluation of uncertainty quantification methods and their hyper-parameters selection is traditionally driven by the negativelog-likelihood metric (NLL) computed over a validation dataset belonging to the training domain [START_REF] Liu | Ensemble learning via negative correlation[END_REF][START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF][START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF]. However, such practice does not account for the epistemic uncertainty out-of-distribution and then does not foster methods which accurately estimate it. This issue is illustrated by the four bottom graphics of Figure 1, the four methods provide almost the same prediction uncertainty on the training domain, their validation NLL is then similar, but their OOD epistemic uncertainty estimation is very different. Therefore, we identify the inability of standard approaches to produce a representative sample of consistent hypotheses as their main limitation. We argue that this limitation is the principal cause of their lack of prediction diversity for OOD data, observed recently [START_REF] Ovadia | Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift[END_REF][START_REF] Liu | The peril of popular deep learning uncertainty estimation methods[END_REF][START_REF] Henning | Are bayesian neural networks intrinsically good at out-of-distribution detection?[END_REF] (cf. Section 5.2).

Maximum Weight Entropy

The main contribution of this work is the development of a practical algorithm to produce a sample of hypotheses that tends to be representative of the whole space of consistent hypotheses. Considering stochastic neural networks, we propose to learn the scale parameters of a distribution over the network weights, centered on a hypothesis of low empirical risk, with the double objective of minimizing the average empirical risk and maximizing the distribution diversity, measured through the weight entropy.

Optimization formulation

We consider the stochastic neural network approach, where samples of hypotheses are produced through a parameterized weight distribution q ϕ in the set Φ = {q ϕ } ϕ∈R D composed of several distributions over W parameterized by ϕ ∈ R D , with D ∈ N the parameter dimension. We propose to penalize the average training risk over q ϕ with the entropy of the weight distribution, leading to the following optimization formulation:

min ϕ∈R D E q ϕ [L S (w)] -λE q ϕ [-log(q ϕ (w))] , (4)
with λ ∈ R + the trade-off parameter.

• The first term: E q ϕ [L S (w)] of the optimization objective in Equation ( 4) is the average empirical risk over the weight distribution. This term induces the increase of the probability mass q ϕ (w) in regions where the weights w ∈ W produce accurate hypotheses on the training dataset, i.e. where L S (w) is small.

• The second term: -λE q ϕ [-log(q ϕ (w))] in Equation ( 4) is a penalty that induces the increase of the weight entropy, which is generally related to expand the support of the weight distribution q ϕ as broad as possible.

It should be underlined that both terms in Equation (4) evolve in opposite direction with respect to the weight distribution: the first term induces a peaked weight distribution around the best performing weight, while the second term induces a uniform distribution over the whole weight space. To solve this trade-off, the weight distribution tends to flatten in regions of little impact on the empirical risk, while remaining concentrated in directions where a small weight perturbation causes an important risk increase. The theoretical analysis in Section 4 shows, indeed, that the distribution spread of the weights is inversely proportional to the neuron activation amplitude. The weight variance is then larger for weights in front of neurons weakly activated by the training data. This theoretical result is supported by numerical results observed on synthetic datasets in Section 7.1 which provide a direct illustration of this link between the neuron activation and the weight variance (cf. Figure 6).

Objective (4) can be understood as a maximum entropy problem [START_REF] Edwin | Information theory and statistical mechanics[END_REF], where, in presence of partial information about the optimal weight, the uncertainty is best described by the distribution of low risk hypotheses with maximal entropy (see Section 5.1). In the Bayesian neural network setting, a similar objective can be derived through the ELBO formulation by using the prior of maximum entropy (Jaynes, 1968), which, in this case, is the uniform distribution over W (see Section 5.3). To highlight the link between our proposed approach and the maximum entropy principle, we call the method: Maximum Weight Entropy (MaxWEnt) in reference to the general maximum entropy modeling framework, commonly named MaxEnt [START_REF] Berger | A maximum entropy approach to natural language processing[END_REF].

Optimization algorithm

Equation ( 4) is solved through stochastic gradient descent with mini-batches. To compute the expectation over q ϕ , we use the reparameterization trick [START_REF] Diederik | Auto-encoding variational bayes[END_REF][START_REF] Jimenez Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF]. We introduce a sampling variable z ∼ Z with Z a distribution over R d and a parameterization function ω : R d × R d → R d such that: w = ω(z, ϕ). Typically, z follows a distribution that can be numerically sampled as the normal or uniform distribution. In case of simple parameterization, the weight entropy can be directly derived from the weight parameters ϕ, such that there exists a function

H : R d → R verifying H(ϕ) = E Z [-log(q ϕ (ω(z, ϕ)))]
. This leads to the following objective function, computed on a mini-batch of data S b ⊂ S of size B > 0:

G(ϕ, S b ) = E Z [L S b (ω(z, ϕ))] -λ H(ϕ).
(5)

By sampling z (1) , ..., z (N ) iid according to Z, we can compute an estimation of the objective function gradient for each mini-batch as follows:

∇ ϕ G(ϕ, S b ) ≃ ∇ ϕ   1 N N j=1 L S b (ω(z (j) , ϕ)) -λ H(ϕ)   . (6)
Note that choosing N = 1 appears to be sufficient, in practice, to obtain efficient results [START_REF] Diederik | Auto-encoding variational bayes[END_REF]. Several gradient updates are performed until convergence to obtain the estimated parameters φ. The training part of the algorithm is summarized in Algorithm 1. For inference on x ∈ X , a set of P predictions (P ∈ N * ) is obtained by sampling multiple z (j) ∼ Z with j ∈ [|1, P |], and computing the corresponding outputs {h w j (x); w j = ω(z (j)

z ∼ Z, S b ∼ U(S B ) 6: ϕ ← ϕ -ν∇ ϕ [L S b (ω(z, ϕ)) -λH(ϕ)]

SCALING PARAMETERIZATION

Obviously, the choice of the weight parameterization ω has an important impact on the resulting weight distribution. In line with the purpose of the MaxWEnt approach, the guidelines for choosing ω should follow these three principles: enable the sampling in regions of accurate hypotheses, foster the increase of the weight entropy and be practical to use. Moreover, one should consider weight parameterizations that provide a tractable formulation of the weight entropy H(ϕ). Following these guidelines, we consider the sampling variable z ∼ Z such that E[z] = 0, V[z] = Id d and propose the "scaling" parameterization defined as follows:

ω(z, ϕ) = w + ϕ ⊙ z. ( 7
)
Where ⊙ is the element-wise product between two vectors, such that ϕ

⊙ z = (ϕ 1 z 1 , ..., ϕ d z d ) with ϕ = (ϕ 1 , ..., ϕ d ) ∈ R d and z = (z 1 , ..., z d ) ∈ R d . The weight vector w ∈ R d is the weight mean E q ϕ [w] = w.
It is typically defined as the weights of a pretrained network h w fitted on the training data. For Z defined as a normal N (0,

Id d ) or uniform distribution U([- √ 3, √ 3] d )
, the parameters ϕ = (ϕ 1 , ..., ϕ d ) act as scaling factors: the higher ϕ k , the wider the distribution

w k ∼ w k + ϕ k z k .
The scaling parameterization (7) meets the three previous requirements for a relevant choice of stochastic model. The mean of the weight distribution verifies E q ϕ [w] = w with w the weights of a pretrained network fitted on S, the weight distribution is then centered in a region of the weight space of low empirical risk. If ϕ ≃ 0, the resulting weight distribution is equivalent to a peaked distribution around w, which meets the first objective to provide samples of accurate hypotheses. Moreover, the weight entropy is directly controlled by the parameters ϕ : when ϕ increases, the weight distribution becomes wider and the entropy increases. We show, indeed, in the next section, that the weight entropy H(ϕ) can be expressed directly as a function of ϕ. Finally, it can be noticed that the scaling parameterization only involves element-wise multiplications, which makes it practical to compute.

We show, through the theoretical analysis developed in Section 4, that the increase of the ϕ parameters is inversely proportional to the neuron activation amplitude. Indeed, if a neuron is weakly activated by the training data, all the weights w k in front of this neuron have little impact on the network predictions in the training domain. Therefore, the parameters ϕ k can be enlarged without degrading the average empirical risk E q ϕ [L S (w)]. In the extreme case, if the neuron is never activated by the training data (it always returns 0), then the parameters ϕ k can go to infinity without impacting the network outputs on the training domain. Based on this theoretical observation, we argue that the weight entropy can be further increased without impacting the training risk by taking into account the correlation between neurons. Indeed, let's consider, for instance, two neurons of the same hidden layer, totally correlated, both with activation amplitude a > 0 on average on the training data. The scales of the weights w k in front of these neurons will verify ϕ k ∝ 1/a. However, by expressing the outputs of these neurons in their singular value decomposition basis, the novel representation is now composed of one component of average amplitude a and the other of null amplitude. In that case, some parameters ϕ k can be further increased without impacting the training risk. Motivated by these arguments, we propose the "SVD" parameterization described in the following subsection.

SVD PARAMETERIZATION

Let's consider a pretrained neural network h w of L hidden layers. We denote ψ (l) (X) ∈ R n×b l the hidden representation of the input data X ∈ R n×b in the l th layer of h w , with b l the hidden layer dimension (i.e. the number of neurons). The singular values decomposition of ψ (l) (X) is written: ψ (l) (X) = U (l) S (l) V (l) with U (l) ∈ R n×n , S (l) ∈ R n×b l and V (l) ∈ R b l ×b l . We propose the SVD parameterization, which consists in "aligning" the weight distribution with the principal components of ψ (l) (X) such that:

w (l) = w (l) + V T (l) (ϕ (l) ⊙ z (l) ), (8)
for any l ∈ [|0, L|]], where w (l) , w (l) , ϕ (l) , z (l) ∈ R b l ×b l+1 are respectively the matrix of weights, average weights, scaling parameters and sampling variables between the l th layer and the next layer.

A compact formulation of the parameterization can be written as follows:

ω(z, ϕ) = w + V (ϕ ⊙ z). ( 9
)
Where V denotes the block matrix:

V = V T (1) , ..., V T (1) , V T (2) , ..., V T (L) of dimension b l × b l+1 .
Similar to the previous one, the SVD parameterization fulfills the guidelines. Indeed, the weight distribution is still centered on w, which ensures to sample in a weight space region of low empirical risk. Moreover, the weight entropy can be increased by enlarging the ϕ parameters. This can be done more efficiently compared to the previous approach due to the integration of the neurons' correlations (cf. Section 4.1.3). The SVD parameterization requires additional computational time compared to the scaling one, due to the SVD decomposition and the matrix multiplication. It should be noticed that the SVD decomposition for each layer is computed only once. Before the stochastic gradient descent, a forward pass of the training data in h w is required to compute each hidden representation ψ (l) (X), then the SVD decomposition of ψ (l) (X) is performed to compute the matrix V (l) . However, the matrix multiplications between V (l) and ϕ (l) ⊙ z (l) are performed at each gradient update, which requires an additional computational burden during the gradient descent compared to the scaling parameterization (cf. Section 5.4 for the complexity calculation). Finally, we show in the next section, that a similar expression of the weight entropy H(ϕ) can be written in function of ϕ for both parameterizations.

Entropy function

The following proposition states that the previous weight parameterizations provide a closed-form expression of the weight entropy H(ϕ):

Proposition 1 (Closed-form expression of the weight entropy) Let q ϕ be a weight distribution described by Equation ( 7) or (9) with z ∼ Z. If Z is defined as the normal N (0, Id d ) or the uniform distribution U([-√ 3, √ 3] d ), there exists two constants C 1 , C 2 such that the weight entropy H(ϕ) is expressed as follows:

H(ϕ) = C 1 d k=1 log(ϕ 2 k ) + C 2 , ( 10
)
with ϕ = (ϕ 1 , ..., ϕ p ) ∈ R d the scaling parameters of the weight distribution q ϕ .

Proof The full proof is reported in Appendix A.1. The proof consists in considering that, for a normal distribution N (0, Σ) or for a uniform distribution defined over a parallelotope described by Σ, the entropy verifies H(ϕ) ∝ log(|det(Σ)|). Then, by showing that for both parameterizations det(Σ) ∝ det(diag(ϕ)), the above result can be derived.

Note that, the C 2 constant can be removed in the objective function of Equation (5) as it does not impact the optimization and the C 1 constant can be integrated in the trade-off parameter λ. This expression of the entropy function is easy to implement. It highlights the direct link between the scale parameter ϕ k and the weight entropy. When ϕ k grows, the weight distribution becomes wider and the entropy increases.

Theoretical Analysis

In this section, we develop a theoretical framework to understand the MaxWEnt approach in the specific case where the loss function is defined by the mean squared error. We first develop theoretical results in the linear regression case, and further extend these results to deep fully-connected neural networks.

Linear Regression

Linear regression can be seen as a particular case of deep fully-connected neural networks where the networks are composed of exactly two layers: the input layer of b neurons and the output layer of 1 neuron with linear activation function. The linear regression case is not representative of the framework considered in this work, as the hypotheses h ∈ H can no longer be considered as universal approximators. However, the following study provides valuable insights on what happened between the neurons of one hidden layer and one neuron of the next layer. In particular, we highlight the link between the scale parameters ϕ and the amplitude of the input features.

NOTATIONS

We consider the linear regression framework, where the learner has access to an input dataset X ∈ R n×b composed of n row data x i ∈ R b drawn iid according to the distribution p(x) and an output vector y ∈ R n such that y = (y 1 , ..., y n ). Each input x i is associated to the scalar output y i ∈ R drawn according to p(y|x i ). We denote S = {(x 1 , y 1 ), ..., (x n , y n )} the set of training observations. We consider the set

H = {x → b k=1 x k w k ; w ∈ R b } of linear hypotheses.
The loss function is the mean squared error, and we define the empirical risk for any weight w ∈ R b as L S (w) =

1 n ||Xw -y|| 2 2 . We denote by a = (a 1 , ..., a b ) ∈ R b + the amplitude of the input features of the training set, such that a 2 j = 1 n ||X j || 2 2 for any j ∈ [|1, b|]
, with X j the j th column of X. We assume that a j > 0 for any j ∈ [|1, b|].

SCALING WEIGHT PARAMETERIZATION

We first consider the weight parameterization defined in Equation ( 7) such that

q ϕ ∼ w + ϕ ⊙ z with z ∼ Z such that Z ∼ N (0, Id b ) or Z ∼ U([- √ 3, √ 3] b ). The weight vector w ∈ R b is the weight mean: E q ϕ [w] = w. Finally, we consider the entropy penalty H(ϕ) defined by H(ϕ) = b k=1 log(ϕ 2 k ).
The optimization problem (4) can then be written:

min ϕ∈R b E Z 1 n ||X(w + ϕ ⊙ z) -y|| 2 2 -λ b k=1 log(ϕ 2 k ). (11)
We show that the MaxWEnt optimization problem of Equation ( 11) has a unique solution, which can be expressed with the following closed-form expression:

Proposition 2 (Closed-form solution for the scaling parameterization) Equation ( 11) has a unique solution ϕ * ∈ R b verifying for any k ∈ [|1, b|]:

ϕ * k 2 = λ a 2 k . ( 12
)
Proof The proof consists in first developing the average risk as follows:

E Z 1 n ||X(w + ϕ ⊙ z) -y|| 2 2 = b k=1 a 2 k ϕ 2 k + 1 n ||Xw -y|| 2 2 . ( 13
)
Optimization ( 11) can then be written:

min ϕ∈R b b k=1 a 2 k ϕ 2 k -λ b k=1 log(ϕ 2 k ). (14)
This is a convex problem, for which the derivative of the objective function with respect to ϕ 2 is null for:

a 2 k -λ/ϕ 2 k = 0. (15)
This closed-form solution of ϕ * is particularly insightful: ϕ * k is inversely proportional to a 2 k , which means that the optimal scale parameters ϕ * k are larger for weights in front of low amplitude features a 2 k . Applied to the hidden layers of a neural network, Proposition (2) states that the weight distribution is wider in front of neurons weakly activated by the training data. As a consequence, if an OOD data activates these neurons, large values are propagated through the network, which produces an important output variance. These statements are formalized in Section 4.2 when considering deep fully connected neural networks.

It can be further noticed that Equation ( 14) is equivalent to a log determinant optimization problem [START_REF] Boyd | Convex optimization[END_REF]. The maximum entropy optimization can then be interpreted as a maximum ellipsoid volume problem, where the volume ϕ 2 k is maximized under the linear constraint k a 2 k ϕ 2 k ≤ λb. If Z is a uniform distribution, this boils down to maximizing the support of the weight distribution while maintaining the average empirical risk on the training data under an acceptable threshold. This is in line with the purpose of the approach to find the weight distribution that covers as many consistent weights as possible.

SVD WEIGHT PARAMETERIZATION

According to Proposition (2), the optimal scale parameters verify ϕ * 2 = λ/a 2 . When injecting this solution in the entropy formulation, we obtain:

H(ϕ) = -log(a 2 k ) + cste.
Considering this formula, it appears clearly that the weight entropy is particularly important if some a 2 k are small, i.e. if some input features have a low amplitude. However, in the presence of correlated features, all amplitudes a 2 k may be high while the input training data may present small variation in some directions of the input space. The SVD parameterization (9) proposes to exploit these directions of small variation by aligning the weight distribution with the singular value components of the input data. For this purpose, we now consider V ∈ R b×b , the matrix of eigenvectors of 1 n X T X and s 2 = (s 2 1 , ..., s 2 b ) ∈ R b + the vector of eigenvalues, and assume that s j > 0 for any j ∈ [|1, b|]. The SVD weight parameterization is written w = w + V (ϕ ⊙ z) with z ∼ Z and the MaxWEnt optimization problem (4) becomes:

min ϕ∈R b E Z 1 n ||X(w + V (ϕ ⊙ z) -y|| 2 2 -λ b k=1 log(ϕ 2 k ). (16)
In comparison to the previous optimization problem in Equation ( 11), there is now the presence of the matrix V between X and ϕ⊙z. By definition of V , the matrix XV is the expression of X in its singular values basis. Thus, the vector ϕ ⊙ z is now aligned with the singular value components. As for the previous parameterization, the optimal parameter vector ϕ * admits a closed-form expression as follows:

Proposition 3 (Closed-form solution for the SVD parameterization) Equation ( 16) has a unique solution ϕ * ∈ R b verifying for any k ∈ [|1, b|]:

ϕ * k 2 = λ s 2 k . ( 17
)
Proof The proof consists in developing the average risk, such that:

E Z 1 n ||X(w + V (ϕ ⊙ z)) -y|| 2 2 = b k=1 s 2 k ϕ 2 k + 1 n ||Xw -y|| 2 2 . (18)
Optimization ( 16) is then written:

min ϕ∈R b b k=1 s 2 k ϕ 2 k -λ b k=1 log(ϕ 2 k ), (19)
which is similar to Equation ( 14) with s 2 k instead of a 2 k .

Proposition (3) states that the optimal parameters ϕ * are now inversely proportional to the singular values of the training data instead of the feature amplitudes. We show, with the next Proposition, that this difference implies a larger weight entropy for the same level of average empirical risk.

Proposition 4 (Comparison between scaling and SVD parameterization) Let q

(1)

ϕ * , q (2)
ϕ * be the respective optimal weight distributions for the scaling and the SVD parameterization. The following propositions hold:

E q (1) ϕ * [L S (w)] = E q (2) ϕ * [L S (w)] (20) E q (1) ϕ * -log(q (1) ϕ * (w)) ≤ E q (2) ϕ * -log(q (2) ϕ * (w)) . ( 21
)
Proof The average empirical risk equality can be derived as follows:

E q (1) ϕ * [L S (w)] = λ b k=1 a 2 k a 2 k + ϵ = λ b + ϵ = λ b k=1 s 2 k s 2 k + ϵ = E q (2) ϕ * [L S (w)] , (22)
with ϵ = 1 n ||Xw -y|| 2 2 . The weight entropy inequality is derived from Hadamard's inequality. In light of Proposition ( 4), it appears that the SVD parameterization leads to a more efficient weight distribution according to the maximum entropy principle. Indeed, for the same level of explanation of the observations (same average empirical risk), the SVD parameterization provides more entropy. Experiments conducted on both synthetic and real datasets show that this last weight parameterization provides, indeed, a better evaluation of the epistemic uncertainty (cf. Section 7) which advocates in favor of the use of the entropy as a measure of weight distribution quality.

Deep fully connected neural network

In this subsection, we extend the previous result to deep fully connected networks under the mean squared error loss. In particular, we formally derive the connection between the neuron activation amplitude and the optimal scaling parameters suggested by Proposition (2).

NOTATIONS

We consider fully-connected neural networks h w ∈ H of L hidden layers with w ∈ W. For the sake of simplicity, we assume that every hidden layer is composed of b neurons with b the dimension of the input data, the last layer is composed of 1 neuron such that the neural networks produce scalar outputs. For any x ∈ X and for any l ∈ [|1, L|], ψ (l) (x) ∈ R b denotes the hidden representation of the input data x in the l th layer; ψ (0) (x) ∈ R b and ψ (L+1) ∈ R are respectively the input and output layer representation, such that ψ (0) (x) = x and ψ (L+1) (x) = h w (x). Notice that the hidden representations depend on w; the notation

ψ (l) (x) is a contraction of ψ (l) (x, w) or ψ (l) w (x).
(x) = ζ ψ (l) (x)
T w (l,j) with ψ (l+1,j) (x) the j th component of the hidden representation ψ (l+1) (x). The weight distributions are denoted q ϕ with ϕ ∈ R d . The loss function ℓ is the mean squared error and the problem to be solved is written:

min ϕ∈R d E q ϕ [L S (w)] -λ d k=1 log(ϕ 2 k ). ( 23
)
We assume that Problem (23) has a unique solution, denoted ϕ * ∈ R d .

SCALING WEIGHT PARAMETERIZATION

We focus our deep neural networks analysis on the scaling parameterization (7) such that

q ϕ ∼ w + ϕ ⊙ z with z ∼ Z where Z ∼ N (0, Id d ) or Z ∼ U([- √ 3, √ 3] d )
and w the weight of a pretrained network h w . In the following, we aim at extending the results of Proposition (2) to the hidden layers of deep neural networks and show that the MaxWEnt optimization leads to scaling parameters inversely proportional to the neuron activation amplitude. For this purpose, we consider the following assumption on the activation function ζ. Assumption (5) states that the order of the first and second moment of the neuron activation are preserved by ζ. This assumption is verified, for instance, for most of the common activation functions, as ReLU or Leaky-ReLU, if the neuron activation follows a centered independent Gaussian distribution.

Assumption 5 (Moments preserving property of the activation function) For any ϕ 1 , ϕ 2 ∈ Φ, l ∈ [|0, L -1|] and any j ∈ [|1, b|], the activation function ζ verifies:

n i=1 E q ϕ 1 [U ij ] ≤ n i=1 E q ϕ 2 [U ij ] =⇒ n i=1 E q ϕ 1 [ζ (U ij )] ≤ n i=1 E q ϕ 2 [ζ (U ij )] (24) n i=1 E q ϕ 1 U i U T i ≼ n i=1 E q ϕ 2 U i U T i =⇒ n i=1 E q ϕ 1 ζ (U i ) ζ (U i ) T ≼ n i=1 E q ϕ 2 ζ (U i ) ζ (U i ) T (25)
Where

U i = (U i1 , ..., U ip ) and U ij = ψ (l) (x i ) T w (l,j) ∀i ∈ [|1, n|], ∀j ∈ [|1, b|].
For two matrices A, B, the notation A ≼ B states that B -A is a positive semi-definite matrix.

Proposition 6 (Optimal scaling parameters) Let ϕ * ∈ R d be the unique solution of Problem ( 23), then ϕ * verifies:

ϕ * = L l=0 b l j=1 ϕ * (l,j,1) , ..., ϕ * (l,j,p) ϕ * (l,j,k) 2 = σ 2 (l,j) b a 2 (l,k) ∀ l ∈ [|1, L|]; j ∈ [|1, b l |]; k ∈ [|1, b|]. (26)
Where is the concatenation operator and for any l

∈ [|0, L|], j ∈ [|1, b l |] and k ∈ [|1, b|]: a 2 (l,k) = 1 n n i=1 E q ϕ * ψ (l,k) (x i ) 2 (27) σ 2 (l,j) = 1 n n i=1 V q ϕ * ψ (l) (x i ) T w (l,j) -w (l,j) . ( 28
)
Proof The full proof is reported in Appendix A.5. The main idea of the proof consists in first dividing Problem (23) by layer and output neurons. The parameters ϕ defined in Equation ( 26) provide the solution for each sub-problem. Then, considering Assumption (5) on the activation function and the uniqueness of the solution, it can be shown that ϕ = ϕ * .

Proposition (6) states that the solution ϕ * of the MaxWEnt optimization ( 23) is the inverse of the average neuron activation amplitude over the training data. We emphasize that the aim of Proposition ( 6) is not to provide an exact solution (as the quantities a 2 (l,k) and σ 2 (l,j) are intractable) but to offer a theoretical understanding of MaxWEnt in the case of deep fully connected neural networks. Numerical observations described in Section 7.1.4 confirm this "inverse proportionality" relationship between the scaling parameters and the neuron activation amplitude. This means that maximizing the weight entropy leads to put more emphasis on the activation of neurons that are weakly activated by the training data. Thus, it can be considered that these neurons act as "detectors" for the out-of-distribution data that activate them.

Discussion

Maximum Entropy

The maximum entropy principle was originally proposed by Jaynes for modeling the uncertainty that one has about a system with a probability distribution [START_REF] Edwin | Information theory and statistical mechanics[END_REF]. It states that one should consider the distribution of maximal entropy which is compatible with the current state of knowledge about the system. This principle provides a practical framework to describe the system uncertainty through distributions [START_REF] Guiasu | The principle of maximum entropy[END_REF] often referred as "MaxEnt" [START_REF] Cortes | Structural maxent models[END_REF], which is used in various research fields as natural language processing [START_REF] Berger | A maximum entropy approach to natural language processing[END_REF], [START_REF] Ratnaparkhi | A maximum entropy model for part-of-speech tagging[END_REF], [START_REF] Rosenfeld | A maximum entropy approach to adaptive statistical language modelling[END_REF], biology [START_REF] Finnegan | Maximum entropy methods for extracting the learned features of deep neural networks[END_REF], as well as ecology, to model the geographic distribution of species [START_REF] Steven | A maximum entropy approach to species distribution modeling[END_REF][START_REF] Elith | A statistical explanation of maxent for ecologists[END_REF].

The MaxWEnt approach developed in this work is built under this framework. In the supervised learning scenario described in Section 2.1, the system is described by the set of hypotheses H (equivalent to the set of weights W), and the observations S. The goal is to model the uncertainty about the best weights w * through a distribution over W. To provide a formal constraint on such distribution, we assume the knowledge of a performance threshold τ ∈ R + , such that w * verifies L S (w * ) ≤ τ . In the absence of further consideration, the maximum entropy principle then states that the uncertainty over w * is best described by the uniform distribution over the set of consistent weights W τ ≡ {w ∈ W; L S (w) ≤ τ }, denoted U(W τ ). However, due to technical limitation, the set of weight distributions considered by the learner, Φ, is generally composed of simple distributions such as independent multi-variate uniform distributions over R d , which offer a poor model for U(W τ ). Moreover, because of the complex structure of W τ , covering consistent weights with q ϕ ∈ Φ generally involves to include some inconsistent weights in the distribution support. To overcome both issues, the technical limitation is taken into account in the maximum entropy framework and the threshold constraint over the empirical risk is relaxed through averaging over q ϕ , leading to the following expression of the problem:

max q ϕ ∈Φ E q ϕ [-log(q ϕ (w))] subject to E q ϕ [L S (w)] ≤ τ. ( 29
)
The MaxWEnt optimization derived in Equation ( 4) is the penalized version of the maximum entropy problem (29).

Formulating the epistemic uncertainty quantification as a maximum entropy problem offers a natural classification among the weight distributions q ϕ ∈ Φ. Between two weight distributions that provide the same level of empirical error on the training data, the learner should select the one of largest entropy. The maximum entropy paradigm also offers an interesting guideline to drive the selection of the weight distribution family Φ: the learner should foster weight parameterization that enables larger increases of the entropy, such as the SVD-parameterization (cf. Proposition (4)) or ensemble of MaxWEnt networks (cf. Section (7.5). Although, this quest of entropy maximization is counter-balanced by the computational efficiency of the weight parameterization.

Finally, It should be underlined that the maximum entropy principle has been applied with deep learning in previous works, as for instance, to the outputs of a classifier in outlier exposure methods [START_REF] Hendrycks | Deep anomaly detection with outlier exposure[END_REF] or to the generator's outputs for energy based generative models [START_REF] Kumar | Maximum entropy generators for energy-based models[END_REF]. These previous methods fundamentally differ from the present approach, as they consider the entropy of the network predictions instead of the weight distribution entropy, as considered in MaxWEnt.

Overfitting, Weight Diversity and Evaluation

In Section 2.3, we identify the main limitation of standard ensemble and Bayesian approaches as their inability to produce a representative sample of the whole consistent hypothesis set. We argue that this limitation is related to over-regularization processes and hyper-parameters selection driven by hold-out validation. Indeed, the use of weight regularization for deep neural network is first designed as a tool to avoid overfitting [START_REF] Krogh | A simple weight decay can improve generalization[END_REF], with the underlying idea that large weights induce the over-specification of the network on the observations. This technique has proven to improve the model accuracy in most cases. However, when applied in ensemble and Bayesian learning, it induces the counter effect of penalizing the diversity of the resulting sample of neural networks. On the contrary, anti-regularization fosters weight diversity [START_REF] De Mathelin | Deep antiregularized ensembles provide reliable out-of-distribution uncertainty quantification[END_REF]. The MaxWEnt optimization can be seen as a form of anti-regularization as it induces the sampling of large weights. Moreover, the use of broad weight distribution avoids overfitting thanks to the marginalization process (Wilson, 2020).

Regarding the use of hold-out validation for hyper-parameters selection, we claim that such a technique fosters narrowed weights distributions. As mentioned in Section 5.1, the covering of a large portion of consistent hypotheses generally comes with the inclusion of inconsistent weights in the support of the weight distribution. As a consequence, the in-distribution performance for distribution of high entropy is usually degraded (confirmed numerically in our experiments). Moreover, for a large number of training data, the in-distribution epistemic uncertainty becomes negligible in front of the aleatoric uncertainty. Its accurate estimation is then not required to obtain good validation NLL. However, for out-of-distribution data, the main source of uncertainty is epistemic, and its estimation is critical. Then, narrowed weights distributions, although improving the validation NLL, fail to produce relevant uncertainty quantification out-of-distribution [START_REF] Ovadia | Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift[END_REF][START_REF] Liu | The peril of popular deep learning uncertainty estimation methods[END_REF][START_REF] Henning | Are bayesian neural networks intrinsically good at out-of-distribution detection?[END_REF].

It should be underlined that, although MaxWEnt tends to enlarge the weight distribution, it cannot fully guarantee to capture the whole set of consistent hypotheses due to the technical limitation of the stochastic model q ϕ . However, the MaxWEnt approach is an important step in this direction. It already provides significant improvements compared to the baselines, as demonstrated by our numerical experiments.

Bayesian Neural Network

In the Bayesian variational inference framework, the learner aims at approximating the posterior distribution p(w|S) with a parameterized distribution q ϕ defined over W. The minimization of the Kullback-Leibler (KL) divergence between p(w|S) and q ϕ leads to the maximization of the evidence lower bound (ELBO) expressed as follows (Wenzel et al., 2020a):

max ϕ∈R D E q ϕ   (x,y)∈S log(p(y|h w (x))   -D KL (q ϕ (w), p(w)) . ( 30
)
Where p(y|h w (x)) is the log likelihood of y with respect to h w (x), D KL is the Kullback-Leibler divergence and p(w) is the prior distribution defined over W.

If we consider a uniform prior over the whole weight space: p(w) ∼ U(W) (assuming W bounded), the second term of the ELBO maximization: D KL (q ϕ (w), p(w)), is equal to the negative entropy of q ϕ (up to a constant). Therefore, if the empirical risk L S (w) can be written as a quantity proportional to the negative log-likelihood, the ELBO maximization ( 30) is equivalent to the MaxWEnt optimization problem (4). This is in line with the application of the maximum entropy principle to the Bayesian framework (Jaynes, 1968), which states that the prior should be selected as the distribution of maximal entropy that integrates prior information. In our case, without any regularity assumption about the optimal hypothesis, the maximum entropy principle then leads to consider a uniform prior over the whole weight space W (bounded), i.e. p(w) ∼ U(W).

The use of "uninformative" parameter priors is considered as the guideline to model epistemic uncertainty in the Bayesian framework (Wilson, 2020). In practice, however, the most commonly used priors for Bayesian neural networks are Dropout [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF][START_REF] Kendall | What uncertainties do we need in bayesian deep learning for computer vision[END_REF][START_REF] Boluki | Learnable bernoulli dropout for bayesian deep learning[END_REF] which has been shown to produce over-confident predictions for out-of-distribution data [START_REF] Liu | The peril of popular deep learning uncertainty estimation methods[END_REF] and the isotropic Gaussian prior p(w) ∼ N (0, σ 2 0 Id d ) [START_REF] Zhang | Noisy natural gradient as variational inference[END_REF][START_REF] Osawa | Practical deep learning with bayesian principles[END_REF][START_REF] Valentin Jospin | Hands-on bayesian neural networks-a tutorial for deep learning users[END_REF], which is recently considered to be often "non-optimal" or "unintentionally informative" (Wenzel et al., 2020a;Fortuin et al., 2021).

When considering a Gaussian isotropic prior p(w) ∼ N (0, σ 2 0 Id d ) with σ 0 ∈ R and an independent multivariate Gaussian stochastic model q ϕ ∼ N (µ, diag(σ 2 )) with µ, σ ∈ R d the mean and scale parameters such that ϕ = (µ, σ), the following expression can be derived for the KL divergence between the approximate posterior and the prior [START_REF] Duchi | Derivations for linear algebra and optimization[END_REF]:

D KL (q ϕ (w), p(w)) = ||µ|| 2 2 2σ 2 0 + 1 2 d k=1 σ 2 k σ 2 0 -log σ 2 k σ 2 0 - d 2 . ( 31
)
From this expression, it appears that the KL divergence operates a "double" regularization regime on the scale parameters σ. When σ 2 k is below σ 2 0 , the term -log(σ 2 k /σ 2 0 ) dominates σ 2 k /σ 2 0 , which induces the increase of the σ 2 k parameter similar to the MaxWEnt penalization. Whereas, for σ 2 k above σ 2 0 , the dominant term becomes σ 2 k /σ 2 0 which stops the increase of the scaling parameter. Then, for σ 0 → +∞, the regularization over σ 2 induced by the KL divergence converges to the maximum entropy penalization. However, as a side effect, the term ||µ|| 2 2 /2σ 2 0 is reduced to zero and no regularization on the mean is operated, which is generally avoided. In many previous works which consider isotropic Gaussian priors, the commonly considered prior bandwidth σ 2 0 are relatively small [START_REF] Zhang | Noisy natural gradient as variational inference[END_REF][START_REF] Osawa | Practical deep learning with bayesian principles[END_REF][START_REF] Ashukha | Pitfalls of in-domain uncertainty estimation and ensembling in deep learning[END_REF], or at least, not designed in a maximum entropy perspective. Moreover, a trade-off parameter λ < 1 is often added between the log likelihood and the KL divergence in optimization (30) (Wenzel et al., 2020a) which further tempers the KL divergence regularization. Our interpretation is that the hyper-parameter selection is often driven by the in-distribution performances (computed on a validation set for instance) which fosters narrowed posterior distributions. Indeed, extending the weight distribution to any consistent weight, generally penalizes the test performances as observed in our experiments (cf. Sections 7.1.3 and 7.2.2). However, we argue that such penalization could be accepted when considering OOD detection.

SVD-parameterization

The SVD-parameterization has been introduced in Section 3.3.2 (cf. Equation ( 9)) with the aim of allowing a larger increase of the weight entropy while limiting the average empirical risk penalty.

We argue, indeed, that using independent weight components in the stochastic model sets the directions of weight distribution expansion to the canonical basis of R d , which seems intuitively suboptimal. We could include correlations between weight components as additional parameters to optimize in ϕ. However, this solution would require the optimization of O(d 2 ) parameters which may become intractable, especially for large neural networks as ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], for instance, for which d > 10 6 . Through the SVD-parameterization, we propose to set the correlation between weight components, at each hidden layer, according to the singular value decomposition of the neuron activation on the training data. Our theoretical analysis in Section 4.1.3 shows, in the case of linear regression, that this weight parameterization provides the same level of average empirical risk as independent weight components but with larger weight entropy.

Previous works consider the use of weight correlations in stochastic model in the form of matrix Gaussian distribution [START_REF] Louizos | Structured and efficient variational deep learning with matrix gaussian posteriors[END_REF][START_REF] Sun | Learning structured weight uncertainty in bayesian neural networks[END_REF] or through more sophisticated models such as weight distributions defined over "well-chosen" subspace of R d [START_REF] Izmailov | Subspace inference for bayesian deep learning[END_REF], as well as normalizing flows [START_REF] Louizos | Multiplicative normalizing flows for variational bayesian neural networks[END_REF] and implicit weight models [START_REF] Pawlowski | Implicit weight uncertainty in neural networks[END_REF]. A notable use of correlation between weights is the Laplace approximations (MacKay, 1992;[START_REF] Andrew Yk Foong | inbetween'uncertainty in bayesian neural networks[END_REF][START_REF] Ritter | A scalable laplace approximation for neural networks[END_REF], where the correlation matrix for a Gaussian model is given by a "closed-form" solution which can be computed using one forward and backward step through the network. Similarities can be observed between the Kronecker Laplace approximation [START_REF] Ritter | A scalable laplace approximation for neural networks[END_REF] and the SVD-parameterization, as both method involve the correlation matrix of the neuron activation, but identifying the link between both methods would require further investigation. In our case, the parameters ϕ are still optimized through stochastic variational gradient descent, whereas the Laplace approximation does not require multiple gradient updates. As we manage to find a closed-form expression for ϕ * in the linear case (cf. Propositions (2) and (3)), interesting future work directions include "Laplace-like" approximation in the MaxWEnt framework, which can potentially speed up the computation of the parameters ϕ * .

Regarding the complexity of the SVD parameterization, we can consider the case of a fully connected neural network with L layers of b neurons each. Computing the SVD decomposition matrix V (cf. Section 3.3.2) requires one forward pass of the training inputs and the computation of the SVD decomposition at each layer with complexity O(Lb 3 ) (Pan and [START_REF] Victor | The complexity of the matrix eigenproblem[END_REF]. Storing the matrices adds O(Lb 2 ) of memory burden, which is equivalent to O(d) with d ∈ N the dimension of the network weight vector. During the variational gradient descent, the matrix multiplication between the matrix V and the vector ϕ ⊙ z has a complexity of order O(Lb 3 ). For comparison, a forward pass with a batch of size B, for the scaling parameterization, is of complexity O(LBb 2 ). If we consider that b ≃ B with B the batch size, we can say that the SVD parameterization requires twice as much computational time as the scaling one, which corresponds approximately to what we observed in our experiments.

Entropy function

In the case of scaling (Equation ( 7)) or SVD parameterization (Equation ( 9)), we manage to provide an expression of the entropy H(ϕ) function of ϕ (cf. Equation ( 10)), which is a convenient property to speed up the MaxWEnt optimization. For other weight parameterizations, one may not be able to derive such a closed-form expression. If the probability density function q ϕ (w) can be computed, one can estimate the entropy through sampling, as done for the empirical risk. An alternative solution is to use a proxy of the entropy which is directly linked to the parameters ϕ. If the entropy is a growing function of ϕ k for any k ∈ [|1, d|], we propose to consider the following general expression for the penalization term related to the entropy:

H(ϕ) = d k=1 g k (ϕ 2 k ), (32)
with g k : R + → R predefined growing functions such that ϕ 2 k grows with H(ϕ). Typical choices are g k (u) = log(u) or g k (u) = √ u. In the case, g k (u) = log(u), Equation (32) matches the entropy expression derived in Proposition (1) within a constant factor. Equation ( 32) can be seen as a "proxy" of the weight "entropy" as it increases with ϕ k as the entropy.

Related Work

The main related works in distance based and ensemble based uncertainty quantification are presented in Section 1. The vast uncertainty estimation literature also includes notable methods as conformal prediction [START_REF] Vovk | Algorithmic learning in a random world[END_REF][START_REF] Lei | Distributionfree predictive inference for regression[END_REF][START_REF] Nikolas | Uncertainty sets for image classifiers using conformal prediction[END_REF], calibration (Guo et al., 2017b;[START_REF] Kuleshov | Accurate uncertainties for deep learning using calibrated regression[END_REF] and evidential learning [START_REF] Sensoy | Evidential deep learning to quantify classification uncertainty[END_REF][START_REF] Amini | Deep evidential regression[END_REF]. Our focus in this present work is on the Bayesian and ensemble approaches, for which we propose a specific improvement through the MaxWEnt algorithm. Readers interested in the alternative approaches will find further details in the following surveys [START_REF] Abdar | A review of uncertainty quantification in deep learning: Techniques, applications and challenges[END_REF][START_REF] Shen | Towards out-of-distribution generalization: A survey[END_REF].

Deep Ensembles and prediction diversity Out-of-distribution

The main challenge, faced by Bayesian and ensemble methods, is the lack of explicit correlation between the prediction diversity and the distance to the training domain, leading to the observation that standard methods in this category often produce over-confident predictions for OOD data [START_REF] Henning | Are bayesian neural networks intrinsically good at out-of-distribution detection?[END_REF][START_REF] Ovadia | Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift[END_REF][START_REF] Liu | The peril of popular deep learning uncertainty estimation methods[END_REF].

As described in Section 1, two main approaches are considered to increase the prediction diversity of deep ensemble, especially out-of-distribution: the first approach works on the diversity of the network outputs, gradients or hidden representations [START_REF] Liu | Ensemble learning via negative correlation[END_REF][START_REF] Shui | Diversity regularization in deep ensembles[END_REF][START_REF] Zhang | The diversified ensemble neural network[END_REF][START_REF] Ross | Ensembles of locally independent prediction models[END_REF][START_REF] Ramé | Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation[END_REF][START_REF] Sinha | Dibs: Diversity inducing information bottleneck in model ensembles[END_REF]. In this category, contrastive approach make use of auxiliary real or synthetic OOD data [START_REF] Pagliardini | Agree to disagree: Diversity through disagreement for better transferability[END_REF][START_REF] Tifrea | Semi-supervised novelty detection using ensembles with regularized disagreement[END_REF][START_REF] Kristiadi | Being a bit frequentist improves bayesian neural networks[END_REF][START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF][START_REF] Alexander Mehrtens | Improving robustness and calibration in ensembles with diversity regularization[END_REF][START_REF] Yu | Unsupervised out-of-distribution detection by maximum classifier discrepancy[END_REF]Wang et al., 2022b). The second approach works on the hypothesis diversity through random initialization and different architectures [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF][START_REF] Wen | Batchensemble: an alternative approach to efficient ensemble and lifelong learning[END_REF]Wenzel et al., 2020b;[START_REF] Zaidi | Neural ensemble search for uncertainty estimation and dataset shift[END_REF] or by imposing the weight diversity [START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF][START_REF] Tagasovska | Single-model uncertainties for deep learning[END_REF][START_REF] Francesco | Repulsive deep ensembles are bayesian[END_REF][START_REF] De Mathelin | Deep antiregularized ensembles provide reliable out-of-distribution uncertainty quantification[END_REF].

These last methods particularly relate to MaxWEnt. In particular, the DARE algorithm (de [START_REF] De Mathelin | Deep antiregularized ensembles provide reliable out-of-distribution uncertainty quantification[END_REF] produces a sample at the edge of the consistent hypothesis set by enlarging the network weights while maintaining the loss under an acceptable threshold. However, DARE presents some limitations when using softmax activation at the end layer, as the use of large weights induces the saturation of the activation for out-of-distribution data. Moreover, the DARE training requires the control of the penalization term to avoid numerical issues when the weights become too large. With the MaxWEnt approach, the training is more stable, as the weight distribution is centered on the weights w of a pretrained network. It also works with softmax activation because of the symmetric increase of the weights. Indeed, enlarging the weight variance causes the prediction of both highly negative and positive network outputs for OOD data.

Bayesian Neural Network Priors and Stochastic Models

Since the seminal work of Jaynes on Bayesian priors (Jaynes, 1968), an ongoing discussion has been opened about the use of the maximum entropy method for assigning priors in Bayesian modeling. This method, considered "thought-provoking" [START_REF] David | Information theory, inference and learning algorithms[END_REF], is generally not recommended [START_REF] Gelman | Prior choice recommendations[END_REF]. With the proposed MaxWEnt approach, we do not plan to further extend this discussion. We do not argue that the maximum entropy method is the "optimal" way to select a prior, as such a statement depends on the considered notion of optimality. Actually, we advocate for the use of MaxWEnt for OOD detection, but do not recommend this method to improve the test accuracy. Enlarging the weight entropy may, indeed, induce a loss of test accuracy due to the large weight variance. However, we show in our experiments that one can always use "shrunk" version of the weight distribution learned by MaxWEnt when looking for accurate inference while sampling over the whole distribution for OOD detection (cf. Section 7.2.2).

The question of the prior choice has been extensively discussed in the Bayesian literature, a recent review provides the main considered approaches (Fortuin et al., 2021). For Bayesian neural networks, two main groups of priors can be distinguished: weight-space priors and function-space priors. The latter includes priors defined in function space, i.e. over H. Many recent works consider this approach [START_REF] Sun | Functional variational bayesian neural networks[END_REF][START_REF] Louizos | The functional neural process[END_REF]Tran et al., 2022;[START_REF] Vincent Fortuin | Priors in bayesian deep learning: A review[END_REF][START_REF] Tim Gj Rudner | Function-space regularization in neural networks: A probabilistic perspective[END_REF], which mainly use Gaussian process priors. These methods can be related to the distance based uncertainty approach, as they make explicit the link between uncertainty and distance to training data through Gaussian processes. The former group corresponds to prior defined over the weights of the neural network, i.e. over W. Our work relates particularly to this approach, as discussed in Section 5.3. The main considered priors in this category are Dropout [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF][START_REF] Gal | Concrete dropout. Advances in neural information processing systems[END_REF][START_REF] Boluki | Learnable bernoulli dropout for bayesian deep learning[END_REF][START_REF] Andre T Nguyen | Out of distribution data detection using dropout bayesian neural networks[END_REF], isotropic Gaussians [START_REF] Zhang | Noisy natural gradient as variational inference[END_REF][START_REF] Osawa | Practical deep learning with bayesian principles[END_REF][START_REF] Valentin Jospin | Hands-on bayesian neural networks-a tutorial for deep learning users[END_REF], mixture of Gaussians [START_REF] Blundell | Weight uncertainty in neural network[END_REF], hierarchical [START_REF] Wu | Deterministic variational inference for robust bayesian neural networks[END_REF] and horseshoe priors [START_REF] Ghosh | Model selection in bayesian neural networks via horseshoe priors[END_REF]. Some methods also propose to define the prior based on empirical observation of the weight distribution of non-Bayesian networks [START_REF] Atanov | The deep weight prior[END_REF]Fortuin et al., 2021).

Regarding the stochastic model of the weight distribution, previous works have considered the use of diagonal Gaussian [START_REF] Graves | Practical variational inference for neural networks[END_REF] and matrix Gaussian to include the weight correlations [START_REF] Louizos | Structured and efficient variational deep learning with matrix gaussian posteriors[END_REF][START_REF] Sun | Learning structured weight uncertainty in bayesian neural networks[END_REF]. In the case of multivariate Gaussian model with fixed mean, approximation methods can be used to derive the posterior distribution without using gradient descent as Laplace approximations (MacKay, 1992;[START_REF] Andrew Yk Foong | inbetween'uncertainty in bayesian neural networks[END_REF][START_REF] Ritter | A scalable laplace approximation for neural networks[END_REF][START_REF] Kristiadi | Being bayesian, even just a bit, fixes overconfidence in relu networks[END_REF] and tractable approximate Gaussian inference (TAGI) [START_REF] Goulet | Tractable approximate gaussian inference for bayesian neural networks[END_REF]. More sophisticated stochastic model have been developed with techniques as normalizing flows [START_REF] Rezende | Variational inference with normalizing flows[END_REF][START_REF] Louizos | Multiplicative normalizing flows for variational bayesian neural networks[END_REF], implicit distribution [START_REF] Pawlowski | Implicit weight uncertainty in neural networks[END_REF] or distribution defined over subspaces of W [START_REF] Izmailov | Subspace inference for bayesian deep learning[END_REF].

Experiments

We conduct several experiments on both synthetic and real datasets. We primarily focus on OOD detection performances to compare the methods. The implementation details for the MaxWEnt algorithm are presented in Section 7.5. The source code of the experiments is available on GitHub1 .

Synthetic Experiments

In this section, we provide a qualitative analysis of the MaxWEnt algorithm on low dimensional synthetic datasets. Specifically, we compare the uncertainty estimation produced by MaxWEnt and standard ensemble and Bayesian methods.

SETUP

We consider both classification and regression experiments, performed respectively on the two following datasets:

• Two Moons Classification : We consider the two-moons classification dataset from scikitlearn2 which simulates a two-dimensional binary classification task with moons like distributed classes. The training set is composed of 200 data points generated from the twomoons generator; 50 additional instances are generated to form a validation dataset. The noise level of the generator is set to 0.1.

• 1D Regression : We reproduce the synthetic univariate regression experiment from [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF] with 100 training and 20 validation instances. The input instances are drawn in X ⊂ R according to the mixture of two Gaussians centered respectively in -0.5 and 0.75 with standard deviation 0.1. The outputs y ∈ Y ⊂ R are drawn according to the conditional distribution: p(y|x) ∼ f * (x) + ϵ with ϵ ∼ N (0, 0.02) the noise variable and f * (x) the "ground truth" defined as:

f * (x) = 0.3 (x + sin(2πx) + sin(4πx)) . (33)
In both experiments, the base estimator is a fully-connected neural network with three layers of 100 neurons, each with ReLU activations. For classification, the end layer is composed of one layer with sigmoid activation to produce probabilistic outputs. The end layer for regression is made of two neurons which respectively encode for the conditional mean µ w (x) and conditional standard deviation σ w (x) of the univariate Gaussian N (µ w (x), σ w (x)) as suggested in [START_REF] David | Estimating the mean and variance of the target probability distribution[END_REF] to produce probabilistic outputs in the regression setting. We consider the five following uncertainty quantification methods:

• Vanilla Network, the baseline, which produces uncertainty estimation based on the network probabilistic outputs h w (x) ∈ [0, 1] for classification and σ w (x) ∈ R + for regression. Notice that an ensemble of Vanilla Networks corresponds to the Deep Ensemble method.

• MC-Dropout [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF], with dropout rate selected through hold-out validation NLL, computed using the validation data, among [0.05, 0.1, 0.2, 0.3, 0.5];

• standard BNN (Bayesian Neural Network) (MacKay, 1992;[START_REF] Graves | Practical variational inference for neural networks[END_REF], trained with stochastic variational inference and reparameterization trick [START_REF] Matthew D Hoffman | Stochastic variational inference[END_REF][START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], we use an independent multivariate Gaussian stochastic model q (µ,σ) ∼ N (µ, diag(σ 2 )) and a Normal prior p(w) ∼ N (0, Id). Following common practices for variational Bayes approach to BNNs, we consider a trade-off parameter λ between the NLL and the KL divergence (Wenzel et al., 2020a). The trade-off parameter is selected in

{10 k } k∈[|-3,3|]
through hold-out validation NLL.

• MaxWEnt, with an independent multivariate uniform stochastic model centered on the resulting weights of the Vanilla Network.

• MaxWEnt-SVD, which uses the "SVD" parameterization of Equation ( 9) in addition to the previous MaxWEnt settings.

We use the Adam optimizer (Kingma and Ba, 2015) with learning rate 0.001 and batch-size 32. 10k iterations are used to train the Vanilla Network and 20k iterations for other methods, as the stochastic variational inference requires more iterations to converge. For both tasks, the loss function is the Negative Log Likelihood (NLL). It can be written for the respective classification and regression settings as follows:

L S (w) = - 1 n (x,y)∈S y log(h w (x)) + (1 -y) log(1 -h w (x)) (Classification) (34) L S (w) = - 1 n (x,y)∈S 1 2 log(σ w (x) 2 ) + (y -µ w (x)) 2 σ w (x) 2 (Regression), (35)
with h w ∈ H the neural network of weights w ∈ W such that, for any x ∈ X , h w (x) = (µ w (x), σ w (x)) for the regression setting (cf. [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF]).

To compute uncertainty estimates, we use the entropy metric for classification and the standard deviation of the "Gaussian mixture approximation" introduced in (Lakshminarayanan et al., 2017) for regression. All uncertainty quantification methods except the Vanilla Network produce stochastic outputs, i.e. for any x ∈ X , h w (x) is a random variable as w follows a stochastic model. To produce uncertainty estimates at inference, we then compute P = 50 predictions {h w i (x)} i∈[|1,P |] with w i drawn iid according to the learned weight distribution. Then, the uncertainty estimates for each setting becomes, for any x ∈ X :

u(x) = -h w (x) log h w (x) -1 -h w (x) log 1 -h w (x) (Classification) (36) u(x) = 1 P P i=1 σ w i (x) 2 + µ w i (x) 2 -µ w (x) 2 (Regression), (37)
with h w (x), µ w (x), the average of the respective sets {h w i (x)} i and {µ w i (x)} i . It should be underlined that, the uncertainty metric for classification in Equation ( 36) is the entropy metric applied to the average predicted output over the P stochastic inferences, while the uncertainty metric for regression in Equation ( 37) is the variance formula for the Gaussian mixture composed of P Gaussians of mean µ w i (x) and variance σ w i (x) 2 (Lakshminarayanan et al., 2017). Notice also that, for the Vanilla Network, the estimated uncertainty is independent of P as the method produces the deterministic outputs h w (x). In the regression case, the Vanilla Network uncertainty is u(x) = σ w (x).

To complete the experiments, we also consider ensembles of the previously mentioned uncertainty quantification methods. We build ensembles of N = 5 networks trained independently with different random weight initialization. In this case, the uncertainty metrics are computed in the same way as in the single-network setting through Equation ( 36) and (37) with P predictions for each network in the ensemble, i.e. with a total of N P = 250 predictions.

RESULTS

The regression experiment results are reported in Figure 2. Predicted uncertainties for each method are presented in the form of confidence intervals in light blue. We observe that the Deep Ensemble, MC-Dropout and BNN methods provide larger uncertainty estimates out-of-distribution than indistribution, which offers an efficient way to detect OODs in this case. However, the three methods fail to capture the full epistemic uncertainty, as a significant part of the ground-truth lies outside the confidence intervals. In contrast, MaxWEnt provides relevant confidence intervals outside the training support when extrapolating on the right and left side of the domain. Although, the predicted uncertainties between the two separated parts of the training domain are still under-estimated. This behavior is corrected by MaxWEnt-SVD which fully manages to produce tight confidence intervals in-distribution and uncertainties as large as possible out-of-distribution. The results of the classification experiment are reported in Figure 3. As for the regression experiment, we observe that Deep Ensemble, MC-Dropout and BNN fail to provide relevant uncertainties estimation whereas MaxWEnt and MaxWEnt-SVD are close to the expected behavior of an ideal uncertainty quantifier. Moreover, in this experiment, the first three methods do not offer a proper discrimination between out-of-distribution and in-distribution data. The produced uncertainties are concentrated in the margin between classes and do not increase in the OOD areas behind the training instances. We observe that MaxWEnt and MaxWEnt-SVD manage to increase the uncertainty outside the margin between classes. 36), are reported in shades of blue (darker areas correspond to larger uncertainties).

DISCUSSION

Both experiments on synthetic data strongly highlight the benefit of using MaxWEnt for uncertainty quantification over standard Bayesian and ensemble methods. As discussed in Section 5.3, the MaxWEnt implementation is related to BNN algorithms, however, the predicted uncertainties of MaxWEnt and BNN are very different (cf. Figures 2 and3). These observed discrepancies between the two methods can be explained by their different paradigms. In standard BNN optimization, the main objective is to produce relevant uncertainty estimation inside the training domain. In this perspective, the prior distribution and the trade-off parameters are selected in order to minimize the validation NLL. Consequently, the expansion of the weight distribution is generally limited. In MaxWEnt optimization, the primary goal is to maximize the entropy of the weight distribution as long as the sampled weights are consistent. Although this approach induces a slight penalization of the validation NLL as suggested in Figure 3 (predicted uncertainties in the training domain are larger for MaxWEnt and MaxWEnt-SVD than for BNN), it significantly improves the predicted epistemic uncertainties outside the training domain. Notice that one can sample from the whole MaxWEnt weight distribution to detect OOD and then from "shrunk" weight distribution to provide more accurate prediction for data identified as in-distribution (cf. Figure 4).

When considering the MaxWEnt-SVD results for both experiments (cf. right side of Figures 2 and3), we might judge that the produced out-of-distribution uncertainties are over-estimated; especially in the regression experiment, where the predicted uncertainties become very large almost instantly at the borders of the training domain. However, this behavior is optimal according to the notion of epistemic uncertainty considered in this work. Indeed, epistemic uncertainty is defined through the set of potential candidates for the best hypothesis h w * . Then, as soon as there exist a neural network h in H which fits the training instances and produces very high outputs out-ofdistribution, the learner has no reason, in absence of further regularity consideration, to exclude that the best hypothesis can be modeled by h. If, for some reason, the learner wants to add some prior information on h w * , such as Lipschitz constraints on the network output, this can be achieved, for example, by clipping the scaling variable ϕ ⊙ z during the MaxWEnt inference as done for the weights of the Wasserstein-GAN to impose the 1-Lipschitz constraint [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF]. This boils down to considering a reduced hypothesis space H, which de facto reduces the epistemic uncertainty, but potentially increases the discrepancy between h w * and f * . We present in Figure 4 the impact of clipping on the predicted uncertainties of MaxWEnt-SVD on the regression dataset. We observe that the clipping parameter enables the interpolation between the behavior of the vanilla probabilistic network and the MaxWEnt-SVD behavior. Notice that clipping is performed at "test time", i.e. after the MaxWEnt optimization, which is convenient as the clipping parameter can be selected "a posteriori". 9)) of the fitted MaxWEnt-SVD network, such that q ϕ ∼ w + V min(ϕ ⊙ z, C), with C the clipping parameter.

C = +∞ C = 1.5 C = 1 C = 0.5 C = 0
The comparison between the regression and classification results suggest that out-of-distribution detection is a more difficult task in the classification setting. Indeed, in this setting, the uncertainty quantification methods do not fully manage to increase uncertainty for OOD data behind the training instances of each class. This behavior can be explained by the use of the sigmoid activation at the end-layer, which hardens the epistemic uncertainty estimation as different large outputs are reduced in the same probabilistic output (close to 1 if positive or 0 if negative). In fact, recent outof-distribution detection methods often abandon the use of softmax and sigmoid activation functions at the end layer in favor of distance-based approaches where class assignment is computed through distance to class prototypes [START_REF] Van Amersfoort | Uncertainty estimation using a single deep deterministic neural network[END_REF]. Notice that, we do not consider distancebased uncertainty methods in these synthetic experiments. For these low dimensional problems, using the Euclidean distance to the training instances would provide an almost perfect OOD detector. However, for high dimensional datasets, ensemble-based approaches generally provide better performances [START_REF] Yang | Openood: Benchmarking generalized out-ofdistribution detection[END_REF].

In both experiments, we observe that MaxWEnt-SVD produces uncertainty estimates of better quality than MaxWEnt. The theoretical analysis in Section 4.1.3 suggests that this improvement is related to the weight entropy increase. To evaluate this theoretical claim, we report the evolution of the predicted uncertainties and the weight entropy H(ϕ) through the epochs for both methods in the regression setting (cf. Figure 5). We observe, for both methods, a strong correlation between the increase of the weight diversity (measured by H(ϕ)) and the increase of the uncertainty estimates out-of-distribution. Moreover, the predicted uncertainties of MaxWEnt-SVD quickly increase around epoch 100 as well as its distribution entropy H(ϕ), which becomes higher than the MaxWEnt entropy (H(ϕ) = -0.03 at epoch 125 for MaxWEnt-SVD while H(ϕ) = -2.51 for MaxWEnt). After this stage, the predicted OOD uncertainties are better for MaxWEnt-SVD than for MaxWEnt, especially in the interpolation regime between the two parts of the training domain.

These observations comfort the idea that higher weight diversity for the same level of in-distribution risk produces better uncertainty quantification out-of-distribution.

NEURON ACTIVATION AMPLITUDE AND SCALING PARAMETERS

In the theoretical analysis in Section 4, we show, in the case of fully-connected neural network, that the scaling parameters ϕ k are inversely proportional to the neuron activation amplitude on the training data denoted a 2 (l,k) for the l th layer (cf. Proposition ( 6)). We aim at comforting this theoretical result with empirical observations. For this purpose, we estimate the activation amplitudes in each layer of the MaxWEnt neural network and compare their values with the average of their corresponding scaling parameters (1/b l ) b l j=1 ϕ (l,j,k) . We report the result in Figure 6. The top three graphics present the scaling parameters as a function of the activation amplitudes in the three layers of the MaxWEnt neural network trained on the two moon dataset. We observe a clear relation of inverse proportionality between the two quantities, in line with the theoretical outcomes. The three graphics below present the results for the standard BNN method. We observe the inverse proportionality relationship for the first layer but to a lesser extent than for MaxWEnt. This relationship is diminished in the two next layers. Moreover, we observe that the scaling parameters in the two first layer are globally larger for MaxWEnt than for BNN.

UCI Regression Datasets

SETUP

In this section, we consider the most common UCI regression datasets used to evaluate uncertainty quantification methods. Most previous works evaluate the methods based on the in-distribution NLL computed on a test set drawn from the same distribution as the training set [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF]. In this work, we focus on the methods' ability to detect whether a data point is outside the training support or not. For this purpose, we build OOD detection problems by splitting each dataset in two distinct parts, with one part modeling the training domain and the other part the OOD data.

reached by every method on every dataset. We then consider 10k iterations for ensemble methods and 50k iterations for Bayesian and Bayesian ensemble methods, as stochastic variational inference converges slower than stochastic gradient descent. A callback process is used to monitor the validation NLL of the model every 100 iterations, the network weights corresponding to the iteration of best validation NLL are restored at the training end. For MaxWEnt, the scale parameters are saved if the validation NLL is below the threshold defined in Section 7.5.

Methods

Abrv. Kind Nets Preds Total Parallel Hyper-parameters

Deep Ensemble DE Ensemble 5 1 5 ✓ None Negative Correlation NC Ensemble 5 1 5 ✓ λ ∈ {10 -k ; k ∈ [|0, 5|]} Maximize-Overall-Diversity MO Ensemble 5 1 5 × λ ∈ {10 -k ; k ∈ [|0, 5|]} Anchored-Networks AN Ensemble 5 1 5 ✓ Σ, λ ∈ [0.
✓ λ ∈ {10 k ; k ∈ [| -2, 2|]} MaxWEnt (×5) ME Bay Ens 5 50 250 ✓ λ = 10, ϕinit = soft(-5) MaxWEnt-SVD (×5) ME+ Bay Ens 5 50 250 ✓ λ = 10, ϕinit = soft(-10)
Table 1: Competitors Summary. The columns "Nets" and "Preds" respectively report the number of networks in the ensemble and the number of predictions at inference for one network. "Total" is the total number of predictions (Nets × Preds). The "Parallel" column reports whether the ensemble can be trained in parallel or not. When a list is given in the "Hyper-parameters" section, the value is selected based on hold-out validation NLL. "soft" is the abbreviation for the "softplus" function: soft(x) = log(1 + exp(x)).

RESULTS

To evaluate the model performances, we use the metric defined in Equation (37) which defines an uncertainty score for each data point, this score is used to compute the AUROC metric between in-distribution and OOD data which is a commonly used metric in the OOD detection setting [START_REF] Yang | Openood: Benchmarking generalized out-ofdistribution detection[END_REF]. All results are reported in Table 2. Each experiment is performed only once to reduce the computational time of the experiments. As many different datasets are used, this is sufficient to obtain statistically significant results. We report the results by kind of methods: ensemble, Bayesian and Bayesian ensemble. The best results for each dataset in each category is emphasized in bold. We report the average AUROC among extrapolation and interpolation experiments and the rank of the methods. Our observations can be summarized as follows:

• MaxWEnt-SVD (ME+) outperforms all other approaches, with or without ensembling.

The second-best non MaxWEnt approach is 11.3 points behind in extrapolation and 18 points in interpolation in terms of average AUROC. Ensembling improves from 4.5 points in extrapolation and 1.2 points in interpolation. Table 2: UCI experiments OOD detection results. AUROC scores for OOD detection are reported. The best score for each category is emphasized in bold (higher scores are better). The three last rows for the extrapolation and interpolation settings report the average AUROC over the eight datasets (Avg AUC), the rank of the method over all methods according to the average score (Rank) and the result of the Poisson Binomial Test (PBT) which reports the probability that the method is better than MaxWEnt-SVD (×5).

• The ensemble version of MaxWEnt (ME) is third best behind the two versions of MaxWEnt-SVD. The single-network MaxWEnt, however, provides poor performances, which advocates for the use of ensembling or SVD parameterization.

• AUROC scores are higher in extrapolation than in interpolation, suggesting that the second task is more difficult. This seems reasonable as the network is conditioned on both sides of the domain in the interpolation case, while being conditioned only in one side of the OOD domain in extrapolation.

• Ensembling of Bayesian methods generally improves the results compared to the singlenet from 7 points on average. However, using Bayesian combined in ensemble increases the training and inference time by the number of members as well as the required memory size. Note that, for these methods, the ensemble training can be conducted in parallel, which can alleviate the training time burden.

Finally, to evaluate the in-distribution performance of the methods, we compute, on the test set, the Negative Log Likelihood (NLL) as well as the Expected Calibration Error (ECE) [START_REF] Levi | Evaluating and calibrating uncertainty prediction in regression tasks[END_REF]. The average metrics computed over the eight datasets are reported in Table 3. To evaluate the impact of clipping on the in-distribution performance, we also report the average metrics for the "clipped" MaxWEnt weight distribution: q ϕ ∼ w + min(ϕ ⊙ z, C) (independent) and q ϕ ∼ w + V min(ϕ ⊙ z, C) (SVD), with C the clipping parameter selected in [+∞, 10, 5, 2, 1, 0.5, 0.2, 0.1, 0] according to the validation NLL performance. We observe that the MaxWEnt algorithms generally penalize the test NLL and ECE compared to the baselines. In particular, the average NLL of MaxWEnt-SVD (x5) is larger than the ones produced by the other methods, suggesting that stronger OOD detection results come with weaker test performances. However, we observe that the use of weight clipping improves the MaxWEnt test performances, which become comparable to those of the baselines. These results suggest that the learner should use the "unclipped" MaxWEnt predicted uncertainties to perform OOD detection and the "clipped" MaxWEnt inferences to provide predictions for data identified as in-distribution. This requires two different inferences: one for OOD detection and one for prediction.

CityCam Regression Datasets

SETUP

This section is dedicated to uncertainty quantification on the real-world dataset CityCam [START_REF] Zhang | Understanding traffic density from large-scale web camera data[END_REF]. This dataset is composed of images gathered from several cameras monitoring the traffic in a city. Each camera records between 1k and 6k images dispatched over several days and hours. The task consists in counting the number of vehicles in the image using a neural network. This task is useful, for instance, to monitor the traffic in the city. To produce in-distribution vs outof-distribution splits, we consider the three following experiments introduced in (de Mathelin et al., 2023):

• Camera-Shift: Images coming from ten different cameras are selected for this experiment. At each round, five cameras are randomly selected to form the training dataset, while the five remaining cameras are used as OOD dataset. On average, both dataset contain around 20k images.

• BigBus-Shift: Images from five cameras are considered in this experiment. Some of them are marked as "big-bus" if a large vehicle mask a significant part of the image (cf. [START_REF] Zhang | Understanding traffic density from large-scale web camera data[END_REF]). These images are selected to form the OOD dataset, while the remaining ones compose the training set. The in-distribution and OOD datasets respectively contain around 17k and 1k images.

• Weather-Shift: For this experiment, we consider the images gathered from three cameras recorded during February the 23 th from 9 am to 6 pm. On this particular day, weather conditions changed considerably between the beginning and end of the day. The dataset is split into two subsets: images recorded before 2 pm are considered as in-distribution, while the others as out-of-distribution. After 4 pm, water drops landed on the cameras blur the images, which causes a clear domain shift (cf. Table 4).

Camera-Shift

BigBus-Shift Weather-Shift Source Target Table 4: CityCam Experiments setup. An example of a webcam image is given for each domain for the three settings: Camera-Shift, BigBus-Shift and Weather-Shift.

The three previous experiments model different out-of-distribution scenarios. OOD data for the BigBus-Shift and Weather-Shift experiments can be considered as "anomalies". When a large vehicle masks an important part of the image or when the images become too blurry due to rain drops, it becomes very difficult to produce accurate predictions even for a human (cf. Table 4). In this case, the learner may expect uncertainty quantification methods to provide large prediction uncertainty in order to detect such abnormal events. The paradigm slightly differs for the Camera-Shift experiment. In this setting, the domain shift essentially lies in the background differences between cameras. Since the model is trained on five different cameras, the learner might expect the model to "generalize" and to provide accurate predictions for the images of the novel cameras.

As preprocessing, we use the features of the last layer of a ResNet50 [START_REF] He | Deep residual learning for image recognition[END_REF] pretrained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. We consider the same setting as for the UCI experiments in terms of base estimator, optimization parameters, callbacks and competitors.

RESULTS

For each experiment, we compute the AUROC metric and the False Positive Rate at 95 percent (FPR@95) using the uncertainty scores given in Equation (37). The computed metrics are reported in Table 5. We observe an important discrepancy between the scores produced by MaxWEnt-SVD and the ones of other methods. The gap is particularly large for the Camera-Shift experiments, where every method produces an average FPR@95 score around 97% while MaxWEnt-SVD provides a false positive rate of 29.4% in the single-net setting and 15.3% with ensembling. Similarly, MaxWEnt-SVD outperforms every other method for the BigBus-Shift and Weather-Shift experiments. The MaxWEnt algorithm without SVD parameterization provides the second best results in the Bayesian and ensemble category, however, the performance gains compared to the baselines are much smaller than the ones obtained with the SVD parameterization. Notice, however, that MaxWEnt-SVD requires more computational time because of the additional matrix multiplication caused by the SVD alignment (cf. Section 5.4). A visualization of the MaxWEnt uncertainty evolution on the Weather-Shift experiment is presented in Figure 7. We compare the evolution of the confidence intervals produced by Deep Ensemble and MaxWEnt (x1) along the day. The left part of Figure 7 corresponds to the images recorded between 2:00 pm and 2:30 pm which are the closest OOD data to the training domain. We observe that, in this time interval, both methods produce tight uncertainty intervals which well cover the ground-truth. The right part of Figure 7 corresponds to the time interval 4:00 pm to 6:00 pm. During this period of time, rain drops progressively land on the camera objective and blur the image. At some point around 5:30 pm, the deterioration of the image becomes critical for the vehicles' counting. We observe that, in this case, the size of the confidence intervals produced by Deep Ensemble do not increase. Paradoxically, the Deep Ensemble method seems to produce more confident predictions around 5:30 pm than before 2:30 pm. Conversely, the MaxWEnt predicted uncertainty progressively grows after 5:00 pm in correlation with the increasing task difficulty. Notice that, at some point, even the ground-truth is not reliable anymore, as the human annotator was not able to accurately count the actual number of vehicles.

IMPACT OF THE TRADE-OFF PARAMETER

We aim at evaluating the impact of the trade-off parameter λ in the MaxWEnt optimization (4). We choose a fixed parameter λ = 10 in all experiments with the underlying idea that λ should not be selected based on validation NLL to not foster small λ values (cf. Section 7.5). We present in Figure 8 the AUROC scores of MaxWEnt (×5) for the OOD detection performed on the three CityCam experiments for different values of λ. We observe that the considered value λ = 10 is always sub-optimal, in particular for the Camera-Shift experiment, where the AUROC score for λ = 10 is more than 10 points below the score obtained with λ = 100. It can be noticed that the MaxWEnt performances are above the Deep Ensemble ones for a large panel of λ values, in particular for the Weather-Shift experiment. The score's decrease for large values of λ in the Camera-Shift and BigBus-Shift experiments can be explained by the instabilities caused by overincreasing the weight entropy. This study of the trade-off parameter impact suggests that future improvements can be reached by finding the proper way of selecting λ.

Camera-Shift

BigBus-Shift Weather-Shift We consider the Open-Set-Recognition (OSR) and Out-of-Distribution detection extensive benchmark (OpenOOD), developed in [START_REF] Yang | Openood: Benchmarking generalized out-ofdistribution detection[END_REF] which compare more than 30 OSR and OOD detection methods on various classification datasets. The source code for the MaxWEnt experiments, conducted within the OpenOOD benchmark, is available on GitHub3 . We focus on the OSR and OOD detection experiments:

• Open-Set-Recognition: For the OSR benchmark, each dataset is divided in two parts by removing the instances corresponding to some classes from the training set. The goal is to detect whether an instance comes from a training class or a removed one. Each experiment is repeated five times with random selection of the training classes. Four datasets are considered: MNIST [START_REF] Deng | The mnist database of handwritten digit images for machine learning research [best of the web[END_REF], CIFAR10, CIFAR100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] and TinyImageNet [START_REF] Torralba | 80 million tiny images: A large data set for nonparametric object and scene recognition[END_REF].

• Out-Of-Distribution Detection: For the OOD detection benchmark, data coming from all classes are used at training time. The goal is then to discriminate between the test set and data coming from other datasets (with no overlapping classes). Two types of OOD datasets are considered: Far-OOD which corresponds to images very different from the training instances (e.g. CIFAR10 vs MNIST) and Near-OOD which corresponds to images close to the training instances (e.g. CIFAR10 vs CIFAR100). This last type of OOD detection is considered more challenging and is closely related to the OSR setting [START_REF] Yang | Openood: Benchmarking generalized out-ofdistribution detection[END_REF]. Three datasets are considered: MNIST, CIFAR10 and CIFAR100.

A summary of the datasets used in each experiment is presented in Table 6. The AUROC score is used to evaluate the discrimination accuracy between test and OOD datasets. To compute the "OOD scores", a variety of algorithms are considered. They can be classified in two main categories:

• post-hoc Methods, defined as methods that can be applied "directly" on a pretrained single network, independently of the training process. These methods are considered practical and model-agnostic [START_REF] Yang | Openood: Benchmarking generalized out-ofdistribution detection[END_REF]. Among them, we can further distinguish the methods that do not require the training data: MSP [START_REF] Hendrycks | A baseline for detecting misclassified and out-of-distribution examples in neural networks[END_REF], MLS (Hendrycks et al., 2022a), ODIN [START_REF] Liang | Enhancing the reliability of out-of-distribution image detection in neural networks[END_REF], EBO [START_REF] Liu | Energy-based out-of-distribution detection[END_REF], GradNorm (Huang et al., 2021a), ReAct [START_REF] Sun | React: Out-of-distribution detection with rectified activations[END_REF], KLM (Hendrycks et al., 2022a) and TempScale (Guo et al., 2017a) and the methods that uses the training set: OpenMax [START_REF] Bendale | Towards open set deep networks[END_REF], MDS (Lee et al., 2018a), Gram (Sastry and Oore, 2020), VIM (Wang et al., 2022a), KNN (Sun et al., 2022), DICE (Sun and Li, 2022). Notice that, except for MSP and MLS, all post-hoc methods at least require the use of a validation dataset to fine-tune their hyper-parameters.

• Non post-hoc Methods, including all methods which do not belong to the previous category, essentially because they require a specific training process (in terms of training loss or data augmentation for instance). This category of methods includes anomaly detection approaches: DeepSVDD [START_REF] Ruff | Deep one-class classification[END_REF], CutPaste [START_REF] Li | Cutpaste: Self-supervised learning for anomaly detection and localization[END_REF], DRAEM [START_REF] Zavrtanik | Draem-a discriminatively trained reconstruction embedding for surface anomaly detection[END_REF]; OOD detection methods with specific training process: ConfBranch (DeVries and Taylor, 2018), G-ODIN [START_REF] Hsu | Generalized odin: Detecting outof-distribution image without learning from out-of-distribution data[END_REF], CSI [START_REF] Tack | Csi: Novelty detection via contrastive learning on distributionally shifted instances[END_REF], ARPL [START_REF] Chen | Adversarial reciprocal points learning for open set recognition[END_REF], MOS (Huang and Li, 2021), OpenGAN [START_REF] Kong | Opengan: Open-set recognition via open data generation[END_REF], VOS [START_REF] Du | Vos: Learning what you don't know by virtual outlier synthesis[END_REF], LogitNorm [START_REF] Wei | Mitigating neural network overconfidence with logit normalization[END_REF]; uncertainty-based approaches: MCdropout (Gal andGhahramani, 2016), Deep Ensemble (Lakshminarayanan et al., 2017); and data augmentation methods: MixUp (Thulasidasan et al., 2019), CutMix [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF], PixMix (Hendrycks et al., 2022b). Table 6: OpenOOD Experiments Summary

According to [START_REF] Yang | Openood: Benchmarking generalized out-ofdistribution detection[END_REF], fair comparison between methods should be done among each category, as non post-hoc methods may benefit from their specific training process. Notice that this classification is not perfect. Post-hoc methods are considered model-agnostic, as they can generally be "plugged" to any pretrained network. However, most post-hoc methods generally require the end-layer of the network to produce logits. post-hoc methods are considered practical because they generally require less computational time than the non post-hoc methods. This computational efficiency is mainly due to the training process economy. It should be mentioned, however, that inference time for some post-hoc methods may become important for large training dataset. For instance, KNN computes the distance between test data and all the training set in the penultimate network layer. This may lead to important memory and computational burden if the training dataset is very large.

The MaxWEnt algorithm can be plugged directly on a pretrained neural network h w . It may not be totally considered as post-hoc, as it requires the additional training of the scale parameters ϕ. However, this training may be done with few epochs and also on a small extract of the training dataset. For our experiments, we trained MaxWEnt with the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with learning rate 5 • 10 -4 and 20 epochs. We also consider an ensemble of five MaxWEnt network. For inference, we use P = 10 predictions.

RESULTS

The results are reported on Figure 9, we compare AUROC scores between MaxWEnt (x1) and MaxWEnt (x5) (in red) to the previously mentioned methods (in blue). Note that we do not include OOD detection methods which require auxiliary OOD datasets during training to the comparison, as MaxWEnt do not use this kind of additional information. post-hoc methods are marked with a dagger †. We group all experiments in the three main categories: OSR, NearOOD and FarOOD as described in Table 6. The reported AUROC scores are averaged over all experiments inside each category and over five different random seeds. We observe that MaxWEnt (x1) is ranked 3 rd , 8 th and 2 nd for respectively the OSR, FarOOD and NearOOD experiments compared to all methods. When restricting the comparison to post-hoc methods, the MaxWEnt (x1) rankings become 1 st , 3 rd and 1 st which demonstrates the effectiveness of the approach. It should be underlined that MaxWEnt (x1) is outperforming all other methods in the particular setting OSR and Near-OOD which are known to be the more challenging. For these two experiments, the MaxWEnt (x1) performance closely match those of Deep Ensemble, which requires the training of five neural networks and thus more computational resources. The ensemble of MaxWEnt networks provide an additional gain of around 2 points of AUROC scores and is then ranked 1 st , 3 rd and 1 st compared to all methods. However, this improvement requires the training of five networks, which increases the computational time.

Implementation Choices

We present hereafter the implementation choices that we consider as "good practice" for MaxWEnt:

• Initialization: In our proposed setup, the weight mean E q ϕ [w] = w is frozen during the MaxWEnt optimization and independent of the parameters ϕ. The weight vector w is derived from a pretrained network h w fitted on the training data. The ϕ parameters are initialized with a small constant value C ≪ 1. Therefore, the weight distribution q ϕ is initialized as a peaked distribution around w, which already provides low empirical risk. Notice that the use of pretrained weights to initialize the mean of q ϕ is similar to the common practice in Laplace approximation [START_REF] Ritter | A scalable laplace approximation for neural networks[END_REF], where the mean of the posterior distribution is set to the maximum a posteriori estimation (MAP). Moreover, in the case where a pretrained network is already available, the use of pretrained weights reduces the computational time. Note, finally, that we also consider a "softplus" activation of the ϕ parameters to smooth the increase of the weight entropy in earlier stages: ϕ = log(1 + exp(u)).

• Trade-off parameter: The MaxWEnt optimization (4) involves a trade-off between empirical risk minimization and entropy maximization, which is controlled by the trade-off parameter λ. A small λ penalizes larger average risks, while a large λ favors the weight distribution expansion. Obviously, the learner has to accept to penalize the empirical risk to offer room for the weight distribution to expand. In this perspective, we do not recommend selecting the trade-off parameter based on validation risk minimization. The λ value should be selected large enough to speed up the increase of the weight entropy, while not too large to avoid optimization instabilities. We observe through numerical experiments that a relatively large range of λ value is acceptable to provide an efficient trade-off (cf. Section 7.3.3). However, we do not find a satisfactory heuristic to set the hyper-parameter value. In all our experiments, we choose to consider a fixed trade-off λ = 104 . Obviously, choosing the same value of λ in any case seems intuitively sub-optimal, as the range of the training risk can vary from one problem to another. However, we observe that, when normalizing the output labels in regression and using logits in classification, the value λ = 10 appears to be a good trade-off.

• restores the weights of the network for the epoch which provides the best validation risk. Of course, we cannot consider such a technique for the MaxWEnt optimization, as increasing the weight entropy generally induces a small degradation of the validation risk. We then propose to save the network weights according to a threshold computed at the beginning of the optimization. Motivated by the maximum entropy framework developed in Section 5.1, we propose to estimate the performance threshold τ by the validation risk of the pretrained network h w plus a statistical error:

τ = L S val (w) + 2 n val (x,y)∈S val (ℓ(h w (x), y) -L S val (w)) 2 . ( 38
)
The second term is proportional to the standard deviation of the errors over the validation dataset.

• Ensemble It should be underlined that the proposed parameterizations (7) and ( 9) limit the range of the weight distribution around a neighborhood of w. A straightforward improvement would be to apply Algorithm (1) on a set of weights w (j) coming from a pretrained deep ensemble [START_REF] Balaji Lakshminarayanan | Simple and scalable predictive uncertainty estimation using deep ensembles[END_REF]. Conceptually, this comes down to describing q ϕ as a mixture with, for any j ∈ [|1, m|], ϕ (j) ∈ R d , z (j) ∼ Z and π ∼ U({1, ..., m}):

q ϕ ∼ m j=1 1(π = j) ω(ϕ (j) , z (j) ), (39)
with ω(ϕ (j) , z (j) ) = w (j) + ϕ (j) ⊙ z (j) or ω(ϕ (j) , z (j) ) = w (j) + V ϕ (j) ⊙ z (j) . In practice, we apply Algorithm (1) to each of the pretrained networks with the scaling parameterization ω(ϕ (j) , z (j) ). Notice that, if there is no overlap between the mixture components, the ensemble parameterization necessarily results in a weight distribution of higher entropy for the same empirical risk level, and then leads to a more efficient parameterization than the single network setting (cf. Section 5.1). A guideline to choose the centers w (j) is then to avoid overlapping, which can be achieved with centers distant from each other. Thus, combining MaxWEnt with techniques as RDE (D'Angelo and Fortuin, 2021), AnchorNet [START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF] or DARE [START_REF] De Mathelin | Deep antiregularized ensembles provide reliable out-of-distribution uncertainty quantification[END_REF] may offer increased performances.

Limitations and Perspectives

In this work, we develop the MaxWEnt algorithm to improve OOD detection with stochastic neural networks. The main goal of MaxWEnt is to produce samples with larger weight diversity compared to standard Bayesian and ensemble methods. Our experiments show that MaxWEnt fulfills its promise, it increases the weight entropy and provides better OOD detection results. Moreover, we show that the more the weight entropy, the better the OOD detection (for the same level of average empirical error).

• Increasing the weight entropy: The weight entropy increase is strongly conditioned by the weight parameterization. We show that the use of the SVD-parameterization is already an important improvement compared to the use of independent scaling parameters. However, more efficient parameterization may be obtained with other techniques as normalizing flows [START_REF] Louizos | Multiplicative normalizing flows for variational bayesian neural networks[END_REF] or weight subspaces [START_REF] Izmailov | Subspace inference for bayesian deep learning[END_REF]. Nevertheless, the maximum entropy framework provides a general guideline for selecting the weight parameterization: an efficient stochastic model should enable large increases of the weight entropy in low empirical risk regions of the weight space.

• Penalized performances in-distribution: We have seen that increasing the entropy penalizes the in-distribution performances. However, this negative result can be mitigated by the use of "shrunk" weight distribution obtained through weight clipping (cf. Section Sections 7.1.3 and 7.2.2). The learner can use the MaxWEnt uncertainties to discriminate between ID and OOD data, and then use the prediction obtained with "shrunk" weight distribution for the data classified as ID.

• SVD-parameterization for Convolutions: For now, the SVD-parameterization is only developed for fully connected neural networks, but it may also be applied to convolutional layers. Convolutions apply the same kernel to multiple windows of one channel. To use the SVD-parameterization in this context, one idea is to concatenate all the windows on which the kernel is applied for all training data and then compute the SVD decomposition of the resulting dataset.

• General Bayesian and ensemble limitations: The developed MaxWEnt approach improves upon Bayesian and ensemble methods in terms of weight diversity. However, it still inherits the other limitations of these approaches, which principally include the computational burden in training and inference. Future work will then consider the use of "Laplace-like" approximation to reduce the computational time of MaxWEnt (cf. Section 5.4).

Conclusion

In this work, we tackle the over-confidence issue encountered with standard Bayesian and ensemble methods outside the training domain. Building on the maximum entropy principle, we show that penalizing the empirical average error with the weight entropy leads to larger hypothesis diversity and, then, to improved OOD detection. Theoretical analysis shows that the behavior of the developed MaxWEnt approach is related to the amplitude of the neuron activation on the training data.

In MaxWEnt neural networks, weakly activated neurons play a more important role in the OOD detection in comparison to vanilla probabilistic networks. Motivated by this quest of entropy maximization and by the outcomes of our theoretical analysis, we propose the SVD parameterization to take advantage of correlations between weights with limited additional complexity. Numerical experiments show the benefit of the method and highlight the link between weight entropy and OOD detection performances. We show that the maximum entropy framework offers a guideline to rank two weight distributions of same empirical risk, the one with the largest entropy should be preferred to improve OOD detection. Moreover, we advocate for the use of stochastic models that foster the increase of the weight entropy, as the SVD parameterization. We are convinced that this approach is a step forward in the safety of deep learning. Although many challenges have to be resolved such as the training and inference computational time.

Proof Let's consider the multivariate Gaussian variable Z ∼ N (0, Σ) with Σ ∈ R d×d . We denote p Z (z) its probability density function such that, for any z ∈ R d :

p Z (z) = 1 (2π) d |det(Σ)| exp - 1 2 z T Σ -1 z . (44)
Then,

-2 log(p Z (z)) = d log(2π) + log(|det(Σ)|) + z T Σ -1 z. ( 45
)
We now consider the eigen-decomposition of Σ -1 , such that

Σ -1 = Q T diag(1/λ)Q with Q ∈ R d×d
an orthogonal matrix and λ the vector of eigenvalues of Σ. The following equality holds:

z T Σ -1 z = (Qz) T diag(1/λ)(Qz) = u T diag(1/λ)u = d k=1 u 2 k λ k . (46)
Moreover, for any z ∼ N (0, Σ), the variable u = Qz follows the distribution N (0, QΣQ T ) = N (0, diag(λ)). We then deduce that:

E[z T Σ -1 z] = d k=1 E[u 2 k ] λ 2 k = d k=1 λ 2 k λ 2 k = d. (47)
Finally, we can derive the following formula for the entropy of Z:

- Let's now consider the variable z ∼ N (0, Id d ). According to Lemma (7), the variable A(z ⊙ ϕ) follows the distribution N (0, A diag(ϕ 2 )A T ). Then, according to Lemma (8) and by invariance of the entropy by translation, the entropy of the distribution q ϕ (w) ∼ w + A(z ⊙ ϕ) is written: (50)

H(ϕ) = -E[log(q ϕ (w))] = C + 1 2 log(|det(A diag(ϕ 2 )A T )|) = C + 1 2 log(|det(A) det(diag(ϕ 2 ))det(A T )|), (49)
The objective function of Problem ( 11) is convex and admits a solution. Moreover, the partial derivative of the objective with respect to ϕ 2 k is written:

∂G(ϕ) ∂ϕ 2 k = a 2 k - λ ϕ 2 k . ( 58
)
As a consequence, the gradient of G is null if and only if

ϕ 2 k = λ a 2 k , (59)
which is well-defined when assuming a 2 k > 0.

A.3 Proof of Proposition 3 Proof Let's consider ϕ ∈ R b , V the matrix of eigenvectors of 1 n X T X with s 2 the corresponding vector of eigenvalues and z ∼ Z. The average training risk for the weight w = w + V (ϕ ⊙ z) can be written as follows:

E Z 1 n ||X(w + V (ϕ ⊙ z)) -y|| 2 2 = E Z 1 n ||XV (ϕ ⊙ z)|| 2 2 + 1 n ||Xw -y|| 2 2 . (60)
We notice that:

1 n ||XV (ϕ ⊙ z)|| 2 2 = 1 n ||XV diag(ϕ)z|| 2 2 = z T diag(ϕ) T V T 1 n X T X V diag(ϕ)z = z T diag(ϕ) T diag(s 2 )diag(ϕ)z = z T diag(s 2 ϕ 2 )z = b k=1 s 2 k ϕ 2 k z 2 k . (61)
Then,

E Z 1 n ||X(w + V (ϕ ⊙ z)) -y|| 2 2 = b k=1 s 2 k ϕ 2 k + 1 n ||Xw -y|| 2 2 . ( 62
)
The continuation of the proof is similar to the proof in Appendix (A.2) with s 2 k instead of a 2 k .

A.4 Proof of Proposition 4

Proof Let q

(1) ϕ * , q

(2)

ϕ * be the respective optimal weight distributions for the scaling and the SVD parameterization. Then, q

(1)

ϕ * ∼ w + λ a ⊙ z (63) q
(2)

ϕ * ∼ w + V ( λ s ⊙ z), (64)
with z ∼ Z. Considering Equations ( 56) and ( 62), both average empirical losses are written: (66)

E q (1)
Then,

E q (1) ϕ * [L S (w)] = E q (2) ϕ * [L S (w)] = λ b + 1 n ||Xw -y|| 2 2 . ( 67
)
Moreover, both entropy can be written:

E q (1)
Let's denote M = 1 n X T X, by definition, we have the following equalities: -log(q

M = V T diag(s 2 )V (
(2)

ϕ * ) ≥ E q (2) ϕ *
-log(q

(1) ϕ * ) .

(74)

A.5 Proof of Proposition 6

The proof consists in first rewriting the optimization problem (23) as a maximum entropy problem with a constraint over the average empirical risk. Then, we show that ϕ * is solution of the optimization problem (OP) augmented with additional equality constraints in the hidden layers. We then remove the constraint over the average empirical risk and show that the solution ϕ † of the resulting OP provides a distribution with higher entropy than ϕ * . By splitting the OP in sub-optimization problems by hidden layer, we show that ϕ † verifies Equation ( 26). Then, using recursively Assumption (5) on the activation function, we show that, for any layer, the first and second moments of the neuron activation are the same for both distribution q ϕ † and q ϕ * . We then prove the equality of empirical risk for ϕ † and q ϕ * , leading to show that ϕ † is solution of Problem ( 23), from which we conclude that ϕ † = ϕ * , as the solution is unique.

From which we deduce that ϕ * * = ϕ * , as the solution of Problem ( 75) is assumed unique. Moreover, ϕ * is the unique solution of Problem (76).

We now remove the constraint over the average empirical risk and consider the following alternative optimization problem: (83)

Considering a similar argument as before, the solution ϕ † of problem ( 83) necessarily provides a distribution of larger or equal entropy than ϕ * , i.e. ).

(84)

Moreover, the optimization problem (83) can be decomposed in multiple sub-problems such that: (86)

ϕ † = L l=0 b l j=1 ϕ † (l,j) , (85)
Then, by writing the Karush-Kuhn-Tucker conditions of the above optimization problem we get the following expression for the solution:

ϕ † (l,j,k) 2 = σ 2 (l,j) b a 2 (l,k) ∀ k ∈ [|1, b|]. (87)
Thus, ϕ † verifies Equation ( 26). We now need to show that ϕ † provides the same empirical risk than ϕ * . For this purpose, we consider l ∈ [|0, L -1|] and assume that the first and the second moments of the neuron activation in layer l are the same for ϕ * and ϕ † , we will then show that this property is true in layer l + 1. Let's then assume that:

b i=1 E q ϕ * ψ (l,j) (x i ) = b i=1 E q ϕ † ψ (l,j) (x i ) ∀ j ∈ [|1, b|] (88) b i=1 E q ϕ * ψ (l) (x i )ψ (l) (x i ) T = b i=1 E q ϕ † ψ (l) (x i )ψ (l) (x i ) T . (89)
Let's define U i = (U i1 , ..., U ip ) with U ij = ψ (l) (x i ) T w (l,j) ∀i ∈ [|1, b|], ∀j ∈ [|1, b|]. Considering Equation ( 88), for any j ∈ [|1, b|], we have:

n i=1 E q ϕ † [U ij ] = n i=1 E q ϕ † ψ (l) (x i ) T w (l,j) = n i=1 E q ϕ * ψ (l) (x i ) T w (l,j) = n i=1 E q ϕ * [U ij ] . (90)
Moreover, for any k, j ∈ [|1, b|] such that k ̸ = j, we have:

n i=1 E q ϕ † U i U T i kj = n i=1 E q ϕ † U ik U T ij = n i=1 E q ϕ † ψ (l) (x i ) T w (l,k) w T (l,j) ψ (l) (x i ) = n i=1 b u=1 b v=1 E q ϕ † ψ (l,u) (x i )ψ (l,v) (x i ) E q ϕ † w (l,k,u) w (l,j,v) = n i=1 b u=1 b v=1 E q ϕ † ψ (l,u) (x i )ψ (l,v) (x i ) w (l,k,u) w (l,j,v) = n i=1 w T (l,k) E q ϕ † ψ (l) (x i )ψ (l) (x i ) T w (l,j) = n i=1
w T (l,k) E q ϕ * ψ (l) (x i )ψ (l) (x i ) T w (l,j) (considering Equation ( 89))

= n i=1 E q ϕ * U i U T i kj . (91)
Then, for any j ∈ [|1, b|], we have:

n i=1 E q ϕ † U i U T i jj = n i=1 E q ϕ † ψ (l) (x i ) T w (l,j) 2 = n i=1
V q ϕ † ψ (l) (x i ) T (w (l,j) -w (l,j) ) + E q ϕ † ψ (l) (x i ) T w (l,j)

2 = n i=1 b k=1 E q ϕ † [ψ (l,k) (x i ) 2 ]ϕ † (l,j,k) 2 + w T (l,j) E q ϕ † ψ (l) (x i )ψ (l) (x i ) T w (l,j) = n i=1 b k=1 E q ϕ * [ψ (l,k) (x i ) 2 ]ϕ †
(l,j,k) 2 + w T (l,j) E q ϕ * ψ (l) (x i )ψ (l) (x i ) T w (l,j) .

(

) 92
Where the last equality is deducted from Equation (89). Moreover, the first term can be developed as follows: V q ϕ * ψ (l) (x i ) T w (l,j) -w (l,j) .

(93)

We then deduce that:

n i=1 E q ϕ † U i U T i jj = n i=1 E q ϕ * U i U T i jj . (94)
Equations ( 91) and ( 94) implies that n i=1 E q ϕ † U i U T i = n i=1 E q ϕ * U i U T i . Considering this last equality, Equation (90) and Assumption (5), we then conclude that:

n i=1 E q ϕ † [ζ (U i )] = n i=1 E q ϕ * [ζ (U i )]
(95)

n i=1 E q ϕ † ζ (U i ) ζ (U i ) T = n i=1 E q ϕ * ζ (U i ) ζ (U i ) T . (96)
Where, ζ (U i ) = ζ ψ (l) (x i )w (l,1) , ..., ζ ψ T (l) (x i )w (l,p) = ψ (l+1) (x i ).

(97)

Then Equations ( 95) and ( 96) are equivalent to the moments' equality in Equations ( 88) and (89) applied to layer l + 1. As these equations are true for l = 0, then, by recurrence, we have Equations ( 88) and ( 89) for l = L + 1, then:

n i=1 E q ϕ † [h(x i )] = n i=1 E q ϕ * [h(x i )] and n i=1 E q ϕ † h(x i ) 2 = n i=1 E q ϕ * h(x i ) 2 . ( 98
)
Moreover, by developing the empirical risk, we have:

L S (w) = n i=1 (h(x i ) -y i ) 2 = n i=1 h(x i ) 2 -2h(x i )y i + y 2 i . ( 99
)
From which we deduce that:

E q ϕ † [L S (w)] = E q ϕ * [L S (w)] . (100)
Then, considering Equation ( 84) and the uniqueness of the solution of Problem (76), we conclude that ϕ † = ϕ * .

Figure 1 :

 1 Figure 1: Uncertainty Estimation Comparison. Above: "two-moons" 2D classification dataset.Below: 1D-regression[START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF]. For classification, uncertainty estimates, in shades of blue, are computed with the average of prediction entropy over multiple predictions (darker areas correspond to higher uncertainty). For regression, the ground-truth is represented in black and the predicted confidence intervals of length 4σ w (x) in light blue, with σ w (x) computed as the standard deviation over multiple predictions. Figure (c) presents the result obtained with MaxWEnt when no regularity assumption is made on the labeling function. Figure (d) presents the result obtained when adding Lipschitz constraint. The full description of the synthetic experiments is presented in Section 7.1.

  Figure 1: Uncertainty Estimation Comparison. Above: "two-moons" 2D classification dataset.Below: 1D-regression[START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF]. For classification, uncertainty estimates, in shades of blue, are computed with the average of prediction entropy over multiple predictions (darker areas correspond to higher uncertainty). For regression, the ground-truth is represented in black and the predicted confidence intervals of length 4σ w (x) in light blue, with σ w (x) computed as the standard deviation over multiple predictions. Figure (c) presents the result obtained with MaxWEnt when no regularity assumption is made on the labeling function. Figure (d) presents the result obtained when adding Lipschitz constraint. The full description of the synthetic experiments is presented in Section 7.1.

  Inputs: Input data x, parameterization ω, scaling vector ϕ, sample size P 2: Outputs: Prediction sample (ŷ 1 , ..., ŷP ) 3: for 1 ≤ i ≤ P do

  The set of network weights verifies W ⊂ R d , with d = Lb 2 + b the number of weights in the network (bias parameters are not considered here). For any weights w ∈ W, w (l,j) ∈ R b denotes the weights between the layer l and the j th components of the layer l + 1 for l ∈ [|0, L|] and j ∈ [|1, b l |], with b l = 1 if l = L and b l = b otherwise. We consider the activation function ζ : R → R such that, for any x ∈ X , any l ∈ [|0, L -1|] and any j ∈ [|1, b|], ψ (l+1,j)

Figure 2 :

 2 Figure 2: 1D-Regression Uncertainty Estimation. The horizontal and vertical axes correspond respectively to the 1D input space X and the 1D output space Y. The black line represents the ground truth f * (x) and the blue line the average predictions µ w (x). Training instances are reported as white dots. Uncertainty estimations are reported in the form of confidence intervals centered around the average prediction (in light blue). The length of the intervals is equal to 4 u(x) with u(x) defined according to Equation (37).

Figure 3 :

 3 Figure 3: Two-Moons Classification Uncertainty Estimation. The horizontal and vertical axes correspond to both dimensions of the input space X . Training instances are represented by white dots. The two "moons" formed by the training instances correspond to two different classes. Predicted uncertainties u(x), computed through Equation (36), are reported in shades of blue (darker areas correspond to larger uncertainties).

Figure 4 :

 4 Figure 4: MaxWEnt-SVD Uncertainties for different clipping parameters. Clipping is performed "a posteriori" on the scaling variable ϕ ⊙ z (cf. Equation (9)) of the fitted MaxWEnt-SVD network, such that q ϕ ∼ w + V min(ϕ ⊙ z, C), with C the clipping parameter.

Figure 5 :

 5 Figure 5: MaxWEnt Uncertainties Evolution through Epochs. One epoch corresponds to 100 iterations. The entropy term H(ϕ) is defined here as H(ϕ) = 1 d

Figure 7 :

 7 Figure 7: Comparison of the uncertainty evolution across time for Deep Ensemble and MaxWEnt on the Weather-Shift OOD dataset. The top images are examples of the camera's recording. The length of confidence intervals (in light blue) is equal to 2σ w (x).

Figure 8 :

 8 Figure 8: Evolution of MaxWEnt AUROC scores as a function of λ. The OOD detection AU-ROC scores of MaxWEnt (×5) are reported for different values of the trade-off parameter λ. The Deep Ensemble performances are reported in dashed black lines. The red dots correspond to the MaxWEnt AUROC scores for λ = 10.

Figure 9 :

 9 Figure 9: OpenOOD benchmark ranking. Each method is ranked according to the average AU-ROC score computed for the three "global" experiments: OSR, Far-OOD, Near-OOD. Each experiment is performed on 3 or 4 different datasets (cf. Table6). The top 23 scores among the 32 competitors are reported. post-hoc methods are marked with daggers, MaxWEnt(x1) can be considered as post-hoc as it applies on a pretrained network, although it requires additional training steps to learn the scaling parameters ϕ.

  R verifying: C = d 2 log(2π) + d 2

  with C ∈ R a constant. Then, as A is an orthogonal matrix, we have |det(A)| = |det(A T )| = 1 and:

  70)M ii = a 2 i ∀i ∈ [|1, b|].(71)Equation (70) implies that M = U U T with U = V T diag(s)V . For any i ∈ [|1, b|], we denoteu i ∈ R b the i th row vector of the matrix U and ||u i || 2 = b j=1 U 2 ij its corresponding Euclidean norm.Applying the Hadamard inequality to the matrix U , we obtain that: formulaU = V T diag(s)V implies that det(U ) = b i=1 s i and the equality M = U U T implies that M ii = b j=1 U 2 ij = ||u i || 2 2 .Considering Equation (71), we then deduce that:

  Proof Let's consider w ∈ R d and, for any ϕ ∈ R d , the distributionq ϕ ∼ w + ϕ ⊙ z with z ∼ Z such that Z ∼ U([-√ 3, √ 3] d ) or Z ∼ N (0, Id d ).The optimization problem (23) is written: that the above optimization problem has a unique solution, denoted ϕ * ∈ R d . Then, there exists τ ∈ R + such that ϕ * verifies the following optimization problem: τ = E q ϕ * [L S (w)], if we denote ϕ * * ∈ R d the solution of problem (76E q ϕ * * [L S (w)] ≤ τ which implies that:E q ϕ * * [L S (w)]

  j) ∀ l ∈ [|0, L|]; j ∈ [|1, b l |].

  with ϕ † (l,j) ∈ R b for any l ∈ [|0, L|], j ∈ [|1, b l |].The operator is the concatenation operator.Each vector ϕ † (l,j) is a solution of the following optimization sub-problem:

E

  q ϕ * [ψ (l,k) (x i ) 2 ]ϕ †

Table 3 :

 3 UCI experiments In-distribution performances. The average Negative Log Likelihood (NLL) and Expected Calibration Error (ECE) over the eight datasets are reported. The scores are computed on the test set, the lower the score the better. The number at the end of the acronyms correspond to the number of networks (ME1 refers to a MaxWEnt single network and ME5 to an ensemble of 5 MaxWEnt networks).

				Baselines			MaxWEnt			MaxWEnt + Clip
		Metric	DE	BN1	BN5	ME1	ME1+	ME5	ME5+	ME1	ME1+	ME5	ME5+
	Extra	Avg NLL Avg ECE	-0.69 -0.61 -0.75 0.37 0.37 0.35	-0.44 0.36	-0.33 0.33	-0.41 0.36	-0.04 0.39	-0.61 0.31	-0.71 0.36	-0.59 0.29	-0.71 0.35
	Intra	Avg NLL Avg ECE	-0.45 -0.49 -0.54 0.33 0.32 0.30	-0.26 0.32	-0.12 0.30	-0.23 0.30	-0.06 0.34	-0.27 0.33	-0.45 0.33	-0.28 0.33	-0.45 0.33

Table 5 :

 5 CityCam Experiments OOD Detection Results. Average AUROC and FPR@95 over three repetitions of the experiment are reported for each dataset and each method.

https://github.com/antoinedemathelin/maxwent-expe

https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_ moons.html

https://github.com/antoinedemathelin/OpenOOD

In practice the entropy is scaled by the number of parameters such that λ = 10/D with D ∈ N the dimension of ϕ

Inspired by [START_REF] Andrew Yk Foong | inbetween'uncertainty in bayesian neural networks[END_REF] and [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF], which propose OOD splits for UCI datasets, we split the dataset along the first component of the input PCA: we define the internal domain with the data between the 25% and 75% percentiles of the input PCA first component while the rest of the data form the external domain. We then consider the two following experimental setup:

• Extrapolation: The training data are defined by the internal domain, while the data from the external domain are considered as OOD.

• Interpolation: The training data are defined by the external domain, while the data from the internal domain are considered as OOD.

In all experiments, we consider as base estimator, a fully-connected network with three hidden layers of 100 neurons each and ReLU activation. The end-layer is composed of two neurons, which respectively predict the conditional mean and standard deviation µ w (x), σ w (x) (cf. Section 7.1.1). We consider 13 different uncertainty quantification approaches: five deep ensemble methods: Deep Ensemble (Lakshminarayanan et al., 2017), Negative Correlation [START_REF] Liu | Ensemble learning via negative correlation[END_REF][START_REF] Shui | Diversity regularization in deep ensembles[END_REF], Maximize-Overall-Diversity (MOD) [START_REF] Jain | Maximizing overall diversity for improved uncertainty estimates in deep ensembles[END_REF], Anchored-Networks [START_REF] Pearce | Uncertainty in neural networks: Bayesian ensembling[END_REF], Repulsive-Deep-Ensemble (RDE) (D'Angelo and Fortuin, 2021) and four "Bayesian" methods: MC-Dropout, BNN, MaxWEnt, MaxWEnt-SVD (described in Section 7.1.1), and ensemble version of these four previous Bayesian methods. The competitor characteristics are summarized in Table 1. We use the Gaussian NLL loss for regression, as defined in Equation ( 35) and the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with learning rate 0.001 and batch size 128. The number of iterations is chosen such that the minimum validation NLL is generally Appendix A. Proofs

A.1 Proof of Proposition 1

Let's consider a matrix A ∈ R d×d and a vector ϕ ∈ R d such that the weights w are written:

with z ∼ Z following either a multivariate normal or a uniform distribution. We demonstrate Proposition (1) for any orthogonal matrix A. Indeed, the weight parameterizations ( 7) and ( 9) correspond respectively to the specific cases A = Id d and A = V T which are both orthogonal matrices.

A.1.1 GAUSSIAN CASE

To demonstrate the result in the Gaussian case z ∼ N (0, Id d ), we first derive the two following preliminary results:

• The entropy of a multivariate Gaussian N (0, Σ) is written C + 1 2 log(|det(Σ)|) with C > 0 a constant (independent of Σ) and det(Σ) the determinant of Σ (cf. Lemma ( 8)).

Lemma 7 For any A ∈ R d×d and any ϕ ∈ R d , we have:

Proof We first notice that linear combinations of Gaussian variables are Gaussians. Then, it appears that:

and:

From which we conclude that

Lemma 8 The entropy of a multivariate Gaussian N (0, Σ) is written C + 1 2 log(|det(Σ)|) with C > 0 a constant (independent of Σ) and det(Σ) the determinant of Σ.

A.1.2 UNIFORM CASE

The probability density function p Z (z) of a uniform distribution defined over the parallelotope P described by the matrix Σ ∈ R d×d is written:

with P the subset of R d defined as P = {Σx ; x ∈ [0, 1] d } and V(P) the volume of P which verifies V(P) = |det(Σ)|.

Let's now consider the variable Z of probability density function p Z (z), the entropy of Z is then written:

As the volume of a subset is invariant by translation, we have V(P) = V( P) with P the parallelotope defined as

We then deduce that the entropy of q ϕ (w) ∼ w + A(z ⊙ ϕ) verifies:

Finally, as A is an orthogonal matrix, we have |det(A)| = 1 and:

A.2 Proof of Proposition 2 Proof Let's consider ϕ ∈ R b and z ∼ Z. The training risk for the weight w = w + ϕ ⊙ z can be written as follows:

When averaging over z ∼ Z, considering that E[z] = 0, we obtain:

The objective function of Problem (11) can then be written, for any ϕ ∈ R b :

For each layer, we define the amplitude of the input neuron activation on average over the training data:

We also define the quantities σ 2 (l,j) , related to the variance of the output neurons, before activation, on average over the training data: , considering the independence between ψ (l) and z (l,j) , we have:

For u ̸ = v, z (l,j,u) ⊥ ⊥ z (l,j,v) and E q ϕ * z (l,j,u) z (l,j,v) = E q ϕ * z (l,j,u) E q ϕ * z (l,j,v) = 0, then:

2 .

(81)

The optimization problem ( 76) is then equivalent to: (82)

Indeed, as problem (82) includes more constraints than problem (76), its solution necessarily provides a distribution of lower or equal entropy than q ϕ * . However, as the additional constraints are verified by ϕ * , ϕ * is the unique solution of problem (82).