Alexandre Ramé

Corentin Dancette

Matthieu Cord

Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization

Learning robust models that generalize well under changes in the data distribution is critical for realworld applications. To this end, there has been a growing surge of interest to learn simultaneously from multiple training domains -while enforcing different types of invariance across those domains. Yet, all existing approaches fail to show systematic benefits under controlled evaluation protocols. In this paper, we introduce a new regularization -named Fishr -that enforces domain invariance in the space of the gradients of the loss: specifically, the domain-level variances of gradients are matched across training domains. Our approach is based on the close relations between the gradient covariance, the Fisher Information and the Hessian of the loss: in particular, we show that Fishr eventually aligns the domain-level loss landscapes locally around the final weights. Extensive experiments demonstrate the effectiveness of Fishr for out-of-distribution generalization. Notably, Fishr improves the state of the art on the DomainBed benchmark and performs consistently better than Empirical Risk Minimization. Our code is available at https: //github.com/alexrame/fishr.

Introduction

The success of deep neural networks in supervised learning [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] relies on the crucial assumption that the train and test data distributions are identical. In particular, the tendency of networks to rely on simple features (Valle-Perez et al., 2019; Geirhos et al., 2020) is generally a desirable behavior reflecting Occam's razor. However, in case of distribution shift, this simplicity bias deteriorates performance when more complex features are needed [START_REF] Tenenbaum | Building machines that learn and think like people[END_REF][START_REF] Shah | The pitfalls of simplicity bias in neural networks[END_REF]. For example, in the To better generalize under distribution shifts, most works [START_REF] Blanchard | Generalizing from several related classification tasks to a new unlabeled sample[END_REF][START_REF] Muandet | Domain generalization via invariant feature representation[END_REF]) assume that the training data is divided into different training domains in which there is a constant underlying causal mechanism [START_REF] Peters | Causal inference by using invariant prediction: identification and confidence intervals[END_REF]. To remove the domain-dependent explanations, different invariance criteria across those training domains have been proposed. [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF]; Sun et al. (2016); Sun & Saenko (2016) enforce similar feature distributions, others [START_REF] Arjovsky | Invariant risk minimization[END_REF][START_REF] Krueger | Out-ofdistribution generalization via risk extrapolation (rex)[END_REF] force the classifier to be simultaneously optimal across all domains. Yet, despite the popularity of this research topic, none of these methods perform significantly better than the classical Empirical Risk Minimization (ERM) when applied with controlled model selection and restricted hyperparameter search [START_REF] Gulrajani | In search of lost domain generalization[END_REF][START_REF] Ye | Ood-bench: Benchmarking and understanding out-ofdistribution generalization datasets and algorithms[END_REF]. These failures motivate the need for new ideas.

To foster the emergence of a shared mechanism with consistent generalization properties, our intuition is that learning should progress consistently and similarly across domains. Besides, the learning procedure of deep neural networks is dictated by the distribution of the gradients with respect to the network weights [START_REF] Yin | Gradient diversity: a key ingredient for scalable distributed learning[END_REF][START_REF] Sankararaman | The impact of neural network overparameterization on gradient confusion and stochastic gradient descent[END_REF] -usually backpropagated in the network during gradient descent. Additionally, individual gradients are expressive representations of the input [START_REF] Fort | Stiffness: A new perspective on generalization in neural networks[END_REF][START_REF] Charpiat | Input similarity from the neural network perspective[END_REF]. Thus, we seek distributional invariance across domains in the gradient space: domain-level gradients should be similar, not only in average direction, but most importantly in statistics such as variance and disagreements.

In this paper, we propose the Fishr regularization for outof-distribution generalization in classification for computer vision -summarized in Fig. 1. We match the domainlevel gradient variances, i.e., the second moment of the gradient distributions. In contrast, previous gradient-based works such as Fish [START_REF] Shi | Gradient matching for domain generalization[END_REF] only match the domainlevel gradients means, i.e., the first moment.

Our strategy is also motivated by the close relations between the gradient variance, the Fisher Information [START_REF] Fisher | On the mathematical foundations of theoretical statistics[END_REF] and the Hessian. This explains the name of our work, Fishr, using gradients as in Fish and related to the Fisher Matrix. Notably, we will study how Fishr forces the model to have similar domain-level Hessians and promotes consistent explanations -by generalizing the inconsistency formalism introduced in Parascandolo et al. (2021).

To reduce the computational cost, we justify an approximation that tackles the gradients only in the classifier, easily implemented with BackPACK [START_REF] Dangel | BackPACK: Packing more into backprop[END_REF].

We summarize our contributions as follows:

• We introduce Fishr, a scalable regularization that brings closer the domain-level gradient variances.

• We theoretically justify that Fishr matches domainlevel risks and Hessians, and consequently, reduces inconsistencies across domains.

Empirically, we first validate that Fishr tackles distribution shifts on the synthetic Colored MNIST [START_REF] Arjovsky | Invariant risk minimization[END_REF]. Then, we show that Fishr performs best on the Do-mainBed benchmark (Gulrajani & Lopez-Paz, 2021) when compared with state-of-the-art counterparts. Critically, Fishr is the only method to perform systematically better than ERM on all real datasets -PACS, VLCS, OfficeHome, TerraIncognita and DomainNet.

Context and Related Work

We first describe our task and provide the notations used along our paper. Then we remind some important related works to understand how our Fishr stands in a rich literature.

Problem definition and notations. We study out-ofdistribution (OOD) generalization for classification. Our model is a deep neural network (DNN) f θ (parametrized by θ) made of a deep features extractor Φ ϕ on which we plug a dense linear classifier w ω : f θ = w ω • Φ ϕ and θ = (ϕ, ω).

In training, we have access to different domains E: for each domain e ∈ E, the dataset D e = x i e , y i e ne i=1 contains n e i.i.d. (input, labels) samples drawn from a domaindependent probability distribution. Combined together, the datasets {D e } e∈E are of size n = e∈E n e . Our goal is to learn weights θ so that f θ predicts well on a new test domain, unseen in training. As described in [START_REF] Koh | A benchmark of in-the-wild distribution shifts[END_REF] and [START_REF] Ye | Ood-bench: Benchmarking and understanding out-ofdistribution generalization datasets and algorithms[END_REF], most common distribution shifts are diversity shifts -where the training and test distributions comprise data from related but distinct domains, for instance pictures and drawings of the same objects -or correlation shiftswhere the distribution of the covariates at test time differs from the one during training. To generalize well despite these distribution shifts, f θ should ideally capture an invariant mechanism across training domains. Following standard notations, ∥M ∥ 2 F denotes the Frobenius norm of matrix M ; ∥v∥ 2 2 denotes the euclidean norm of vector v; 1 is a column vector with all elements equal to 1.

The standard Empirical Risk Minimization (ERM) [START_REF] Vapnik | An overview of statistical learning theory[END_REF] framework simply minimizes the average empirical risk over all training domains, i.e.,

|E|

e∈E R e (θ) where R e (θ) = 1 ne ne i=1 ℓ f θ x i e , y i e and ℓ is the negative log-likelihood loss. Many approaches try to exploit some external source of knowledge [START_REF] Xie | In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness[END_REF], in particular the domain information. As a side note, these partitions may be inferred if not provided [START_REF] Creager | Environment inference for invariant learning[END_REF]. Some works explore data augmentations to mix samples from different domains [START_REF] Wang | Heterogeneous domain generalization via domain mixup[END_REF][START_REF] Wu | Dual mixup regularized learning for adversarial domain adaptation[END_REF], some re-weight the training samples to favor underrepresented groups (Sagawa et al., 2020a;b;[START_REF] Zhang | Deep stable learning for out-of-distribution generalization[END_REF] and others include domain-dependent weights [START_REF] Ding | Deep domain generalization with structured low-rank constraint[END_REF][START_REF] Mancini | Best sources forward: domain generalization through sourcespecific nets[END_REF]). Yet, most recent works promote invariance via a regularization criterion and only differ by the choice of the statistics to be matched across training domains. They can be categorized into three groups: these methods enforce agreement either (1) in features (2) in predictors or (3) in gradients.

First, some approaches aim at extracting domain-invariant features and were extensively studied for unsupervised domain adaptation. The features are usually aligned with adversarial methods [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF][START_REF] Gong | Domain adaptation with conditional transferable components[END_REF]Li et al., 2018b;c) or with kernel methods [START_REF] Muandet | Domain generalization via invariant feature representation[END_REF][START_REF] Long | Transfer joint matching for unsupervised domain adaptation[END_REF]). Yet, the simple covariance matching in CORAL (Sun et al., 2016;Sun & Saenko, 2016) performs best on various tasks for OOD generalization [START_REF] Gulrajani | In search of lost domain generalization[END_REF]. With Z ij e the jth dimension of the features extracted by Φ ϕ for the ith example x i e of domain e

∈ E = {A, B}, CORAL minimizes ∥Cov(Z A ) -Cov(Z B )∥ 2 F where Cov(Z e ) = 1 ne-1 (Z ⊤ e Z e -1 ne 1 ⊤ Z e ⊤ 1 ⊤ Z e )
is the feature covariance matrix. CORAL is more powerful than mere feature A third and most recent line of work promotes agreements between gradients with respect to network weights. Gradient agreements help batches from different tasks to cooperate, and have been previously employed for multitasks [START_REF] Du | Adapting auxiliary losses using gradient similarity[END_REF][START_REF] Yu | Gradient surgery for multi-task learning[END_REF], continual (Lopez-Paz & Ranzato, 2017), meta [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF][START_REF] Zhang | Adaptive risk minimization: A metalearning approach for tackling group distribution shift[END_REF] and reinforcement [START_REF] Zhang | Learning novel policies for tasks[END_REF] 2021) try to find minimas in the loss landscape that are shared across domains. Specifically, these works tackle the domain-level expected gradients: Along with the increased computation cost, the main limitation of previous gradient-based methods is the per-domain batch averaging of gradients: this removes more granular statistics, in particular the information from pairwise interactions between gradients from samples in a same domain. In opposition, our new regularization for OOD generalization keeps extra information from individual gradients and matches across domains the domain-level gradient variances. In a nutshell, Fishr is similar to the covariance-based CORAL (Sun et al., 2016;Sun & Saenko, 2016) but in the gradient space rather than in the feature space.

matching 1 n A 1 ⊤ Z A -1 n B 1 ⊤ Z B
g e = E (xe,ye)∼De ∇ θ ℓ (f θ (x e ), y e ) . (1)

Fishr

Gradient variance matching

The individual gradient g i e = ∇ θ ℓ f θ (x i e ), y i e is the firstorder derivative for the i-th data example x i e , y i e from domain e ∈ E with respect to the weights θ. Previous methods have matched the gradient means g e = 1 ne ne i=1 g i e for each domain e ∈ E. These gradient means capture the average learning direction but can not capture gradient disagreements [START_REF] Sankararaman | The impact of neural network overparameterization on gradient confusion and stochastic gradient descent[END_REF][START_REF] Yin | Gradient diversity: a key ingredient for scalable distributed learning[END_REF]. With G e = [g i e ] ne i=1 of size n e × |θ|, we compute the domain-level gradient variance vectors of size |θ|:

v e = Var(G e ) = 1 n e -1 ne i=1 g i e -g e 2 , (2)
where the square indicates an element-wise product. To reduce the distribution shifts in the network f θ across domains, we bring the domain-level gradient variances {v e } e∈E closer. Hence, our Fishr regularization is:

L Fishr (θ) = 1 |E| e∈E ∥v e -v∥ 2 2 , (3)
the square of the Euclidean distance between the gradient variance from the different domains e ∈ E and the mean gradient variance v = 1 |E| e∈E v e . Balanced with a hyperparameter coefficient λ > 0, this Fishr penalty complements the original ERM objective, i.e., the empirical training risks:

L(θ) = 1 |E| e∈E R e (θ) + λL Fishr (θ). (4)
Remark 3.1. Gradients g i e can be computed on all network weights θ. Yet, to reduce the memory and training costs, they will often be computed only on a subset of θ, e.g., only on classification weights ω. This approximation is discussed in Section 4.2.2 and Appendix D.3.2.

Theoretical analysis

We theoretically motivate our Fishr regularization by leveraging the domain inconsistency score introduced in ANDmask [START_REF] Parascandolo | Learning explanations that are hard to vary[END_REF]. We first derive a generalized upper bound for this score. Then, we show that Fishr minimizes this upper bound by matching simultaneously domain-level risks and Hessians. A,θ * contains weights θ for which R A (θ) is low (≤ 0.2) but R B (θ) is high (≥ 0.9). This inconsistency is due to conflicting domain-level loss landscapes, specifically gaps between domain-level risks and curvatures at θ * . This is visible in the disagreements across the variances of gradients

INCONSISTENCY FORMALISM

{g i A } n A i=1 and {g i B } n B i=1 .
Parascandolo et al. (2021) argues that "patchwork solutions sewing together different strategies" for different domains may not generalize well: good weights should be optimal on all domains and "hard to vary" [START_REF] Deutsch | The beginning of infinity: Explanations that transform the world[END_REF]. They formalize this insight with an inconsistency score:

I ϵ (θ * ) = max (A,B)∈E 2 max θ∈N ϵ A,θ * |R B (θ) -R A (θ * )| , (5)
where θ ∈ N ϵ A,θ * if there exists a path in the weights space between θ and θ * where the risk R A remains in an ϵ > 0 interval around R A (θ * ). I increases with conflicting geometries in the loss landscapes around θ * as in Fig. 2: i.e., when another 'close' solution θ is equivalent to the current solution θ * in a domain A but yields different risks in B.

For e ∈ E, the second-order Taylor expansion of R e around θ * = 0 (with a change of variable) gives:

R e (θ) = R e (θ * ) + θ ⊤ ∇ θ R e (θ * ) + 1 2 θ ⊤ H e θ + O(∥θ∥ 2 2 ),
where the Hessian H e = ∇ 2 θ R e (θ * ) approximates the local curvature of the loss landscape. Moreover, we assume simultaneous convergence, i.e., θ * is a local minima across all domains: ∇ θ R e (θ * ) = 0. Thus, locally around θ * :

max θ∈N ϵ A,θ * |R B (θ) -R A (θ * )| ≈ max |R A (θ)-R A (θ * )|≤ϵ |R B (θ) -R A (θ * )| ≈ max 1 2 |θ ⊤ H A θ|≤ϵ R B (θ * ) + 1 2 θ ⊤ H B θ -R A (θ * ) ⪅ |R B (θ * ) -R A (θ * )| + max 1 2 |θ ⊤ H A θ|≤ϵ 1 2 θ ⊤ H B θ , (6)
where we deduced the last line from the triangle inequality. Appendix A.1 formally demonstrates following equality.

Proposition 1. Under the quadratic bowl Assumption A.1 with positive definite Hessians, for small ϵ (see Eq. 11):

I ϵ (θ * ) = max (A,B)∈E 2 (R B (θ * ) -R A (θ * ) + max 1 2 θ ⊤ H A θ≤ϵ 1 2 θ ⊤ H B θ). (7)
The Hessian being positive definite is a standard hypothesis, notably used in The first term in the RHS of Proposition 1 is the difference between domain-level risks, whose square is the criterion minimized in V-REx [START_REF] Krueger | Out-ofdistribution generalization via risk extrapolation (rex)[END_REF]. We will prove and show that Fishr forces this term to be small in Section 3.2.

max 1 2 θ ⊤ H A θ≤ϵ 1 2 θ ⊤ H B θ = max ∥ θ∥ 2 2 ≤ϵ i θ2 i λ B i /λ A i = ϵ × max i λ B i /λ A i . (8)
This is large when exists i such that λ A i is small but λ B i is large: indeed, a small weight perturbation in the direction of the associated eigenvector would change the loss slightly in the domain A but drastically in domain B. Thus, this second term decreases when H A and H B have similar eigenvalues. This result holds when Hessians are co-diagonalizable. In conclusion, this explains why forcing H A = H B reduces inconsistencies in the loss landscape and thus improves generalization. AND-mask matches Hessians by zeroing out gradients with inconsistent directions across domains; however, this masking strategy introduces dead zones [START_REF] Shahtalebi | Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization[END_REF] in weights where the model could get stuck, ignores gradient magnitudes and empirically performs poorly with real datasets from DomainBed. As shown in Section 3.2.3, Fishr proposes a new method to align domain-level Hessians leveraging the close relations between the gradient variance, the Fisher Information and the Hessian.

FISHR MATCHES THE DOMAIN-LEVEL RISKS

Gradients take into account the label Y , which appears as an argument for the loss ℓ. Hence, gradient-based approaches are 'label-aware' by design. In contrast, feature-based methods were shown to fail in case of label shifts, because they do not consider Y [START_REF] Johansson | Support and invertibility in domain-invariant representations[END_REF][START_REF] Zhao | On learning invariant representations for domain adaptation[END_REF].

The fact that the label and the loss appear in the formula of the gradients has another important consequence: matching gradient distributions also matches training risks, as motivated in V-REx [START_REF] Krueger | Out-ofdistribution generalization via risk extrapolation (rex)[END_REF]. We confirm this insight in Table 2: matching gradient variances with Fishr induces

|R A -R B | 2 → 0 when E = {A, B}.
Intuitively, gradient amplitudes are directly weighted by the loss values: multiplying the loss by a constant will also multiply the gradients by the same constant. Thus roughly, if the domain-level empirical training risks are different, then the domain-level gradient norms should also differ.

Theoretically, we prove in Appendix A.2 that Fishr regularization component with reference to the classification bias is exactly the difference between domain-level mean squared errors. We recover the objective from V-REx [START_REF] Krueger | Out-ofdistribution generalization via risk extrapolation (rex)[END_REF], with a different loss (squared error instead of negative log likelihood). More generally, we show in this Appendix that Fishr in the classifier w ω acts as a feature-adaptive version of V-REx: the components in Fishr adaptively force the risks to be similar across domains.

FISHR MATCHES THE DOMAIN-LEVEL HESSIANS

The Hessian matrix H = n i=1 ∇ 2 θ ℓ f θ (x i ), y i is of key importance in deep learning. Yet, H cannot be computed efficiently in general. Recent methods [START_REF] Izmailov | Averaging weights leads to wider optima and better generalization[END_REF][START_REF] Parascandolo | Learning explanations that are hard to vary[END_REF][START_REF] Foret | Sharpness-aware minimization for efficiently improving generalization[END_REF] tackled the Hessian indirectly by modifying the learning procedure. In contrast, we use the fact that the diagonal of H is approximated by the gradient variance Var(G); this is confirmed in Table 1. This result is derived below from 3 individual and standard approximation steps. The 'empirical' FIM and the gradient covariance. Critically, F is nothing else than the unnormalized uncentered covariance matrix when ℓ is the negative loglikelihood. Thus, the gradient covariance matrix C = )). The gradient variance, computable efficiently with a unique backpropagation, serves as a proxy for the Hessian. Details and more experiments in Section 4.1 (notably Fig. 3) and in Appendix C.2.1.

1 n-1 G ⊤ G -1 n 1 ⊤ G ⊤ 1 ⊤ G of

ERM Fishr

∥Var(G 90% ) -Var(G 80% )∥ 2 F 1.6 4.1 × 10 -5 |R 90% -R 80% | 2 1.0 × 10 -2 3.8 × 10 -6 ∥Diag (H 90% -H 80% )∥ 2 F 2.9 × 10 -1 2.7 × 10 -4
Conclusion. Fishr efficiently matches (1) domain-level empirical risks and (2) domain-level Hessians across the training domains, using gradient variances as a proxy. This will align domain-level loss landscapes, reduce domain inconsistencies and increase domain generalization. In particular, the domain-level Hessian matching illustrates that Fishr is more than just a generalization of gradient-mean approaches such as Fish [START_REF] Shi | Gradient matching for domain generalization[END_REF].

Finally, we refer the readers to Appendix A.3 where we leverage the Neural Tangent Kernel (NTK) [START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF] theory to further motivate the gradient variance matching during the optimization process -and not only at convergence. In brief, as F and the NTK matrices share the same non-zero eigenvalues, similar {C e } e∈E during training reduce the simplicity bias by preventing the learning of different domain-dependent shortcuts at different training speeds: this favors a shared mechanism that predicts the same thing for the same reasons across domains.

Experiments

We prove Fishr effectiveness on Colored MNIST [START_REF] Arjovsky | Invariant risk minimization[END_REF] and then on the DomainBed benchmark (Gulrajani & Lopez-Paz, 2021). To facilitate reproducibility, the code is available at https://github.com/ alexrame/fishr. Moreover, we show in Appendix B that Fishr is effective in the linear setting.

Proof of concept on Colored MNIST

The task in Colored MNIST [START_REF] Arjovsky | Invariant risk minimization[END_REF] is to predict whether the digit is below or above 5. Moreover, the labels are flipped with 25% probability (except in Appendix C.2.2). Critically, the digits' colors spuriously correlate with the labels: the correlation strength varies across the two training domains E = {90%, 80%}. To test whether the model has learned to ignore the color, this correlation is reversed at test time. In brief, a biased model that only considers the color would have 10% test accuracy whereas an oracle model that perfectly predicts the shape would have 75%. As previously done in V-REx [START_REF] Krueger | Out-ofdistribution generalization via risk extrapolation (rex)[END_REF], we strictly follow the IRM implementation and just replace the IRM penalty by our Fishr penalty. This means that we use the exact same MLP and hyperparameters, notably the same two-stage scheduling selected in IRM for the regularization strength λ, that is low until epoch 190 and then jumps to a large value, which was optimized via a gridsearch for IRM. More experimental details are provided in Appendix C.1.

Table 3 reports the accuracy averaged over 10 runs with standard deviation. Fishr θ (i.e., applying Fishr on all weights θ) obtains the best trade-off between train and test accuracies; notably in test, it reaches 71.2%, or 70.2% when digits are grayscale. Moreover, computing the gradients only in the classifier w ω performs almost as well (69.5% in test for Fishr ω ) while reducing drastically the computational cost. Finally, Fishr ϕ only in the features extractor ϕ works best in test, though it has lower train accuracy. This last experiment shows that we can reduce domain shifts without explicitly forcing the predictors to be simultaneously optimal. These results highlight the effectiveness of gradient variance matching -even with standard hyperparameters -at different layers of the network.

The main advantage of this synthetic dataset is the possibility of empirically validating some theoretical insights. For example, the training dynamics in Fig. 3 show that the domain-level empirical risks get closer once the Fishr θ gradient variance matching loss is activated after step 190 (|R 90% -R 80% | → 0), even though predicting accurately on the domain 90% is easier than on the domain 80%. This confirms insights from Section 3.

IMPLEMENTATION DETAILS

We systematically apply Fishr only in the classifier w ω in DomainBed. Indeed, keeping individual gradients in memory for ϕ from a ResNet-50 was impossible for computational reasons. Fishr θ and Fishr ω performed similarly in previous Section 4.1. This is partly because the gradients in ω still depend on Φ ϕ . Additionally, as highlighted in Appendix D.3.2, this relaxation may improve results for real-world datasets. Indeed, while Colored MNIST is a correlation shift challenge, the other datasets mostly demonstrate diversity shifts where "each domain represents a certain spectrum of diversity in data" [START_REF] Ye | Ood-bench: Benchmarking and understanding out-ofdistribution generalization datasets and algorithms[END_REF]. Then, as the pixels distribution are quite different across domains, low-level layers may need to adapt to these domain-dependent peculiarities. Moreover, if we used all weights θ = (ϕ, ω) to compute gradient variances, the invariance in w ω may be ERM was carefully tuned in DomainBed and thus remains a strong baseline. Moreover, all previous methods are far from the best score on at least one dataset. Invariant predictors (IRM, V-REx) and gradient masking (AND-mask) approaches perform poorly on real datasets. Additionally, CORAL not only performs worse than ERM on TerraIncognita, but most importantly fails to detect correlation shifts on Colored MNIST: this is because feature-based approaches do not take into account the label, as previously stated in Section 3.2.2.

Contrarily, Fishr is the only method to efficiently tackle correlation and diversity shifts, as defined in [START_REF] Ye | Ood-bench: Benchmarking and understanding out-ofdistribution generalization datasets and algorithms[END_REF]. Indeed, not only Fishr outperforms ERM on Colored MNIST (68.8% vs. 57.8%), but Fishr also systematically performs better than ERM on all real datasets: the differences are over standard errors on VLCS (78.2% vs. 77.6%), OfficeHome (68.2% vs. 66.4%) and on the larger-scale Do-mainNet (41.8% vs. 41.3%). Appendix D.3.2 shows that Fishr performs even better when combined with gradientmean matching. In summary, Fishr consistently beats ERM (despite the restricted hyperparameter search): this is the main point to validate the effectiveness of our method.

Additionally, Fishr performs best after averaging: Firshr reaches 70.8% vs. 69.2% for the second best CORAL. When ignoring the Colored MNIST task, averaging over the 6 other datasets leads to a similar ranking: 1.Fishr(avg=71.1), 2.CORAL(71.0), 3.Mixup(70.8) and 4.ERM(70.5). This arguably partial metric is confirmed by the more robust ranking information; Fishr's median ranking of second reflects that Fishr is consistently among the best methods. Overall, Fishr is the state-of-the-art approach, not only in average accuracy, but most importantly in average ranking.

Conclusion

In this paper, we addressed the task of out-of-distribution generalization for classification in computer vision. We derive a new and simple regularization -Fishr -that matches the gradient variances across domains as a proxy for matching domain-level risks and Hessians. We prove that this reduces inconsistencies across domains. Fishr reaches state-of-the-art performances on DomainBed when samples from the test domain are available for model selection. Our experiments -reproducible with our open-source implementation -suggest that Fishr would consistently improve a deep classifier for real-world usages when dealing with data from multiple domains. We hope to pave the way towards new gradient-based regularization to improve the generalization abilities of deep neural networks.

A. Additional Theoretical Analysis

A.1. Demonstration of Proposition 1 from Section 3.2.1

Assumption A.1. We make the quadratic bowl assumption around the local minima θ * on all domains : ∀e ∈ E,

R e (θ) = R e (θ * ) + 1 2 (θ -θ * ) ⊤ H e (θ -θ * ), (9)
where H e is positive definite of eigenvalues Proposition 2. (Reformulation of Proposition 1, illustrated in Fig. 4). Let ϵ > 0, weights θ * . ∀(A, B) ∈ E 2 , with N ϵ A,θ * the largest path-connected region of weights space where the risk R A remains in an ϵ interval around R A (θ * ), we note:

λ e 1 ≥ • • • ≥ λ e h > 0. Remark A.
I ϵ (A, B) = max θ∈N ϵ A,θ * |R B (θ) -R A (θ * )| , R(A, B) = R B (θ * ) -R A (θ * ), H ϵ (A, B) = max 1 2 (θ-θ * ) ⊤ H A (θ-θ * )≤ϵ 1 2 (θ -θ * ) ⊤ H B (θ -θ * ). (10)
If ∀(A, B) ∈ E 2 such as R(A, B) < 0, we have:

ϵ ≤ -R(A, B) × λ A h λ B 1 , (11)
then under previous Assumption A. 1,

max (A,B)∈E 2 I ϵ (A, B) = max (A,B)∈E 2 (R(A, B) + H ϵ (A, B)) (12)
Proof We first prove that, under quadratic Assumption A.1,

∀A ∈ E, N ϵ A,θ * = {θ| |R A (θ) -R A (θ * )| ≤ ϵ}.
Indeed, the former is always included in the latter by definition. Reciprocally, be given θ in the latter, {λθ * + (1 -λ)θ|λ ∈ [0, 1]} linearly connects θ * to θ in parameter space with the risk R A remaining in an

ϵ interval around R A (θ * ) because ∀µ ∈ [0, 1] we have |R A (µθ * + (1 -µ)θ) -R A (θ * )| = (1 -µ) 2 |R A (θ) -R A (θ * )| ≤ (1 -µ) 2 ϵ ≤ ϵ. Therefore ∀(A, B) ∈ E 2 : I ϵ (A, B) = max |R A (θ)-R A (θ * )|≤ϵ |R B (θ) -R A (θ * )| = max 1 2 (θ-θ * ) ⊤ H A (θ-θ * )≤ϵ R(A, B) + 1 2 (θ -θ * ) ⊤ H B (θ -θ * ) (13)
As the Hessians are positive, H ϵ (A, B) > 0. We now need to split the analysis based on the sign of R(A, B).

I ϵ (A, B) = max (A,B)∈E 2 |R(A,B)≥0 (R(A, B) + H ϵ (A, B)) . ( 14
)
Case R(A, B) < 0 Leveraging λ B 1 the largest eigenvalue from H B and λ A h the lowest eigenvalue from H A , we upper bound:

H ϵ (A, B) ≤ max λ A h 2 ∥θ-θ * ∥ 2 2 ≤ϵ λ B 1 2 ∥θ -θ * ∥ 2 2 = ϵ × λ B 1 λ A h . ( 15
)
Then Eq. 11 gives H ϵ (A, B) < -R(A, B). Thus the number inside the absolute value from the RHS of Eq. 13 is negative. This leads to:

I ϵ (A, B) = -R(A, B) -H ϵ (A, B) < -R(A, B) = R(B, A) < I ϵ (B, A).
Thus the max over E 2 of function (A, B) → I ϵ (A, B) can not be achieved for (A, B) with R(A, B) < 0. We obtain:

max (A,B)∈E 2 I ϵ (A, B) = max (A,B)∈E 2 |R(A,B)≥0 I ϵ (A, B) (16)
Similarly, R(A, B)

+ H ϵ (A, B) ≤ 0 < R(B, A) + H ϵ (B, A). Thus the max over E 2 of function (A, B) → (R(A, B) + H ϵ (A, B))
can not be achieved for (A, B) with R(A, B) < 0. We obtain:

max (A,B)∈E 2 (R(A, B) + H ϵ (A, B)) = max (A,B)∈E 2 |R(A,B)≥0 (R(A, B) + H ϵ (A, B)) (17)
Conclusion Combining Eq. 14, Eq. 16 and Eq. 17, we conclude the proof.

A.2. Fishr as a feature-adaptive version of V-REx

We delve into the theoretical analysis of the Fishr regularization in the classifier w ω , that leverages p features extracted from ϕ. We note z i e ∈ R p the features for the i-th example from the domain e, ŷi e ∈ [0, 1] the predictions after sigmoid and y i e ∈ {0, 1} the one-hot encoded target. The linear layer W is parametrized by weights {w k } p k=1 and bias b. The gradient of the loss for this sample with respect to the bias b is ∇ b ℓ(y i e , ŷi e ) = (ŷ i e -y i e ). Thus, the uncentered gradient variance in b for domain e is: v b e = 1 ne ne i=1 (ŷ i e -y i e ) 2 , which is exactly the mean squared error (MSE) between predictions and targets in domain e. Thus, matching gradient variances in b will match risks across domains. This is the objective from V-REx [START_REF] Krueger | Out-ofdistribution generalization via risk extrapolation (rex)[END_REF], where the squared error has replaced the negative log likelihood.

We can also look at the gradients with respect to the weight w k : ∇ w k ℓ(y i e , ŷi e ) = (ŷ i e -y i e )z i e [k]. Thus, the uncentered gradient variance in w k for domain e is:

v w k e = 1 ne ne i=1 (ŷ i e -y i e )z i e [k]
2 . This is the squared error, weighted for each sample (z i e , y i e ) by the square of the k-th feature z i e [k]: matching gradient variances directly matches these weighted squared errors, with k different weighting schemes, that depend on the features distribution. This describes Fishr as a feature-adaptive version of V-REx (Krueger et al., 2021). An intuitive example is when features are binary (z i e ∈ {0, 1}); in that case, Fishr matches domain-level risks on groups of samples having a shared feature.

More exactly in Fishr, we match centered gradient variances, equivalent to the uncentered variance gradient matching at convergence under the assumption g e ≈ 0. Experiments in Table 5 and in Appendix C.2.4 confirm that centering or not the variances perform similarly.

A.3. Neural Tangent Kernel perspective

In this Section we motivate the matching of gradient covariances with new arguments from the Neural Tangent Kernel (NTK) [START_REF] Jacot | Neural tangent kernel: Convergence and generalization in neural networks[END_REF] theory. As a reminder, the NTK K ∈ R n×n is the gramian matrix with entries K[i, j] = ∇ θ f θ (x i ) ⊤ • ∇ θ f θ (x j ) that measure the gradients similarity at two different input points x i and x j . This kernel dictates the training dynamics of the DNN and remains fixed in the infinite width limit. Most importantly, as stated in Yang & Salman (2019), "the simplicity bias of a wide neural network can be read off quickly from the spectrum of K: if the largest eigenvalue [λ max ] of K accounts for most of Tr(K), then a typical random network looks like a function from the top eigenspace of K": this holds for ReLu networks. In summary, gradient descent mostly happens in a tiny subspace (Gur-Ari et al., 2018) whose directions are defined by the main eigenvectors from K. Moreover, the learning speed is dictated by λ max , which can be used to estimate a condition for a learning rate η to converge: η < 2/λ max [START_REF] Karakida | Pathological spectra of the fisher information metric and its variants in deep neural networks[END_REF].

In a multi-domain framework, having similar spectral decompositions across {K e } e∈E during the optimization process would improve OOD generalization for two reasons:

B. Experiments on a Linear Example

We experimentally prove that Fishr is effective in the linear setting. To do so, we consider the binary classification dataset introduced in the Section 3.2 from Fish [START_REF] Shi | Gradient matching for domain generalization[END_REF]. Each example is composed of 4 static features (f 1 , f 2 , f 3 , f 4 ). While f 1 is invariant across the two train domains and the test domain, the three other features are spurious: their correlations with the label vary in each domain. The model is a linear logistic regression, with trainable weights W and bias b. As f 2 and f 3 have higher correlations with the label than f 1 in training, ERM relies mostly on f 2 and f 3 . This is indicated in the first line of Table 5 This is also visible in Fig. 7, which is equivalent to Fig. 3, but for ERM (without the Fishr regularization). The distance between domain-level gradient variances (red) keeps increasing across domains E = {90%, 80%}: so does the distance across Hessians (purple). The distance across risks (pink) decreases, but slower than with Fishr regularization. Overall, the network still predicts the digit's color while only slightly using the digit's shape. That's why the test accuracy (blue) remains low. To further validate that Fishr can tackle distribution shifts, we investigate Colored MNIST but without the 25% label flipping. In Table 6, the label is then fully predictable from the digit shape. Using hyperparameters defined previously in Appendix C.1, we recover that IRM (82.2%) fails when the invariant feature is fully predictive [START_REF] Ahuja | Invariance principle meets information bottleneck for out-of-distribution generalization[END_REF]): indeed, it performs worse than ERM (91.8%). In contrast, V-REx and Fishr ω perform better (95.3%): in conclusion, Fishr works even without label noise.

C.2.2. COLORED MNIST WITHOUT LABEL FLIPPING

C.2.3. GRADIENT VARIANCE OR COVARIANCE ?

We have justified ignoring the off-diagonal parts of the covariance to reduce the memory overhead. For the sake of completeness, the second line in Table 7 shows results with the full covariance matrix. This experiment is possible only when considering gradient in the classifier w ω for memory reasons. Overall, results are similar (or slightly worse) as when using only the diagonal: the slight difference may be explained by the approaches' different suitability to the hyperparameters (that were optimized for IRM). In conclusion, this preliminary experiment suggests that targeting the diagonal components is the most critical. We hope future works will further investigate this diagonal approximation or provide new methods to reduce the computational costs, such as K-FAC approximations [START_REF] Heskes | On "natural" learning and pruning in multilayered perceptrons[END_REF][START_REF] Martens | Optimizing neural networks with kronecker-factored approximate curvature[END_REF]. The convolutional neural network architecture used for the MNIST experiments is the one introduced in DomainBed: note that this is not the same MLP (described in Appendix C.1) as in our proof of concept in Section 4.1. All real datasets leverage a 'ResNet-50' pretrained on ImageNet, with a dropout layer before the newly added dense layer and fine-tuned with frozen batch normalization layers.

D.2. 'Training-domain' model selection

In the main paper, we focus on the 'Test-domain' model selection, where the validation set follows the same distribution as the test domain. This is important to adapt the degree of model invariance according to the test domain. For Fishr, if the domain-dependant correlations are useful in test, the selected λ would be small and Fishr would behave like ERM; in contrast, if the domain-dependant correlations are detrimental in test, the selected λ would be large, and Fishr would improve over ERM by enforcing invariance. In Table 9, we use the 'Training-domain' model selection: the validation set is formed by randomly collecting 20% of each training domain. Fishr performs better than ERM on all real datasets (over standard errors for OfficeHome and DomainNet), except for PACS where the two reach 85.5%. In average, Fishr (67.1%) finishes third and is above most methods such as V-REx (65.6%). Fishr median ranking is fifth, with a mean ranking of 5.6. These additional results were not included in the main paper due to space constraints and also because this 'Training-domain' model selection has three clear limitations.

First, learning causal mechanisms can be useless in this 'Training-domain' setup. Indeed, when the correlations are more predictive in training than the causal features, the variant model may be selected over the invariant one. This explains the poor results for all methods in 'Training-domain' Colored MNIST, where the color information is more predictive than the shape information in training. The best model on this task is ARM (Zhang et al., 2020) that uses test time adaptation -thus in a sense uses information from the test-domain -and whose contribution is mostly complementary to ours. Third, the 'Test-domain' model selection is more realistic for real applications. Indeed, one user would easily label some samples to validate the efficiency of its algorithm. It's not realistic to believe that the users would simply deploy their new algorithm without at least checking that the performances are correct. We recall that the 'Test-domain' setup in DomainBed benchmark is quite restricting, allowing only one evaluation per choice of hyperparameters, without early-stopping.

That's why Teney et al. (2021) even states that "OOD performance cannot, by definition, be performed with a validation set from the same distribution as the training data". Both opinions being reasonable and arguable, we included 'Trainingdomain' results for the sake of completeness, where Fishr remains stronger than ERM. Yet, our state-of-the-art results on the 'Test-domain' setup from Table 4 alone are sufficient to prove the usefulness of our approach for real-world applications.

D.3. Fishr component analysis on DomainBed

D.3.1. FOCUS ON THE EXPONENTIAL MOVING AVERAGE

Following Le Roux et al. (2011), we use an exponential moving average (ema) parameterized by γ for computing gradient variances in DomainBed: the closer γ is to 1, the longer a batch will impact the variance from later steps. We now further analyze the impact of this strategy, which is not specific to Fishr and was used previously in other works (Nam et + (1 -γ)R t e when E = {A, B}. Thus, we compare V-REx and Fishr, with γ = 0 (✗) or with γ ∼ Uniform(0.9, 0.99) (✓, as described in Table 8). On the synthetic Colored MNIST in Table 10, the ema is critical for Fishr -notably when training on E = {90%, 80%} and the dataset 10% is in test (from ✗34.0% to ✓58.9% in 'Test-domain'). V-REx also benefits from ema. On the 'real' dataset OfficeHome in Table 11, the ema is less beneficial (from ✗67.5% to ✓68.2% in 'Test-domain' for Fishr). Notably, it worsens V-REX. Overall, Fishr -with and without ema -outperforms V-REx on OfficeHome.

We speculate that ema mainly helps when the batch size is not sufficiently large to detect 'slight' correlation shifts in the training datasets: e.g., when batch size ∼ 2 Uniform (3,9) and training datasets E = {90%, 80%} in Colored MNIST. We remind that when the batch size was 25,000 in the Colored MNIST setup from IRM, Fishr reached 69.5% (without ema) in Table 3 from Section 4.1. On the contrary, when the shift is more prominent as in OfficeHome, the ema may be less necessary. Most importantly, Fishr -with and without ema -improves over ERM on these datasets.

D.3.2. COMPONENT ANALYSIS BY COMPARING GRADIENT VARIANCE VERSUS GRADIENT MEAN MATCHING

As a reminder from the Section 2, IGA [START_REF] Koyama | Out-of-distribution generalization with maximal invariant predictor[END_REF]) is an unpublished gradient-based approach that matches gradient means across domains, i.e., minimizes ||g A -g B || 2 2 when E = {A, B} and where g e = 1 ne ne i=1 ∇ θ ℓ (f θ (x e ), y e ). Scores for IGA are not available publicly and thus were not included in Section 4.2.1. Moreover, IGA is very costly and impractical: IGA is approximately (|E| + 1) times longer to train than ERM. Yet, we ran the DomainBed implementation of IGA on one 'synthetic' and one 'real' dataset. Table 12 shows that the IGA has little effect on Colored MNIST (58.0% vs. 57.8% for ERM in 'Test-domain'). Moreover, on OfficeHome in Table 13, IGA hinders learning (56.9% vs. 66.4% for ERM in 'Test-domain'). In brief, the seminal "IGA [. . .] could completely fail when generalizing to unseen domains", as stated in Fish [START_REF] Shi | Gradient matching for domain generalization[END_REF].

In the rest of this Section, we include IGA in Fishr codebase so that both methods leverage the same implementation choices: this enables fairer comparisons between gradient mean matching and gradient variance matching. These experiments provide further insights regarding Fishr main components: specifically, enforcing invariance (1) only in the classifier's weights ω (2) after a warmup period and (3) with an exponential moving average.

First, Fishr only considers gradient variances in the classifier's weights ω. Similarly, we try to apply IGA's gradient mean matching but only in w ω rather than in f θ . This new method works significantly better (67.2% when g e = 1 ne ne i=1 ∇ ω ℓ (f θ (x e ), y e ) vs. 56.9% when g e = 1 ne ne i=1 ∇ θ ℓ (f θ (x e ), y e ) for 'Test-domain' OfficeHome in Table 13) while reducing the computational overhead. This further motivates the invariance in the classifier rather than in the low-level layers (which need to adapt to shifts in pixels for instance). We have done this analysis on IGA and not on Fishr because keeping all individual gradients for a ResNet-50 in the GPU memory was not possible on our hardware.

Second, Fishr uses a double-stage scheduling inherited from IRM [START_REF] Arjovsky | Invariant risk minimization[END_REF]: the DNN first learns predictive features with standard ERM (λ = 0) until a given epoch, at which λ takes its true (high) value to then force domain invariance. This warmup strategy slightly increases 'Test-domain' results on Colored MNIST (from 58.6% to 59.8% for Fishr, from 58.3% to 59.2% for IGA) but does not seem critical: in particular, it reduces IGA 'Test-domain' scores on OfficeHome.

Third, the estimation of gradient variances was improved with an exponential moving average (see Section 4.2.1 and Appendix D.3.1). We now use this strategy with domain-level gradient means for IGA in ω: ḡt e = γ ḡt-1 e + (1 -γ)g t e . This improves IGA (from 67.0% to 67.2% in 'Test-domain' on OfficeHome): yet, these scores remain consistently worse than Fishr's (from 67.5% to 68.2%).

In conclusion, this complements the experiments in Section 4.2.1 which showed that tackling gradient variance does better than tackling gradient mean: indeed, Fishr performed better than Fish [START_REF] Shi | Gradient matching for domain generalization[END_REF], AND-mask [START_REF] Parascandolo | Learning explanations that are hard to vary[END_REF] and SAND-mask [START_REF] Shahtalebi | Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization[END_REF]. As a final note, Fishr + IGA -i.e., matching simultaneously gradient means (the first moment) and variances (the second moment) -performs best. Future works may further analyze the complementary of these gradient-based methods. This Section is a preliminary introduction to a meta-discussion, not about the methodology to select the best hyperparameters, but about the methodology to select the hyperparameter distributions in DomainBed. This question has not been discussed in previous works (as far as we know).

D.3.3. HYPERPARAMETER DISTRIBUTIONS

After few initial iterations on the main idea of the paper, we had to select the distributions to sample our three hyperparameters from, as described in Table 8. First, to select the ema γ distribution, we knew that the authors from Le Roux et al. ( 2011) have not noticed "any significant difference in validation errors" for different values higher than 0.9. Moreover γ should remain strictly lower than 1. Thus, sampling from Uniform(0.9, 0.99) seemed appropriate. Second, sampling the number of warmup iterations uniformly along training from Uniform(0, 5000) seemed the most natural and neutral choice. Lastly, the choice of the λ distribution was more complex. As a reminder, a low λ inactivates the regularization while an extremely high λ may destabilize the training.

In Table 14, we investigate two distributions: λ ∼ 10 Uniform (1,4) (eventually chosen for Fishr) and λ ∼ 10 Uniform (1,5) . First, we observe that results are mostly similar: it confirms that Fishr is consistently better than ERM (where λ = 0), and in average is the best approach with the 'Test-domain' model selection and among the best approaches with the 'Trainingdomain' model selection. Second, the existence of consistent differences in results suggests that the best hyperparameter distribution depends on the dataset at hand and that the performance gap depends on the selection method.

While out of the scope of this paper, we believe these results were important for transparency (along with publishing our code), and may motivate the need for new protocols -for example with bayesian hyperparameter search (Turner et al., 2021) -that future benchmarks may introduce. ERM 95.9 ± 0.1 98.9 ± 0.0 98.8 ± 0.0 98.9 ± 0.0 98.9 ± 0.0 96.4 ± 0.0 98.0 2 IRM 95.5 ± 0.1 98.8 ± 0.2 98.7 ± 0.1 98.6 ± 0.1 98.7 ± 0.0 95.9 ± 0.2 97.7 15 GroupDRO 95.6 ± 0.1 98.9 ± 0.1 98.9 ± 0.1 99.0 ± 0.0 98.9 ± 0.0 96.5 ± 0.2 98.0 2 Mixup 95.8 ± 0.3 98.9 ± 0.0 98.9 ± 0.0 98.9 ± 0.0 98.8 ± 0.1 96.5 ± 0.3 98.0 2 MLDG 95.8 ± 0.1 98.9 ± 0.1 99.0 ± 0.0 98.9 ± 0.1 99.0 ± 0.0 95.8 ± 0.3 97.9 8 CORAL 95.8 ± 0.3 98.8 ± 0.0 98.9 ± 0.0 99.0 ± 0.0 98.9 ± 0.1 96.4 ± 0.2 98.0 2 MMD 95.6 ± 0.1 98.9 ± 0.1 99.0 ± 0.0 99.0 ± 0.0 98.9 ± 0.0 96.0 ± 0.2 97.9 8 DANN 95.0 ± 0.5 98.9 ± 0.1 99.0 ± 0.0 99.0 ± 0.1 98.9 ± 0.0 96.3 ± 0.2 97.8 13 CDANN 95.7 ± 0.2 98.8 ± 0.0 98.9 ± 0.1 98.9 ± 0.1 98.9 ± 0.1 96.1 ± 0.3 97.9 8 MTL 95.6 ± 0.1 99.0 ± 0.1 99.0 ± 0.0 98.9 ± 0.1 99.0 ± 0.1 95.8 ± 0.2 97.9 8 SagNet 95.9 ± 0.3 98.9 ± 0.1 99.0 ± 0.1 99.1 ± 0.0 99.0 ± 0.1 96.

D.4. Full DomainBed results

Tables

Figure 1 :

 1 Figure 1: Fishr principle. Fishr considers the individual (per-sample) gradients of the loss in the network weights θ. Specifically, Fishr matches the domain-level gradient variances of the distributions across the two training domains: A ({g i A } n A i=1 in orange) and B ({g i B } n B i=1 in blue). We will show how this regularization during the learning of θ improves the out-of-distribution generalization properties by aligning the domain-level loss landscapes at convergence.

2 2

 2 as in Deep Domain Confusion (DDC) (Tzeng et al., 2014). Yet, Johansson et al. (2019) and Zhao et al. (2019) show that these approaches are insufficient to guarantee good generalization. Motivated by arguments from causality (Pearl, 2009) and the idea that statistical dependencies are epiphenomena of an underlying structure, Invariant Risk Minimization (IRM) (Arjovsky et al., 2019) explains that the predictor should be invariant (Peters et al., 2016; Rojas-Carulla et al., 2018), i.e., simultaneously optimal across all domains. Yet, recent works point out pitfalls of IRM (Guo et al., 2021; Kamath et al., 2021; Ahuja et al., 2019), that does not provably work with non-linear data (Rosenfeld et al., 2021) and could not improve over ERM when hyperparameter selection is restricted (Koh et al., 2020; Gulrajani & Lopez-Paz, 2021). Among many suggested improvements (Chang et al., 2020; Idnani & Kao, 2020; Teney et al., 2020; Ahmed et al., 2021), Risk Extrapolation (V-REx) (Krueger et al., 2021) argues that training risks from different domains should be similar and thus penalizes |R A -R B | 2 when E = {A, B}.

  learning. In OOD generalization, Koyama & Yamaguchi (2020); Parascandolo et al. (2021); Shi et al. (

When E =

 = {A, B}, IGA (Koyama & Yamaguchi, 2020) minimizes ||g A -g B || 2 2 ; Fish (Shi et al., 2021) increases g A • g B ; AND-mask (Parascandolo et al., 2021) and others (Mansilla et al., 2021; Shahtalebi et al., 2021) update weights only when g A and g B point to the same direction.

Figure 2 :

 2 Figure 2: Loss landscapes around inconsistent weights θ * at convergence. N 0.2A,θ * contains weights θ for which R A (θ) is low (≤ 0.2) but R B (θ) is high (≥ 0.9). This inconsistency is due to conflicting domain-level loss landscapes, specifically gaps between domain-level risks and curvatures at θ * . This is visible in the disagreements across the variances of gradients {g i A } n A i=1 and {g i B } n B i=1 .

2 .

 2 In contrast, Parascandolo et al. (2021) made the strong assumption: R A (θ * ) = R B (θ * ) = 0. While Parascandolo et al. (2021) ignored this first term, we follow their diagonal approximation of the Hessians to analyze the second term. In that case, H e = diag (λ e 1 , • • • , λ e h ) with ∀i ∈ {1, . . . , h} , λ e i > 0. Then:

F

  and F were shown to share the same structure and to be similar up to a scalar factor (Thomas et al., 2020). They also have analogous properties: Tr( F ) ≈ Tr(F ). This was discussed in Li et al. (2020) and further highlighted even at early stages of training (before overfitting) in the Fig. 1 and the Appendix S3 of Singh & Alistarh (2020).

Figure 3 :

 3 Figure 3: Colored MNIST dynamics. At epoch 190, λ strongly steps up: then, the Fishr θ regularization matches the domain-level gradient variances (red) across domains E = {90%, 80%}, and consequently, the training empirical risks (dotted pink) and Hessians (purple). This reduces train accuracy (orange) but increases test accuracy (blue) as the network learns to predict the digit's shape. As shown in Fig. 7, training dynamics are different for ERM.

Algorithm 1

 1 Training procedure for Fishr on DomainBed. Input: DNN f θ , observations D e = x i e , y i e ne i=1 for domains e ∈ E, regularization weight λ, warmup iteration i warmup , exponential moving average γ and batch size b s Initialize: moving averages: ∀e ∈ E, v mean e ← 0 for iter from 1 to #iters do {# Step 1: standard ERM procedure} for e ∈ E do Randomly select batch: {(x i e , y i e )} i∈B of size b s Compute predictions: ∀i ∈ B, ŷi e ← f θ (x i e ) Compute empirical risks: R e (θ) ← i∈B ℓ ŷi e , y i e end for L(θ) = 1 |E| e∈E R e (θ) {# Step 2: gradient variances in classifier} for e ∈ E do Compute individual gradients in w ω with Back-PACK: ∀i ∈ B, g i e ← ∇ ω ℓ ŷi e , y i e Compute domain gradient variances v e (Eq. 2) Update v mean e = v e ← γv mean e + (1 -γ)v iter e end for if iter ≥ i warmup then L(θ) += λL Fishr (θ) (Eq. 3) end if {# Step 3: gradient descent in the whole network} Backpropagate gradients ∇ θ L(θ) in the network f θ with standard PyTorch end for for the same number of steps. Results are averaged over three trials. This experimental setup is further described in Appendix D.1. By imposing the datasets, the training procedure and controlling the hyperparameter search, DomainBed is arguably the fairer open-source benchmark to rigorously compare the different strategies for OOD generalization.

2 2

 2 2. Assumption A.1 is milder on N ϵ e,θ * for low ϵ. Indeed, when ϵ → 0, then max θ∈N ϵ e,θ * ∥θ -θ * ∥ → 0 and the quadratic approximation coincides with the second-order Taylor expansion around θ * . Moreover, this approximation is common in optimization[START_REF] Schaul | No more pesky learning rates[END_REF][START_REF] Jastrzebski | Three factors influencing minima in SGD[END_REF].

Figure 4 :

 4 Figure 4: Inconsistency I ϵ (A, B) between domains A and B, decomposed into R(A, B) depending on domain-level risks and H ϵ (A, B) depending on domain-level curvatures at θ * .

Fig. 3 .

 3 We leveraged the DiagHessian method from BackPACK to compute Hessian diagonals, in all network weights θ. Notably, Hessians are impractical in a training objective as computing "Hessian is an order of magnitude more computationally intensive" (see Fig.9in[START_REF] Dangel | BackPACK: Packing more into backprop[END_REF]). This Appendix further analyzes the Hessian trajectory during training.

Fig. 5

 5 Fig.5illustrates the dynamics for Fishr θ : following the scheduling previously described in Appendix C.1, λ jumping to a high value at epoch 190 activates the regularization. After this epoch, the domain-level Hessians are not only close in Frobenius distance, but also have similar norms and directions. On the contrary, when using only ERM in Fig.6, the distance between domain-level Hessians keeps increasing with the number of epochs. As a side note, flatter loss landscapes in ERM -as reflected by the Hessian norms in orange -do not correlate with improved generalization[START_REF] Dinh | Sharp minima can generalize for deep nets[END_REF].

Figure 5 :

 5 Figure 5: Hessian dynamics on Colored MNIST with Fishr: at epoch 190, λ steps up. Then domain-level Hessians are matched across domains (purple). More precisely, they take similar directions -high cosine similarity (red) -and similar norms (blue). The Hessians' norms (orange) remain quite high thus the loss landscapes are rather sharp.

Figure 6 :

 6 Figure 6: Hessian dynamics on Colored MNIST with ERM: λ = 0 along training. The Frobenius distance between domain-level Hessians (purple) keeps increasing: so does the distance between their norms (blue). Their cosine similarity (red) steadily decreases. The loss landscapes are flat at convergence (low Hessian norms in orange).

Figure 7 :

 7 Figure 7: Colored MNIST dynamics with ERM.

  Second, the 'Training-domain' setup suffers from underspecification: "predictors with equivalently strong held-out performance in the training domain [...] can behave very differently" in test (D'Amour et al., 2020). This underspecification favors low regularization thus low values of λ. To select the model with the best generalization properties, future benchmarks may consider the training calibration (Wald et al., 2021) rather than merely selecting the model with the best training accuracy.

  Parascandolo et al. (2021), that is empirically reasonable (Sagun et al., 2018): "in only very few steps . . . large negative eigenvalues disappear" (Ghorbani et al., 2019).

Table 1 :

 1 Cosine similarity between Hessian diagonals and gradient variances cos (Diag (H e ) , Var(G e )), for an ERM at convergence on Colored MNIST with the two training domains e ∈ {90%, 80%}.ŷ∼P θ (•|x i ) ∇ θ log p θ ( ŷ|x i )∇ θ log p θ ( ŷ|x i ) ⊤ log p θ (y i |x i )∇ θ log p θ (y i |x i ) ⊤[START_REF] Martens | New insights and perspectives on the natural gradient method[END_REF] where p θ (•|x) is the density predicted by f θ on input x.While F uses the model distribution P θ (•|X), F uses the data distribution P (Y |X). Despite this key difference,

				e = 90%	e = 80%
		On classifier weights w	0.9999980 0.9999905
		On all network weights θ 0.9971040 0.9962264
	The	Hessian	and	the	Fisher	Informa-
	tion	Matrix	(FIM). The	FIM	F	=
	n i=1 E (Fisher, 1922; C.R., 1945) approximates the Hessian H

Table 2 :

 2 Diagonal approximation. The empirical similarities between C and H motivate using gradient variance rather than gradient covariance, which scales down the number of targeted components from |θ| 2 to |θ|. Indeed, diagonally approximating the Hessian is common: e.g., for OOD generalization(Parascandolo et al., 2021), optimization (LeCun et al., 2012; Kingma & Ba, 2014), continual learning (Kirkpatrick et al., 2017) and pruning (LeCun et al., 1990; Theis et al., 2018). This is based on the empirical evidence (Becker & Le Cun, 1988) that Hessians are diagonally dominant at the end of training. Our diagonal approximation is also motivated by the critical importance of Tr(C) (Jastrzebski et al., 2021; Faghri et al., 2020) to analyze the generalization properties of DNNs. We confirm empirically in Appendix C.2.3 that considering the off-diagonal parts of C performs no better that just matching the diagonals. Invariance

	size |θ| × |θ| and

analysis at convergence on Colored MNIST across the two training domains E = {90%, 80%}. Compared to ERM, Fishr matches the gradient variance (Diag(C 90% ) ≈ Diag(C 80% )) in all network weights θ. Most importantly, this enforces invariance in domainlevel risks (R 90% ≈ R 80% ) and in domain-level Hessians (Diag(H 90% ) ≈ Diag(H 80%

Table 3 :

 3 Colored MNIST results. All methods use hyperparameters optimized for IRM.

	Method Train acc.	Test acc.	Gray test acc.
	ERM	86.4 ± 0.2 14.0 ± 0.7	71.0 ± 0.7
	IRM	71.0 ± 0.5 65.6 ± 1.8	66.1 ± 0.2
	V-REx 71.7 ± 1.5 67.2 ± 1.5	68.6 ± 2.2
	Fishr

θ 69.6 ± 0.9 71.2 ± 1.1 70.2 ± 0.7 Fishr ω 71.0 ± 0.9 69.5 ± 1.0 70.2 ± 1.1 Fishr ϕ 65.6 ± 1.3 73.8 ± 1.0 70.0 ± 0.9

Table 4 :

 4 DomainBed benchmark. We format first, second and worse than ERM results.[START_REF] Blanchard | Domain generalization by marginal transfer learning[END_REF]. The memory overhead is (|E| × |ω|). We study by ablation the importance of this warmup strategy and this γ in Appendices D.3.1 and D.3.2. Fishr is simple to implement (see the Algorithm 1) using the BackPACK (Dangel et al., 2020) package. While PyTorch (Paszke et al., 2019) can compute efficiently batch gradients, BackPACK optimizes the computation of individual gradients, sample per sample, at almost no time overhead.Thus, Fishr is also at low computational costs. For example, on PACS (7 classes and |ω| = 14, 343) with a ResNet-50 and batch size 32, Fishr induces an overhead in memory of +0.2% and in training time of +2.7% (with a Tesla V100) compared to ERM; on the larger-scale DomainNet (345 classes and |ω| = 706, 905), the overhead is +7.0% in memory and +6.5% in training time. As a side note, keeping the full covariance of size |ω| 2 ≈ 5 × 10 8 on DomainNet would not have been possible. In contrast, Fish[START_REF] Shi | Gradient matching for domain generalization[END_REF] leverages a meta-learning algorithm that is impractical as |E| times longer to train than ERM.

					Accuracy (↑)					Ranking (↓)
	Algorithm	CMNIST RMNIST	VLCS	PACS	OfficeHome TerraInc DomainNet Avg	Arith. mean	Geom. mean	Median
	ERM	57.8 ± 0.2	97.8 ± 0.1 77.6 ± 0.3 86.7 ± 0.3	66.4 ± 0.5	53.0 ± 0.3	41.3 ± 0.1	68.7	9.1	8.1	8
	IRM	67.7 ± 1.2	97.5 ± 0.2 76.9 ± 0.6 84.5 ± 1.1	63.0 ± 2.7	50.5 ± 0.7	28.0 ± 5.1	66.9	14.7	12.4	16
	GroupDRO	61.1 ± 0.9	97.9 ± 0.1 77.4 ± 0.5 87.1 ± 0.1	66.2 ± 0.6	52.4 ± 0.1	33.4 ± 0.3	67.9	8.6	7.5	8
	Mixup	58.4 ± 0.2	98.0 ± 0.1 78.1 ± 0.3 86.8 ± 0.3	68.0 ± 0.2	54.4 ± 0.3	39.6 ± 0.1	69.0	5.3	3.9	4
	MLDG	58.2 ± 0.4	97.8 ± 0.1 77.5 ± 0.1 86.8 ± 0.4	66.6 ± 0.3	52.0 ± 0.1	41.6 ± 0.1	68.7	9.1	8.2	9
	CORAL	58.6 ± 0.5	98.0 ± 0.0 77.7 ± 0.2 87.1 ± 0.5	68.4 ± 0.2	52.8 ± 0.2	41.8 ± 0.1	69.2	4.6	3.4	3
	MMD	63.3 ± 1.3	98.0 ± 0.1 77.9 ± 0.1 87.2 ± 0.1	66.2 ± 0.3	52.0 ± 0.4	23.5 ± 9.4	66.9	7.0	4.9	6
	DANN	57.0 ± 1.0	97.9 ± 0.1 79.7 ± 0.5 85.2 ± 0.2	65.3 ± 0.8	50.6 ± 0.4	38.3 ± 0.1	67.7	11.9	9.6	15
	CDANN	59.5 ± 2.0	97.9 ± 0.0 79.9 ± 0.2 85.8 ± 0.8	65.3 ± 0.5	50.8 ± 0.6	38.5 ± 0.2	68.2	9.6	7.4	10
	MTL	57.6 ± 0.3	97.9 ± 0.1 77.7 ± 0.5 86.7 ± 0.2	66.5 ± 0.4	52.2 ± 0.4	40.8 ± 0.1	68.5	8.4	7.8	7
	SagNet	58.2 ± 0.3	97.9 ± 0.0 77.6 ± 0.1 86.4 ± 0.4	67.5 ± 0.2	52.5 ± 0.4	40.8 ± 0.2	68.7	8.0	7.2	6
	ARM	63.2 ± 0.7	98.1 ± 0.1 77.8 ± 0.3 85.8 ± 0.2	64.8 ± 0.4	51.2 ± 0.5	36.0 ± 0.2	68.1	9.9	7.5	12
	V-REx	67.0 ± 1.3	97.9 ± 0.1 78.1 ± 0.2 87.2 ± 0.6	65.7 ± 0.3	51.4 ± 0.5	30.1 ± 3.7	68.2	7.7	5.5	5
	RSC	58.5 ± 0.5	97.6 ± 0.1 77.8 ± 0.6 86.2 ± 0.5	66.5 ± 0.6	52.1 ± 0.2	38.9 ± 0.6	68.2	9.9	9.4	9
	AND-mask	58.6 ± 0.4	97.5 ± 0.0 76.4 ± 0.4 86.4 ± 0.4	66.1 ± 0.2	49.8 ± 0.4	37.9 ± 0.6	67.5	13.4	13.1	12
	SAND-mask 62.3 ± 1.0	97.4 ± 0.1 76.2 ± 0.5 85.9 ± 0.4	65.9 ± 0.5	50.2 ± 0.1	32.2 ± 0.6	67.2	14.3	13.5	15
	Fish	61.8 ± 0.8	97.9 ± 0.1 77.8 ± 0.6 85.8 ± 0.6	66.0 ± 2.9	50.8 ± 0.4	43.4 ± 0.3	69.1	8.4	6.6	7
	Fishr	68.8 ± 1.4	97.8 ± 0.1 78.2 ± 0.2 86.9 ± 0.2	68.2 ± 0.2	53.6 ± 0.4	41.8 ± 0.2	70.8	3.9	2.8	2

overshadowed by Φ ϕ due to |ω| ≪ |ϕ|. Finally, it's worth noting that this last-layer approximation is consistent with the IRM condition

[START_REF] Arjovsky | Invariant risk minimization[END_REF]

and is common for unsupervised domain adaptation

[START_REF] Ganin | Domain-adversarial training of neural networks[END_REF]

.

Fishr relies on three hyperparameters. First, the λ coefficient controls the regularization strength: with λ = 0 we recover ERM while a high λ may cause underfitting. We show that Fishr is robust to the choice of the sampling distribution for hyperparameter λ in Appendix D.3.3. Second the warmup iteration defines the step at which we activate the regularization. This warmup strategy is taken from previous works such as IRM

[START_REF] Arjovsky | Invariant risk minimization[END_REF]

, V-REx

[START_REF] Krueger | Out-ofdistribution generalization via risk extrapolation (rex)[END_REF]

or Spectral Decoupling

[START_REF] Pezeshki | Gradient starvation: A learning proclivity in neural networks[END_REF]

. Before that step, the DNN is trained with ERM to learn predictive features. After that step, the Fishr regularization encourages the DNN to have invariant gradient variances. Lastly, the domain-level gradient variances are more accurate when estimated over more data points.

Table 4

 4 summarizes the results on DomainBed using the 'Test-domain' model selection: the validation set (to select the best hyperparameters) follows the same distribution as the test domain. Appendix D.2 reports results with the 'Training-domain' model selection while results are detailed per dataset in Appendix D.4.

  1. Having similar top eigenvectors across {K e } e∈E would delete detrimental domain-dependent shortcuts and favor the learning of a common mechanism. Indeed, truly informative features should remain consistent across domains. 2. Having similar top eigenvalues across {K e } e∈E would improve the optimization schema for simultaneous training at the same speed. Indeed, it would facilitate the finding of a learning rate for simultaneous convergence on all domains. It's worth noting that if we quickly overfit on a first domain using spurious explanations, invariances will then be hard to learn due to the gradient starvation phenomena (Pezeshki et al., 2021).

Directly matching K e would require assuming that each domain coincides and contains the same samples; for example, with different pose angles

[START_REF] Ghifary | Domain generalization for object recognition with multitask autoencoders[END_REF]

. To avoid such a strong assumption, we leverage the fact that the 'true' Fisher Information Matrix F and the NTK K share the same non-zero eigenvalues since F is dual to K (see Appendix C.1 in Maddox et al. (2019), notably for classification tasks). Moreover, their eigenvectors are strongly related (see Appendix C in Kopitkov & Indelman (2019)). Thus, having similar {F e } e∈E encourages {K e } e∈E to have similar spectral decomposition. Based on the close relations between C and F (see Section 3.2.3), this further motivates the need to match gradient variances during the SGD trajectory -and not only at convergence as in Section 3.2.

Table 5 :

 5 Performances comparison on the linear dataset from[START_REF] Shi | Gradient matching for domain generalization[END_REF] In test, green digits have a 10% chance of being in 5-9. Due to this modification in correlation, a model should ideally ignore the color information and only rely on the digits' shape: this would obtain a 75% test accuracy.In the experimental setup from IRM, the network is a 3 layers MLP with ReLu activation, optimized with Adam[START_REF] Kingma | A method for stochastic optimization[END_REF]. IRM selected the following hyperparameters by random search over 50 trials: hidden dimension of 390, l 2 regularizer weight of 0.00110794568, learning rate of 0.0004898536566546834, penalty anneal iters (or warmup iter) of 190, penalty weight (λ) of 91257.18613115903, 501 epochs and batch size 25,000 (half of the dataset size). We strictly keep the same hyperparameters values in our proof of concept in Section 4.1. The code is almost unchanged from https://github.com/facebookresearch/InvariantRiskMinimization. Li et al., 2020; Singh & Alistarh, 2020; Thomas et al., 2020), we argue in Section 3.2.3 that gradient covariance C can be used as a proxy to regularize the Hessian H -even though the proper approximation bounds are out of scope of this paper. This was empirically validated at convergence in Table 2 and during training in

	Method	Matched statistics	Train acc. Test acc.	W	b
	ERM	N/A	97 %	57 %	[2.8,3.3,3.3,0.0] -2.7
	Fish	Gradient means	93 %	93 %	[0.4,0.2,0.2,0.0] -0.4
	Fishr	Centered gradient variances	93 %	93 %	[2.0,1.2,1.2,0.0] -0.6
	Fishr	Uncentered gradient variances	93 %	93 %	[1.9,0.9,0.9,0.0] -0.6
	C. Colored MNIST in the IRM Setup

by the large values

(3.3)

for weights associated to f 2 and f 3 ; this induces low test accuracy (57%). On the contrary, Fishr forces the linear model to rely mostly on the invariant feature f 1 , as indicated by the lower values (1.2) for weights associated to f 2 and f 3 ; in accuracy, Fishr performs similarly in test and train (93%).

C.1. Description of the Colored MNIST experiment

Colored MNIST is a binary digit classification dataset introduced in IRM (

[START_REF] Arjovsky | Invariant risk minimization[END_REF]

. Compared to the traditional MNIST (LeCun et al., 2010), it has 2 main differences. First, 0-4 and 5-9 digits are each collapsed into a single class, with a 25% chance of label flipping. Second, digits are either colored red or green, with a strong correlation between label and color in training. However, this correlation is reversed at test time. Specifically, in training, the model has access to two domains E = {90%, 80%}: in the first domain, green digits have a 90% chance of being in 5-9; in the second, this chance goes down to 80%.

Table 6 :

 6 Colored MNIST experiments without label flipping.

	Method Train acc.	Test acc.	Gray test acc.
	ERM	99.0 ± 0.0 91.8 ± 0.2	95.0 ± 0.4
	IRM	96.4 ± 0.2 82.2 ± 0.1	92.6 ± 0.2
	V-REx 97.1 ± 0.2 95.3 ± 0.4	94.1 ± 0.4
	Fishr θ	97.9 ± 0.2 93.6 ± 0.4	94.8 ± 0.4
	Fishr ω	97.0 ± 0.2 95.3 ± 0.4	94.1 ± 0.4
	Fishr ϕ	97.9 ± 0.1 93.5 ± 0.3	94.8 ± 0.4

Table 7 :

 7 Colored MNIST experiments with different statistics matched. All hyperparameters were optimized for IRM.In Section 3.2.3, we argue that the gradient centered covariance C and the empirical Fisher Information Matrix (or uncentered covariance) F are highly related and equivalent when the DNN is at convergence and the gradient means are zero. So, we could have tackled the diagonals of the domain-level { Fe } e∈E across domains, i.e., without centering the variances.We omitted the recent weight averaging approaches(Cha et al., 2021; Rame et al., 2022) whose contribution is complementary to others, that uses a custom hyperparameter search and does not report scores with the 'Test-domain' model selection. L38, L43, L46}, with 24,788 examples of dimension (3, 224, 224) and 10 classes. 7. DomainNet (Peng et al., 2019) has six domains d ∈ {clipart, infograph, painting, quickdraw, real, sketch}, with 586,575 examples of size (3, 224, 224) and 345 classes.

		Method			25% label flipping		No label flipping
	Gradients in	Name	Matched statistics Train acc.	Test acc.	Gray test acc.	Train acc.	Test acc.	Gray test acc.
		Centered variance (= Fishrω)	Var(Ge)	71.0 ± 0.9 69.5 ± 1.0	70.2 ± 1.1	97.0 ± 0.2	95.3 ± 0.4	94.1 ± 0.4
	ω	Centered covariance	Ce	70.7 ± 1.0 69.1 ± 1.1	69.9 ± 1.1	97.0 ± 0.2	95.3 ± 0.4	94.0 ± 0.4
		Uncentered variance	Diag( 1 ne Fe)	71.3 ± 0.9 69.5 ± 1.0	70.3 ± 1.0	97.0 ± 0.2	95.3 ± 0.4	94.1 ± 0.4
		Centered variance (= Fishrθ)	Var(Ge)	69.6 ± 0.9 71.2 ± 1.1	70.2 ± 0.7	97.9 ± 0.1	93.5 ± 0.3	94.7 ± 0.4
	θ	Centered covariance	Ce	Not	possible	for	computational (memory)	reasons
		Uncentered variance	Diag( 1 ne Fe)	71.0 ± 0.8 70.0 ± 1.1	70.1 ± 0.9	97.9 ± 0.0	93.5 ± 0.3	94.8 ± 0.4
		Centered variance (= Fishrϕ)	Var(Ge)	65.6 ± 1.3 73.8 ± 1.0	70.0 ± 0.9	97.9 ± 0.1	93.5 ± 0.3	94.8 ± 0.4
	ϕ	Centered covariance	Ce	Not	possible	for	computational (memory)	reasons
		Uncentered variance	Diag( 1 ne Fe)	71.5 ± 0.8 69.1 ± 1.1	70.0 ± 1.0	97.9 ± 0.1	93.5 ± 0.3	94.8 ± 0.4
	C.2.4. CENTERED OR UNCENTERED VARIANCE ?

Empirically, comparing the first and third lines in Table

7

shows that centering or not the variance are almost equivalent. This holds true when applying Fishr on all weights θ (as lines fourth and six are also very similar). This was empirically confirmed in DomainBed: for example, Fishr with either centered or uncentered variances reach 67.8. Still, it's worth noting that explicitly matching simultaneously the gradient centered variances along with the gradient means performs best in Appendix D.3.2.

DomainBed includes seven multi-domain computer vision classification

datasets: 1. Colored MNIST (Arjovsky et al., 2019) is a variant of the MNIST handwritten digit classification dataset (LeCun et al., 2010). As described previously in Appendix C.1, domain d ∈ {90%, 80%, 10%} contains a disjoint set of digits colored: the correlation strengths between color and label vary across domains. The dataset contains 70,000 examples of dimension (2, 28, 28) and 2 classes. Most importantly, the network, the hyperparameters, the image shapes, etc. are not the same as in the IRM setup from Section 4.1. 2. Rotated MNIST (Ghifary et al., 2015) is a variant of MNIST where domain d ∈ {0, 15, 30, 45, 60, 75} contains digits rotated by d degrees, with 70,000 examples of dimension (1, 28, 28) and 10 classes. 3. VLCS (Fang et al., 2013) includes photographic domains d ∈ {Caltech101, LabelMe, SUN09, VOC2007}, with 10,729 examples of dimension (3, 224, 224) and 5 classes. 4. PACS (Li et al., 2017) includes domains d ∈ {art, cartoons, photos, sketches}, with 9,991 examples of dimension (3, 224, 224) and 7 classes. 5. OfficeHome (Venkateswara et al., 2017) includes domains d ∈ {art, clipart, product, real}, with 15,588 examples of dimension (3, 224, 224) and 65 classes. 6. TerraIncognita (Beery et al., 2018) contains photographs of wild animals taken by camera traps at locations d ∈ {L100,

Table 9 :

 9 DomainBed with 'Training-domain' model selection. We format first, second and worse than ERM results.

					Accuracy (↑)					Ranking (↓)
	Algorithm	CMNIST RMNIST	VLCS	PACS	OfficeHome TerraInc DomainNet Avg	Arith. mean	Geom. mean	Median
	ERM	51.5 ± 0.1	98.0 ± 0.0 77.5 ± 0.4 85.5 ± 0.2	66.5 ± 0.3	46.1 ± 1.8	40.9 ± 0.1	66.6	7.0	5.9	7
	IRM	52.0 ± 0.1	97.7 ± 0.1 78.5 ± 0.5 83.5 ± 0.8	64.3 ± 2.2	47.6 ± 0.8	33.9 ± 2.8	65.4	10.7	8.5	14
	GroupDRO	52.1 ± 0.0	98.0 ± 0.0 76.7 ± 0.6 84.4 ± 0.8	66.0 ± 0.7	43.2 ± 1.1	33.3 ± 0.2	64.8	11.3	8.4	14
	Mixup	52.1 ± 0.2	98.0 ± 0.1 77.4 ± 0.6 84.6 ± 0.6	68.1 ± 0.3	47.9 ± 0.8	39.2 ± 0.1	66.7	5.7	4.2	3
	MLDG	51.5 ± 0.1	97.9 ± 0.0 77.2 ± 0.4 84.9 ± 1.0	66.8 ± 0.6	47.7 ± 0.9	41.2 ± 0.1	66.7	8.0	7.0	8
	CORAL	51.5 ± 0.1	98.0 ± 0.1 78.8 ± 0.6 86.2 ± 0.3	68.7 ± 0.3	47.6 ± 1.0	41.5 ± 0.1	67.5	3.6	2.5	2
	MMD	51.5 ± 0.2	97.9 ± 0.0 77.5 ± 0.9 84.6 ± 0.5	66.3 ± 0.1	42.2 ± 1.6	23.4 ± 9.5	63.3	12.3	11.8	10
	DANN	51.5 ± 0.3	97.8 ± 0.1 78.6 ± 0.4 83.6 ± 0.4	65.9 ± 0.6	46.7 ± 0.5	38.3 ± 0.1	66.1	10.3	8.8	12
	CDANN	51.7 ± 0.1	97.9 ± 0.1 77.5 ± 0.1 82.6 ± 0.9	65.8 ± 1.3	45.8 ± 1.6	38.3 ± 0.3	65.6	11.1	10.7	10
	MTL	51.4 ± 0.1	97.9 ± 0.0 77.2 ± 0.4 84.6 ± 0.5	66.4 ± 0.5	45.6 ± 1.2	40.6 ± 0.1	66.2	10.9	10.2	10
	SagNet	51.7 ± 0.0	98.0 ± 0.0 77.8 ± 0.5 86.3 ± 0.2	68.1 ± 0.1	48.6 ± 1.0	40.3 ± 0.1	67.2	4.0	3.0	3
	ARM	56.2 ± 0.2	98.2 ± 0.1 77.6 ± 0.3 85.1 ± 0.4	64.8 ± 0.3	45.5 ± 0.3	35.5 ± 0.2	66.1	8.7	5.6	9
	V-REx	51.8 ± 0.1	97.9 ± 0.1 78.3 ± 0.2 84.9 ± 0.6	66.4 ± 0.6	46.4 ± 0.6	33.6 ± 2.9	65.6	8.3	7.7	8
	RSC	51.7 ± 0.2	97.6 ± 0.1 77.1 ± 0.5 85.2 ± 0.9	65.5 ± 0.9	46.6 ± 1.0	38.9 ± 0.5	66.1	11.4	10.6	9
	AND-mask	51.3 ± 0.2	97.6 ± 0.1 78.1 ± 0.9 84.4 ± 0.9	65.6 ± 0.4	44.6 ± 0.3	37.2 ± 0.6	65.5	13.6	12.7	15
	SAND-mask 51.8 ± 0.2	97.4 ± 0.1 77.4 ± 0.2 84.6 ± 0.9	65.8 ± 0.4	42.9 ± 1.7	32.1 ± 0.6	64.6	13.4	12.7	13
	Fish	51.6 ± 0.1	98.0 ± 0.0 77.8 ± 0.3 85.5 ± 0.3	68.6 ± 0.4	45.1 ± 1.3	42.7 ± 0.2	67.1	5.6	3.8	3
	Fishr	52.0 ± 0.2	97.8 ± 0.0 77.8 ± 0.1 85.5 ± 0.4	67.8 ± 0.1	47.4 ± 1.6	41.7 ± 0.0	67.1	5.6	4.8	5

Table 10 :

 10 Importance of the exponential moving average (ema) on DomainBed's Colored MNIST.

	Model selection Algorithm ema	+90%	+80%	10%	Avg
		ERM	N/A 71.8 ± 0.4 72.9 ± 0.1 28.7 ± 0.5 57.8
	Test-domain	V-REx	✗ ✓	72.8 ± 0.3 73.0 ± 0.3 55.2 ± 4.0 67.0 73.0 ± 0.2 73.0 ± 0.3 59.9 ± 2.6 68.6
		Fishr	✗ ✓	72.7 ± 0.3 72.8 ± 0.1 34.0 ± 4.5 59.8 74.1 ± 0.6 73.3 ± 0.1 58.9 ± 3.7 68.8
		ERM	N/A 71.7 ± 0.1 72.9 ± 0.2 10.0 ± 0.1 51.5
	Training-domain	V-REx	✗ ✓	72.4 ± 0.3 72.9 ± 0.4 10.2 ± 0.0 51.8 72.6 ± 0.5 73.3 ± 0.1 9.8 ± 0.1 51.9
		Fishr	✗ ✓	71.1 ± 0.6 73.6 ± 0.1 10.1 ± 0.2 51.6 72.3 ± 0.9 73.5 ± 0.2 10.1 ± 0.2 52.0

Table 11 :

 11 Importance of the exponential moving average (ema) on DomainBed's OfficeHome.

	Model selection Algorithm ema	A	C	P	R	Avg
		ERM	N/A 61.7 ± 0.7 53.4 ± 0.3 74.1 ± 0.4 76.2 ± 0.6 66.4
	Test-domain	V-REx	✗ ✓	59.6 ± 1.0 53.3 ± 0.3 73.2 ± 0.5 76.6 ± 0.4 65.7 59.0 ± 0.7 52.8 ± 0.8 74.6 ± 0.4 75.5 ± 0.3 65.5
		Fishr	✗ ✓	63.6 ± 0.4 53.2 ± 0.5 75.4 ± 0.5 77.8 ± 0.3 67.5 63.4 ± 0.8 54.2 ± 0.3 76.4 ± 0.3 78.5 ± 0.2 68.2
		ERM	N/A 61.3 ± 0.7 52.4 ± 0.3 75.8 ± 0.1 76.6 ± 0.3 66.5
	Training-domain	V-REx	✗ ✓	60.7 ± 0.9 53.0 ± 0.9 75.3 ± 0.1 76.6 ± 0.5 66.4 59.2 ± 1.0 51.7 ± 0.5 75.2 ± 0.2 76.6 ± 0.3 65.7
		Fishr	✗ ✓	62.2 ± 1.0 53.5 ± 0.2 76.6 ± 0.2 77.8 ± 0.4 67.5 62.4 ± 0.5 54.4 ± 0.4 76.2 ± 0.5 78.3 ± 0.1 67.8

Table 12 :

 12 Fishr (gradient variance) vs. IGA (gradient mean) on DomainBed's Colored MNIST.

	Model selection Algorithm	Gradients in Warmup ema	+90%	+80%	10%	Avg
		ERM	N/A	N/A	N/A 71.8 ± 0.4 72.9 ± 0.1 28.7 ± 0.5 57.8
			θ = ω ⊕ ϕ	✗	✗	71.8 ± 0.5 73.0 ± 0.3 29.2 ± 0.5 58.0
	Test-domain	IGA	ω ω ω	✗ ✓ ✓	✗ ✗ ✓	72.4 ± 0.1 73.3 ± 0.2 29.3 ± 0.6 58.3 72.5 ± 0.2 73.3 ± 0.1 31.8 ± 0.7 59.2 72.6 ± 0.3 72.9 ± 0.2 50.0 ± 1.2 65.2
				✗	✗	73.0 ± 0.3 73.2 ± 0.1 29.5 ± 1.1 58.6
		Fishr	ω	✓	✗	72.7 ± 0.3 72.8 ± 0.1 34.0 ± 4.5 59.8
				✓	✓	74.1 ± 0.6 73.3 ± 0.1 58.9 ± 3.7 68.8
		Fishr + IGA ω	✓	✓	73.3 ± 0.0 72.6 ± 0.5 66.3 ± 2.9 70.7
		ERM	N/A	N/A	N/A 71.7 ± 0.1 72.9 ± 0.2 10.0 ± 0.1 51.5
			θ = ω ⊕ ϕ	✗	✗	71.8 ± 0.3 73.2 ± 0.2 9.8 ± 0.0 51.6
	Training-domain	IGA	ω ω ω	✗ ✓ ✓	✗ ✗ ✓	71.8 ± 0.1 73.2 ± 0.2 10.1 ± 0.0 51.7 71.8 ± 0.2 73.1 ± 0.2 10.1 ± 0.0 51.7 72.5 ± 0.4 73.3 ± 0.2 10.1 ± 0.1 52.0
				✗	✗	71.6 ± 0.1 73.2 ± 0.1 9.9 ± 0.0 51.6
		Fishr	ω	✓	✗	71.1 ± 0.6 73.6 ± 0.1 10.1 ± 0.2 51.6
				✓	✓	72.3 ± 0.9 73.5 ± 0.2 10.1 ± 0.2 52.0
		Fishr + IGA ω	✓	✓	72.4 ± 0.4 73.1 ± 0.1 10.1 ± 0.1 51.8

Table 13 :

 13 Fishr (gradient variance) vs. IGA (gradient mean) on DomainBed's OfficeHome.

	Model selection Algorithm	Gradients in Warmup ema	A	C	P	R	Avg
		ERM	N/A	N/A	N/A 61.7 ± 0.7 53.4 ± 0.3 74.1 ± 0.4 76.2 ± 0.6 66.4
			θ = ω ⊕ ϕ	✗	✗	50.1 ± 2.5 49.6 ± 1.6 59.5 ± 6.7 68.5 ± 1.2 56.9
	Test-domain	IGA	ω ω ω	✗ ✓ ✓	✗ ✗ ✓	62.3 ± 0.3 53.9 ± 0.2 75.2 ± 0.4 77.4 ± 0.1 67.2 61.9 ± 0.4 52.6 ± 0.6 76.0 ± 0.8 77.5 ± 0.3 67.0 62.3 ± 1.0 53.4 ± 0.3 76.0 ± 0.7 77.0 ± 0.1 67.2
				✗	✗	61.8 ± 0.9 53.8 ± 0.4 76.6 ± 0.6 77.7 ± 0.2 67.5
		Fishr	ω	✓	✗	63.6 ± 0.4 53.2 ± 0.5 75.4 ± 0.5 77.8 ± 0.3 67.5
				✓	✓	63.4 ± 0.8 54.2 ± 0.3 76.4 ± 0.3 78.5 ± 0.2 68.2
		Fishr + IGA ω	✓	✓	63.6 ± 1.0 54.6 ± 0.5 76.6 ± 0.2 78.4 ± 0.4 68.3
		ERM	N/A	N/A	N/A 61.3 ± 0.7 52.4 ± 0.3 75.8 ± 0.1 76.6 ± 0.3 66.5
			θ = ω ⊕ ϕ	✗	✗	51.7 ± 1.3 49.3 ± 1.5 58.6 ± 7.1 69.0 ± 1.1 57.1
	Training-domain	IGA	ω ω ω	✗ ✓ ✓	✗ ✗ ✓	61.9 ± 0.0 53.6 ± 0.9 75.7 ± 0.5 76.0 ± 0.1 66.8 61.2 ± 0.1 52.2 ± 0.5 76.1 ± 0.2 77.2 ± 0.3 66.7 61.7 ± 0.5 52.4 ± 0.7 75.9 ± 0.4 77.1 ± 0.2 66.8
				✗	✗	63.8 ± 0.6 52.5 ± 0.5 76.7 ± 0.6 77.1 ± 1.0 67.5
		Fishr	ω	✓	✗	62.2 ± 1.0 53.5 ± 0.2 76.6 ± 0.2 77.8 ± 0.4 67.5
				✓	✓	62.4 ± 0.5 54.4 ± 0.4 76.2 ± 0.5 78.3 ± 0.1 67.8
		Fishr + IGA ω	✓	✓	63.3 ± 1.0 54.1 ± 0.3 76.5 ± 0.4 78.2 ± 0.6 68.0

Table 14 :

 14 Impact of the λ distribution from Table8.

	Model selection λ distribution	CMNIST RMNIST	VLCS	PACS	OfficeHome TerraInc DomainNet Avg
		Constant(0) (= ERM) 57.8 ± 0.2 97.8 ± 0.1 77.6 ± 0.3 86.7 ± 0.3	66.4 ± 0.5	53.0 ± 0.3	41.3 ± 0.1	68.7
	Test-domain	10 Uniform (1,4)	68.8 ± 1.4 97.8 ± 0.1 78.2 ± 0.2 86.9 ± 0.2	68.2 ± 0.2	53.6 ± 0.4	41.8 ± 0.1	70.8
		10 Uniform (1,5)	68.7 ± 1.3 97.8 ± 0.0 78.7 ± 0.3 87.5 ± 0.1	68.0 ± 0.4	52.2 ± 0.5	42.0 ± 0.1	70.7
		Constant(0) (= ERM) 51.5 ± 0.1 98.0 ± 0.0 77.5 ± 0.4 85.5 ± 0.2	66.5 ± 0.3	46.1 ± 1.8	40.9 ± 0.1	66.6
	Training-domain	10 Uniform (1,4)	52.0 ± 0.2 97.8 ± 0.0 77.8 ± 0.1 85.5 ± 0.4	67.8 ± 0.1	47.4 ± 1.6	41.7 ± 0.0	67.1
		10 Uniform (1,5)	51.8 ± 0.3 97.9 ± 0.0 77.9 ± 0.1 85.5 ± 0.6	67.4 ± 0.3	47.2 ± 1.0	41.8 ± 0.1	67.1

  below detail results for each dataset with 'Test-domain' and 'Training-domain' model selection methods. We format first and second best accuracies. Note that the per-dataset results for Fish[START_REF] Shi | Gradient matching for domain generalization[END_REF] are not available.

	D.4.2. ROTATED MNIST
	Rotated MNIST. Model selection: 'Test-domain' validation set
	Algorithm	0	15	30	45	60	75	Avg Ranking
	ERM	95.3 ± 0.2 98.7 ± 0.1 98.9 ± 0.1 98.7 ± 0.2 98.9 ± 0.0 96.2 ± 0.2 97.8	12
	IRM	94.9 ± 0.6 98.7 ± 0.2 98.6 ± 0.1 98.6 ± 0.2 98.7 ± 0.1 95.2 ± 0.3 97.5	16
	GroupDRO	95.9 ± 0.1 99.0 ± 0.1 98.9 ± 0.1 98.8 ± 0.1 98.6 ± 0.1 96.3 ± 0.4 97.9	5
	Mixup	95.8 ± 0.3 98.7 ± 0.0 99.0 ± 0.1 98.8 ± 0.1 98.8 ± 0.1 96.6 ± 0.2 98.0	2
	MLDG	95.7 ± 0.2 98.9 ± 0.1 98.8 ± 0.1 98.9 ± 0.1 98.6 ± 0.1 95.8 ± 0.4 97.8	12
	CORAL	96.2 ± 0.2 98.8 ± 0.1 98.8 ± 0.1 98.8 ± 0.1 98.9 ± 0.1 96.4 ± 0.2 98.0	2
	MMD	96.1 ± 0.2 98.9 ± 0.0 99.0 ± 0.0 98.8 ± 0.0 98.9 ± 0.0 96.4 ± 0.2 98.0	2
	DANN	95.9 ± 0.1 98.9 ± 0.1 98.6 ± 0.2 98.7 ± 0.1 98.9 ± 0.0 96.3 ± 0.3 97.9	5
	CDANN	95.9 ± 0.2 98.8 ± 0.0 98.7 ± 0.1 98.9 ± 0.1 98.8 ± 0.1 96.1 ± 0.3 97.9	5
	MTL	96.1 ± 0.2 98.9 ± 0.0 99.0 ± 0.0 98.7 ± 0.1 99.0 ± 0.0 95.8 ± 0.3 97.9	5
	SagNet	95.9 ± 0.1 99.0 ± 0.1 98.9 ± 0.1 98.6 ± 0.1 98.8 ± 0.1 96.3 ± 0.1 97.9	5
	ARM	95.9 ± 0.4 99.0 ± 0.1 98.8 ± 0.1 98.9 ± 0.1 99.1 ± 0.1 96.7 ± 0.2 98.1	1
	V-REx	95.5 ± 0.2 99.0 ± 0.0 98.7 ± 0.2 98.8 ± 0.1 98.8 ± 0.0 96.4 ± 0.0 97.9	5
	RSC	95.4 ± 0.1 98.6 ± 0.1 98.6 ± 0.1 98.9 ± 0.0 98.8 ± 0.1 95.4 ± 0.3 97.6	15
	AND-mask	94.9 ± 0.1 98.8 ± 0.1 98.8 ± 0.1 98.7 ± 0.2 98.6 ± 0.2 95.5 ± 0.2 97.5	16
	SAND-mask 94.7 ± 0.2 98.5 ± 0.2 98.6 ± 0.1 98.6 ± 0.1 98.5 ± 0.1 95.2 ± 0.1 97.4	18
	Fish							97.9	11
	Fishr	95.8 ± 0.1 98.3 ± 0.1 98.8 ± 0.1 98.6 ± 0.3 98.7 ± 0.1 96.5 ± 0.1 97.8	12
	Rotated MNIST. Model selection: 'Training-domain' validation set
	Algorithm	0	15	30	45	60	75	Avg Ranking

  ± 0.3 64.4 ± 1.4 74.1 ± 0.4 76.2 ± 1.3 78.3 4 RSC 97.9 ± 0.1 62.5 ± 0.7 72.3 ± 1.2 75.6 ± 0.8 77.1 17 AND-mask 97.8 ± 0.4 64.3 ± 1.2 73.5 ± 0.7 76.8 ± 2.6 78.1 5 SAND-mask 98.5 ± 0.3 63.6 ± 0.9 70.4 ± 0.8 77.1 ± 0.8 77.4 ± 1.0 81.3 ± 0.6 96.2 ± 0.3 82.7 ± 1.1 86.7 8 IRM 84.2 ± 0.9 79.7 ± 1.5 95.9 ± 0.4 78.3 ± 2.1 84.5 18 GroupDRO 87.5 ± 0.5 82.9 ± 0.6 97.1 ± 0.3 81.1 ± 1.2 87.1 3 Mixup 87.5 ± 0.4 81.6 ± 0.7 97.4 ± 0.2 80.8 ± 0.9 86.8 6 MLDG 87.0 ± 1.2 82.5 ± 0.9 96.7 ± 0.3 81.2 ± 0.6 86.8 6 CORAL 86.6 ± 0.8 81.8 ± 0.9 97.1 ± 0.5 82.7 ± 0.6 87.1 3 MMD 88.1 ± 0.8 82.6 ± 0.7 97.1 ± 0.5 81.2 ± 1.2 87.2 1 DANN 87.0 ± 0.4 80.3 ± 0.6 96.8 ± 0.3 76.9 ± 1.1 85.2 17 CDANN 87.7 ± 0.6 80.7 ± 1.2 97.3 ± 0.4 77.6 ± 1.5 85.8 14 MTL 87.0 ± 0.2 82.7 ± 0.8 96.5 ± 0.7 80.5 ± 0.8 86.7 8 SagNet 87.4 ± 0.5 81.2 ± 1.2 96.3 ± 0.8 80.7 ± 1.1 86.4 10 ARM 85.0 ± 1.2 81.4 ± 0.2 95.9 ± 0.3 80.9 ± 0.5 85.8 14 V-REx 87.8 ± 1.2 81.8 ± 0.7 97.4 ± 0.2 82.1 ± 0.7 87.2 1 RSC 86.0 ± 0.7 81.8 ± 0.9 96.8 ± 0.7 80.4 ± 0.5 86.2 12 AND-mask 86.4 ± 1.1 80.8 ± 0.9 97.1 ± 0.2 81.3 ± 1.1 86.4 10 SAND-mask 86.1 ± 0.6 80.3 ± 1.0 97.1 ± 0.3 80.0 ± 1.3 85.9 ± 0.6 80.8 ± 0.5 97.9 ± 0.4 81.1 ± 0.8 86.9 5 PACS. Model selection: 'Training-domain' validation set ± 0.8 77.4 ± 0.8 97.3 ± 0.4 73.5 ± 2.3 83.6 16 CDANN 84.6 ± 1.8 75.5 ± 0.9 96.8 ± 0.3 73.5 ± 0.6 82.6 18 MTL 87.5 ± 0.8 77.1 ± 0.5 96.4 ± 0.8 77.3 ± 1.8 84.6 10 SagNet 87.4 ± 1.0 80.7 ± 0.6 97.1 ± 0.1 80.0 ± 0.4 86.3 1 ARM 86.8 ± 0.6 76.8 ± 0.5 97.4 ± 0.3 79.3 ± 1.2 85.1 7 V-REx 86.0 ± 1.6 79.1 ± 0.6 96.9 ± 0.5 77.7 ± 1.7 84.9 8 RSC 85.4 ± 0.8 79.7 ± 1.8 97.6 ± 0.3 78.2 ± 1.2 85.2 6 AND-mask 85.3 ± 1.4 79.2 ± 2.0 96.9 ± 0.4 76.2 ± 1.4 84.4 14 SAND-mask 85.8 ± 1.7 79.2 ± 0.8 96.3 ± 0.2 76.9 ± 2.0 84.6 ± 1.4 51.8 ± 0.7 73.4 ± 0.5 75.5 ± 0.9 65.3 17 CDANN 57.9 ± 0.2 52.1 ± 1.2 74.9 ± 0.7 76.2 ± 0.2 65.3 14 MTL 60.7 ± 0.8 53.5 ± 1.3 75.2 ± 0.6 76.6 ± 0.6 66.5 8 SagNet 62.7 ± 0.5 53.6 ± 0.5 76.0 ± 0.3 77.8 ± 0.1 67.5 10 ARM 58.8 ± 0.5 51.8 ± 0.7 74.0 ± 0.1 74.4 ± 0.2 64.8 14 V-REx 59.6 ± 1.0 53.3 ± 0.3 73.2 ± 0.5 76.6 ± 0.4 65.7 1 RSC 61.7 ± 0.8 53.0 ± 0.9 74.8 ± 0.8 76.3 ± 0.5 66.5 12 AND-mask 60.3 ± 0.5 52.3 ± 0.6 75.1 ± 0.2 76.6 ± 0.3 66.1 10 SAND-mask 59.9 ± 0.7 53.6 ± 0.8 74.3 ± 0.4 75.8 ± 0.5 65.9 ± 0.8 54.2 ± 0.3 76.4 ± 0.3 78.5 ± 0.2 68.2 5 ± 1.4 51.4 ± 0.3 74.8 ± 1.1 75.1 ± 1.3 65.5 16 ANDMask 59.5 ± 1.2 51.7 ± 0.2 73.9 ± 0.4 77.1 ± 0.2 65.6 15 SAND-mask 60.3 ± 0.5 53.3 ± 0.7 73.5 ± 0.7 76.2 ± 0.3 65.8 ± 0.9 49.3 ± 0.6 60.1 ± 1.1 43.2 ± 0.5 53.0 3 IRM 56.5 ± 2.5 49.8 ± 1.5 57.1 ± 2.2 38.6 ± 1.0 50.5 16 GroupDRO 60.4 ± 1.5 48.3 ± 0.4 58.6 ± 0.8 42.2 ± 0.8 52.4 6 Mixup 67.6 ± 1.8 51.0 ± 1.3 59.0 ± 0.0 40.0 ± 1.1 54.4 1 MLDG 59.2 ± 0.1 49.0 ± 0.9 58.4 ± 0.9 41.4 ± 1.0 52.0 9 CORAL 60.4 ± 0.9 47.2 ± 0.5 59.3 ± 0.4 44.4 ± 0.4 52.8 4 MMD 60.6 ± 1.1 45.9 ± 0.3 57.8 ± 0.5 43.8 ± 1.2 52.0 9 DANN 55.2 ± 1.9 47.0 ± 0.7 57.2 ± 0.9 42.9 ± 0.9 50.6 15 CDANN 56.3 ± 2.0 47.1 ± 0.9 57.2 ± 1.1 42.4 ± 0.8 50.8 13 MTL 58.4 ± 2.1 48.4 ± 0.8 58.9 ± 0.6 43.0 ± 1.3 52.2 7 SagNet 56.4 ± 1.9 50.5 ± 2.3 59.1 ± 0.5 44.1 ± 0.6 52.5 5 ARM 60.1 ± 1.5 48.3 ± 1.6 55.3 ± 0.6 40.9 ± 1.1 51.2 12 V-REx 56.8 ± 1.7 46.5 ± 0.5 58.4 ± 0.3 43.8 ± 0.3 51.4 11 RSC 59.9 ± 1.4 46.7 ± 0.4 57.8 ± 0.5 44.3 ± 0.6 52.1 8 AND-mask 54.7 ± 1.8 48.4 ± 0.5 55.1 ± 0.5 41.3 ± 0.6 49.8 18 SAND-mask 56.2 ± 1.8 46.3 ± 0.3 55.8 ± 0.4 42.6 ± 1.2 50.2 ± 0.9 50.3 ± 0.3 58.8 ± 0.5 44.9 ± 0.5 53.6 2 TerraIncognita. Model selection: 'Training-domain' validation set ± 1.9 41.3 ± 4.8 54.9 ± 1.7 39.8 ± 2.3 45.8 11 MTL 49.3 ± 1.2 39.6 ± 6.3 55.6 ± 1.1 37.8 ± 0.8 45.6 12 SagNet 53.0 ± 2.9 43.0 ± 2.5 57.9 ± 0.6 40.4 ± 1.3 48.6 1 ARM 49.3 ± 0.7 38.3 ± 2.4 55.8 ± 0.8 38.7 ± 1.3 45.5 13 V-REx 48.2 ± 4.3 41.7 ± 1.3 56.8 ± 0.8 38.7 ± 3.1 46.4 9 RSC 50.2 ± 2.2 39.2 ± 1.4 56.3 ± 1.4 40.8 ± 0.6 46.6 8 AND-mask 50.0 ± 2.9 40.2 ± 0.8 53.3 ± 0.7 34.8 ± 1.9 44.6 15 SAND-mask 45.7 ± 2.9 31.6 ± 4.7 55.1 ± 1.0 39.0 ± 1.8 42.9

	3 ± 0.1 98.0 96.7 ± 0.2 99.1 ± 0.0 99.0 ± 0.0 99.0 ± 0.1 99.1 ± 0.1 96.5 ± 0.4 98.2 95.9 ± 0.2 99.0 ± 0.1 98.9 ± 0.1 98.9 ± 0.1 98.7 ± 0.1 96.2 ± 0.2 97.9 94.8 ± 0.5 98.7 ± 0.1 98.8 ± 0.1 98.8 ± 0.0 98.9 ± 0.1 95.9 ± 0.2 97.6 94.8 ± 0.2 98.8 ± 0.1 98.9 ± 0.0 98.7 ± 0.0 98.7 ± 0.1 95.5 ± 0.4 97.6 SAND-mask 94.5 ± 0.4 98.6 ± 0.1 98.8 ± 0.1 98.7 ± 0.1 98.6 ± 0.0 95.5 ± 0.2 97.4 ARM V-REx RSC AND-mask Fish 98.0 Fishr 95.0 ± 0.3 98.5 ± 0.0 99.2 ± 0.1 98.9 ± 0.0 98.9 ± 0.1 96.5 ± 0.0 97.8 D.4.3. VLCS VLCS. Model selection: 'Test-domain' validation set Algorithm C L S V Avg Ranking ERM 97.6 ± 0.3 67.9 ± 0.7 70.9 ± 0.2 74.0 ± 0.6 77.6 12 IRM 97.3 ± 0.2 66.7 ± 0.1 71.0 ± 2.3 72.8 ± 0.4 76.9 16 GroupDRO 97.7 ± 0.2 65.9 ± 0.2 72.8 ± 0.8 73.4 ± 1.3 77.4 15 Mixup 97.8 ± 0.4 67.2 ± 0.4 71.5 ± 0.2 75.7 ± 0.6 78.1 4 MLDG 97.1 ± 0.5 66.6 ± 0.5 71.5 ± 0.1 75.0 ± 0.9 77.5 14 CORAL 97.3 ± 0.2 67.5 ± 0.6 71.6 ± 0.6 74.5 ± 0.0 77.7 10 MMD 98.8 ± 0.0 66.4 ± 0.4 70.8 ± 0.5 75.6 ± 0.4 77.9 6 DANN 99.0 ± 0.2 66.3 ± 1.2 73.4 ± 1.4 80.1 ± 0.5 79.7 2 CDANN 98.2 ± 0.1 68.8 ± 0.5 74.3 ± 0.6 78.1 ± 0.5 79.9 1 MTL 97.9 ± 0.7 66.1 ± 0.7 72.0 ± 0.4 74.9 ± 1.1 77.7 10 SagNet 97.4 ± 0.3 66.4 ± 0.4 71.6 ± 0.1 75.0 ± 0.8 77.6 12 ARM 97.6 ± 0.6 66.5 ± 0.3 72.7 ± 0.6 74.4 ± 0.7 77.8 7 V-REx 98.4 ± 0.2 66.4 ± 0.7 72.8 ± 0.1 75.0 ± 1.4 78.1 4 RSC 98.0 ± 0.4 67.2 ± 0.3 70.3 ± 1.3 75.6 ± 0.4 77.8 7 AND-mask 98.3 ± 0.3 64.5 ± 0.2 69.3 ± 1.3 73.4 ± 1.3 76.4 17 SAND-mask 97.6 ± 0.3 64.5 ± 0.6 69.7 ± 0.6 73.0 ± 1.2 76.2 18 Fish 77.8 7 Fishr 97.6 ± 0.7 67.3 ± 0.5 72.2 ± 0.9 75.7 ± 0.3 78.2 3 VLCS. Model selection: 'Training-domain' validation set Algorithm C L S V Avg Ranking ERM 97.7 ± 0.4 64.3 ± 0.9 73.4 ± 0.5 74.6 ± 1.3 77.5 10 IRM 98.6 ± 0.1 64.9 ± 0.9 73.4 ± 0.6 77.3 ± 0.9 78.5 3 GroupDRO 97.3 ± 0.3 63.4 ± 0.9 69.5 ± 0.8 76.7 ± 0.7 76.7 18 Mixup 98.3 ± 0.6 64.8 ± 1.0 72.1 ± 0.5 74.3 ± 0.8 77.4 13 MLDG 97.4 ± 0.2 65.2 ± 0.7 71.0 ± 1.4 75.3 ± 1.0 77.2 15 CORAL 98.3 ± 0.1 66.1 ± 1.2 73.4 ± 0.3 77.5 ± 1.2 78.8 1 MMD 97.7 ± 0.1 64.0 ± 1.1 72.8 ± 0.2 75.3 ± 3.3 77.5 10 DANN 99.0 ± 0.3 65.1 ± 1.4 73.1 ± 0.3 77.2 ± 0.6 78.6 2 CDANN 97.1 ± 0.3 65.1 ± 1.2 70.7 ± 0.8 77.1 ± 1.5 77.5 10 MTL 97.8 ± 0.4 64.3 ± 0.3 71.5 ± 0.7 75.3 ± 1.7 77.2 15 SagNet 97.9 ± 0.4 64.5 ± 0.5 71.4 ± 1.3 77.5 ± 0.5 77.8 6 ARM 98.7 ± 0.2 63.6 ± 0.7 71.3 ± 1.2 76.7 ± 0.6 77.6 9 V-REx 98.4 13 Fish 77.8 6 Fishr 98.9 ± 0.3 64.0 ± 0.5 71.5 ± 0.2 76.8 ± 0.7 77.8 D.4.4. PACS PACS. Model selection: 'Test-domain' validation set Algorithm A C P S Avg Ranking ERM 86.5 13 Fish 85.8 14 Fishr 87.9 Algorithm A C P S Avg Ranking ERM 84.7 ± 0.4 80.8 ± 0.6 97.2 ± 0.3 79.3 ± 1.0 85.5 3 IRM 84.8 ± 1.3 76.4 ± 1.1 96.7 ± 0.6 76.1 ± 1.0 83.5 17 GroupDRO 83.5 ± 0.9 79.1 ± 0.6 96.7 ± 0.3 78.3 ± 2.0 84.4 14 Mixup 86.1 ± 0.5 78.9 ± 0.8 97.6 ± 0.1 75.8 ± 1.8 84.6 10 MLDG 85.5 ± 1.4 80.1 ± 1.7 97.4 ± 0.3 76.6 ± 1.1 84.9 8 CORAL 88.3 ± 0.2 80.0 ± 0.5 97.5 ± 0.3 78.8 ± 1.3 86.2 2 MMD 86.1 ± 1.4 79.4 ± 0.9 96.6 ± 0.2 76.5 ± 0.5 84.6 10 DANN 86.4 10 Fish 85.5 3 Fishr 88.4 ± 0.2 78.7 ± 0.7 97.0 ± 0.1 77.8 ± 2.0 85.5 D.4.5. OFFICEHOME OfficeHome. Model selection: 'Test-domain' validation set Algorithm A C P R Avg Ranking ERM 61.7 ± 0.7 53.4 ± 0.3 74.1 ± 0.4 76.2 ± 0.6 66.4 8 IRM 56.4 ± 3.2 51.2 ± 2.3 71.7 ± 2.7 72.7 ± 2.7 63.0 18 GroupDRO 60.5 ± 1.6 53.1 ± 0.3 75.5 ± 0.3 75.9 ± 0.7 66.2 3 Mixup 63.5 ± 0.2 54.6 ± 0.4 76.0 ± 0.3 78.0 ± 0.7 68.0 6 MLDG 60.5 ± 0.7 54.2 ± 0.5 75.0 ± 0.2 76.7 ± 0.5 66.6 6 CORAL 64.8 ± 0.8 54.1 ± 0.9 76.5 ± 0.4 78.2 ± 0.4 68.4 3 MMD 60.4 ± 1.0 53.4 ± 0.5 74.9 ± 0.1 76.1 ± 0.7 66.2 1 DANN 60.6 13 Fish 66.0 12 Fishr 63.4 OfficeHome. Model selection: 'Training-domain' validation set Algorithm A C P R Avg Ranking ERM 61.3 ± 0.7 52.4 ± 0.3 75.8 ± 0.1 76.6 ± 0.3 66.5 7 IRM 58.9 ± 2.3 52.2 ± 1.6 72.1 ± 2.9 74.0 ± 2.5 64.3 18 GroupDRO 60.4 ± 0.7 52.7 ± 1.0 75.0 ± 0.7 76.0 ± 0.7 66.0 11 Mixup 62.4 ± 0.8 54.8 ± 0.6 76.9 ± 0.3 78.3 ± 0.2 68.1 3 MLDG 61.5 ± 0.9 53.2 ± 0.6 75.0 ± 1.2 77.5 ± 0.4 66.8 6 CORAL 65.3 ± 0.4 54.4 ± 0.5 76.5 ± 0.1 78.4 ± 0.5 68.7 1 MMD 60.4 ± 0.2 53.3 ± 0.3 74.3 ± 0.1 77.4 ± 0.6 66.3 10 DANN 59.9 ± 1.3 53.0 ± 0.3 73.6 ± 0.7 76.9 ± 0.5 65.9 12 CDANN 61.5 ± 1.4 50.4 ± 2.4 74.4 ± 0.9 76.6 ± 0.8 65.8 13 MTL 61.5 ± 0.7 52.4 ± 0.6 74.9 ± 0.4 76.8 ± 0.4 66.4 8 SagNet 63.4 ± 0.2 54.8 ± 0.4 75.8 ± 0.4 78.3 ± 0.3 68.1 3 ARM 58.9 ± 0.8 51.0 ± 0.5 74.1 ± 0.1 75.2 ± 0.3 64.8 17 V-REx 60.7 ± 0.9 53.0 ± 0.9 75.3 ± 0.1 76.6 ± 0.5 66.4 8 RSC 60.7 13 Fish 68.6 2 Fishr TerraIncognita. Model selection: 'Test-domain' validation set Algorithm L100 L38 L43 L46 Avg Ranking ERM 59.4 17 Fish 50.8 13 Fishr 60.4 Algorithm L100 L38 L43 L46 Avg Ranking ERM 49.8 ± 4.4 42.1 ± 1.4 56.9 ± 1.8 35.7 ± 3.9 46.1 10 IRM 54.6 ± 1.3 39.8 ± 1.9 56.2 ± 1.8 39.6 ± 0.8 47.6 4 GroupDRO 41.2 ± 0.7 38.6 ± 2.1 56.7 ± 0.9 36.4 ± 2.1 43.2 16 Mixup 59.6 ± 2.0 42.2 ± 1.4 55.9 ± 0.8 33.9 ± 1.4 47.9 2 MLDG 54.2 ± 3.0 44.3 ± 1.1 55.6 ± 0.3 36.9 ± 2.2 47.7 3 CORAL 51.6 ± 2.4 42.2 ± 1.0 57.0 ± 1.0 39.8 ± 2.9 47.6 4 MMD 41.9 ± 3.0 34.8 ± 1.0 57.0 ± 1.9 35.2 ± 1.8 42.2 18 DANN 51.1 ± 3.5 40.6 ± 0.6 57.4 ± 0.5 37.7 ± 1.8 46.7 7 CDANN 47.0 17 Fish 45.1 14 62.4 ± 0.5 54.4 ± 0.4 76.2 ± 0.5 78.3 ± 0.1 67.8 D.4.6. TERRAINCOGNITA Fishr 50.2 ± 3.9 43.9 ± 0.8 55.7 ± 2.2 39.8 ± 1.0 47.4	2 1 8 16 16 18 2

ACKNOWLEDGMENTS

This work was granted access to the HPC resources of IDRIS under the allocation A0100612449 made by GENCI. We acknowledge the financial support by the ANR agency in the chair VISA-DEEP (ANR-20-CHIA-0022-01).

These Appendices complement the main paper.

1. We first detail some theoretical points. Appendix A.1 demonstrates our Proposition 1. Appendix A.2 shows that Fishr acts as a feature-adaptive V-REx. Appendix A.3 motivates Fishr with intuitions from the Neural Tangent Kernel theory.

2. Appendix B proves the effectiveness of our approach for a linear toy dataset.

3. Appendix C enriches the Colored MNIST experiment in the IRM setup. In detail, we first describe the experimental setup in Appendix C.1. We then validate in Appendix C.2 some insights provided in the main paper; in particular, Appendix C.2.3 motivates the diagonal approximation of the gradient covariance. We now further detail our experiments on the DomainBed benchmark. Scores from most baselines are taken from the DomainBed (Gulrajani & Lopez-Paz, 2021) paper. Scores for AND-mask and SAND-mask are taken from the SAND-mask paper [START_REF] Shahtalebi | Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization[END_REF]. Scores for IGA [START_REF] Koyama | Out-of-distribution generalization with maximal invariant predictor[END_REF] are not yet available: yet, for the sake of completeness, we analyze IGA in Appendix D.3.2. Missing scores will be included when available.

The same procedure was applied for all methods: for each domain, a random hyperparameter search of 20 trials over a joint distribution, described in Table 8, is performed. We discuss the choice of these distributions in Appendix D.3.3. The learning rate, the batch size (except for ARM), the weight decay and the dropout distributions are shared across all methodsall trained with Adam (Kingma & Ba, 2014). Specific hyperparameter distributions for concurrent methods can be found in the original work of Gulrajani & Lopez-Paz (2021). The data from each domain is split into 80% (used as training and testing) and 20% (used as validation for hyperparameter selection) splits. This random process is repeated with 3 different seeds: the reported numbers are the means and the standard errors over these 3 seeds. We clarify a subtle point (omitted in the Algorithm 1) concerning the hyperparameter γ that controls: vt e = γ vt-1 e +(1-γ)v t e at step t. We remind that vt-1 e from previous step t -1 is 'detached' from the computational graph. Thus when L from Eq. 4 is differentiated during SGD, the gradients going through v t e are multiplied by (1 -γ). To compensate this and decorrelate the impact of γ and of λ (that controls the regularization strength), we match 1 1-γ vt e . Finally, with this (1 -γ) correction, the gradients' strength backpropagated in the network is independent of γ.

Here we list all concurrent approaches.

• ERM: Empirical Risk Minimization [START_REF] Vapnik | An overview of statistical learning theory[END_REF] • IRM: Invariant Risk Minimization (