Rodrigue Siry

Louis Hémadou

Loïc Simon

Frédéric Jurie

On the inductive biases of deep domain adaptation

Even if it is not always stated explicitly, the majority of recent approaches to domain adaptation are based on theoretically unjustified assumptions, on the one hand, and (often hidden) inductive biases on the other. This paper first point out that feature alignment which is often misrepresented as a minimizer of some theoretical upper-bounds on risk in the target domain is neither necessary nor sufficient to obtain low target risk. This paper also demonstrates, through numerous experiments, that deep domain adaptation methods, in fact, rely heavily on some hidden inductive biases found in common practices, such as model pretraining or encoder architecture design. In a third step, the paper argues that using handcrafted priors might not be sufficient to bridge distant domains: powerful parametric priors can be instead learned from data, leading to a large improvement in target accuracy. A meta-learning strategy allowing to find inductive biases that solve specific transfers is proposed. It shows superior performance to that of handcrafted priors on several tasks. *

Introduction

Deep learning models achieve impressive performance on image classification tasks, when a large number of training samples is provided. However, in many situations, there is not enough labeled data available for the task of interest (the target domain), while there is some in related domains (source domains) but with a statistical bias which makes them impossible to use as is to learn a model. The objective of domain adaptation is to exploit the data in the source dataset to obtain a model that performs well in the target domain. In this article, we focus on unsupervised domain adaptation, a task for which only unlabeled examples of the target domain are available to train the model.

This work focuses on domain adaptation under the covariate shift assumption, which assumes that the same input data is associated with a single label, regardless of the domain to which it belongs (see Section 3 for a formal definition of the covariate shift). Indeed, transfer learning usually assumes that the labeling function is consistent across all domains and only accounts for qualitative shifts in the distribution of input data.

Following [START_REF] Bouvier | Robust domain adaptation: Representations, weights and inductive bias[END_REF], we analyze domain adaptation through the scope of inductive biases. An inductive bias is a set of hidden assumptions that condition the behavior of the model on unseen data (in that case, the target domain). Indeed, the vast majority of domain adaptation domains. [START_REF] Zhang | Bridging theory and algorithm for domain adaptation[END_REF] further refined the bound by relaxing the divergence to a single adversarial hypothesis. [START_REF] Mansour | Domain adaptation: Learning bounds and algorithms[END_REF] provides another classifier-based bound that does not require the covariate shift assumption. [START_REF] Shen | Adversarial representation learning for domain adaptation[END_REF] proposes a bound based on the Wasserstein distance that is robust to disjoint supports. Several bounds were also proposed for the PAC-bayesian setting [START_REF] Li | A bayesian divergence prior for classiffier adaptation[END_REF][START_REF] Germain | Pac-bayesian theorems for domain adaptation with specialization to linear classifiers[END_REF][START_REF] Germain | A new pac-bayesian perspective on domain adaptation[END_REF]. [START_REF] Simon | Equivalence of several curves assessing the similarity between probability distributions[END_REF] present a divergence based on PR curves. More details about the bounds will be provided in the next section, but we also advise the reader to refer to [START_REF] Kouw | An introduction to domain adaptation and transfer learning[END_REF], [START_REF] Redko | A survey on domain adaptation theory: learning bounds and theoretical guarantees[END_REF] for an exhaustive list of existing domain adaptation bounds.

Domain-Invariant feature learning

The large majority of domain adaptation methods aim at producing a domain-invariant feature representation at the output of an encoder model by aligning source and target feature distributions. To perform this feat, early works minimize some closed-form measure of discrepancy between distributions; DeepCORAL [START_REF] Sun | Deep coral: Correlation alignment for deep domain adaptation[END_REF] trains the encoder model to align the distribution means and variance-covariance matrices. [START_REF] Tzeng | Deep domain confusion: Maximizing for domain invariance[END_REF] uses the more powerful Maximum Mean Discrepancy (MMD) metric, wich was then extended by [START_REF] Long | Learning transferable features with deep adaptation networks[END_REF] with multiple kernels. With the advent of GANs [START_REF] Goodfellow | Generative adversarial nets[END_REF], adversarial training was quickly used for domain-alignment, leading to the DANN algorithm [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF]. A wide set of variants of DANN followed: ADDA [START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF] uses a separate feature encoder for source and target, [START_REF] Shen | Adversarial representation learning for domain adaptation[END_REF] uses a Wasserstein critic instead of the standard Jensen-Shannon. PixelDA [START_REF] Bousmalis | Unsupervised pixel-level domain adaptation with generative adversarial networks[END_REF] aligns distributions in image-space by using an image translation model. As domain-alignment was proven to be insufficient for good target performance, new improvements were found by the community to better condition the process of distribution alignment: [START_REF] Shu | A dirt-t approach to unsupervised domain adaptation[END_REF][START_REF] Häusser | Associative domain adaptation[END_REF][START_REF] Xu | d-sne: Domain adaptation using stochastic neighborhood embedding[END_REF] exploit a cluster assumption in feature space to stabilize alignment. [START_REF] Cicek | Unsupervised domain adaptation via regularized conditional alignment[END_REF][START_REF] Chen | Progressive feature alignment for unsupervised domain adaptation[END_REF][START_REF] Lv | Pareto domain adaptation[END_REF][START_REF] Zhang | Bridging theory and algorithm for domain adaptation[END_REF][START_REF] Saito | Maximum classifier discrepancy for unsupervised domain adaptation[END_REF][START_REF] Kang | Contrastive adaptation network for unsupervised domain adaptation[END_REF][START_REF] Des Combes | Domain adaptation with conditional distribution matching and generalized label shift[END_REF]) make use of pseudo-labels to condition alignment to class information. [START_REF] Kumar | Coregularized alignment for unsupervised domain adaptation[END_REF] performs alignment on an ensemble of feature-spaces and forces a single classifier to agree on all of them.

Inspired by the representation learning literature, other methods add auxiliary modeling objectives to the features to improve their robustness, interpretability and transferability: [START_REF] Peng | Domain agnostic learning with disentangled representations[END_REF][START_REF] Bousmalis | Domain separation networks[END_REF] learn a feature space where class and style information are disentangled. [START_REF] Sun | Unsupervised domain adaptation through selfsupervision[END_REF] completes alignment with a self-supervision objective.

Controversies

Alignment methods seek theoretical justification in the learning bounds derived from [START_REF] Ben-David | A theory of learning from different domains[END_REF] by applying them in feature space. However, further analyses from [START_REF] Shu | A dirt-t approach to unsupervised domain adaptation[END_REF][START_REF] Zhao | On learning invariant representations for domain adaptation[END_REF][START_REF] Johansson | Support and invertibility in domain-invariant representations[END_REF][START_REF] Bouvier | Robust domain adaptation: Representations, weights and inductive bias[END_REF][START_REF] Siry | A study of alignment mechanisms in adversarial domain adaptation[END_REF] proved that domainalignment only partially minimizes those bounds and is not sufficient for provable good target accuracy. We will adopt the same critical approach in the next section and give additional examples.

Inductive bias

We refer as an inductive bias any set of assumptions we can use to help a model generalize to unseen data. The importance of inductive bias in domain adaptation has been first mentioned in [START_REF] Bouvier | Robust domain adaptation: Representations, weights and inductive bias[END_REF]. In this paper, we consolidate this work by unveiling various instances of inductive bias that were already used in the literature and crucially underpin good target performance. A fact that has not been highlighted much by the domain adaptation community.

Meta-learning

Meta-learning, or learning to learn, is a recent paradigm in deep learning which can be viewed as a data-centric way to learn a parametric inductive bias that works for some distribution of tasks. First introduced to address few-shot generalization [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF], meta learning has then been used in domain adaptation, especially in the multi-source domain generalization setting: [START_REF] Li | Learning to generalize: Meta-learning for domain generalization[END_REF] meta-learns an encoder representation that generalizes to test domains, this was then improved by Balaji et al. (2018) which meta-learns regularization weights optimized for transferable fine-tuning. [START_REF] Wei | Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation[END_REF] uses meta-learning to synchronize the learning dynamics of domain alignment and supervision on source.

Two common assumptions that are nevertheless flawed

In this section, we attempt to highlight two commonly held assumptions in the domain adaptation literature and show that these assumptions are in some cases unfounded, thus leaving most of the practical algorithms presented in the literature without any proper theoretical guarantees.

We refer as source and target domains two distributions S and T over the space of labeled data X × Y . We define S X and T X the marginals of S and T over X and S Y , T Y the marginals over Y . Finally, we call domain shift the discrepancy existing between S and T .

Provided labeled samples from S and unlabeled samples from T , we would like to exploit labeled knowledge from S to obtain a model that minimizes the target risk.

Obviously, this can only be possible if a relationship between S and T exists.

The following general cases are often described in the literature: Most deep learning domain adaptation contributions indeed invoke the covariate shift assumption, and, among them, several works build upon theoretical upper bounds on the target error. For example, this is the case of DANN [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF] which built upon the upper bounds developed in [START_REF] Ben-David | A theory of learning from different domains[END_REF].

The most commonly used upper bounds usually take the following form (see e.g., [START_REF] Ben-David | A theory of learning from different domains[END_REF]; [START_REF] Mansour | Domain adaptation: Learning bounds and algorithms[END_REF]; [START_REF] Shen | Adversarial representation learning for domain adaptation[END_REF] or [START_REF] Redko | A survey on domain adaptation theory: learning bounds and theoretical guarantees[END_REF] for a more exhaustive list).

T (h) ≤ S (h) + δ(S X , T X ) + λ (1)
where h ∈ H is a hypothesis, S (h) and T (h) represent the error associated with h in the two domains.

The second term of the right hand side of the bound, δ(S X , T X ) represents a divergence between the marginal distribution of X under both domains, and this divergence involves the class of hypotheses H.

The last term of Eq.( 1), λ, underpins the adaptability in the sense that λ is small only if a common hypothesis h * can perform well on both domains. This term is not amenable to be computed without the labels in the target, but it is expected to be small in well-posed problems.

The first term, S (h), can be easily minimized to zero by training h with a standard cross-entropy cost as the source domain is assumed to be fully labeled. However, for a given hypothesis space, δ(S X , T X ) cannot be brought close to zero, because the input distributions are fixed. Furthermore, in the majority of domain changes that are considered in the computer vision literature, the domains actually have supports that are disjoint, i.e. S X ⊥ T X .

For instance, pictures of an object taken during the day (the source domain) will have zero probability under the distribution of images taken at night (the target domain). Another example is the MNIST / SVHN pair of domains. Both are well-known datasets of digit recognition: while MNIST displays white-on-black handwritten digits with a moderate variety of shapes and sizes, the richer SVHN is a dataset of house numbers, hence displaying a wide variety of colors, fonts and sizes. Also in that case, it is straightforward to determine with 100% accuracy from which dataset a digit image was drawn.

In such cases, the notion of conditional distribution, say P S (Y |X), is undetermined outside of the support of S X . As a result, the constraint P S (Y |X) = P T (Y |X) is nonbinding, and the notion of covariate shift no longer makes sense. This statement is the first untruth we would like to point out.

Although this fact is often disregarded in domain adaptation theoretical works, which still advocate for the covariate shift assumption, the usual practice is to introduce a feature extractor Ψ : X → Z hence embedding both distributions in latent space Z where the supports may overlap (e.g. [START_REF] Sun | Deep coral: Correlation alignment for deep domain adaptation[END_REF]; [START_REF] Long | Learning transferable features with deep adaptation networks[END_REF]; [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF]; [START_REF] Häusser | Associative domain adaptation[END_REF]; [START_REF] Zhang | Bridging theory and algorithm for domain adaptation[END_REF]). When the bound is applied on feature-space, δ(S X , T X ) can be trivially minimized: to bring the divergence term close to zero, it is sufficient to output a domain-invariant representation; indeed, a feature representation is domain-invariant if the features from the source domain follow the same distribution than features from the target domain. Most methods hence train the encoder to satisfy a dual objective: i) give sufficiently informative features to get a good classification accuracy on source and ii) minimize a measure of discrepancy between source and target feature distributions. The most prominent domain-alignment technique is DANN, which makes use of adversarial learning: it makes the feature extractor compete with a domain classifier in a minimax game; the domain classifier trains itself to recognize the domain from which feature samples come from by minimizing a classification error on domain labels, then the feature extractor trains itself to maximize this error, hence fooling the classifier and bringing source and target distributions closer to each other. After several steps of alternate optimization, the algorithm eventually reaches equilibrium and source and target distributions are aligned. At this point, the H∆H-divergence of the Ben-David bound from which DANN draws its inspiration is close to zero.

Since the last term λ cannot be evaluated without knowing the target labels, only the first two terms of the RHS are considered in the minimization, and when reasoning in feature space, the prior assumption that λ is small does not hold anymore.

This fact has been highlighted by at least three recent papers, namely [START_REF] Zhao | On learning invariant representations for domain adaptation[END_REF][START_REF] Johansson | Support and invertibility in domain-invariant representations[END_REF][START_REF] Bouvier | Robust domain adaptation: Representations, weights and inductive bias[END_REF] in a form of a no-free-lunch theorem. The gist of this result relies on the observation that while the first two terms of the upper-bound are minimized, the third one can go out of control. In particular, the feature extractor can map different regions of T X of different classes to the same region of Z leading to a larger target Bayes risk. This label mixing will be further strengthened by minimizing the divergence in scenarios where classes are balanced differently in the source and target domains, a situation known as a prior shift which is not ruled out by the covariate shift assumption). Worst, nothing prevents regions of T X of a given class to be embedded in regions corresponding to different classes of S X . We illustrate this case in Figure 2.

Both phenomenons are due to the non-invertibility of Ψ [START_REF] Johansson | Support and invertibility in domain-invariant representations[END_REF], which is necessary for the support of S Z and T Z to overlap.

Despite the previous caveat, domain alignment approaches based on upper-bounds are still often successful in practice. On the one hand, this success can be explained by the way in which published results are selected. For instance, while the transfer from SVHN to MNIST is often reported, it is almost never the case of the inverse (MNIST to SVHN -see Fig 3 for a visual comparison of DANN latent representation in both cases). Indeed, the SVHN to MNIST should be considered the "easy" way: as SVHN is richer than MNIST in terms of degrees of freedom (fonts, scales, colors, background textures), the features learned thanks to supervision on SVHN are already robust Figure 3: t-SNE plot of feature space after DANN adaptation (labels are color-coded, plain dots are source samples, crosses are target samples). Left: SVHN→MNIST, Right: MNIST→ SVHN. SVHN has many more degrees of freedom (digit fonts, scales, colors) than MNIST, it is therefore not surprising that the features learnt from raw SVHN supervision transfer consistently on MNIST, making the joint feature distributions close. Left: one can see that the alignment obtained with DANN is non-degenerate. Right: The inverse transfer MNIST → SVHN is much harder. Features learnt with MNIST supervision do not extrapolate consistently to the richer SVHN. In that case, applying the DANN algorithm leads to a degenerate alignment.

enough to generalize slightly on MNIST, and thanks to the adjustment brought by the domain alignment, we can see in the left plot of Fig 3 that not only the marginal feature distributions are aligned, but so is the joint feature-label distribution; on each source class cluster is a target cluster made of samples from the same class. The MNIST to SVHN case is much harder: supervision on MNIST does not naturally produce features that are transferable to SVHN, and aligning domains leads to a degenerate case in which marginals are aligned but not the joint distribution. In the right plot of Fig 3, on each source class cluster is a target cluster made of target samples from arbitrary classes, indicating that the encoder used its large capacity to exploit arbitrary sources of information found in the target region of image-space to match the source distribution. The few works reporting results on the MNIST to SVHN transfer use ad-hoc tricks such as augmenting MNIST with random colors [START_REF] Häusser | Associative domain adaptation[END_REF], which qualitatively brings it closer to SVHN, or use of InstanceNorm [START_REF] Kumar | Coregularized alignment for unsupervised domain adaptation[END_REF], which helps the encoder model to ignore colors and contrasts, thus introducing a good handcrafted invariance that helps conditional alignment. In fact, these "tricks" are part of the inductive bias which will be the subject of the next section.

On the other hand, the analysis of their success is restricted to reasoning on bounds such as Eq 1, while the no-free-lunch theorem should raise concerns on these justifications. It is therefore clear that other explanations must be put forward to understand what makes a transfer susceptible to favorable alignment. In this article, we elaborate on the role of (hidden) inductive biases, as advocated by [START_REF] Bouvier | Robust domain adaptation: Representations, weights and inductive bias[END_REF].

Inductive (implicit) biases found in domain adaptation practice

Since theoretical bounds alone cannot fully enlighten what makes domain adaptation work, we speculate that its relative success is essentially conditioned by the existence of some inductive bias. For reference, in machine learning, inductive bias is the set of assumptions (implicit or explicit) used to improve a model in addition to the training data. For example, one of such biases arises from use of pre-trained models, a design choice that is ubiquitous in the common practice of domain adaptation [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF], [START_REF] Zhang | Bridging theory and algorithm for domain adaptation[END_REF], [START_REF] Shen | Adversarial representation learning for domain adaptation[END_REF], especially when dealing with difficult, low-shot transfers such as Office-31 [START_REF] Gong | Geodesic flow kernel for unsupervised domain adaptation[END_REF]. In domain-adaptation-related publications, such choices are often pushed into the background in the "experiment" section and presented as subsidiary information [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF][START_REF] Lv | Pareto domain adaptation[END_REF]. Their overall contribution on transfer success with respect to some domain adaptation method is rarely mentioned nor discussed. This lack of clarity introduces many uncertainties when comparing methods or reproducing results. We therefore propose here a set of experiments to complete the understanding of several cases of inductive bias. We organize the different biases in four main categories namely, i) biases involved in the domain alignment per se, ii) biases in the data augmentation process, iii) implicit biases of the feature extractor and iv) biases related to the network architectures. All the experiments involve training a classifier of the form f = g • Ψ, where Ψ is a feature extractor and g is a classifier. The experiments presented in this section are based on different datasets commonly used in domain adaptation, i.e. MNIST (LeCun et al., 1998), MNIST-M [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF], SVHN [START_REF] Netzer | Reading digits in natural images with unsupervised feature learning[END_REF] and PACS [START_REF] Li | Deeper, broader and artier domain generalization[END_REF]. Moreover, we use the default ResNet-18 encoder for Ψ in all experiments except for the BatchNorm ablation one (in that case, a simpler VGG-16 convolutional architecture is used). When applicable, we use the default ImageNet pre-trained weights provided by PyTorch. The classifier g is a simple feedforward network with one hidden layer of width 1024.

Inductive biases in the alignment method

Our first experiment is designed to evaluate the response of several domain alignment strategies with respect to prior shift. Indeed, we proved in the previous section that perfect marginal feature alignment is not sufficient to align domains; we will now further demonstrate that it is not necessary as well. Worse, it can have a negative impact on target accuracy on some cases when compared to the source-only baseline. To introduce such a shift while keeping the experiment compatible with the covariate shift assumption, we consider two versions of MNIST for both domains. In the source domain the classes are represented in a balanced ratio (i.e. 10%), while in the target we purposely make the ratios uneven, with class partitions ranging from 5 to 20%. Therefore, the transfer problem falls strictly speaking within the covariate shift assumption, i.e. P S (Y |X) = P T (Y |X).

In this experiment, we consider three forms of domain adaptation. The first one corresponds to training g • Ψ without any explicit alignment. In this case, referred to as source only (SO), the cost function is computed solely from the cross entropy on source samples. The second and third approaches correspond to DANN Ganin et al. (2016) and associative DA [START_REF] Häusser | Associative domain adaptation[END_REF]. Associative-DA is an alignment technique which only partially aligns source and target distributions to avoid some limitations of DANN. Indeed, if the distribution of classes is not the same in source and target domains, the perfect feature alignment obtained through adversarial techniques is likely to degrade the performance: in such a case, to satisfy a perfect distribution matching criterion, samples from one mode would be moved to another mode to restore the balance and hence take the risk of being misclassified. To avoid this, Associative-DA relies on the assumption that, in the source and target distributions, samples of same class will cluster together, and proceeds to align clusters centroïds without aligning cluster population. We therefore expect Associative-DA to be more robust to prior shifts. Indeed, in the case of prior shift, aligning marginal distributions in the latent representation Z = Ψ(X) is detrimental. This can be seen in hand, the source-only training performs almost perfectly (0.99 target accuracy). On the other hand, the DANN approach tries to strictly impose the alignment of marginals and deteriorates the performance by a large margin (0.67 target accuracy). Last, the associative DA approaches do the alignment in a looser manner, by relying on the aforementioned cluster assumption and reaches a target accuracy of 0.97. In fine, even though the alignment remains slightly harmful to the performance, this experiment exhibits a form of inductive bias that renders alignment more robust to prior shift.

Data Augmentation

A straightforward and well-known way to enhance feature generalization is to explicitly augment input images with a family of class-preserving image transformations. For example, it can be constructed by a composition of random flips, scaling, rotation, translation, blur or changes in color statistics. Such transformations represent a subset of all the possible class-preserving transformations and can hence partly explain the domain shifts found in real life. Therefore, augmentation of the source images is another inductive bias and its effect on transferability must be measured in the domain adaptation setting. On Figure 2, we show that a simple color randomization of source MNIST digits and backgrounds helps solving the hard MNIST→SVHN transfer, usually out of reach even for pre-trained models, with a drastic 45% increase of the target performance. This shows how a simple but ad-hoc transform can help bridging the complex domain shift that exists between MNIST and SVHN. It provides a good, general and robust feature space that considerably fastens convergence to almost all downstream classification tasks, and helps generalizing when training data is scarce. This tendency to generalize easily extends naturally to domain adaptation: in practice, pre-trained features are crucial for successful adaptation of low-shot domains of the Office-31 benchmark and make simpler transfers such as SVHN→ MNIST easier. Even the simplest "source-only" baseline, (fine-tune on source, test on target) already gives convincing results when pre-training is employed. If the same model started from a random initialization scheme, its target accuracy would drop dramatically.

In our experiments, pre-training is performed on the widely known ImageNet dataset, containing more that 1000 categories and a million samples. Solving this large-scale classification problem encourages the discovery of features that are invariant to all class-preserving transforms. It is therefore not surprising to see that the so obtained models can more easily bridge domains.

To characterize the role of pre-training in domain adaptation, we envision the following two hypotheses (the latter being a refined version of the former): i) pretraining the feature extractor leads to an operating point where source and target samples are grouped together in a consistent way in the latent embedding ii) pretraining the feature extractor and the optimization bias of the source cross-entropy leads to such a consistently clustered embedding. To assess the likelihood of these two potential explanations, we conducted two experiments as described below.

Assessing feature transferability with KNN: in order to validate the first hypothesis, we design a simple KNN classifier to predict the label of the feature samples from the target distribution. This classifier works as follow: it finds the top-K (K=50) labeled source nearest feature vectors of the unlabeled target feature embedding, according to a base metric in feature-space (e.g., the L2norm), then returns the majority vote of their K labels. Good accuracy of this classifier would demonstrate that the representations of the classes in the target domain are close to the ones of the same classes in the source domain. To say it differently, good KNN performance implies natural class equivariance and domain invariance of the considered feature space. However, as mentioned earlier, expecting the property to be verified for the initial ImageNet features might be too restrictive since the source-only training baseline also involves end-to-end tuning of the encoder model on source samples, which subsequently modifies the feature space. To account for this alternate explanation, the effects of the source-only fine-tuning have to be reported as well. Therefore, we evaluate the KNN classifier on ImageNet pre-trained ResNet-18 features, before (PT) and after (PT+FT) sourceonly fine-tuning. We also evaluate non-pretrained features obtained after source-only (NoPT+FT) as a baseline. We do so for several transfers and report results in Table 3. On digit classification transfers, we observe better-than-random but low KNN accuracy with not finetuned features, for example 19% for MNIST→ MNIST-M and 25% for SVHN→ MNIST. However, the source and target feature distributions become significantly closer as the encoder is tuned on source, with KNN accuracies rising to 56% and 84% respectively on those two transfers. The exact same trend can be observed on PACS transfers. The benefits of pre-training hence cannot be fully explained by the geometry of pre-trained output features: pre-trained hidden ResNet-18 layers also condition the dynamics of source-only fine tuning that eventually leads to good source and target representations thanks to an implicit optimization bias.

Dead Pixel on CIFAR dataset:

To complement this first experiment on the influence of pretraining and source-only finetuning, we have designed an experiment to show that the presence of confusing factors in the source domain can make pretraining detrimental. Note that this experiment is deliberately extreme: it does not follow the prior assumptions of natural classification problems, to a point of completely defeating the commonly useful bias of ImageNet pretraining. For this experiment, we build a synthetic transfer tailored to make pre-training less efficient than random initialization. In both domains, we construct an image sample by taking a random CIFAR-10 image and set its i th upper-left pixel to a fixed color value (grey), the pixel index i is chosen between 1 and 10. In the source domain, i matches the original CIFAR-10 image class. On the contrary, in the target domain, the dead pixel index and the original image class are completely decorrelated, and the image label is given by the dead pixel and not by the object contained in the image. The task is to evaluate to which extend the classifier is tied to the so-called Dead Pixel.

This experiment exposes one of the main obstacles in domain adaptation: the presence of confounding factors that are present in the source domain but absent from the target domain. When supervised on source, we expect pretraining to introduce a bias associating the content of image with the label. It is therefore highly probable that the pre-trained model will ignore the dead pixel and exploit features from the objects, as the initial parameters already make this information salient and filter out pixel-level details. In the target domain, where the CIFAR-10 object is not useful anymore to predict the image label, the pre-trained model should fail even if source-only fine-tuning is performed. On the contrary, a randomly initialized classifier should quickly identify the dead pixel as the safest, easiest way to predict the image label and should completely ignore the CIFAR-10 object. Consequently, the classifier trained from scratch on source should achieve a good accuracy on target. Results for the dead pixel CIFAR transfer (D-Pix) can be found on Table 4: as expected, pre-training performs worse, but still manages to exploit some Dead-Pixel information, leading to reasonable performance on target. The model trained from scratch performs consistently in both domains and reaches maximum accuracy on target. Also, the explicit alignment with DANN hardly improves over source only in this scenario.

We conducted additional experiments in Table 4 to sum-up the influence of pre-training on transfers that satisfy the properties of realistic domain shifts. We observe a consistent performance increase when pre-training is used, in both source-only and DANN adaptation. transfers, DANN provides an additional increase in performance even in the non-pretrained case (+46% on MNIST → MNIST-M, +6% on SVHN→MNIST without pre-training). Those transfers fall in the "easy" category, in which supervision on source naturally builds features that are transferable to target without pre-training. The only exception is the very hard MNIST → SVHN in which DANN fails both in non-pre-trained and pre-trained regimes. This degenerate case has already been illustrated in Figure 3. On PACS transfers, we observe that using DANN is very beneficial in the pre-trained case, with large increases in target performance. However DANN still improves performance on average in the non-pre-trained regime.

In this section, we have confirmed that pre-trained representations usually allows to obtain features displaying a high degree of transferability. We have shown that pretraining alone is not always sufficient and is mainly useful if combined with source-only fine-tuning. Our synthetic experiment corroborates that pre-training should be considered as an inductive bias that helps with domain shifts similar to those found in real life, and fails in cases where confounding factors make this bias counter-effective.

Network architecture components

The update of features during source-only tuning is a complex, non-linear process that does not depend solely on the initialization of the feature encoder. To have a finer understanding of how transferable task-relevant features emerge, one must also take into account the architecture of the encoder. In this section, we conduct additional experiments to show how transfer learning can be sensitive to these architecture components.

The BatchNorm is mainly used to accelerate training of deep models: it rescales all activations in the effective range of the subsequent non-linearity, avoiding flat regions in the loss landscape on which gradient descent makes little progress. Furthermore, this multiplicative operation might give rise to descriptors that display higher degrees of invariance to certain transformations, for example variability in colorization or contrast. To illustrate this tendency to give rise to such invariant representations, we perform a very simple transfer, the purpose of which is to adapt MNIST to its color-inverted counterpart, Inv-MNIST, and show in Table 5 2014) is a simple yet very effective technique to regularize the capacity of neural networks by setting activations to zero with some probability p during training. This avoids co-adaptation of neurons and performs implicit model averaging. As dropout is ubiquitous in standard classification to increase generalization, we would like to measure its contribution to domain-invariance. To do so, we run all experiments of Table 4 again with and without dropout in the penultimate layer. We noticed that the impact of dropout was not the same depending whether pre-training is applied as well or not. We therefore gather in Table 7

Optimization of Inductive Biases

So far, we have looked for inductive biases controlled by a fixed set of hyperparameters. Although some of them, like pre-training, seem to provide a steady increase in target accuracy, there is no principled way to determine how such design choices should be chosen, adjusted and combined to address a given transfer in the general case. Moreover, there is no guarantee that fixed priors and assumptions produce an optimal solution.

In this section, we will explore several methods that add a degree of flexibility to the model by enabling optimization of parametric inductive biases. This optimization can be either heuristic or designed to minimize the actual objective (target risk). In the latter case, we fall in the metalearning case.

Choosing Augmentation Parameters by Optimizing a Proxy Objective

In the usual domain adaptation experimental setting, the true objective cannot be computed nor optimized directly, as target labels are not available. Therefore, we must choose a proxy objective according to which our hyperparameters controlling the inductive bias can be optimized. The SDA method [START_REF] Ilse | Designing data augmentation for simulating interventions[END_REF] falls within this category: it aims at learning the augmentation parameters leading to the best adaptation between a pair of domains. Given a set of augmentation functions (e.g., random rotation, gaussian blur, color jitter, etc.), SDA greedily chooses the augmentation function that maximizes domain confusion. It does so by training a classifier to distinguish between the two augmented domains and select the augmentation for which the accuracy is minimal (i.e. domain confusion is highest). The process is then repeated with all the remaining augmentations, composed with the previous one and so on until a stopping criterion is met. At test time, we train the model on the source domain augmented with the final chain of chosen augmentations and evaluate the performance on the target domain.

To study the soundness of this proxy objective, we evaluate in Figure 4 the domain confusion induced by 4 augmentation types (random rotation, random color jitter, gaussian noise and random crop) on 5 transfers. For each experimental configuration (augmentation function, transfer), we gradually increase the parameter controlling the augmentation intensity from its minimal value (corresponding to the application of no augmentation) towards its maximum value. For example, in the case of rotations, we span from ±0 • to ±180 • . For each degree of intensity, we report domain classification accuracy as a measure of confusion, and class accuracy on source as a measure of how the augmentation removes the useful class information.

Except for the pacs-photo to pacs-painting transfer, in which we observe a slight decrease in domain classification accuracy as the augmentation becomes too severe, the domains can be perfectly distinguished with a simple pre-trained ResNet-18. However, we observe a consistent drop in classification accuracy in most transfers.

To facilitate the emergence of domain confusion, we tried to weaken the domain classifier by using the older AlexNet architecture and SGD instead of Adam and report the results in Figures 5 and6. We observed that for those regularized classifiers, domain classification remains trivial even on heavily corrupted images. However, the class accuracy is significantly lower than with ResNet-18.

Our conclusions on SDA is twofold: 1) the domain classification accuracy given by a deep model is not a reliable criterion for measuring confusion, as augmented source and target distributions do not overlap. 2) the heuristic chosen by SDA to choose the right set of augmentations do not have any consideration for the corruption of relevant class information, and hence does not lead to a dependable increase in target accuracy.

Meta-Learning

Formalism and definition

Meta-learning, or "learning to learn", is a recent trend in machine learning that aims at learning the inductive bias from data itself, by directly optimizing the purpose it should serve. In this section, we propose to meta-learn a parametric inductive bias that helps better transfer some given domain shift, namely the initial weights of the network.

We define a task T as a N -way classification problem with its corresponding training set T train and test set T test . A meta learning model defines an inner learning procedure that aims at solving any task T sampled from a task distribution p(T ), which means that, on average, for any T ∼ p(T ), it should display a good performance on the test set after training itself on the training set. Indeed, tasks from p(T ) are assumed to share some common information to justify the learning of a common inductive bias. A meta-learning model is defined by two sets of parameters θ and Φ:

• θ, also called "fast weight" contains the parameters that are specific to the learning of a task T i provided on-the-fly, That means that it is authorized to be trained into θ i after observing the task training data T train i . θ i is obtained during an imposed heuristic called "inner-loop training". In practice, we may initialize θ from only Φ and/or a random source of information and then train θ into θ i .

• Φ is the task-agnostic parameter, also called "meta parameter" or "slow weight" : it embeds the common information shared among all T ∼ p(T ). It is not allowed to change when a specific T i is provided, but it is meta-optimized to a fixed value that maximizes expected performance of (θ i , Φ) on every T i . This process is called "meta-optimization" or "outer-loop training".

We can sum-up the key principles of meta-learning by the following two equations :

Φ = argmin Φ E Ti∼p(T ) [L(θ i , Φ, T test i )] θ i = innerloop(Φ, T train i )
Any prototype of meta-learning model can hence be defined by a parametrization of θ, Φ, some functional innerloop and a loss function L. We expect a clever meta-learning design to beat on average any heuristic optimization/regularization on T i once meta-trained. However, to demonstrate its usefulness, a meta-learning method must be trained on tasks T i that are different from test tasks. We therefore define two sets of tasks that are mutually exclusive, called "metatraining" and "meta-test" sets. We hope that some Φ that is optimal for the meta-train set will also optimal on the meta-test set. Similarly to standard machine learning, there is a risk of "metaoverfitting" [START_REF] Rajendran | Meta-learning requires meta-augmentation[END_REF] on meta-training tasks that we must mitigate, for example by increasing the diversity of meta-training tasks, or by choosing a clever parametrization of θ and Φ.

Application of Mela-Learning to Domain Adaptation

Early works on meta-learning focused on few-shot learning problems, which is an extreme case of standard generalization. However, meta-learning is an extremely flexible framework that we can apply to any scenario, including domain adaptation.

We propose to use meta-learning to maximize the target-domain performance of a "Source-Only" training. In other words, for any task T i , we set T train i = S and T test i = T . During its inner loop, the model will hence optimize θ i by exploiting samples from the source domain, and will be meta-optimized so that the tuple (Φ, θ i ) performs well on the same classification task, but on the target domain. We favor the Source-Only setting for its simplicity and flexible experimental conditions. Indeed, it does not require unlabeled target samples during inner-loop training. Note that several other meta-learning methods for domain adaptation do not use the Source-only setting for their inner loop [START_REF] Wei | Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation[END_REF]. We now describe in detail the two parts of our meta-learning method.

Inner loop : We use the same parametrization than MAML [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF] : the goal is to meta-train the initialisation θ 0 of a chosen neural network architecture, so that K gradient descent steps from this initialization computed on T train i lead to a final parameter θ K = θ i that performs well on the target domain.

To catch up with the formalism presented in part 5.2.1, we can consider the equivalence Φ = θ 0 and θ i = θ K .

Outer loop optimization : To optimize θ 0 into θ 0 , we unroll the full graph of derivatives of innerloop, evaluate the meta-objective, compute the gradient of this objective w.r.t. θ 0 and finally perform a gradient descent step on θ 0 . We repeat this process until convergence to θ 0 .

≤ i < n iters do Y Ti ← sample 10 classes(Y M T rain ) T train i ∼ get samples of said classes(S, Y Ti ) T test i ∼ get samples of said classes(T, Y Ti ) θ ← θ 0 for 0 ≤ j < K do θ ← θ -γ ∂L(θ,T train i ) ∂θ end for θ 0 ← θ 0 -η ∂L(θ,T test i ) ∂θ 0
end for Definition of meta-train and meta-test sets : Until now, our training loop is identical to the one proposed by [START_REF] Li | Learning to generalize: Meta-learning for domain generalization[END_REF]. However, we still have to define exactly what will be our meta-train set and, most importantly, to which meta-test set of tasks we expect our learned inductive bias to generalize. There are several possibilities :

• The algorithm of [START_REF] Li | Learning to generalize: Meta-learning for domain generalization[END_REF] solves a problem of multi-source domain generalization with meta-learning. In this setting, we have P -1 source domains and only 1 target domain. In this case, to build a meta-train task, we select 2 source domains among the P -1, and use them as T train and T test . Only one task is evaluated in meta test : it uses the union of the P -1 source domains as T train and the unique target domain as T test . Note that the same classes are used during meta-training and meta-testing. In this case, meta-training is exploited as a means of cleverly merging the information from different source domains to better solve the same task on an unseen target domain.

• In our contribution, we address a totally different use-case. We have only two domains from which we can sample tasks. In these domains, we have P meta-training classes and Q metatesting classes. To build a meta-training task, we choose N classes among the P , and define T train , T test as the sets of samples from domain 1 (resp. from domain 2) belonging to these classes. To build a meta-test task, we proceed similarly, but by picking among the Q meta-test classes instead. Here, the goal is to learn an inductive bias that implements the ability to transfer from one fixed domain to another fixed domain regardless of the classes encountered in the task. This has a strong connection with the discussions of section 4. We define this training and evaluation setting by the acronym MAML-2DOM.

The two nested training loops of MAML-2DOM are summarized in pseudo-code in Algorithm 1.

Experiments

Dataset: To satisfy the experimental setting described above, we use images from the VisDA dataset. VisDA is a dataset that includes 6 domains and subsequent domain shifts of varying difficulty : "Real", "Painting", "Sketch", "Quickdraw", "Clipart" and "Infograph". All domains contain the same 345 classes. Hence, making this dataset ideal to synthesize various transfer tasks.

In our experiments, we define the first 200 classes as meta-training classes and the remaining 145 classes as meta-test classes. All tasks are 10-way classification problems.

Training MAML-2DOM: Given a pair of domains S and T chosen in VisDA, we metatrain an instance of MAML-2DOM to solve this transfer, and this transfer only. To do that, we generate tasks by sampling 10 classes among the 200 meta-train tasks, simulate inner source-only trainings from the source samples, measure the loss on target samples and minimize it w.r.t. the meta-parameter until convergence.

Implementation details: Training a meta-learning model can be costly in terms of memory and compute. Indeed, we must unroll and store the whole graph of derivatives at each realization of the inner loop. To save compute and memory, we propose to reduce considerably model size by avoiding working directly on image space. Indeed, doing so requires the use of a deep convolutional architecture, involving many operations and storing high-dimensional feature maps during the inner loop. We rather define the meta-model on top of a feature space produced by the pretrained ResNet-18, that are assumed to be informative enough to solve our downstream tasks. We can therefore 1) pre-encode the whole VisDA database into features and 2) use a smaller model architecture : in our case a perceptron with two hidden layers.

Baselines: To demonstrate the superiority of our approach based on learned inductive bias, we must compare it to several reference baselines. Unless otherwise specified, these baselines also use the same architecture and work on the same pre-trained feature space.

• Random initialization (Random): A classifier initialized with the standard random initialization, that is then trained on source-domain samples of the 10-way task.

• Pre-training followed by fine-tuning (PT+SO): Random initialization might not be a fair baseline, as it could not exploit labeled information from the 200 meta-train classes from both domains. We hence propose to pre-train the classifier model to solve the 200-way classification problem for both meta-train domains simultaneously, then replace the last layer and fine-tune on the downstream 10-way test task.

• DANN: We perform the 10-way transfer with both labeled source images and unlabeled target images, we add the DANN adversarial constraint in the classifier to enhance transferability. • CAN and ASAN [START_REF] Kang | Contrastive adaptation network for unsupervised domain adaptation[END_REF][START_REF] Raab | Bridging adversarial and statistical domain transfer via spectral adaptation networks[END_REF] are state-of-the-art single-source domain adaptation methods. We use the default implementation provided by the authors : those methods hence work directly on image space. Note that when using their respective codebases, we nonetheless choose the pre-trained ResNet extractor and perform hyperparameter tuning to improve on our VisDA experiments.

We favor the baselines as much as possible by doing hyperparameter search on optimizer, learning rate and number of training iterations during fine-tuning.

Results : We study the transfers from "Real" towards any of the 5 other domains and display our results in table 8. Except for the transfer real→infograph, the corresponding MAML-2DOM beats all baselines by a large margin, especially on real→quickdraw, that is peculiarly difficult and on which the descriptors from ResNet-18 does not seem to transfer well, necessitating a correction from the subsequent classifier through a learned inductive bias. Concerning baselines, DANN generally provides an improvement of several percents compared to Source-Only both in the pretrained and non-pre-trained cases. Those gains, however, are not comparable to those brought by MAML-2DOM. Last, pre-training the classifier on the full 200-way meta-train task does not seem to improve over random initialization.

Conclusion

In this paper, we pursued the analysis from [START_REF] Zhao | On learning invariant representations for domain adaptation[END_REF]; [START_REF] Johansson | Support and invertibility in domain-invariant representations[END_REF] to defend a meta-learning approach for unsupervised domain-adaptation. As a starting point, we consider the following question. How much of the success of deep domain alignment approaches can be unraveled through theoretical upper-bounds from domain adaptation theory. Despite their appealing prospects and their prescriptive significance in terms of modern approaches, a no-free-lunch theorem can be stated which invalidates a universal benefit of the domain alignment strategy. This surprising fact calls for a refined analysis of the key ingredients of successful domain alignment transfers.

We therefore investigate the role of various inductive biases to give a more appropriate account of the situation. We illustrate four kinds of inductive biases ranging from those inherent to the alignment approach itself to more generic ones such as the network architecture or data augmentation. In particular, the role of pretraining on a general purpose database such as imagenet is insidious for several reasons. First, without this step, current alignment methods often fail which suggests that pre-training helps moving the supports of source and target distributions to a close match from which alignment methods can catch up. It appears nonetheless that the role of pretraining is sometimes more tangled as was evidenced by our kNN classifier experiment. In a nutshell, in such situations it is the combined effect of pretraining and source-driven optimization biases that are responsible for successful alignment.

The latter evidence motivated our search for efficient ways to design good inductive biases in a principled way. We have conducted an illustrative experiment in which we meta-learned parametric inductive biases that perform better than usual domain-adaptation heuristics on a given transfer. Given the impact of pre-training indicated by our former analysis, we proposed to meta-learn the initialization of the network. Although this choice revealed very effective in our set of experiments, it is not the only meaningful avenue. For instance, it is also possible to approach meta-learning on a regularization perspective or an optimization one. We hope to see more future work exploring this direction. On a closing note, we would like to point out that if our analysis sheds some light on the inner working of domain alignment, it is only in an empirical way: much remains to be done on the theoretical counterpart.

1.

  Covariate shift / inductive transfer: S X = T X and P S (Y |X) = P T (Y |X), 2. Concept shift / transductive transfer: S X = T X and P S (Y |X) = P T (Y |X), 3. Unsupervised transfer: S X = T X and P S (Y |X) = P T (Y |X).

Figure 1 :

 1 Figure 1: Well-behaved domain-alignment: both divergence between feature marginals and joint error λ are low. Colors denotes domain. Continuous boundaries denotes class 1, dashed boundaries denotes class 2.

Figure 2 :

 2 Figure 2: Degenerate domain-alignment: divergence between feature marginals is low but joint error λ is high. Colors denotes domains. Continuous boundaries denotes class 1, dashed boundaries denotes class 2.

Figure 4 :

 4 Figure 4: Increase in domain-confusion and class-confusion for a set of transfers and augmentations.Despite the increase of the augmentation parameter, a simple ResNet-18 model easily manages to distinguish the domains perfectly. As a sanity check, we also provide the source class as a measurement of image corruption.

Figure 5 :

 5 Figure 5: Same experiment than in figure 4, but with an AlexNet trained by Adam

Figure 6 :

 6 Figure 6: Same experiment than in figure 4, but with an AlexNet trained by SGD

Table 1

 1

	: on the one

Table 1 :

 1 Effect of alignment in a prior-shifted transfer; M = balanced MNIST, MI = MNIST with class imbalance

Table 3 :

 3 The next question is probably the most subtle one. It has been noted by[START_REF] Siry | A study of alignment mechanisms in adversarial domain adaptation[END_REF] that the target accuracy of a source-only trained model is predictive of a high chance of success in the alignment itself. This observation can lead to several interpretations in terms of inductive biases.Indeed, most image classification tasks share some common priors: sample images contain a single (or a few very salient) object to classify. Similarly, domain shifts found in real life consist in Target accuracies of the KNN classifier on static pre-trained features (PT) and features obtained after source-only fine-tuning (PT+FT). We also give non-pretrained source-only features as a reference (NoPT+FT); P=PACS-Photo, A=PACS-Art-Painting, Sk=PACS-Sketch, C=PACS-Clipart, M=MNIST, MM=MNIST-M, S=SVHN class-preserving transformations, which are a narrow subset of all possible image transforms (see previous section). A realistic domain shift can hence be modeled as a random combination of such transforms. Taking those priors into account defines a narrower family of possible domains and transfers, for which it is easier to design domain-invariant encoding functions.ImageNet pre-training is widely used in the Deep Learning community.

		DANN w/o augment DANN w/ augment
	M→ S	0.15	0.6
	P→Sk	0.38	0.42
	P→C	0.22	0.24
	P→A	0.58	0.64
	Table 2: Effects of data augmentation; M=MNIST, S=SVHN, P=PACS-Photo, A=PACS-Art-
	Painting, Sk=PACS-Sketch, C=PACS-Clipart; Model is pre-trained
	4.3 Pretraining and optimizer implicit biases

Table 4 :

 4 Target accuracies for the source-only baseline and the DANN alignment algorithm; Left: No pretraining, Right: pretraining; P=PACS-Photo, A=PACS-Art-Painting, Sk=PACS-Sketch, C=PACS-Clipart, M=MNIST, MM=MNIST-M, S=SVHN, D-Pix=Dead Pixel CIFAR transfer

	On most digits

Table 5 :

 5 that in the non-pretrained case, adding BatchNorm entirely conditions the success of domain adaptation for both source-only and DANN. Source-only accuracy for two different encoder architectures (with and without batch normalization); M=MNIST, InvM=MNIST with inverted colors Global Pooling is an operation that collapses a whole feature map into a single vector by averaging across all spatial dimensions. The mean operator does not retain the original location of features and is therefore translation-invariant. Popular backbone architectures VGG-16, ResNet-18 or DenseNet have a 7x7 AvgPool layer. We conduct another transfer to evaluate the capacity to learn translation-invariance without being explicitly supervised to do so ; the source domain is MNIST, while the target domain is MNIST-T. MNIST-T is generated from MNIST samples, augmented by a random translation/scale/rotation. We report the performance of this transfer in Table6. Results show, again, how a simple but appropriate architectural inductive bias such as Average Pooling can modify the extrapolation behavior of a model on an unseen target domain.

	M→ InvM	SO DANN
	Encoder-BN	0.5	0.95
	Encoder-NoBN 0.05	0.04

Interestingly, when this positive bias is used, DANN brings further improvement over SO, while without global pooling, DANN performs even worth than SO.

Dropout

Srivastava et al. (

Table 7 :

 7 Target accuracy without / with dropout; Left table: Non-pre-trained Right table: Pre-trained; P=PACS-Photo, A=PACS-Art-Painting, Sk=PACS-Sketch, C=PACS-Clipart, M=MNIST, MM=MNIST-M, S=SVHN

	M→ MT	SO DANN
	NoGlobalAvgPool 0.51	0.48
	GlobalAvgPool	0.8	0.95
	Table 6: Source only accuracies for two different encoder architectures; M=MNIST, MT=MNIST
	with random translation/scale/rotation augmentation

a detailed sets of results (with/without dropout and with/without pretraining). Comparison with the non-dropout baseline shows a negligible impact in both non-pretrained and pre-trained cases.

  Algorithm 1 MAML-2DOM Require: γ: inner learning rate, η: outer learning rate, S source domain, T target domain, Y M T rain set of all meta-training classes for 0

Table 8 :

 8 ± 0.036 0.197 ± 0.053 0.191 ± 0.033 0.300 ± 0.082 0.28 ± 0.033 0.26 ± 0.023 0.542 ± 0.010 real→painting 0.708 ± 0.065 0.641 ± 0.065 0.731 ± 0.067 0.728 ± 0.050 0.50 ± 0.051 0.61 ± 0.043 0.767 ± 0.009 real→sketch 0.501 ± 0.053 0.447 ± 0.043 0.536 ± 0.054 0.576 ± 0.082 0.58 ± 0.063 0.68 ± 0.016 0.681 ± 0.015 real→clipart 0.620 ± 0.059 0.543 ± 0.061 0.656 ± 0.082 0.640 ± 0.071 0.62 ± 0.054 0.70 ± 0.033 0.746 ± 0.008 real→infograph 0.359 ± 0.086 0.313 ± 0.032 0.446 ± 0.067 0.415 ± 0.062 0.34 ± 0.027 0.71 ± 0.017 0.502 ± 0.005 Average target accuracies and standard deviations for 10-way domain adaptation test tasks, computed over 10 runs; we outline best performance in bold • Pre-training followed by DANN (PT+DANN): We combine pre-trained initialization and DANN training. Note that none of the transfers we build are prior-shifted as we use artificially balanced source and target batches, and that in all cases, we favor the baselines by choosing on it the best optimizer, learning rate and number of fine-tune iterations.

	Random+SO Pretrain+SO	DANN	Pretrain+DANN	CAN	ASAN	MAML-2DOM
	real→quickdraw 0.185

Acknowledgments

Research reported in this publication was supported by the Agence Nationale pour la Recherche (ANR) under award number ANR-19-CHIA-0017.