Yassine Ouali
email: yassine.ouali@centralesupelec.fr

Céline Hudelot
email: celine.hudelot@centralesupelec.fr

Myriam Tami
email: myriam.tami@centralesupelec.fr

Spatial Contrastive Learning for Few-Shot Classification

In this paper, we explore contrastive learning for few-shot classification, in which we propose to use it as an additional auxiliary training objective acting as a data-dependent regularizer to promote more general and transferable features. In particular, we present a novel attention-based spatial contrastive objective to learn locally discriminative and class-agnostic features. As a result, our approach overcomes some of the limitations of the cross-entropy loss, such as its excessive discrimination towards seen classes, which reduces the transferability of features to unseen classes. With extensive experiments, we show that the proposed method outperforms state-of-the-art approaches, confirming the importance of learning good and transferable embeddings for few-shot learning Code:

Introduction

Few-shot learning [START_REF] Lake | One shot learning of simple visual concepts[END_REF] has emerged as an alternative to supervised learning to simulate more realistic settings that mimic human capabilities, and in particular, it consists of reproducing the learner's ability to rapidly and efficiently adapt to novel tasks. In this paper, we tackle the problem of few-shot image classification, which aims to equip a learner with the ability to learn novel visual concepts and recognize unseen classes with limited supervision.

A popular paradigm to solve this problem is meta-learning [START_REF] Thrun | Lifelong learning algorithms[END_REF][START_REF] Naik | Meta-neural networks that learn by learning[END_REF] consisting of two disjoint stages, meta-training and meta-testing. During metatraining, the goal is to acquire transferable knowledge from a set of tasks sampled from the meta-training tasks so that the learner is equipped with the ability to adapt to novel tasks quickly. This fast adaptability to unseen classes is evaluated at test time by the average test accuracy over several meta-testing tasks. Such transferable knowledge can be acquired from the meta-training tasks with optimizationbased methods [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF][START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF] or metric-based methods [START_REF] Vinyals | Matching networks for one shot learning[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Sung | Learning to compare: Relation network for few-shot learning[END_REF].

Similarity Similarity

Spatial Features

Embedding Model

Alinged Spatial Features

Spatial Contrastive Loss Spatial Alignement Figure 1. Spatial Contrastive Learning (SCL). To learn more locally class-independent discriminative features, we propose to measure the similarity between a given pair of samples using their spatial features as opposed to their global features. We first apply an attention-based alignment, aligning each input with respect to the other. Then, we measure the one-to-one spatial similarities and compute the Spatial Contrastive (SC) loss.

Recently, a growing line of works [START_REF] Chen | A closer look at few-shot classification[END_REF][START_REF] Dhillon | A baseline for few-shot image classification[END_REF]Tian et al., 2020b) show that learning good representations results in fast adaptability at test time, suggesting that feature reuse [START_REF] Raghu | Rapid learning or feature reuse? towards understanding the effectiveness of maml[END_REF] plays a more important role in few-shot classification than the meta-learning aspect of existing algorithms. Such methods consider an extremely simple transfer learning baseline, in which the model is first pre-trained using the standard cross-entropy (CE) loss on the meta-training set. Then, at test time, a linear classifier is trained on the meta-testing set on top of the pre-trained model. The pre-trained model can either be finetuned [START_REF] Dhillon | A baseline for few-shot image classification[END_REF][START_REF] Afrasiyabi | Associative alignment for few-shot image classification[END_REF] together with the classifier, or fixed and used as a feature extractor [START_REF] Chen | A closer look at few-shot classification[END_REF]Tian et al., 2020b). While promising, we argue that using the CE loss during the pre-training stage hinders the quality of the learned representations since the model only acquires the necessary knowledge to solve the classification task over seen classes at train time. As a result, the learned visual features are excessively discriminative against the training classes, rendering them sub-optimal for test time classification tasks constructed from an arbitrary set of unseen and novel classes.

To alleviate these limitations, we propose to leverage contrastive representation learning (Wu et al., 2018b;[START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] as an auxiliary objective, where instead of only mapping the inputs to fixed targets, we also optimize the features, pulling together semantically similar (i.e., positive) samples in the embedding space while pushing apart dissimilar (i.e., negative) samples. By integrating the contrastive loss into the learning objective, we give rise to discriminative representations between dissimilar instances while maintaining an invariance towards visual similarities. Subsequently, the learned representations are more transferable and capture more prevalent patterns outside of the seen classes. Additionally, by combining both losses, we leverage the stability of the CE loss and its effectiveness on small datasets and small batch sizes, while taking benefit of the contrastive loss as a data-dependent regularizer promoting more general-purpose embeddings. Additionally, by combining both losses, we leverage the stability of the CE loss and its effectiveness on small datasets and small batch sizes, in addition to taking benefit of the contrastive loss as a data-dependent regularizer promoting more general-purpose embeddings.

Specifically, we propose a novel attention-based spatial contrastive loss (see Fig. 1) as the auxiliary objective to further promote class-agnostic visual features and avoid suppressing local discriminative patterns. It consists of measuring the local similarity between the spatial features of a given pair of samples after an attention-based spatial alignment mechanism, instead of the global features (i.e., avg. pooled spatial features) used in the standard contrastive loss. We also adopt the supervised formulation [START_REF] Khosla | Supervised contrastive learning[END_REF] of the contrastive loss to leverage the provided label information when constructing the positive and negative samples. However, directly optimizing the features and promoting the formation of clusters of similar instances in the embedding space might result in extremely disentangled representations. Such an outcome can be undesirable for few-shot learning, where the testing tasks can be notably different from the tasks encountered during training, e.g., training on generic categories, and testing on fine-grained sub-categories. To solve this, we propose contrastive distillation to reduce the compactness of the features in the embedding space and provide additional refinement of the representations.

Contributions. To summarize, our contributions are:

• We explore contrastive learning as an auxiliary pretraining objective to learn more transferable features.

• We propose a novel Spatial Contrastive (SC) loss with an attention-based alignment mechanism to spatially compare a pair of features, further promoting classindependent discriminative patterns.

• We employ contrastive distillation to avoid excessive disentanglement of the learned embeddings and improve the performances.

• We demonstrate the effectiveness of the proposed method with extensive experiments on standard and cross-domain few-shot classification benchmarks, achieving state-of-the-art performances.

• We show the universality of the proposed method by applying it to a standard metric learning approach, resulting in a notable performance boost.

Preliminaries

Following a similar notation as (Tian et al., 2020b;[START_REF] Lee | Meta-learning with differentiable convex optimization[END_REF], we start by introducing the meta-learning formulation and the standard transfer learning baseline of (Tian et al., 2020b) in §2.1 and §2.2. Then, in §2.3, we analyze the quality of the learned features in such a setting, motivating the need for an alternative pre-training objective in order to learn more transferable embeddings. To this end, the objective of meta-training algorithms is to learn a good embedding model f φ so that the average test error of the classifier f θ is minimized. This usually involves two stages: first, a meta-training stage inferring the parameters φ of the embedding model using the metatraining set T , followed by a meta-testing stage evaluating the embedding model's performance on meta-testing set S.

Problem Definition

Transfer Learning Baseline

In this work, we consider the simple transfer learning baseline of (Tian et al., 2020b), in which the embedding model f φ is first pre-trained on the merged tasks from the metatraining set using the CE loss. Then, the model is carried over to the meta-testing stage and fixed during evaluation. (1)

Then, during the meta-training stage, the embedding model f φ can be pre-trained on the resulting set of seen classes using the standard CE loss L CE :

φ = arg min φ L CE (D new ; φ). (2)
The pre-trained model f φ is then fixed (i.e., no fine-tuning is performed) and leveraged as a feature extractor during the meta-testing stage. For a given task (D train q , D test q ) sampled from S, a linear classifier f θ is first trained on top of the extracted features to recognize the unseen classes using the training dataset D train

q : θ = arg min θ L CE (D train q ; θ, φ) + R(θ), (3)
where R is a regularization term, and the parameters θ = {W, b} consist of weight and bias terms, respectively. The predictor f θ can then be used on the features of the test dataset D test q to obtain the class predictions and evaluate f φ .

Analysis of the Learned Representations

Although the baseline of §2.2 delivers impressive results, we hypothesis that the usage of the CE loss during the metatraining stage can hinder the performances. Our intuition is that the learned representations lack general discriminative visual features since the CE loss induces embeddings tailored for solving the classification task over the seen classes.

As a results, their transferability to novel domains with unseen classes is reduced, and especially if the domain gap between the training and testing stages is significant.

To empirically validate such a hypothesis, we conduct a k-nearest neighbor search [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF] on the learned embedding space. First, we train a model with the CE loss on the meta-training set of mini-ImageNet [START_REF] Vinyals | Matching networks for one shot learning[END_REF] as in Eq. (2). Then, for a given test image, we search for its neighbors from the meta-testing set. The results are shown in Fig. 2. For a fast test-time adaptation of the predictor f θ , the desired outcome is to have visually and semantically similar images adjacent in the embedding space. However, we observe that the neighboring images are semantically dissimilar. Using Grad-CAM [START_REF] Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF], we notice that dominant discriminative features acquired during training might not be useful for discriminating between unseen classes at test time. In the case of mini-ImageNet, this observation is reinforced by the fact that the meta-training and meta-testing sets are closely related, in which better transferability of the learned features in expected when compared to other benchmarks. We note that similar behavior was also observed by [START_REF] Doersch | Crosstransformers: spatially-aware few-shot transfer[END_REF] for metric-learning based approaches.

To further investigate this behavior, we conduct a spectral analysis of the learned features. As shown in Fig. 3 maximal singular values are significantly larger than the remaining ones, diminishing the amount of informative signal that can be captured.

Methodology

Based on the observations presented in §2.3, in this section, we explore an alternative pre-training objective in order to learn a more transferable embedding model f φ . First, in §3.1, we present the standard supervised contrastive loss. Then, in §3.2, we introduce a novel spatial contrastive learning objective followed by the pre-training objective in §3.3. Finally, an optional contrastive distillation step in §3.4.

Contrastive Learning

We explore contrastive learning as an auxiliary pre-training objective to learn general-purpose visual embeddings capturing discriminative features usable outside of the metatraining set. It thus facilitate the test time recognition of unseen classes. Specifically, given that in a few-shot classification setting we are provided with the class labels, we examine the usage of the supervised formulation [START_REF] Khosla | Supervised contrastive learning[END_REF] of the contrastive loss which leverages the label information to construct the positive and negative samples.

Formally, let f φ be an embedding model mapping the inputs x to spatial features z s ∈ R HW ×d , followed by an average pooling operation to obtain the global features z g ∈ R d , which are then mapped into a lower dimensional space using a projection head p, i.e., f = p(z g ) with f ∈ R d , and let a global similarity function sim g be denoted as the cosine similarity between a pair of projected global features f i and f j (i.e., dot product between the 2 normalized features). First, we sample a batch of N pairs of images and labels from the merged meta-training set D new and augment each example in the batch, resulting in 2N data points. Then, the supervised contrastive loss [START_REF] Khosla | Supervised contrastive learning[END_REF], referred to as the Global Contrastive (GC) loss, can be computed as follows:

L GC = 2N i=1 1 2N yi -1 2N j=1 1 i =j • 1 yi=yj • ij , (4) where ij = -log exp(sim g (f i , f j )/τ ) 2N k=1 1 i =k • exp(sim g (f i , f k )/τ )
, with 1 cond ∈ {0, 1} as an indicator function evaluating to 1 iff cond is satisfied, N yi as the total number of images with the same label y i , and τ as a scalar temperature parameter. By using the GC loss of Eq. ( 4) as an additional pre-training objective with the CE loss, we push the embedding model f φ to learn the visual similarities between instances of the same class, instead of only maintaining the useful features for the classification task over the seen classes, which results in more useful and transferable embeddings.

Spatial Contrastive Learning

Although the GC loss is capable of producing good embeddings, using the global features z g might suppress some local discriminative features present in the spatial features z s that can be informative at the meta-testing stage (e.g., suppressing object specific features while overemphasizing the irrelevant background features). Additionally, encoding the relevant spatial information into the learned representations can play a critical role in increasing the robustness of the embeddings and reducing their sensitivity to domain changes, which is a highly desirable property for few-shot tasks. To this end, we propose a novel SC loss as an alternative objective, leveraging the spatial features z s to compute the similarity between a given pair of examples. However, to locally compare a pair of spatial features z s i and z s j and compute the SC loss, we first need to define a mechanism to align them spatially. To this end, we employ the attention mechanism [START_REF] Vaswani | Attention is all you need[END_REF] to compute the spatial attention weights to align the features z s i with respect to z s j and vice-versa. Then, we measure the one-to-one spatial similarity as illustrated in Fig. 4, and finally, compute the SC loss. a given pair of examples. However, to locally compare a pair of spatial features z s i and z s j and compute the SC loss, we first need to define a mechanism to align them spatially. To this end, we employ the attention mechanism [START_REF] Vaswani | Attention is all you need[END_REF] to compute the spatial attention weights to align the features z s i with respect to z s j and vice-versa. Then, we measure the one-to-one spatial similarity as illustrated in Fig. 4, and finally, compute the SC loss.

Attention-based Spatial Alignment. Let h v , h q and h k denote the value, query and key projection heads, taking as input the spatial features z s and outputting the value v, query q and key k of d -dimensional features, i.e., v, q, k ∈ R HW ×d . Given a pair of spatial features z s i and z s j of two To compute the spatial similarity between a pair of features (purple and blue), we first spatially align the first features (purple) with respect to the second (blue) features with the attention mechanism (see Eq. ( 5)). Then we can compare the aligned value of the first features with the value of the second features. Note that the same process is applied in reverse to compute the final spatial similarity (see Eq. ( 6)).

instances i and j, we want to compute the aligned values of i with respect to j, denoted as v i|j . Such an alignment can be obtained using the key k i and the query q j to compute the attention weights a ij ∈ R HW ×HW , which can then be applied to v i to obtain v i|j . Concretely, this can be computed as follows:

v i|j = a ij v i where a ij = softmax q j k i √ d . (5)
Similarly, we compute v j|i aligning the value of j with respect to i using the key k j and the query q i . Time Complexity. The spatial alignment mechanism has a time complexity of O(N 2 H 2 W 2 d 2 ), which varies with the batch size, the size of the spatial features and the dimensionality of the values v. To avoid excessive cost, for large input images, we apply an adaptive average pooling to reduce the size of the spatial features, in addition to using a small dimensionality d and relatively small batches.

Spatial Similarity. Given a pair of values v i and v j , together with their two aligned versions v i|j and v j|i computed using the attention mechanism detailed above, and with v r * denoting a feature vector at a spatial location r ∈ [1, HW ], we first perform an 2 normalization step of the values v r * at each spatial location r. Then, we compute the total spatial similarity sim s (z s i , z s j ) between a pair of spatial features as follows:

sim s (z s i , z s j ) = 1 HW HW r=1 (v r i ) v r j|i + (v r j ) v r i|j . (6)
Spatial Contrastive Learning. With the spatial similarity function sim s defined in Eq. ( 6), and similar to the GC loss in Eq. ( 4), the SC loss can be computed as follows: The plot shows the evolution of the intra-class variation using the Davies-Bouldin index [START_REF] Davies | A cluster separation measure[END_REF] during the course of training on mini-ImageNet when using the contrastive loss. We see that the learned embeddings of each class are significantly over-clustered, an outcome that might not be desired in some cases.

L SC = 2N i=1 1 2N yi -1 2N j=1 1 i =j • 1 yi=yj • ij , (7) where ij = -log exp(sim s (z s i , z s j )/τ ) 2N k=1 1 i =k • exp(sim s (z s i , z s k )/τ ) , 0 50
with τ as a scalar temperature parameter.

Pre-training Objective

Based on the contrastive objectives in Eq. ( 4) and Eq. ( 7), the pre-training objective can take different forms. We mainly consider the case where the pre-training objective L T is the summation of the CE and SC losses, with λ CE and λ SC as scaling weights to control the contribution of each term:

L T = λ CE L CE + λ SC L SC . (8)
However, we also explore other alternatives such as replacing L SC with L GC or training with both L GC and L SC as auxiliary losses with their corresponding weighting terms. Additionally, we also consider the self-supervised formulations of the GC and SC losses, where the label information is discarded and the only positives considered are the augmented versions of each example (i.e., y i = i mod N ). We refer to them as SS-GC and SS-SC (Self-Supervised Global and Spatial Contrastive) losses respectively.

Using the total loss L T , the embedding model f φ can be trained together with the projection head and the attention modules during the meta-training stage. Specifically, let ψ represent the parameters of the projection head p and the attention modules h v , h q and h k . The parameters are obtained as follows:

{φ, ψ} = arg min {φ,ψ} L T (D new ; {φ, ψ}). ( 9
)
After the pre-training stage, the parameters ψ are discarded, and the embedding model f φ is then fixed and carried over from meta-training to meta-testing.

Avoiding Excessive Disentanglement

Since the contrastive objectives encourage closely aligned embeddings of instances of the same class while distributing all of the normalized features uniformly on the hypersphere (Wang & Isola, 2020), we have to consider a possible overclustering of the features of the same class (see Fig. 5). Such an outcome can be desired for closed-set recognition, but in a few-shot setting, in which the discrepancy between the meta-training and meta-testing domain might differ greatly from one case to the other (e.g., training on coarse seen categories, and testing on fine-grained unseen sub-categories), this might lead to sub-optimal performances. As such, to avoid an excessive disentanglement of the learned features and to further improve the generalization of the embedding model, we propose Contrastive Distillation (CD) to reduce the compactness of the features in embeddings space.

Contrastive Distillation. Given a teacher model f φt pretrained with the objective in Eq. ( 8), we transfer its knowledge to a student model f φs using the standard knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] objective L KL (i.e., the Kullback-Leibler (KL) divergence between the student's predictions and the soft targets predicted by the teacher), but with an additional contrastive distillation loss L CD . This loss consists of maximizing the inner dot product between the 2 normalized global features of the teacher z gt and that of the student z gs , which corresponds to minimizing the squared Euclidean distance, formally:

L CD = 1 N N i=1 z gt i -z gs i 2 2 . ( 10
)
To summarize, the student's parameters are learned as follows:

φ s = arg min φs λ CD L CD (D new ; φ s , φ t ) + λ KL L KL (D new ; φ s , φ t ). (11)
As result, and different from the standard contrastive distillation loss (Tian et al., 2019b) that leverages negative samples, only maximizing the similarity between the pairs of features without using any negatives relaxes the uniformity constraint of the contrastive loss, which in turns reduces the disentanglement of the learned embeddings.

Experiments

For the experimental section, we base our implementation on the publicly available code of (Tian et al., 2020b) and conduct experiments on four popular few-shot classification benchmarks: mini-ImageNet [START_REF] Vinyals | Matching networks for one shot learning[END_REF], tiered-ImageNet [START_REF] Ren | Meta-learning for semi-supervised few-shot classification[END_REF], CIFAR-CS [START_REF] Bertinetto | Metalearning with differentiable closed-form solvers[END_REF] and FC100 [START_REF] Oreshkin | Task dependent adaptive metric for improved few-shot learning[END_REF]. Additionally, we present experiments on cross-domain few-shot benchmarks introduced by [START_REF] Tseng | Crossdomain few-shot classification via learned feature-wise transformation[END_REF]. We note that additional experimental details and results are presented in the supplementary material.

Experimental Details

Architecture. For the embedding model f φ , we follow (Tian et al., 2020b) and use a ResNet-12 consisting of 4 residual blocks with Dropblock as a regularizer and 640dimensional output features (i.e., d = 640). For the projection head and the attention modules, we use an MLP with one hidden layer and a ReLU non-linearity similar to SimCLR, outputting 80-dimensional features (i.e., d = 80).

Training Setup. For optimization, we use SGD with a momentum of 0.9, a weight decay of 5 × 10 -4 , a learning rate of 5 × 10 -2 and a batch size of 64. For the loss functions, we set the temperature parameters τ and τ to 0.1 and the scaling weights λ CE , λ SC , and λ GC to 1.0, except for CIFAR-FS where we set them to 0.5. For distillation, we set λ CD to 10.0 and λ KL to 1.0 and use a temperature of 4.0 for the KL loss. Evaluation Setup. During meta-testing, and given a pretrained embedding model f φ , we follow (Tian et al., 2020b) and consider a linear classifier as the predictor f θ , implemented in scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF] and trained on the 2 normalized features produced by f φ . Specifically, we sample a number of C-way K-shot testing classification tasks constructed from the unseen classes of the meta-testing set, with C as the number of classes and K as the number of training examples per class. After training f θ on the train set, the predictor is then applied to the features of the test set to obtain the prediction and compute the accuracy. In our case, we evaluate the model over 600 randomly sampled tasks and report the median accuracy over 3 runs with 95% confidence intervals, where in each run, the accuracy is the mean accuracy of the 600 sampled tasks.

Ablation Studies

We start by conducting detailed ablation studies to analyze the contribution of each component of the proposed method, from the choices of the loss function to the hyperparameters of the SC loss.

Loss Functions. To investigate the effect of the contrastive losses when used as auxiliary training objectives, we evalu-Loss Function Aug. mini-ImageNet, 5-way CIFAR-CS, 5-way 1-shot 5-shot 1-shot 5-shot CE 61.8 ± 0.7 79.7 ± 0.6 71.3 ± 0.9 86.1 ± 0.6 CE 61.8 ± 0.8 78.6 ± 0.5 71.9 ± 0.9 86.3 ± 0.5 CE + SS-GC 62.7 ± 0.7 81.0 ± 0.6 70.9 ± 0.9 84.5 ± 0.6 CE + SS-SC 64.0 ± 0.8 81.5 ± 0.5 72.1 ± 0.8 86.2 ± 0.6 CE + SS-GC + SS-SC 62.8 ± 0.8 81.1 ± 0.6 69.0 ± 0.9 85.0 ± 0.6 CE + GC 65.0 ± 0.8 81.6 ± 0.5 74.0 ± 0.8 87.3 ± 0.6 CE + SC 65.7 ± 0.8 82.5 ± 0.5 75.0 ± 0.9 87.4 ± 0.6 CE + GC + SC 65.0 ± 0.8 81.3 ± 0.5 76.0 ± 0.7 87.5 ± 0.5 Features Used mini-ImageNet, 5-way CIFAR-CS, 5-way 1-shot 5-shot 1-shot 5-shot Spatial 64.5 ± 0.8 82.1 ± 0.5 75.0 ± 0.9 87.1 ± 0.6 Global 65.7 ± 0.8 82.5 ± 0.5 75.0 ± 0.9 87.4 ± 0.6 Glo. & Spa. (Max) 65.6 ± 0.8 82.1 ± 0.5 74.2 ± 0.8 87.3 ± 0.5 Glo. & Spa. (Sum) 65.7 ± 0.8 83.1 ± 0.5 75.6 ± 0.9 87.6 ± 0.6 ate the performances obtained with various loss functions as detailed in §3.3. The results are shown in Table 1. We observe a notable gain in performance when adopting auxiliary contrastive losses, be it supervised or self-supervised, Loss Function mini-ImageNet, 5-way CIFAR-CS, 5-way 1-shot 5-shot 1-shot 5-shot Teacher 65.7 ± 0.8 82.5 ± 0.5 75.0 ± 0.9 87.4 ± 0.6 KL 66.0 ± 0.8 82.5 ± 0.5 75.9 ± 0.9 87.4 ± 0.6 KL+CD 67.4 ± 0.8 82.7 ± 0.5 76.5 ± 0.9 87.6 ± 0.6 with better gains when using the supervised formulation, highlighting the benefits of using the label information when constructing the positives and negatives samples. More importantly, the SC loss outperforms the standard GC loss, confirming the effectiveness of using the spatial features rather than the global features. Additionally, using both the SC and GC losses does not result in distinct gains over the SC loss. Thus, for the rest of this section, we adopt the SC as a sole auxiliary loss.

Spatial Contrastive Loss. In this section, we examine different variations and hyperparameters of the SC loss when used as an auxiliary objective along the CE loss. In particular, we consider the following variations:

-Hyperparameters. To inspect the SC loss's hyperparameter stability, we conduct experiments with different batch sizes and temperature values. As seen in Fig. 6, by combining the CE and the SC losses, we leverage the stability of the CE and obtain consistent results across several batch sizes, circumventing the need for very large batches when training with only the contrastive losses as it is the case in the unsupervised representation learning setting. As for the temperatures, disregarding the low temperatures in which the SC loss is dominated by the small distances, rendering the actual distances between widely separated representations almost irrelevant, we see comparable performances for temperatures above 0.05, further confirming the stability of the approach.

-Augmentations. Although we mainly use SimCLR type augmentations to produce the additional N augmented examples within a given batch, other augmentations can also be used. Specifically, we consider the standard augmentations used when training with only the CE loss, AutoAugment [START_REF] Cubuk | Learning augmentation strategies from data[END_REF] and Stacked RandAugment (Tian et al., 2020a). [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF] 16.70 ± 0.23 MAML [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF] 16.49 ± 0.58 Meta-SGD [START_REF] Li | Meta-sgd: Learning to learn quickly for few-shot learning[END_REF] 17.56 ± 0.64 LGM-Net [START_REF] Li | LGM-net: Learning to generate matching networks for fewshot learning[END_REF] 26.14 ± 0.34 Ours 36.81 ± 0.38 Ours-Distill 37.47 ± 0.32

Table 8. Comparison with prior few-shot classification works on 20-way 1-shot miniImageNet classification. We show the mean acc. and 95% confidence interval.

for the standard augmentation, without any novel transformations that the model is forced to be invariant under, the gains are minimal. As for strong augmentations (i.e., Au-toAugment and Stacked RandAugment), the augmented inputs might be substantially deformed, making the spatial alignment insufficient and reducing the effect of the SC loss.

-Aggregation Function. Table 3 presents the results obtained with various aggregation functions used to aggregate the one-to-one spatial similarities into an overall measure.

We observe that when using the mean as the aggregate, we obtain overall better performances across the different datasets and settings.

Distillation.

To improve the generalization of the embedding model, we investigate the effect of knowledge distillation by training a new (i.e., student) model using a pre-trained (i.e., teacher) network with various training objectives. Table 5 shows a clear performance gain with the proposed CD objective as an additional loss term, confirming the benefits of optimizing the learned features and relaxing the compactness of the embedding space.

Additionally, we explore sequential self-distillation similar to Born-again networks [START_REF] Furlanello | Born again neural networks[END_REF], where we consider the student model as the teacher and repeat the distillation process. As detailed in Fig. 7, we notice a clear drop in performances beyond a single distillation step. We suspect this might be a result of an over disentanglement of the features induced by the CD loss. As such, for the rest of Method CUB, 5-way Cars, 5-way Places, 5-way Plantae, 5-way 1-shot 5-shot 1-shot 5-shot 1-shot 5-shot 1-shot 5-shot MatchingNet [START_REF] Vinyals | Matching networks for one shot learning[END_REF] 35.89 ± 0.5 51.37 ± 0.7 30.77 ± 0.5 38.99 ± 0.6 49.86 ± 0.8 63.16 ± 0.8 32.70 ± 0.6 46.53 ± 0.6 MatchingNet w/ FT [START_REF] Tseng | Crossdomain few-shot classification via learned feature-wise transformation[END_REF] 36.61 ± 0.6 55.23 ± 0.8 29.82 ± 0.4 41.24 ± 0.6 51.07 ± 0.7 64.55 ± 0.7 34.48 ± 0.5 41.69 ± 0.6 RelationNet [START_REF] Sung | Learning to compare: Relation network for few-shot learning[END_REF] 42.44 ± 0.7 57.77 ± 0.7 29.11 ± 0.6 37.33 ± 0.7 48.64 ± 0.8 63.32 ± 0.8 33.17 ± 0.6 44.00 ± 0.6 RelationNet w/ FT [START_REF] Tseng | Crossdomain few-shot classification via learned feature-wise transformation[END_REF] 44.07 ± 0.7 59.46 ± 0.7 28.63 ± 0.6 39.91 ± 0.7 50.68 ± 0.9 66.28 ± 0.7 33.14 ± 0.6 45.08 ± 0.6 GNN [START_REF] Garcia | Few-shot learning with graph neural networks[END_REF] 45.69 ± 0.7 62.25 ± 0.6 31.79 ± 0.5 44.28 ± 0.6 53.10 ± 0.8 70.84 ± 0.6 35.60 ± 0.5 52.53 ± 0.6 GNN w/ FT [START_REF] Tseng | Crossdomain few-shot classification via learned feature-wise transformation[END_REF] 47 Table 9. Comparison with prior works on cross-domain few-shot classification benchmarks. We train the model on the mini-ImageNet domain and evaluate the trained model on other domains. We show the mean acc. and 95% confidence interval.

the paper, we only apply a single distillation step to refine the features further while preserving the learned structures.

Evaluation. Up until now, we primarily trained a linear classifier on top of the global features during the meta-testing stage. Nonetheless, given that we explicitly optimize the spatial features during training, which increases their discriminability, we investigate their usage as inputs to the linear classifier. To this end, we compare the performance when training over the global features, the spatial features, or both, where we train two classifiers and aggregate their predictions. Table 4 shows the evaluation results. Overall, using the global features to train the linear classifier offers slightly better results than the spatial features. We suspect this might result from slight overfitting of the classifier given that the spatial features increase the number of parameters to be learned, which negatively impacts the performances. However, when leveraging both the spatial and global features, we obtain better results confirming the usefulness of the spatial feature even during the meta-testing stage.

Few-shot Classification

In this section, and based on the results of the ablation studies, we fix the training objective as SC+CE during the meta-training stage and use both the spatial and global feature during the meta-testing stage with a sum aggregate, and compare our approach with other popular few-shot classification methods.

ImageNet derivatives. The mini-ImageNet benchmark is a standard dataset used for few-shot image classification. It consists of 100 randomly selected classes from ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. Following [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF], the classes are split into 64, 16, and 20 classes for meta-training, meta-validation, and meta-testing respectively. Each class contains 600 images of size 84×84. The tiered-ImageNet [START_REF] Ren | Meta-learning for semi-supervised few-shot classification[END_REF] benchmark presents a larger subset of ImageNet, with 608 classes and images of size 84×84 assembled into 34 super-categories. These are split into 20 categories for meta-training and 6 categories for both meta-validation and meta-testing aiming to minimize the semantic similarity between the split.

CIFAR derivatives. CIFAR-CS [START_REF] Bertinetto | Metalearning with differentiable closed-form solvers[END_REF] and FC100 [START_REF] Oreshkin | Task dependent adaptive metric for improved few-shot learning[END_REF] are both CIFAR-100 [START_REF] Krizhevsky | Cifar-10[END_REF] derivatives, containing 100 classes and images of size 32×32. For CIFAR-CS, the classes are divided into 64, 16 and 20 classes for meta-training, metavalidation, and meta-testing respectively. As for FC100, the classes are grouped into 20 super-categories, which are split into 12 categories for meta-training and 4 categories for both meta-validation and meta-testing.

Results. The results of 5-way classification are summarized in Table 6 and Table 7 for ImageNet and CIFAR derivatives respectively, in addition to 20-way classification results on mini-ImageNet in Table 8. Our method outperforms previous works and achieves state-of-the-art performances across different datasets and evaluation settings. This suggests that our attention-based SCL approach coupled with the CE loss improves the transferability of the learned embeddings without any meta-learning techniques, with additional improvements using a contrastive distillation step. These results also show the potential of integrating contrastive losses as auxiliary objectives for various few-shot learning scenarios.

Cross-Domain Few-shot Classification

To further affirm the improved transferability of the learned embedding with our approach, we explore the effects of an increased domain difference between the seen and unseen classes, i.e., the discrepancy between the meta-training and meta-testing stages. Precisely, we follow the same procedure as [START_REF] Tseng | Crossdomain few-shot classification via learned feature-wise transformation[END_REF] where we first train on the whole mini-ImageNet dataset using the same setting as detailed above. Then, we evaluate the embeddings model on four different domains: CUB [START_REF] Welinder | [END_REF], Cars [START_REF] Krause | 3d object representations for fine-grained categorization[END_REF], Places [START_REF] Zhou | Places: A 10 million image database for scene recognition[END_REF]Plantae (Van Horn et al., 2018). We show the obtained results in Table 9, and see a notable gain in performance using the proposed method, from 2% gain on CUB dataset, up to 7% gain on Cars dataset, indicating a clear enhancement in terms of the generalization of the embedding model. Table 10. Comparison with prior works on mini-ImageNet. We report the mean acc. for 5-shot 5-way classification with implementation details including image size, backbone model and auxiliary training losses for each method.

ProtoNet Experiments

To demonstrate the generality of the proposed approach and its applicability in different settings, in this section, we provide additional metric-learning based experiments in which we integrate the contrastive losses into the ProtoNet [START_REF] Snell | Prototypical networks for few-shot learning[END_REF] framework. ProtoNet is a distance-based learner trained in an episodic manner, so that both the metatraining and meta-testing stages have matching conditions. . At each training iteration, after sampling a given task from T , we first compute the class prototypes for classification using the support set. Then, the embeddings model is trained to min-imize the CE loss where each query example is classified based on the distances to the class prototypes. In order to add the contrastive objectives as auxiliary losses to the Pro-toNet training objective, we simply merge the query and support set, augment each exampled within it, and compute the contrastive losses detailed in §3 over this merged and augmented set.

Experimental Details. For the experimentation, we follow [START_REF] Chen | A closer look at few-shot classification[END_REF] and base our implementation on their few-shot learning code base. In particular, we use a ResNet-18 network as the embedding model with 512-dimensional output features. We train on ImagetNet derivatives using ADAM optimizer with a learning rate of 10 -3 for 60,000 episodes and use 5-way (classes) 5-shot (examples per-class) with 16 query images. For contrastive learning, similar to §4, we use and a two-layer MLP for the projection head and the attention modules with an output dimensionality of 64, and set λ CE to 1.0, and λ GC and λ SC to 0.5. For meta-testing, we report the mean accuracy and 95% confidence interval over 600 randomly sampled tasks, where each class consists of 5 support images and 16 query images.

Results. To investigate the impact of the contrastive losses on the performances of ProtoNet, we report the mean acc. for 5-way 5-shot classification on ImageNet derivatives with different training objectives. The results in Fig. 8 show a notable performance gain over the ProtoNet baseline. Surprisingly, when disregarding the labels and training with the self-supervised formulation of the contrastive objectives, we obtain better results. The SS-SC and SS-GC losses perform comparatively on mini-ImageNet with a 3.2% gain, and with the SS-SC loss performing slightly better on tiered-ImageNet with a 2.2% gain. We suspect that the obtained gain when using the self-supervised formulation might be a result of using a larger number of negatives as opposed to the supervised formulation, since each batch of examples only contains 5 unique classes. Additionally, we compare the performances of our approach with other self-supervised auxiliary losses, i.e., rotation prediction [START_REF] Gidaris | Unsupervised representation learning by predicting image rotations[END_REF] and jigsaw puzzle [START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF], for which [START_REF] Su | When does self-supervision improve few-shot learning[END_REF] provided their integration into the ProtoNet framework. As shown in Table 10, we observe that a larger performance gain can be obtained with the contrastive objectives as auxiliary losses compared to other self-supervised objectives, especially when using both the SS-SC and SS-GC losses with a 3.6% gain over the baseline, which further confirms the effectiveness of the proposed SC loss.

Related Work

Few-Shot Classification. In few-shot classification, the objective is to learn to recognize unseen novel classes with few labeled example in each class. Meta-learning remains the most popular paradigm to tackle this problem.

Roughly, such approaches can be divided into two categories. Optimization-based, or learning to learn methods [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF][START_REF] Andrychowicz | Learning to Spatial Contrastive Learning for Few-Shot Classification learn by gradient descent by gradient descent[END_REF][START_REF] Wang | Learning to learn: Model regression networks for easy small sample learning[END_REF][START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF][START_REF] Sun | Meta-transfer learning for few-shot learning[END_REF][START_REF] Lee | Meta-learning with differentiable convex optimization[END_REF][START_REF] Rusu | Meta-learning with latent embedding optimization[END_REF], that integrate the fine-tuning process in the meta-training algorithm to rapidly adapt to model to the unseen classes with limited supervision. And metricbased, or learning to compare methods [START_REF] Vinyals | Matching networks for one shot learning[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Sung | Learning to compare: Relation network for few-shot learning[END_REF][START_REF] Oreshkin | Task dependent adaptive metric for improved few-shot learning[END_REF][START_REF] Scott | Adapted deep embeddings: A synthesis of methods for k-shot inductive transfer learning[END_REF][START_REF] Doersch | Crosstransformers: spatially-aware few-shot transfer[END_REF], that learn a common embedding space in which the similarities between the data can help distinguish between different novel categories with a given distance metric. Most relevant to our work are the methods that follow the standard transfer learning strategy [START_REF] Chen | A closer look at few-shot classification[END_REF][START_REF] Dhillon | A baseline for few-shot image classification[END_REF]Tian et al., 2020b;[START_REF] Afrasiyabi | Associative alignment for few-shot image classification[END_REF]. Consisting of two stages, a pre-training stage with the CE loss on the meta-training set, then a fine-tuning stage on the meta-testing set. Despite their apparent simplicity, (Tian et al., 2020b) showed that such a strategy yields state-of-the-art results on standard benchmarks.

Cross-Entropy Loss. The CE loss continues to be the prominent supervised learning objective used for training deep networks, in which the model is trained to predict the corresponding class label in the form of a one-hot vector. However, despite its success, some works showed many possible drawbacks [START_REF] Khosla | Supervised contrastive learning[END_REF], e.g., noise sensitivity [START_REF] Sukhbaatar | Training convolutional networks with noisy labels[END_REF][START_REF] Zhang | Generalized cross entropy loss for training deep neural networks with noisy labels[END_REF], adversarial examples [START_REF] Nar | Crossentropy loss and low-rank features have responsibility for adversarial examples[END_REF], and suboptimal margins [START_REF] Elsayed | Large margin deep networks for classification[END_REF][START_REF] Cao | Learning imbalanced datasets with label-distribution-aware margin loss[END_REF]. While other works proposed some alternative approaches, such as changing the label distribution [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF][START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF][START_REF] Müller | When does label smoothing help[END_REF]Zhang et al., 2018) or leveraging the contrastive losses [START_REF] Khosla | Supervised contrastive learning[END_REF].

Contrastive Learning. Instead of training the network to match to the input to a fixed target. Contrastive learning acts directly on the low-dimensional representations with contrastive losses [START_REF] Hadsell | Dimensionality reduction by learning an invariant mapping[END_REF][START_REF] Gutmann | Noise-contrastive estimation: A new estimation principle for unnormalized statistical models[END_REF][START_REF] Salakhutdinov | Learning a nonlinear embedding by preserving class neighbourhood structure[END_REF], that measure the similarities of different samples in the embedding space. Recently, contrastive learning based methods have emerged as the state-of-the-art approaches for self-supervised representation learning. The main difference between them is the way they construct and choose the positive samples. In this work, we differentiate between self-supervised contrastive methods [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF]Wu et al., 2018b;[START_REF] Hjelm | Learning deep representations by mutual information estimation and maximization[END_REF][START_REF] Henaff | Data-efficient image recognition with contrastive predictive coding[END_REF]Tian et al., 2019a;[START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] that leverage data augmentations to construct the positive pairs, and supervised contrastive methods [START_REF] Salakhutdinov | Learning a nonlinear embedding by preserving class neighbourhood structure[END_REF]Wu et al., 2018a;[START_REF] Kamnitsas | Semisupervised learning via compact latent space clustering[END_REF][START_REF] Khosla | Supervised contrastive learning[END_REF] that leverage the provided labels to sample the positive examples.

Self-Supervised Learning and Few-Shot Classification. Relevant to our work are methods that try to build on the insights and advances in contrastive learning, or more broadly self-supervised learning, to improve the few-shot classification task. Such methods [START_REF] Gidaris | Boosting few-shot visual learning with self-supervision[END_REF][START_REF] Medina | Self-supervised prototypical transfer learning for few-shot classification[END_REF][START_REF] Su | When does self-supervision improve few-shot learning[END_REF][START_REF] Doersch | Crosstransformers: spatially-aware few-shot transfer[END_REF][START_REF] Gao | Contrastive prototype learning with augmented embeddings for few-shot learning[END_REF] integrate various types of self-supervised training objectives into different few-shot learning frameworks in order to learn more transferable features and improve the few-shot classification performance. In this paper, we propose a novel contrastive learning objective based on the spatial features to further promote general purpose and robust representations suited for few-shot classification. In this context, a similar idea was proposed in [START_REF] Doersch | Crosstransformers: spatially-aware few-shot transfer[END_REF]. In their approach, a constrastive pre-training stage is first conducted followed by the standard ProtoNet [START_REF] Snell | Prototypical networks for few-shot learning[END_REF] finetuning stage where spatial features are used to compute the similarity between the training and testing instances. In our work, contrary to [START_REF] Doersch | Crosstransformers: spatially-aware few-shot transfer[END_REF], we integrate the spatial information directly into the contrastive learning loss. The proposed loss is then integrated into the training as an auxiliary loss, resulting in a far more effective, flexible and general framework usable in various few-shot learning scenarios.

Conclusion

In this paper, we investigated contrastive losses as auxiliary training objectives along the CE loss to compensate for its drawbacks and learn richer and more transferable features. With extensive experiments, we showed that integrating contrastive learning into existing few-shot learning frameworks results in a notable boost in performances, especially with our spatial contrastive learning objective. Future work could investigate the spatial contrastive method extension for other few-shot learning scenarios and adapt it for other visual tasks such as unsupervised representation learning.

B. Additional Experiments

B.1. Embedding Model

In the transfer learning experiments, we mainly used a ResNet-12 backbone as the embedding model. Since the backbone also has a significant impact on the quality of the produced embedding, we experiment with various backbones. In particular, we compare ResNet-12 to two other alternative: 4 layer convolution network (i.e., 64-64-64-64) and ResNet-12 with sequeeze-and-excitation [START_REF] Hu | Squeeze-and-excitation networks[END_REF] layers that play the role of a channel-wise attention module. See Fig. D.1 for an illustration of these backbones. The results are presented in Table B.1, where we train the models either on the meta-training set only or on both the meta-training and the meta-validation set. In both cases with and without a contrastive distillation step. As expected, we observe that, (1) better models improve the performances, and (2) more training data yield better results, further emphasizing the importance of learning a well performing and transferable embedding model for an effective few-shot classification.

B.2. Weight Imprinting

During the ablation experiments, and when training a base classifier on top of the spatial features, a case that requires a larger classifier, we observed a slight decrease in the performances, which we suspect might be due to the overfitting of the model, especially at low shot setting. To overcome this, we investigate using imprinted weights [START_REF] Qi | Low-shot learning with imprinted weights[END_REF] to initialize the base classifier and help stabilize the convergence. Weight imprinting consists of directly setting the base classifier's weights from novel training examples during low-shot learning i.e., the 2 normalized class prototypes constructed from the training (i.e., support) features. This process is called weight imprinting since it directly sets the weights of the base classifier based on an the scaled features produced by the embedding model for a given training example at test time. Table B.2 summarizes the results, although the weight imprinting does slightly help the performances, the results are not conclusive and need further investigation.

B.3. Contrastive Distillation

As detailed in the paper, in order to relax the placement of the embeddings when applying a distillation step, we also optimize a contrastive loss L CD loss in addition to the standard KL loss. The loss consists of aligning the global features of the teacher and the student by reducing their cosine similarity. However, we can also align the spatial features rather than the global features, or both. In this case, the contrastive loss is written as follows:

L CD = α 1 N N i=1 z gt i -z gs i 2 2 + β 1 N N i=1 z st i -z ss i 2 2 . ( 12
)
The results are shown in Table B.3. We observe while contrasting only the global features works slightly better, the obtained We train a linear classifier on either the spatial, the global features, or both, where two classifiers are used and their predictions are aggregated by either taking their sum or maximum per class. We also investigate the effect of weight imprinting to improve the initialization of the classifier at test time. Note that the presented results use a PyTorch implementation of the classifiers instead of scikit-learn, resulting in slight decrease in performances. results are overall similar.

C. Additional Experimental Details

Training. As stated in the paper, during training, we reduce the learning rate with a factor of 0.1 at different training iterations. For mini-ImageNet, we train for 90 epochs and we decay the learning rate at 60 and 80 epochs, for tiered-ImageNet, we train for 60 epochs and decay the learning rate rate three times, at 30, 40 and 50 epochs. As for CIFAR-100 derivatives, we train for 90 epochs and decay the learning rate three times at 45, 60 and 75 epochs for CIFAR-FS, while for FC-100, we train for 65 epochs with single learning decay step at 60 epochs. For distillation, we change to learning rate to 10 -2 and train with similar settings.

For the weights of the loss functions, while the weight of the cross-entropy loss is always set to 1, when training with only one contrastive loss as an auxiliary loss, we set its weight to 1, i.e., λ GC = λ SC = 1, be it the supervised of self-supervised formulations, except for CIFAR-FS where we set λ GC = λ SC = 0.5. Additionally, when training with both objectives, we set λ SC = λ GC = 0.5.

For transfer learning experiments, we train on either 84×84 sized images for ImageNet derivatives, resulting in spatial features of spatial dimensions of 5×5, or 32×32 sized images for CIFAR-100 derivatives, resulting in spatial features of spatial dimensions of 2×2. For ProtoNet experiments, we train on 224×224 sized images with ResNet-18 backbone, resulting in spatial features of spatial dimensions of 7×7. Thus, in order to reduce the computational requirement when applying the attention based spatial alignment, we apply an adaptive average pooling, reducing the dimensions to 3×3 instead of 7×7 resulting in both better results and faster training time.

Evaluation. During evaluation, we use a multivariate logistic regression implemented in scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF] trained on the 2 normalized global features. Additionally, when using the spatial features during evaluation, and in order to reduce the number of parameters of the linear classifier and avoid overfitting, we first apply a max pooling operation over these features, reducing their spatial dimensions to 2 × 2, and then feed them to the linear classifier. As for cross-domain experiments, we found that augmenting each training (i.e., support) samples 10 times instead of 5 produces slightly better results.

When conducting ablation studies with weight imprinting (i.e., Table B.2), we use our own implementation of the multivariate logistic regression with an L-BFGS optimizer and an 2 penalty. While the performances are overall similar, this gives us more degrees of freedom when implementing the base classifier. Specifically, when using the spatial features as input to the base classifier, we implement the classifier as a convolutional layer, in which the filter size matches the dimensions of the spatial features.

D. Architectures

Few-shot classification

  usually involves a meta-training set T and a meta-testing set S with disjoint label spaces. The meta-training set discerns seen classes, while the metatesting set discerns novel and unseen classes. Each one of the meta sets consists of a number of classification tasks where each task describes a pair of training (i.e., support) and testing (i.e., query) sets with few examples, i.e., T = {(D train t , D test t )} T t=1 and S = {(D train q , D test q )} Q q=1 , with each dataset containing pairs of images x and their ground-truth labels y. The goal of few-shot classification is to learn a classifier f θ parametrized by θ capable of exploiting the few training examples provided by the dataset D train to correctly predict the labels of the test examples from D test for a given task. However, given the high dimensionality of the inputs and the limited number of training examples, the classifier f θ suffers from high variance. As such, the training and testing inputs are replaced with their corresponding features, which are produced by an embedding model f φ parametrized by φ and then used as inputs to the classifier f θ .

Figure 2 .

 2 Figure 2. Analysis of the Learned Representations. (a) k-Nearest Neighbors Analysis. For a given test image from mini-ImageNet dataset, we compute the nearest neighbors in the embedding space on the test set, and we observe that they are semantically dissimilar. This suggests that the learned embeddings are excessively discriminative towards features used to solve the training classification tasks (e.g., the beer bottles in the second test image), which are not useful to recognize the novel classes at test time. (b) GradCAM results. To obtain the class activation maps (CAMs) explaining such an outcome, we train a linear classifier on the whole test set on top of the frozen embedding model and compute the CAMs. We see that the dominant discriminative features are not the ones useful for test-time classification.

,

  Concretely, we start by merging all the meta-training tasks D train t from T into a single training set D new of seen classes: D new = ∪{D train 1

Figure 4 .

 4 Figure4. Attention-based Spatial Alignment. To compute the spatial similarity between a pair of features (purple and blue), we first spatially align the first features (purple) with respect to the second (blue) features with the attention mechanism (see Eq. (5)). Then we can compare the aligned value of the first features with the value of the second features. Note that the same process is applied in reverse to compute the final spatial similarity (see Eq. (6)).

Figure 5 .

 5 Figure5. Degree of Clustering. The plot shows the evolution of the intra-class variation using the Davies-Bouldin index[START_REF] Davies | A cluster separation measure[END_REF] during the course of training on mini-ImageNet when using the contrastive loss. We see that the learned embeddings of each class are significantly over-clustered, an outcome that might not be desired in some cases.

Figure 7 .

 7 Figure 7. Comparison of the mean acc. obtained on mini-ImageNet and CIFAR-FS with sequential distillation.

Figure 8 .

 8 Figure8. The obtained improvement when adding the contrastive objectives as auxiliary losses. We show the mean acc. and 95% confidence interval for 5-way 5-shot classification across Imaget-Net derivatives.

)

  During meta-training, for a C-way K-shot setting, we construct a meta-training set T = {(D train t depicts C randomly chosen classes from the seen classes, with K images per class for the training (i.e., support) set D train t , and M images per classes for the test (i.e., query) set D test t

  Comparison of contrastive distillation losses on 5-way classification on mini-ImageNet, CIFAR-FS and FC100. We show the mean acc. and 95% confidence interval.

Figure D. 1 .

 1 Figure D.1. Architectural details. The architecture of the backbones used as embeddings models, and the projection module used both as the projection head for GC loss, and for the attention modules (i.e., head, query and value heads) for the SC loss. "seq & exc" refers to squeeze and excitation modules.

  Data Augmentation. During meta-training, for a given augmented batch of 2N examples, and consistent with other approaches(Tian et al., 2020b;[START_REF] Lee | Meta-learning with differentiable convex optimization[END_REF], the first N instances are obtained using standard augmentations, i.e., random crop, color jittering and random horizontal flip. The remaining N instances are obtained with SimCLR type augmentations, i.e., random resized crop, color jittering, random horizontal flip and random grayscale conversion. During the meta-testing stage, we follow(Tian et al., 2020b) and create 5 augmented versions of each training image to overcome the problem of data insufficiency and train the linear classifier f θ .

Table 1 .

 1 Comparison of the mean acc. obtained on mini-ImageNet and CIFAR-FS with different training objectives. "Aug." indicates the usage of SimCLR type augmentations. Comparison of the mean acc. obtained on mini-ImageNet across various batch sizes and SC loss temperatures. ± 0.7 82.1 ± 0.5 74.0 ± 0.9 86.7 ± 0.6 Stacked RandAug. 64.9 ± 0.8 81.6 ± 0.6 75.0 ± 0.9 87.6 ± 0.6

							1-shot	5-shot
	Meta-Test Accuracy (%)	60 65 70 75 80 85					Meta-Test Accuracy (%)	60 65 70 75 80 85
		55						55
		32	64	96	128	196	256	0.02 0.04 0.08 0.12 0.16	0.2
				Batch Size				Temperature
	Figure 6. Augmentation		mini-ImageNet, 5-way 1-shot 5-shot	CIFAR-CS, 5-way 1-shot 5-shot
	Standard			64.3 ± 0.7 80.6 ± 0.5 74.9 ± 0.8 86.3 ± 0.6
	SimCLR			65.7 ± 0.8 82.5 ± 0.5 75.0 ± 0.9 87.4 ± 0.6
	AutoAugment		65.2

Table 2 .

 2 Comparison of the mean acc. obtained on mini-ImageNet and CIFAR-FS with different augmentation strategies, which are used to obtain the additional N augmented instances within a minibatch.

	Aggregation	mini-ImageNet, 5-way 1-shot 5-shot	CIFAR-CS, 5-way 1-shot 5-shot
	Sum	65.2 ± 0.8 81.2 ± 0.5 75.3 ± 0.8 87.3 ± 0.5
	Mean	65.7 ± 0.8 82.5 ± 0.5 75.0 ± 0.9 87.4 ± 0.6
	Maximum	65.5 ± 0.7 82.0 ± 0.5 73.4 ± 0.8 86.4 ± 0.6
	LogSumExp	64.8 ± 0.8 81.7 ± 0.6 74.2 ± 0.8 87.0 ± 0.6

Table 3 .

 3 Comparison of the mean acc. obtained on mini-ImageNet and CIFAR-FS with different aggregation functions, which are used to amount the total similarity from the one-to-one spatial similarities.

Table 4 .

 4 Comparison of the mean acc. obtained on mini-ImageNet and CIFAR-FS with different evaluation settings, in which we use either the global features, the spatial features, or both.

Table 5 .

 5 Comparison of the mean acc. obtained on mini-ImageNet and CIFAR-FS with different distillation objectives.

Table 6 .

 6 Comparison with prior few-shot classification works on ImageNet derivatives. We show the mean acc. and 95% confidence interval. † results obtained by training on both train and validation sets.

	mini-ImageNet, 5-way	tiered-ImageNet, 5-way

Table 2 shows that the SimCLR type augmentations yield the best results overall. We speculate that

Table 7 .

 7 Comparison with prior few-shot classification works on CIFAR-10 derivatives. We show the mean acc. and 95% confidence interval. † results obtained by training on both train and validation sets.

	Method	mini-ImageNet
		20-way 1-shot
	Matching networks (Vinyals et al., 2016)	17.31 ± 0.22
	Meta-LSTM

  .47 ± 0.6 66.98 ± 0.7 31.61 ± 0.5 44.90 ± 0.6 55.77 ± 0.8 73.94 ± 0.7 35.95 ± 0.5 53.85 ± 0.6

	Ours	49.58 ± 0.7 67.64 ± 0.7 34.46 ± 0.6 52.22 ± 0.7 59.37 ± 0.7 76.46 ± 0.6 40.23 ± 0.6 59.38 ± 0.6
	Ours-Distill	50.09 ± 0.7 68.81 ± 0.6 34.93 ± 0.6 51.72 ± 0.7 60.32 ± 0.8 76.51 ± 0.6 39.75 ± 0.8 59.91 ± 0.6

Table A .

 A 1. Datasets. Additional details about the datasets used in the experiments.

Table B

 B

.2. Comparison of different evaluation setups.

Acknowledgements

The first author is supported by Randstad corporate research chair, in collaboration with CentraleSupélec, Université Paris-Saclay. This work was performed using HPC resources from the Mésocentre computing center of Cen-traleSupélec and École NormaleSupérieure Paris-Saclay supported by CNRS and Région Île-de-France (http:// mesocentre.centralesupelec.fr/).

Supplementary Material

A. Datasets

In this paper, we experimented with different datasets used for various few-shot classification settings. For the standard few-shot classification setting, we used the four popular benchmarks: mini-ImageNet, tiered-ImageNet, CIFAR-FS and FC100. As for cross-domain few-shot classification, following [START_REF] Tseng | Crossdomain few-shot classification via learned feature-wise transformation[END_REF], we train on the whole mini-ImageNet (i.e., train, val and test sets), and then we evaluate on one of the following datasets: CUB, Cars, Places or Plantae. The details about each dataset are presented in

E. Quality of the Learned Representations

In this section, and similar to the analysis conducted in the paper, we conduct an empirical analysis of the embeddings to assess the quality of the learned features in two cases: (1) when the model is trained with only the CE loss, and (2) when adding the SC as an additional auxiliary objective. Figs. E.1 and E.2 show the results. We observe a clear improvement in terms of the obtained nearest neighbors when using the SC loss and also an increase in terms of the amount of informative signal retrained within the embedding matrix, both indicating an enhancement in the quality of the learned embeddings. For a given test image from mini-ImageNet dataset, we compute the nearest neighbors in the embedding space on the test set when a model is train with either, (a) the standard CE, or, (b) with the proposed SC as an additional auxiliary objective. We observe that the neighboring images in the embedding space found when the SC loss is used are more semantically similar then the standard case with the CE loss. It suggests that the quality of the learned embeddings is increased with the usage of the SC as an auxiliary loss as a result of optimizing for more general-purpose features.