Alexandre, Raphaël, Tiapo, Sylvie Maman

Mamie Christophe

Nathalie Gilles

Alain Cathy

Frédéric Je

Few-Annotation Learning Few-Shot Learning Labeled Data Unlabeled Data Pretraining Fine-Tuning Meta

Keywords: Learning Aux, Labeled Data Labeled Data Aux, or Target Unlabeled Data Target Labeled Data

In this thesis, we develop theoretical, algorithmic and experimental contributions for Machine Learning with limited labels, and more specifically for the tasks of Image Classification and Object Detection in Computer Vision. In a first contribution, we are interested in bridging the gap between theory and practice for popular Meta-Learning algorithms used in Few-Shot Classification. We make connections to Multi-Task Representation Learning, which benefits from solid theoretical foundations, to verify the best conditions for a more efficient meta-learning. Then, to leverage unlabeled data when training object detectors based on the Transformer architecture, we propose both an unsupervised pretraining and a semi-supervised learning method in two other separate contributions. For pretraining, we improve Contrastive Learning for object detectors by introducing the localization information. Finally, our semi-supervised method is the first tailored to transformer-based detectors.

Je voudrais avant tout remercier le service du SIALV et en particulier le labo LVA pour m'avoir accueilli pendant ces dernières années. J'ai eu la chance de travailler avec des personnes remarquables et bienveillantes qui ont été d'une aide précieuse. Je tiens également à remercier tous les autres doctorants avec qui j'ai eu la chance de partager des moments de discussions enrichissants, notamment mes co-bureaux Julien et Rémi. Vous avez été une source de motivation quotidienne.

Je voudrais exprimer ma gratitude envers mes encadrants du CEA, Romaric et Angélique, qui m'ont guidé et encadré depuis 4 ans ! On a commencé à travailler ensemble avant la thèse et je vous remercie d'avoir cru en moi. Je vais maintenant voler de mes propres ailes, mais je garderai toujours vos précieux conseils en tête. Je souhaite d'ailleurs beaucoup de courage à Angélique pour son nouveau poste de cheffe et de maman. Je voudrais également remercier Ievgen, qui a certes quitté l'encadrement officiellement à la moitié de la thèse, mais qui est resté de bon conseil tout du long ! Même avec la distance, en France ou à l'autre bout de l'Europe, tu m'as accompagné et es resté disponible. Un grand merci à mon directeur de thèse Amaury, qui malgré les difficultés et la distance a toujours été présent et disponible pour moi. Je n'ai pas pu venir à Saint-Etienne aussi souvent que je l'aurais voulu à cause du contexte sanitaire, mais je te suis reconnaissant de m'avoir fait confiance toi aussi.

Je voudrais aussi exprimer ma gratitude envers les membres du jury, je suis honoré d'avoir pu présenter mon travail devant vous. J'ai une pensée pour Diane et Devis qui ont pris de leur temps pour évaluer mon travail même à distance. David qui a pris la responsabilité de la présidence du jury. Enfin, je souhaite exprimer toute ma reconnaissance envers les rapporteurs Céline et Nicolas, qui ont consacré du temps à la lecture de mon manuscrit de thèse. J'ai été touché par leurs retours pertinents et constructifs.

Résumé

Nous développons dans cette thèse des contributions théoriques, algorithmiques et expérimentales pour l'apprentissage automatique avec peu d'annotations, et plus spécifiquement pour les tâches de classification d'images et de détection d'objets en vision par ordinateur. Dans une première contribution, nous nous intéressons à combler le fossé entre la théorie et la pratique concernant les algorithmes de méta-apprentissage les plus répandus, utilisés pour de la classification avec peu d'exemples. Nous établissons des liens avec l'apprentissage de représentation multi-tâches, qui bénéficie de bases théoriques solides, afin de vérifier les meilleures conditions amenant à un méta-apprentissage plus efficace. Ensuite, afin d'exploiter les données non étiquetées pour entraîner des détecteurs d'objets basés sur l'architecture Transformer, nous proposons à la fois un pré-entraînement non supervisé et une méthode d'apprentissage semisupervisée dans deux autres contributions distinctes. Pour le pré-entraînement, nous améliorons l'apprentissage contrastif des détecteurs d'objets en introduisant les informations de localisation. Enfin, notre méthode semi-supervisée est la première adaptée aux détecteurs d'objets utilisant des modèles type Transformers.

-ii - This chapter provides an introduction to the main problems addressed in this thesis, as well as the presentation of the structure and contributions of this manuscript.

Context and Background

Context

Learning something new in real life does not necessarily mean going through a lot of examples in order to capture the essence of it. Even though it is said that it takes 10,000 hours to master a new skill [Gladwell, 2009], it is also true that it only takes 20 hours to learn it [Kaufman, 2013]. Humans are able to build upon prior experience and have the ability to adapt, allowing to combine previous observations with only little evidence for fast learning. This is particularly the case for recognition tasks, for which we are often capable of differentiating between two distinct objects after having seen only a few examples of them. We give an illustrative example from paintings samples presented in Figure 1.1. It is quite obvious that one would easily guess and recognize the painter of the last painting below after having seen just one example of each painter's styles above. This idea has found its application in machine learning in a more general few-shot learning paradigm that wants to mimic the human capability to quickly learn how to solve a new problem. The above example is a direct illustration of one-shot learning.

Background

Learning models, i.e. estimating parameters from data, has long proven to be an effective approach in the literature for solving different kinds of applications, ranging from computer vision to natural language processing. This is what is commonly referred as Machine Learning.

Ever since AlexNet [Krizhevsky, 2009], the rise of Deep Learning (DL) [START_REF] Lecun | Deep learning[END_REF], Goodfellow et al., 2016] models for Representation Learning has led to near-human performance on many vision tasks. Current best performing architectures are based on Convolutional Neural Networks (CNN) [START_REF] Lecun | Handwritten digit recognition with a backpropagation network[END_REF], such as the popular ResNet architecture [START_REF] He | Deep residual learning for image recognition[END_REF], and more recently on Transformers [START_REF] Vaswani | Attention is all you need[END_REF], Dosovitskiy et al., 2021, Carion et al., 2020].

Training DL models consists in optimizing an objective, or loss function w.r.t. to some experience, or previously acquired information. Learning algorithms considered throughout this thesis are allowed to experience an entire dataset. A dataset is a collection of many examples, also called data samples. These examples are usually measurements, and more specifically in the case of Computer Vision, they most commonly take the form of images, annotated or not.

Learning frameworks

Training paradigms can be broadly separated into either Supervised Learning (SL), Unsupervised Learning (UL), or Semi-supervised Learning (SSL) depending on the amount of annotated information available during the learning process.

Supervised Learning (SL)

The most common form of machine learning, deep or not, is supervised learning [START_REF] Shalev-Shwartz | Understanding machine learning: From theory to algorithms[END_REF], Mohri et al., 2018, LeCun et al., 2015]. During training, SL algorithms are shown examples associated with a label, or target. The term supervised learning comes from the view that an instructor, or teacher, is providing supervision through the labels [START_REF] Goodfellow | Deep Learning[END_REF]. Indeed, the training optimizes supervised loss functions that take into account the additional information contained in the labels.

Unsupervised learning (UL) Unlike SL, UL algorithms [START_REF] Shalev-Shwartz | Understanding machine learning: From theory to algorithms[END_REF], Mohri et al., 2018, LeCun et al., 2015] experience a dataset without labels, i.e. without additional information. The objective is to learn useful properties about the structure of the input dataset, by, e.g. learning the entire probability distribution that generated the dataset, either explicitly or implicitly [START_REF] Goodfellow | Deep Learning[END_REF]. This problem is generally seen as more difficult than SL, but also potentially more impactful since it requires less information on the data provided by the user [START_REF] Lecun | Deep learning[END_REF]. The recent Self-Supervised learning (Self-SL) paradigm is a particular case of UL, in which the data provides its own supervision to learn a practical representation.

Semi-Supervised Learning (SSL) In addition to data, SSL algorithms [START_REF] Mohri | Foundations of machine learning[END_REF] are supervised with labels, but not for all examples. This paradigm can be seen as halfway between supervised and unsupervised learning [START_REF] Chapelle | Semisupervised learning[END_REF]. The dataset can be usually divided into labeled data, for which labels are provided, and unlabeled data, the labels of which are not known. [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]. The pipeline is split into 3 primary tasks: (a) labeling the categories (class of object) present in the image, (b) locating and marking all instances of the labeled categories, and (c) segmenting each object instance.

Figure 1.2. Illustration of the annotation pipeline from the COCO dataset

When having access to large datasets, SSL is often a good compromise between the two other frameworks in practice. Resulting models can achieve strong performance while requiring limited supervision. Even though SL is the most common form of DL, it is also the most restrictive one for practical use, as it requires to have access to a large number of labels.

Challenges and Objectives

The Problem of Labeling Data

Supervised Learning requires a lot of labeled data to be effective. The ImageNet dataset [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF], one of the most popular datasets for image recognition, contains from 1.2M (ILSVRC 2012 or ImageNet-1k) to 14M labeled images . However, the process of annotating data can be hard, time-consuming and thus, expensive. For a highly accurate dataset, humans must verify and manually annotate each collected example. Then, to reduce human errors, the process is repeated independently by multiple persons and the final label of each sample is taken as the convincing majority of labels given. Each year from 2010 to 2017, ImageNet employed 20,000 to 30,000 people to annotate the dataset through the Amazon Mechanical Turk (AMT) 1 service, paying a few cents for each image labeled2 . Furthermore, more dense computer vision tasks, i.e. problems that require a deeper understanding of images and thus more complex annotations per image, take more time to annotate. Figure 1.2 is an illustration of the full annotation process for the MS COCO (COCO) dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]. COCO is one of the most popular largescale dataset used for the Object Detection (OD) task, which consists in locating and classifying every instance of object in an image. The dataset is a collection of over 200k labeled images for a total of about 2.5M object instances. As we can see in Figure 1.2, the annotation pipeline is separated in three tasks, (a) Category labeling, (b) Instance spotting and (c) Instance segmentation. Here again, multiple AMT workers were asked to manually annotate and verify images, for a total of over 85,000 worker hours [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], i.e. over 9 worker years.

The success stories of the above two examples of dataset labeling heavily rely on a collaborative annotation process to reduce the overall cost. However, not all datasets can be annotated in the same way. While anyone can label images of dogs and cats, data from more complex domains might require precious time from qualified workers. For instance, labeling medical or borehole images requires insights from health specialists and geophysicists, increasing even more the cost of annotation. In addition, in some cases, sensitive data or private datasets cannot be shared with anyone. For all these reasons, we often do not have access in practice to large-scale labeled datasets.

Training data conditions in practice

We can separate learning problems in practice in three categories, depending on the amount of target data, i.e. data linked to the task that we want to solve, but also depending on the labels available on this data. We give an illustration of these different categories in Figure 1.3.

Many-Shot Learning (MSL)

The most common category, MSL, corresponds to learning problems for which we have access to a lot of annotated data. They usually arise when the data is easy to acquire and to annotate. We will not delve into the Target Labeled and Unlabeled Data FAL FSL or Figure 1.5. A tentative general pipeline encompassing approaches of FSL and FAL, coping with limited target labels. The semi-supervised phase (dotted lines) is possible if unlabeled target data are also available (FAL).

data on the target task to better learn on the few labeled data available. Methods in this setting either rely on an unsupervised pretraining phase on the unlabeled data followed by a supervised fine-tuning phase on the labeled data (left), or semi-supervised learning to leverage unlabeled data along with labeled data during training (right).

In this thesis, we are specifically interested in problems with limited labels available and explore both settings of FSL and FAL. In this case, from approaches in the literature, we can draw a general learning pipeline that distinguishes three different phases, illustrated in Figure 1.5, which are not always present. First, a pretraining phase either unsupervised or supervised on auxiliary labeled data, learns a general representation. Then, the finetuning phase adapts the representation learned to the target labeled data. Additionally, a semi-supervised phase can complete the training with both labeled and unlabeled target data.

While there is a large literature in MSL, research in FSL and, even more, in FAL, have been much more sparsely addressed. Yet, they represent the most common settings in industry because of the cost of data annotation, and sometimes even its collection. This discrepancy between research and industry interests motivated the work presented in this thesis.

Issues addressed

Throughout this thesis, we are interested in applications to Computer Vision problems, with a focus on Image Classification and Object Detection tasks. As detailed above, while the easiest learning conditions and the most effective methods require having access to a large amount of labeled images, learning with scarcely annotated data remains a challenging problem even though it corresponds to the most common setting in practice. We address the issues of learning with limited labels from different perspectives:

• On the one hand, our objective is to better understand the inner workings of learning algorithms in FSL settings from a theoretical point of view. Conditions and guarantees for a given algorithm to learn well are given by theoretical analyses and are expressed by learning bounds. These bounds often take the following form [Vapnik, 1999]:

True Risk -Empirical Risk ≤ Model Complexity F (Number of training data) , ( 1.1)
where the True and Empirical Risk correspond to the expected value of the loss function, respectively computed on the full data distribution and the available dataset, the Model Complexity is a measure of the representation capacity of the model used, and F is a function of the number of training data used. However, in FSL where the number of training data available is small, this last specific term can hinder the training process. Through connections to frameworks with solid theoretical foundations, we aim to find the best training conditions for this setting and investigate this problem in Chapter 4, with the settings considered illustrated in Figure 1.6a.

• On the other hand, we want to improve Object Detection methods from an algorithmic point of view in FAL settings. The specificities of Object Detection have been barely addressed in the context of FAL, even though it represents a task with many practical applications. We are specifically interested in unsupervised pretraining and semi-supervised learning methods, investigated respectively in Chapter 5 and Chapter 6, with their respective settings illustrated in Figure 1.6b and Figure 1.6c.

In the following section, we discuss the overall structure of the manuscript and our contributions addressing the issues introduced above.

Outline and Contributions

We present in this section an overview of this thesis and the diverse associated publications.

Structure of the manuscript

This manuscript is organized as follows. The present Chapter 1 is meant as a motivation to the work presented in the following chapters. We introduced the different learning paradigms and the main settings encountered in practice. We illustrated the timecost problem of manual data labeling with examples of the annotation process from popular datasets. This problem is at the heart of all the questions that will interest us throughout this thesis, since it dictates the amount of target annotated data available.

In Chapter 2, we delve more in depth into the different concepts in Machine Learning that will be important to understand this work. We present the learning process and theoretical notions with simple models, then deeper models and their specificities. Finally, since we will focus on using images as our data throughout this thesis, we introduce the popular tasks of Image Classification and Object Detection, and their respective recent literature.

In Chapter 3, we review related work in learning with limited labels. These works span the topics of Data Augmentation, Meta-Learning, Transfer Learning and Semisupervised Learning. We introduce recent work in each topic, and conclude with the research directions outlined for this thesis.

Then, the following chapters present our contributions, each positioned w.r.t. the general training pipeline from Figure 1.5,in Figures 1.6a,1.6b and 1.6c. Chapter 4 is intended to bridge the gap between Multi-Task Representation Learning theory and practices in Meta-Learning for FSL. We investigate the behavior and theory behind popular meta-learning algorithms through the lens of theoretical assumptions from recent learning bounds for Multi-Task Representation Learning. This leads us to better understand these methods and improve their performance by forcing them to follow the conditions for a more efficient learning in FSL.

Chapter 5 focuses on pretraining strategies for learning Object Detector with limited labels. In this chapter, we propose a novel pretraining method for recent transformerbased object detectors, based on a Proposal-Contrastive Learning paradigm. Throughout the chapter we advocate for consistency in the level of information learned during pretraining, which forms the foundations of our method.

In Chapter 6, we propose a novel semi-supervised method tailored to transformerbased object detectors. We are interested in improving performance with scarcely

The Basics of Learning from images

This chapter presents the preliminary knowledge to understand this thesis. The main objective is to describe basic notions and challenges in Machine

Learning and Deep Learning, with a focus on using images as our input data. We begin by explaining the theoretical context of Machine Learning problems in Section 2.1, then we move to Deep Learning, with a justification of the necessity of deeper models as well as a presentation of the specificities and techniques in Section 2.2. Finally, we give a general overview of the field of Computer Vision with a focus on the Image Classification and Object Detection tasks in Section 2.3.

-13 - We illustrate our problem using images of famous paintings, and groundtruth labels of the corresponding painters (left). We want to find a model (right, the green line) that can correctly predict the painters, which can be easily done by a human.

Machine Learning

Objective

In this section, we want to solve a task for which we have a dataset containing a ground truth label associated to each input samples. We illustrate the problem with images in Figure 2.1. Using Machine Learning (ML), the goal is to learn a model that can correctly predict the label given an input data. This model is generally a function, potentially complex depending on the problem to solve, and this function can be represented by the set of its parameters, that we estimate from the data during the learning process. Throughout this thesis we will be specifically interested in the case of images. However, input data could correspond to any vector of values.

Learning Process

We formally describe the process of learning from data [Vapnik, 1999, Goodfellow et al., 2016] in this section. Suppose that we have N training data samples represented as d 1 -dimensional input vectors from an input space X: {x i } N i=1 ∈ X N ⊆ R N ×d 1 with their associated ground-truths, or labels, represented as d 2 -dimensional vectors from an output space Y: {y i } N i=1 ∈ Y N ⊆ R N ×d 2 , corresponding to N observations of random vectors x and y that were drawn independently from fixed but unknown true data distributions P (x) and P (y|x). We group these observations into matrices X ∈ R N ×d 1 and Y ∈ R N ×d 2 to form the training set D train , such that D train := (X, Y) = ((x 1 , y 1 ), . . . , (x N , y N )) ∼ P (x, y),

(2. [START_REF] Gladwell | Outliers. Back Bay Books[END_REF] with the joint probability distribution P (x, y) = P (y|x)P (x), called the data-generating distribution. We typically make the assumption that the data samples are independent and identically distributed (i.i.d.). We also further assume that we have a testing set of data D test , obtained by the same underlying distribution P (x, y), that will be used to evaluate our future model and is never used during the training process that we describe in the next section.

Risk Minimization

The goal of Machine Learning is to find a function f θ : X → Y, or model, that gives a good approximation of the labels associated to entries of X w.r.t. P (y|x). f θ is parameterized by a set of weights θ ∈ Θ. The model f θ is selected by a learning algorithm from a set of predefined models, called the hypothesis space:

H ⊆ {f θ |f θ : X → Y, θ ∈ Θ}.
The learning is driven by a loss function L, which provides a measure of the quality of a prediction given by the model outputs f θ (x) w.r.t. the corresponding ground-truth y, and the true risk R given by the expected value of the loss: R(θ) := L(y, f θ (x))dP (x, y).

(2.2)

Then, the optimal parameters θ * can be found by minimizing the risk functional. However, since the true data-generating distribution is unknown, the only available information is contained in the training set D train defined in Equation (2.1). Thus, in practice, we compute the empirical risk E from the training set as follows:

E(θ) := 1 N N i=1
L(y i , f θ (x i )).

(2.3)

Finally, the empirical parameters θ are obtained by Empirical Risk Minimization (ERM):

θ := arg min θ∈Θ E(θ).

(2.4)

The induction principle of ERM assumes that the empirical model f θ, which minimizes E, results in a true risk close to its minimum as well [Vapnik, 1999]. Finally, the process of minimizing E is solved as an optimization problem, and it is referred to as training a model.

Learning Bounds

The soundness of the ERM principle largely depends on the uniform convergence of the empirical risk E to the true risk R, given > 0:

P {sup θ∈Θ |R(θ) -E(θ)| > } → 0 when N → ∞, (2.5)
and the rate of convergence, i.e. how fast f θ converges towards f θ * in terms of number of samples N , which is also called the sample efficiency. The more sampleefficient a learning algorithm is, the more interesting in practice it becomes.

Rates of convergence for a given class of function H are obtained by theoretical analysis, but mainly depend on the capacity C(H) of the models. Informally, a model's capacity indicates its ability to fit a wide variety of functions, and can be quantified in different ways, with the most known ones being the Vapnik-Chervonenkis dimension [START_REF] Vapnik | On the uniform convergence of relative frequencies of events to their probabilities[END_REF] and the Rademacher complexity [START_REF] Bartlett | Rademacher and gaussian complexities: Risk bounds and structural results[END_REF]. However, these are mainly theoretical quantities useful for upper-bounding purposes. Accurately measuring the capacity of a class of model in practice is still an open problem. Statistical learning theory [Vapnik, 1999, Hastie et al., 2009, Shalev-Shwartz and Ben-David, 2014, Mohri et al., 2018] gives us learning bounds providing justification of the learning process, as well as guarantees on the rate of convergence and generalizability of the models. We discuss the latter in the next section.

Generalization Problem

The question of the generalizability of a model arises when evaluating on the testing data D test . The ability of a model to perform well on unseen data is called generalization.

When training a model, we can compute an error measure on the training set, that we call the training error, and this error is usually reduced during the optimization process of the ERM. However, we want our model to be effective on the full underlying distribution of inputs P (x) to be useful in practice, not only on the training observations X. Thus, we also measure the test error on D test and want it to be low as well, to estimate the generalization error on unseen data. The need for generalizability is what separates machine learning from optimization.

The capacity of a model has a great influence on its generalization capabilities, as can be seen in Figure 2.2. A model with low capacity will likely underfit, i.e. not predict accurate labels, which leads to a high training and test error. On the other hand, a model with a too big capacity will likely overfit, i.e. memorize training data leading to a low training error but a high test error. [START_REF] Goodfellow | Deep Learning[END_REF]. Since we only have access to training data, training and test error behave differently. At the left end of the graph, training error and generalization error are both high. This is the underfitting regime. As we increase capacity, training error decreases, but the gap between training and generalization error increases. Eventually, the size of this gap outweighs the decrease in training error, and we enter the overfitting regime, where capacity is too large, above the optimal capacity.

Figure 2.2. Illustration of the relationship between capacity and error from

To avoid overfitting to the training data, we can either restrict the hypothesis space to low capacity models, or more elegantly guide the optimization process towards low capacity models through regularization, which we explain in the next section.

Regularization

Regularization consists in defining a penalty function, called regularizer, to introduce in the learning problem, that will express a preference for specific functions in the hypothesis space. For instance, a common regularizer used is the weight decay [START_REF] Krogh | A simple weight decay can improve generalization[END_REF]] Ω: Ω(θ) := θ 2 2 , (2.6) defined as the squared L 2 norm of the parameters, that influences the optimization towards small weights and reduces overfitting. We can introduce the weight decay regularization in the training process by considering

J := L + λΩ, (2.7)
as our regularized loss function for the ERM, which is then called Regularized ERM (RERM). In the above equation, we add the parameter λ to control the strength of the regularization.

Types of Models

The most simple Machine Learning algorithms are based on linear models, such that

H = {f (x) = w x + b, w ∈ R d 1 , b ∈ R},
with the most influential approach being the Support Vector Machine (SVM) [Boser et al., 1992, Cortes andVapnik, 1995]. These models are easy to learn since the training can be easily transformed into a convex optimization problem thanks to a formulation based on a maximum margin principle:

ŵ, b = arg min w,b w 2 , subject to y i (w x i -b) ≥ 1, ∀i ∈ {1, . . . , N }. (2.8)
To overcome the limitation of the representational capacity of linear models, which is most apparent when dealing with high dimensional data such as images, one key innovation from SVM is the kernel trick [START_REF] Boser | A training algorithm for optimal margin classifiers[END_REF]. The main idea comes from observing that linear models can be rewritten as a dot product between examples:

w x + b = b + N i=1 α i x x i .
(2.9)

The kernel trick consists in using a non-linear function φ to represent the data samples x by φ(x), and replacing the dot product by a kernel function k(x, x i ) = φ(x), φ(x i ) . The kernel trick is powerful since it allows to learn a model that is nonlinear as a function of x using efficient convex optimization methods, but the mapping φ and the associated kernel are usually handcrafted or generic. The computations also become dependent on the number of training examples, which can be highly expensive when the training dataset is large. Deep Learning represents another family of methods that allow to overcome these limitations by learning efficiently non-linear models through backpropagation and stochastic gradient descent.

Deep Learning

Simple machine learning algorithms generally struggle with high-dimensional data, such as natural images in practice, since the number of possible configurations for the model increases exponentially with the number of dimensions. This is known as the curse of dimensionality in Machine Learning [START_REF] Goodfellow | Deep Learning[END_REF]. These high-dimensional data often lie in specific and complicated regions of the representation space. For instance, sampling random values from a uniform distribution for each pixel will result in an image very different from a real photo, which is illustrated in Figure 2.3.

All these problems motivated the design of more complex models to generalize to central applications such as computer vision. The strategy of Deep Learning is to learn a non-linear mapping φ providing a representation of the data, instead of using generic or handcrafted functions with the kernel trick.

Figure 2.3.

Although there is a non-zero probability to generate a real photo by randomly sampling values for pixels, we never actually observe this in practice. This suggests that natural images lie in specific and narrow regions of the image space, which can be difficult to represent with a simple model.

Simple Neural Networks

Deep Learning architectures are composed of several layers of artificial neurons. By stacking layers and adding more units within layers, deep neural networks can represent complex, non-linear functions.

Artificial Neurons

A neuron is represented as a single parametric function f w : X → R which takes inputs x ∈ X and applies an activation function ϕ : R → R:

f w (x) = ϕ w x .
(2.10) Artificial neurons were popularized by the Perceptron algorithm [Rosenblatt, 1958] for supervised binary classification, which used a step function as activation: ϕ = 1 R + . They are at the basis of most deep learning architectures.

Multilayer Perceptrons

Multilayer Perceptrons (MLP) are the simplest deep learning models, composed of multiple layers of neurons. The number of neurons in a layer determine its width, i.e. its dimensionality. The first and last layers are respectively the input layer and output layer, and the dimensionality of the input and output data respectively define their widths. All the other layers are called hidden layers, and their respective widths are hyperparameters specified by design. Each layer describes a function, and the full model f θ can be seen as the composition of all the functions:

f θ = f (d) θ (d) • • • • • f (1) θ (1) , (2.11)
where

f (i)
θ (i) corresponds to the function describing the i-th layer. The resulting parameters θ of the whole model are defined as the combination of parameters θ (i) of each layer. The number of layers gives the depth d of the model. An illustration of an MLP with two hidden layers is given in Figure 2.4. A specificity of this architecture is that for a given layer, each neuron is connected to every node of the next layer, leading to many connections, and these layers are also called fully connected. Nonlinearities can be introduced between layers through activation functions, but they make the optimization of most loss functions a non-convex problem. Popular choice of activation functions for hidden layers are the Rectified Linear Unit (ReLU) [START_REF] Jarrett | What is the best multi-stage architecture for object recognition?[END_REF] functions.

Gradient-Based Learning

Loss Functions and Predictive Modeling Problems

An important aspect of the design of neural networks is the choice of the loss function, which is tightly coupled with the choice of the activation function of the output layer. Suppose that the network provides a set of hidden features h, from the last hidden layer. Depending on the prediction problem that the network must perform, the role of the output layer is to provide an additional transformation to the features h to follow the correct framing of the problem. The loss function then computes the error depending on the predictions and labels.

Regression Problems

In a regression problem, the network must predict an ndimensional vector y ∈ R n , which can be seen as the mean of a Gaussian distribution. Affine functions are often used in this case:

ŷ = θ (d) h + b.
(2.12)

The error between the output and ground-truth can then be measured using the Mean Squared Error (MSE) loss function:

L M SE (y, ŷ) = y -ŷ 2 2 .
(2.13)

Binary Classification Problems In a binary classification problem, the model has to classify samples as belonging to one of two possible classes. The two classes are often labeled 0 and 1 for simplicity, such that y ∈ {0, 1}. These problems can be cast as predicting a Bernoulli distribution: ŷ = P (y = 1). A popular choice is then to use a logistic sigmoid activation function defined by: (2.15) and the output ŷ is then computed as:

σ : R → R (2.14) σ(x) = 1 1 + e -x ,
ŷ = σ θ (d) h + b . (2.16)
It can be seen as the combination of a linear layer to compute z = θ (d) h + b, also called logit, with a sigmoid activation to convert z into a probability. The usual loss function is then the Binary Cross Entropy (BCE):

L BCE (y, ŷ) = -(y • log ŷ + (1 -y) • log ŷ).
(2.17)

Multi-Class Classification Problems

In a multi-class classification problem, the model must predict a class for an input sample over C possible classes. For a sample with ground-truth class c, the target label is usually either the class number y = c or a vector y ∈ {0, 1} C such that y c = 1, also called one-hot label vector. These problems can be represented as predicting a probability distribution over a discrete variable with C possible values, also called a Multinoulli distribution. The output is a vector ŷ such that ŷc represents the predicted probability of class c: ŷc = P (y = c). A softmax activation function is often used for these problems:

softmax : R C → R C (2.18) softmax(x) i = exp(x i ) C j=1 exp(x j ) . (2.19)
The output ŷ is obtained such that:

∀i ∈ [1, . . . , C], ŷi = softmax θ (d) h + b i .
(2.20)

Similarly to binary classification, the output layer is the combination of a linear layer to compute z = θ (d) h + b with a softmax activation. The Cross Entropy (CE) loss function is used for these problems:

L CE (y, ŷ) = - C i=1 y i • log ŷi . (2.21)
While the affine, sigmoid and softmax activation functions described above are the most common, neural networks can generalize to almost any kind of output layer. The total cost function used to train a neural network will often combine one of a loss function described above with additional regularization terms, such as the weight decay discussed in Section 2.1.6.

Optimization by Backpropagation

Deep Learning models are usually trained iteratively using gradient-based optimizers [Bottou et al., 1998, Kingma and Ba, 2015, Loshchilov and Hutter, 2019] on the training data. The loss functions being non-convex because of non-linear activation functions, the optimization problems can have several local and global minima. The optimization process aims to drive the loss function to a very low value, but has no convergence guarantees and is very sensitive to initialization.

The optimization algorithms typically compute the updates of the parameters based on an expected value of the loss function computed on randomly sampled subsets, or minibatch, of the training data. We usually call batch size the size of each minibatch in the training process. In practice, the choice of the batch size is important and is driven by several factors. On the one hand, large minibatch leads to more accurate gradients, and more different data seen for the same number of training iterations. On the other hand, a small batch size induces a noise that has also a regularization effect during training. However, the batch size is ultimately limited by the hardware capacities.

Stochastic Gradient Descent (SGD) [Bottou et al., 1998] and its variants, with the popular Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], are the most used optimization algorithms in machine learning in general and deep learning in particular. A crucial parameter in these algorithms is the learning rate, i.e. the step size in the direction given by the gradient. In practice, it is necessary to define a function, called scheduler, to decay the learning rate over the course of the training process. Popular choices of scheduler include linear, cosine or step functions.

Computing an analytical expression for the gradient of the model can be straightforward, but numerically evaluating it can be computationally expensive. The backpropagation algorithm [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF]] is a simple procedure to efficiently compute an approximation of the gradients. The general idea of the algorithm is to recursively apply the chain rule of calculus to compute the derivatives of the model as a function formed by the composition of other functions. In practice, back-propagation is integrated and computed by the deep learning frameworks implementing automatic differentiation, such as Pytorch1 or Tensorflow2 . However, gradient-based learning methods with backpropagation also suffers from instabilities in the training process. During each training iteration, the updates of the weights are scaled depending on the current weights. With deep architectures, the scaling accumulates between layers because of backpropagation, and updates can become either very small (vanishing gradients) or very large (exploding gradients) [START_REF] Hochreiter | Gradient flow in recurrent nets: the difficulty of learning long-term dependencies[END_REF]. Vanishing gradients effectively prevent the weights from changing their values, while exploding gradients can result in an unstable training.

Deeper Architectures

Convolutional Neural Networks

Convolutional Neural Networks [START_REF] Lecun | Handwritten digit recognition with a backpropagation network[END_REF], or CNNs, are a specialized kind of MLP for data with a grid-like topology and local information. CNNs are composed of convolutional layers which apply convolution on the input using different filters, or kernels, and pass the result to the next layer. For instance, with two-dimensional input I and kernel K, the convolution operation is defined as:

(I * K)(i, j) = m n I(m, n)K(i -m, j -n).
(2.22)

We usually assume that these kernels are 0 everywhere but a finite set of points. See Figure 2.5 for an illustration of the convolution operation. CNNs have the advantage of working with inputs of variable sizes and implement by design three important concepts:

Sparse connectivity: with a kernel smaller than the input, there are fewer parameters and connections than in MLP layers.

Parameter sharing: the same parameters in the kernel are used for different locations in the input feature map. [START_REF] Goodfellow | Deep Learning[END_REF]. The input and output are also called feature maps. A kernel matrix slides across the input feature map to compute the output feature map.

Figure 2.5. An illustration of the 2D convolution operation used in CNNs from

Equivariance to translation: the convolution operation is invariant to a translation in the inputs since the kernel is translated at different location in the input feature map.

The CNN architectures have been at the heart of many advances in deep learning, their widespread application leading to deep CNNs reaching state of the art in many vision problems starting with AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], with the architecture illustrated in Figure 2.6. Among the most popular ones are the Residual Networks or ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], that implement residual connections illustrated in Figure 2.7, allowing to go deeper in the architectures design and represent even more complex spaces, also called embeddings. Notably, residual connections allow to limit the problem of vanishing gradients [START_REF] He | Deep residual learning for image recognition[END_REF]. They are now an important building block used in almost all recent architectures, notably the Transformers [START_REF] Vaswani | Attention is all you need[END_REF] detailed hereafter. [Krizhevsky, 2009]. Dimensions of the resulting features are written below each layers. The architecture is composed of 5 convolutional layers (conv) followed by 3 fully-connected layers (fc). [START_REF] He | Deep residual learning for image recognition[END_REF]. This is an important building block of the ResNet architecture.

Transformers

The Transformer architecture [START_REF] Vaswani | Attention is all you need[END_REF], illustrated in Figure 2.8, was originally introduced for Natural Language Processing (NLP), but has been recently used and adapted to Computer Vision tasks as well [START_REF] Carion | End-to-end object detection with transformers[END_REF], Dosovitskiy et al., 2021]. Transformer are designed to represent set-to-set functions, i.e. they take a fixed number of tokens as input and outputs the same number of processed tokens. However, there is no particular association between the input tokens and the output tokens.

Before being used by the encoder and decoder, input and output tokens are converted to embedding vectors through simple representation functions such as a linear projection [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF]. For the model to make use of the order of the input sequence, information about the position of each token are injected through -25 -Figure 2.8. The Transformer encoder-decoder architecture from [START_REF] Vaswani | Attention is all you need[END_REF] used for NLP tasks. The left and right halves respectively represent the Transformer encoder and Transformer decoder architectures. positional encoding. There are many choices of positional encodings, learned or fixed [START_REF] Gehring | Convolutional sequence to sequence learning[END_REF]. The original Transformer architecture [START_REF] Vaswani | Attention is all you need[END_REF] is decomposed into two parts, an encoder and a decoder.

The encoder is made of a stack of several (N ) identical layers, with each layer split into two sub-layers: a multi-head self-attention, illustrated in Figure 2.9b, and a position-wise MLP. Each sub-layers are surrounded by residual connections [START_REF] He | Deep residual learning for image recognition[END_REF], followed by layer normalization [START_REF] Ba | Layer normalization[END_REF]. The inputs of the layer are projected into three matrices called Key (K), Value (V ), and Query (Q). The attention operation corresponds to a scaled dot-product between these three matrices:

Attention(Q, K, V ) = softmax( QK √ d )V, (2.23)
with d the dimension of the keys K. The main objective of attention is to focus on the important elements of the task by allowing for an adequate weighting of the inputs. An illustration of the sequencing of operations in attention can be found in Figure 2.9a. [START_REF] Vaswani | Attention is all you need[END_REF]. These operations are computed between the keys K, queries Q and values V matrices.

The projection and computation of the attention are repeated on several sub-layers in parallel, hence the name multi-head. Using different K, Q and V matrices for each head, allows considering different possible representations for the same inputs. The term self-attention is also used in the literature, since the attention function is computed between the input vectors. The outputs of the Multi-head attention layer is then sent to the position-wise MLP. In each layers of the encoder and decoder, the position-wise MLP is applied to each position separately but identically. While the architectures of the MLP are the same across different positions, they use different parameters.

The decoder part is also made of a stack of N identical layers. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. The multi-head attention is also modified by a masking, to prevent positions from attending to subsequent positions.

For Computer Vision, Transformers have been successfully adapted for Image Classification [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] but also for Object Detection [START_REF] Carion | End-to-end object detection with transformers[END_REF], and are now strong contenders to CNNs. We present these computer vision tasks in the next section.

Deep Computer Vision

Image Classification

In Computer Vision, the task of Image Classification consists in predicting a class for a whole image, among multiple predefined classes. Classification is one of the goto problems for evaluating and comparing Deep Learning models, thanks to several benchmarks from black and white handwritten digits in MNIST [LeCun et al., 1998] in the early days of Deep Learning, to tiny natural images in CIFAR [Krizhevsky, 2009], and now large-scale image recognition challenges with ImageNet [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF]. These benchmarks have allowed to quantitatively compare results and improvements between different methods and relatively to baselines. With the advent of large-scale datasets such as the full ImageNet dataset [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF], containing 14M images divided into 21k classes [START_REF] Ridnik | Imagenet-21k pretraining for the masses[END_REF], the Image Classification task on ImageNet serves as the main initialization of models before training on other downstream tasks [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], He et al., 2019]. Improving performance on the classification tasks often serves as a proxy metric for general applicability [START_REF] Kornblith | Do better imagenet models transfer better?[END_REF].

However, this task has its limitations. It does not take into account multiple different objects in a single image, nor their locations, illustrated in Figure 2.10. The model only learns general information from the images, rather than local information as it is done in more dense tasks, i.e. tasks that require a deeper understanding of the image, such as Object Detection.

Object Detection

The goal of Object Detection is to locate and classify every single object appearing in an image, from a pool of known classes objects. The locating task is done by predicting a Bounding Box around the object. Early Object Detectors were based on the computation of local features such as SIFT [Lowe, 2004] or HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] along with more classic ML paradigms, but they are now largely outperformed by Deep Learning models. These more recent object detectors can be separated into three categories, of which we give a brief review below. We refer the reader to [START_REF] Jiao | A survey of deep learning-based object detection[END_REF], Zaidi et al., 2022] for more thorough surveys of modern deep-learning based object detection methods. They all use a backbone network as a feature extractor, typically a ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], then inserts specific additional networks. The backbone is usually pretrained on Image Classification or an unsupervised task [START_REF] He | Rethinking imagenet pretraining[END_REF], before finetuning the full network, i.e. updating the parameters through training, on the Object Detection downstream task. Illustrations of typical object detection models are given in Figures 2.11 and 2.12.

Similarly to Image Classification, Object Detection has several large scale benchmark datasets to compare improvements between methods. The most notorious are the PASCAL VOC [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] and MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] benchmarks.

Two-stage Detectors

These networks [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF], Girshick, 2015, Ren et al., 2015, He et al., 2015, Dai et al., 2016, Lin et al., 2017a], illustrated in Figure 2.11 (a), have a separate module to generate Region Proposals [START_REF] Uijlings | Selective search for object recognition[END_REF]. They find an arbitrary number of objects proposals in the input image during the first stage, and then classify and refine their location in the second. The most representative detector from this category is Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], in which the first stage proposes candidate object bounding boxes using a Region Proposal Network (RPN), and the second stage extracts features from each candidate box by the RoI (Region of Interest) pooling operation for the following classification and regression tasks. As these methods have two separate steps, they generally take more time to generate and process the region proposals, but they have strong detection performance.

One-stage Detectors

These detectors [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF], Redmon and Farhadi, 2017, Liu et al., 2016[START_REF]Focal loss for dense object detection[END_REF], Law and Deng, 2018, Zhou et al., 2019, Tian et al., 2019] classify and locate in a single shot using a dense sampling. They either predefine boxes, or anchors, of various scales and aspect ratios in the feature maps, that are then refined to locate and classify objects [Redmon et al., 2016, Redmon Figure 2.11. Illustration from [START_REF] Jiao | A survey of deep learning-based object detection[END_REF] of the basic architectures of (a) twostage detectors and (b) one-stage detectors, based on deep convolutional networks. The cls and loc outputs correspond respectively to the class and location predictions of objects. The detectors are composed of a backbone network (in yellow), with additional detection-specific layers (in blue).

Figure 2.12. Illustration from [START_REF] Carion | End-to-end object detection with transformers[END_REF] of DETR, the original architecture based on Transformers for end-to-end object detection. The model is also composed of a backbone network (top left), followed by detection-specific transformer encoders and decoders. and Farhadi, 2017, [START_REF] Liu | Ssd: Single shot multibox detector[END_REF], or predict bounding boxes according to reference points in the image [START_REF] Law | Cornernet: Detecting objects as paired keypoints[END_REF], Zhou et al., 2019, Tian et al., 2019]. Onestage methods have a much simpler design than two-stage ones, and have the benefit of real-time inference speed at the cost of lower detection performance.

Transformer-based Detectors Since the introduction of Detection Transformers (DETR) [START_REF] Carion | End-to-end object detection with transformers[END_REF], transformer-based detectors [START_REF] Carion | End-to-end object detection with transformers[END_REF], Zhu et al., 2021, Dai et al., 2021a, Meng et al., 2021, Wang et al., 2021e, Liu et al., 2022a, Yang et al., 2022, Li et al., 2022] have been an increasingly popular architecture. They allow end-to-end detection with a simpler overall architecture and without the need for hand-crafted heuristics such as the Non-Maximal Suppression (NMS) used in Faster R-CNN, or the predefined boxes locations in one-stage detectors. However, Transformers being set-to-set functions, these detectors rely on the Hungarian algorithm [Munkres, 1957] to find the optimal matching between predicted object proposals and the groundtruths, which can be costly to compute at each iterations.

In the next chapter, we now focus on the main problem of interest of this thesis, learning from few labels. Some methods and problems presented in this chapter will be revisited according to this setting.

-31 - This chapter aims at discussing related work in the context of learning with limited labels. The main goal is to give a general overview of the recent background related to this topic while allowing us to present our contributions in the following chapters. The organization of the chapter is as follows. First, we present previous work on data augmentation in Section 3.1, to alleviate label scarcity by increasing diversity. Then we focus on learning frameworks adapted to the limited data setting, Meta-Learning in Section 3.2, specifically designed for FSL, Transfer Learning by Pretraining in Section 3.3 which includes discussion on both pretraining and fine-tuning strategies, and finally Semi-Supervised Learning with few annotations in Section 3.4, that leverages unlabeled data along with few labeled data. In each section, we discuss applications for image classification and object detection separately. [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF].

Figure 3.1. A general taxonomy of data augmentations techniques from

Data Augmentation with few images

The most direct way to deal with the problem of having too few samples, is to artificially increase the diversity in the training dataset through data augmentation procedures. Data augmentation has been a staple since the early days of deep learning [START_REF] Yaeger | Effective training of a neural network character classifier for word recognition[END_REF]. It helps to guide the model into learning invariance properties or have a better estimate of the true data distribution. For a comprehensive overview of data augmentation techniques for computer vision applications, we refer the reader to [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF], and to [START_REF] Wang | Generalizing from a few examples: A survey on few-shot learning[END_REF] for more details in the specific case of few-shot learning. A taxonomy of the different techniques is given in Figure 3.1.

Hand-crafted augmentations

Choosing the best augmentations for a given problem can require guidance from domain experts. Indeed, depending on the data and tasks to solve, some augmentations can be important while others can become inadequate [START_REF] Cubuk | Autoaugment: Learning augmentation strategies from data[END_REF]. Furthermore, Figure 3.2. Illustration from [Chen et al., 2020b] of common hand-crafted augmentations.

the magnitude of the transformation must be chosen carefully. Indeed, every image processing function can result in a label changing transformation at some distortion magnitude [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF]. Common hand-crafted augmentations are based on basic image transformations, illustrated in Figure 3.2, that can be grouped into geometric transformations such as flipping, cropping, rotation, translation, and photometric transformations with color jittering, gaussian blur or grayscale conversion [START_REF] Taylor | Improving deep learning with generic data augmentation[END_REF]. A counterintuitive but effective augmentation technique is mixing images together, illustrated in Figure 3.3. The combination can be done linearly [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF], Tokozume et al., 2018, Inoue, 2018], with non-linear functions [START_REF] Summers | [END_REF]Dinneen, 2019, Takahashi et al., 2019], or through random cropping, patching and pasting training images together [START_REF] Yun | Cutmix: Regularization strategy to train strong classifiers with localizable features[END_REF], Takahashi et al., 2019, Bochkovskiy et al., 2020, Li et al., 2021].

Another interesting procedure is the Random erasing, also called CutOut, augmentation [Zhong et al., 2020, DeVries and[START_REF] Devries | Improved regularization of convolutional neural networks with cutout[END_REF]] and illustrated in Figures 3.2 and 3.4. It works by randomly masking a patch in an image with random pixel values, preventing the model from overfitting on particular visual features in images.

Learning augmentations

In contrast to the above-mentioned transformations that are applied in the input space, augmentations can also be applied in the feature space [Chawla et al., 2002, DeVries andTaylor, 2017a]. To do so, recent methods learn several layers of neural networks for the transformation, using additional unlabeled data [START_REF] Fu | Transductive multi-view zero-shot learning[END_REF], or online during training [Hariharan andGirshick, 2017, Wang et al., 2018b, Schwartz et al., Figure 3.3. Different ways of mixing images. (left) Non-linear vs linear mixing from [START_REF] Summers | Improved mixed-example data augmentation[END_REF]. (right) Mixing with random patch and cropping [START_REF] Takahashi | Data augmentation using random image cropping and patching for deep cnns[END_REF]. 2018, [START_REF] Tseng | Cross-domain few-shot classification via learned feature-wise transformation[END_REF], with one of such method illustrated in Figure 3.5. However, it is very difficult to interpret the augmented data after feature space augmentations. Finally, if transformations are plausible in the image space, they provide greater benefit for improving performance and reducing overfitting than generic augmentations in the feature space [START_REF] Wong | Understanding data augmentation for classification: when to warp?[END_REF].

Limitations of augmentation procedures

One important consideration when choosing augmentation procedures is the intrinsic bias in the initial dataset, since augmented samples are always based on the original samples. While augmentations are designed to alleviate specific biases and prevent overfitting by modifying limited datasets with characteristics of larger data, they cannot overcome all biases present in small datasets and will always reproduce some of them. An illustration of this problem is given in Figure 3.6. Over-extensive use of augmentations can even lead to more overfitting by emphasizing specific features [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF]. Therefore, one cannot only rely on data augmentations for few-shot learning. In FSC, an episode is an Nway K-shot classification problem. We illustrate here 3-way 2-shot episodes. The meta-learning algorithm learns each task at the inner-level and between tasks at the outer-level.

Original image

Meta-Learning

Meta-learning, or learning to learn [START_REF] Thrun | Learning to learn: Introduction and overview[END_REF]], refers to the process of improving a learning algorithm over multiple learning episodes, i.e. a distribution of related tasks, as opposed to multiple data samples in conventional learning frameworks. Meta-learning is separated into an inner-level and outer-level learning phase. In the inner-level, a learning algorithm solves a task, such as image classification, defined by a small dataset and an objective function. In the outer-level, the inner algorithm is updated by an outer algorithm to improve an outer objective, such as generalization performance or learning speed, i.e. the number of iterations and samples required for convergence to a good solution.

More formally, we consider that the model learns over a distribution of tasks P (T ), also called episodes in practice, where each task T t corresponds to K i.i.d. training observations called support set S t = {(x (t,j) , y (t,j) } K j=1 ∈ X K × Y K and q i.i.d. testing observations called query set Q t = {(x (t,j) , y (t,j) )} q j=1 ∈ X q × Y q , all generated by the corresponding distribution µ t over the joint data space X × Y. During meta-training, a task T t is sampled from P (T ) and the model is trained at the inner level to learn the new task from the K support samples. The model is then improved at the outer-level by considering the test error on the unseen query samples from T t . In effect, the test error on episode T t serves as the meta-training error of the meta-learning process. For metatesting, new unseen episodes are sampled from P (T ) and the model's performance is measured on the query samples after learning from the K support samples. In practice meta-learning is usually done through a bi-level optimization procedure. We give an illustration of meta-learning episodes for Few-Shot Classification (FSC) in Figure 3.7.

While meta-learning has become a popular paradigm that successfully applied in areas spanning reinforcement learning [START_REF] Alet | Meta-learning curiosity algorithms[END_REF], hyperparameter optimization [START_REF] Franceschi | Bilevel programming for hyperparameter optimization and meta-learning[END_REF] and neural architecture search [START_REF] Liu | DARTS: Differentiable architecture search[END_REF], we will focus on the few-shot image recognition applications [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF], Snell et al., 2017]. Meta-learning-based approaches are increasingly powerful to train CNNs on small datasets for many vision problems. By learning to solve many tasks with only few training samples, the meta-learned model can more easily adapt to a novel unseen task with few examples as well. We refer the reader to [START_REF] Hospedales | Meta-learning in neural networks: A survey[END_REF] for a more general presentation of deep meta-learning and its applications, with its respective taxonomy given in Figure 3.8, and to [START_REF] Wang | Generalizing from a few examples: A survey on few-shot learning[END_REF] for a detailed review of the FSL applications. Note more details will be provided in Chapter 4 describing our first contribution on FSC. In the following sections, we detail related work applying meta-learning for Few-Shot Image Recognition and Object Detection.

Meta-Learning for Few-Shot Image Recognition

The most common application of meta-learning in computer vision is multi-class FSC, using cross-entropy as the inner and outer loss functions [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], Ravi and Larochelle, 2017, Li et al., 2017, Antoniou et al., 2019, Ye and Chao, 2021, Vinyals et al., 2016, Satorras and Estrach, 2018, Rusu et al., 2018, Lee et al., 2019, Yin et al., 2020, Bertinetto et al., 2019, Raghu et al., 2020]. Meta-learning methods are usually separated in optimization-based [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF], Raghu et al., 2020, Oh et al., 2021], model-based [START_REF] Santoro | Meta-learning with memory-augmented neural networks[END_REF], Qiao et al., 2018a] or metric-based [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], Allen et al., 2019] categories that are briefly presented below.

Figure 3.8.

A taxonomy of Meta-Learning and its applications going beyond FSL, from [START_REF] Hospedales | Meta-learning in neural networks: A survey[END_REF]. Figure 3.9. Illustrations from [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF] of (a) the Optimization-based method MAML, (b) the corresponding pseudo-code.

Optimization-based

Optimization-based methods are literally solving the meta-learning objective as an optimization problem. The most representative example is Model Agnostic Meta-Learning (MAML) [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF], illustrated in Figure 3.9, which aims to learn initialization parameters for novel tasks such that a small number of inner steps produces a classifier that performs well. In this case, the optimization is done end-to-end by gradient descent, but other alternatives also considers SVM [START_REF] Lee | Metalearning with differentiable convex optimization[END_REF], Ridge Regression [START_REF] Bertinetto | Meta-learning with differentiable closed-form solvers[END_REF] or recurrent networks [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF]. However, optimization through multiple inner-steps leads to several computation and memory challenges which make the models difficult to train [START_REF] Antoniou | How to train your MAML[END_REF] and to analyze theoretically [Finn andLevine, 2018, Arnold et al., 2021].

Model-based

Model-based methods, also called black-box methods, wrap the inner learning step in a forward pass of a model designed specifically for learning with few training steps. Typical architectures include convolutional networks [START_REF] Mishra | A simple neural attentive meta-learner[END_REF], hyper-networks [Qiao et al., 2018a, Gidaris and[START_REF] Gidaris | [END_REF] or memory-augmented neural networks [START_REF] Graves | Neural turing machines[END_REF], Santoro et al., 2016, Kaiser et al., 2017, Munkhdalai and Yu, 2017, Cai et al., 2018] which are illustrated in Figure 3.10. While these methods enjoy simpler optimization without second-order gradients compared to optimizationbased methods, they are usually less able to generalize to out-of-distributions tasks and asymptotically weaker as they struggle to embed large training set [START_REF] Finn | Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm[END_REF].

Metric-based

Metric-based methods [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], Allen et al., 2019, Vinyals et al., 2016, Koch et al., 2015, Sung et al., 2018, Bateni et al., 2020] perform non-parametric learning at the inner-level by comparing query points with support samples through similarity functions, and metric learning at the outer-level to find a good embedding space suited for comparison. The most popular representative is Prototypical Networks (ProtoNet) [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], illustrated in Figure 3.11, which uses Euclidean distance to compare query samples with class prototypes, represented by the average features of support examples for each class. The non-parametric inner-level computations make these methods simpler and faster to train than other methods, but they are also limited to -41 - Training examples taken from the various datasets forming Meta-Dataset from [START_REF] Triantafillou | Meta-dataset: A dataset of datasets for learning to learn from few examples[END_REF].

FSL applications compared to other meta-learning methods that could be potentially applied for other settings.

Datasets used

We present here common datasets used for FSC problems. The first dataset introduced for this problem is Omniglot [START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF], a dataset of 20 instances of 1623 characters from 50 different alphabets. Each image was hand-drawn by different people using Amazon Mechanical Turk. This dataset is still popular in the literature for benchmarks and has the advantage of being lightweight, but it is now considered too simple for modern few-shot learning methods, with state-of-the-art performance of about 99% of accuracy. For more complicated datasets, recent methods are comparing performance on miniImageNet [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF], tiered-ImageNet [START_REF] Ren | Meta-learning for semisupervised few-shot classification[END_REF] or CIFAR-FS [START_REF] Bertinetto | Meta-learning with differentiable closed-form solvers[END_REF] datasets. Both miniImageNet and tiered-ImageNet are constructed by taking subsets of the ImageNet dataset [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF]. The former consists of 100 randomly chosen classes and 600 images for each class, while the latter is chosen such that training classes are semantically unrelated to testing classes to be more complicated, and is composed of 779 165 images divided into 608 classes. CIFAR-FS, which stands for CIFAR100 few-shot, corresponds to classes from CIFAR-100 [Krizhevsky, 2009] randomly separated and with 600 images for each class, to obtain a dataset harder than Omniglot but lighter than miniImageNet.

Recent work introduced Meta-Dataset [START_REF] Triantafillou | Meta-dataset: A dataset of datasets for learning to learn from few examples[END_REF] for a large-scale benchmark consisting of data from 10 different image datasets. Examples from each Figure 3.13. The Few-shot object detection protocol proposed by [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF], as illustrated by [START_REF] Huang | A survey of self-supervised and few-shot object detection[END_REF]. The model is trained on base classes, then finetuned on a combination of base and novel classes from a limited support set.

During testing, the model is evaluated on base and novel class detection.

dataset are shown in Figure 3.12. This benchmark aims to introduce realistic class imbalance with varying number of classes and training set sizes, to test the robustness across the spectrum of low-shot learning and domain discrepancy.

Meta-learning approaches and the episodic formulation of the problem have steadily improved performance in FSC compared to early methods [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF], Koch et al., 2015, Vinyals et al., 2016], leading to the recent interest in this line of work. However, the benefit over simpler methods based on transfer learning is also being challenged [Chen et al., 2020a, Tian et al., 2020c, Chen et al., 2021b], as we will see in Section 3.3.

Meta-Learning for Few-Shot Object Detection

Few-Shot Object Detection (FSOD) is the task of learning to detect new categories of objects in an image using only few training examples per class. The categories of objects are separated into two disjoint sets: the base classes, for which we have access to many training examples, and novel classes, for which we only have few examples, or shots, per class. We give an illustration of the framework introduced by [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF] in Figure 3.13. With Meta-Learning, methods are trained on query images, i.e. the image to detect objects from, using annotated support images, i.e. reference examples of each class to detect. Support images generally have a single bounding-box around the object to detect and are randomly sampled from all base classes during training. Then, a predefined set of base and novel images are used during a finetuning phase and in the evaluation phase. An example of evaluation in one-shot object detection with support and query images is given in Figure 3.14. We refer the reader to [START_REF] Huang | A survey of self-supervised and few-shot object detection[END_REF] for a more detailed survey on FSOD.

Metric-based methods

Similarly to FSC, meta-learning methods for FSOD are based on metric learning. They compute prototypes for each object categories using support examples, and then classifies object proposals according to a similarity function to each representative. The prototypes can be obtained through linear operations [START_REF] Wu | Meta-rcnn: Meta learning for few-shot object detection[END_REF][START_REF]FGN: fully guided network for few-shot instance segmentation[END_REF], Han et al., 2022, Xiao and Marlet, 2020, Qiao et al., 2021] or learned with a separated module [START_REF] Karlinsky | Repmet: Representativebased metric learning for classification and few-shot object detection[END_REF], Kang et al., 2019, Li and Li, 2021, Yan et al., 2019], and the similarity can be computed through an explicit distance function [START_REF] Karlinsky | Repmet: Representativebased metric learning for classification and few-shot object detection[END_REF][START_REF] Wu | Meta-rcnn: Meta learning for few-shot object detection[END_REF], Qiao et al., 2021] or from an attention operation [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF], Li and Li, 2021, Yan et al., 2019, Fan et al., 2020a[START_REF]FGN: fully guided network for few-shot instance segmentation[END_REF], Han et al., 2022, Xiao and Marlet, 2020]. Even though the large majority of the methods presented could technically be used without finetuning, by directly conditioning on the support examples at few-shot evaluation time, in practice, most works find it beneficial to finetune, and in fact many do not even report numbers without finetuning. It shows the necessity found in finetuning, and it is thus difficult to really assess the benefit of the methods without finetuning on target data. [Fan et al., 2020a].

Statistics

Datasets used

We describe here the dominant FSOD benchmarks used in the literature. PASCAL VOC [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] is one of the most popular smaller benchmarks for traditional object detection, and has naturally been adapted for FSOD [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF]. The object categories are separated into 15 base classes and 5 novel classes, and three different splits between classes are usually considered, denoted splits 1, 2 and 3. For base training, training and validation images of the base classes from (PASCAL) VOC2007 and (PASCAL) VOC2012 are used (trainval sets). For finetuning, a fixed subset of the VOC2007 and VOC2012 trainval sets is taken as the support set. The benchmarks consider between 1 and 10 shots which correspond to between 15 and 150 base bounding boxes and between 5 and 50 novel bounding boxes. For evaluation, about 5k images from VOC2007 test set are used. Another popular dataset for object detection, Microsoft Common Object in COntext (COCO) [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], has also been adapted to FSOD [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF]. The object categories are split between 20 novel classes, shared with PASCAL VOC, and 60 base classes. A subset of 5k images from COCO validation set are used for evaluation, denoted val5k, and the remaining training and validation images of COCO are used for training and finetuning, denoted trainvalno5k. A novel dataset, FSOD [Fan et al., 2020a], that we will refer to FSODdataset to avoid confusion, has been introduced specifically for the FSOD problem. The dataset contains 1000 categories with 800 training classes and 200 testing classes, unambiguously separated. The total number of images is around 66k with about 182k bounding boxes. Detailed statistics are given in Table 3.1.

Similarly to FSC, recent work [Wang et al., 2020a, Bar et al., 2022] has shown that simply finetuning a classic detection model through supervised learning on the fewshot dataset after a pretraining phase on a bigger dataset achieves competitive results, compared with more complicated methods tailored for few-shot learning. Furthermore, even with meta-learning, the models learned are mainly doing feature reuse between tasks rather than a rapid learning of each task [START_REF] Raghu | Rapid learning or feature reuse? Towards understanding the effectiveness of MAML[END_REF]. These papers reignited research on Transfer Learning with limited labeled data on the target dataset, both for FSC and FSOD.

Transfer Learning by Pretraining

Transfer Learning is one of the oldest practice in deep learning to deal with novel unseen tasks with limited labeled data [START_REF] Pan | A survey on transfer learning[END_REF]. We focus in this section more specifically on pretraining-based methods, a classical learning paradigm which aims to reuse features learned on an extensive source dataset to a potentially limited target dataset.

Pretraining and Fine-tuning for Few-Shot Learning

Transfer learning can be divided into a first pretraining stage, in which we aim to learn a general representation of the data encoded in a feature extractor, and then a finetuning stage, during which the model is fully or partially updated using the few labeled data available for the target task. The general process in the case of FSC is illustrated in Figure 3.15. Even though early works on FSC had shown large improvements against this simple baseline [START_REF] Vinyals | Matching networks for one shot learning[END_REF], Ravi and Larochelle, 2017, Finn et al., 2017], later works have found that they achieve competitive performance when compared on common grounds, with same training details and also with deeper networks [START_REF] Chen | A closer look at few-shot classification[END_REF], Nakamura and Harada, 2019, Wang et al., 2019], while others have refined the fine-tuning stage specifically for FSL [Gidaris and Komodakis, 2018, Lifchitz et al., 2019, Dhillon et al., 2020, Tian et al., 2020c, Shen et al., 2021]. Several recent works are initializing meta-learning with a pretraining stage, showing that they also benefit from strong pretrained feature extractors [START_REF] Rusu | Meta-learning with latent embedding optimization[END_REF], Ye et al., 2020, Ye and Chao, 2021]. In [Tian et al., 2020c], authors have also investigated using different type of pretrained models, from classical supervised pretraining, to early self-supervised pretraining methods [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], Tian et al., 2020a], and obtained similar results between them, showing the potential of strong pretraining for FSL. More recent works in self-supervised learning are evaluating the fine-tuning performance on limited labeled data [Chen et al., 2020c]. Self-supervised learning have also been used in parallel during the fine-tuning stage to further boost FSL [START_REF] Gidaris | Boosting few-shot visual learning with self-supervision[END_REF]. Using an unsupervised pretraining stage would also alleviate even more the need of labels.

Self-supervised Pretraining

Self-Supervised Learning (Self-SL) is an unsupervised learning procedure in which the data provides its own supervision to learn a good representation. The representation learned can then be transferred and fine-tuned on various downstream tasks such as classification or object detection. Several works have shown that a self-supervised pretrained model can outperform its supervised counterpart [START_REF] Caron | Unsupervised learning of visual features by contrasting cluster assignments[END_REF], Grill et al., 2020, Caron et al., 2021]. Self-SL is based on pretext tasks, i.e. unsupervised tasks that apply transformations to the data, or part of it, in order to provide its supervision.

Early pretext tasks

Early works proposed different pretext tasks from different set of augmentations to learn good data representations, such as instance discrimination [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with exemplar convolutional neural networks[END_REF], patch localization [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF], colorization [START_REF] Zhang | Colorful image colorization[END_REF], solving jigsaw puzzle [START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF], counting [START_REF] Noroozi | Representation learning by learning to count[END_REF] or angle rotation prediction [Gidaris et al., 2018]. These tasks are currently outperformed by Contrastive Learning, which represents now the majority of recent work in Self-SL.

Contrastive Learning

Contrastive Learning is a learning paradigm relying on instance discrimination using a pair of positive views from the same input contrasted with all other negative instances in the batch of images, called negatives [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF], Wu et al., 2018, He et al., -48 -2020, Chen et al., 2020b, Misra and Maaten, 2020, Grill et al., 2020, Caron et al., 2020, Chen et al., 2020c, Chen et al., 2021a, Denize et al., 2023]. The popular InfoNCE loss function [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF] is widely used for contrastive learning:

L InfoNCE = - 1 N N i=1 log exp(z i • z i /τ ) N j=1 exp(z i • z j /τ ) . (3.1)
This loss function computes the pairwise similarity scaled by a temperature parameter τ from a batch of feature vectors B = {(z i , z i ), i ∈ N }, and maximizes agreement between positive pairs of features (z i , z i ) while pushing negative pairs {(z i , z j ), j = i} away. However, InfoNCE [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF] requires a large amount of negative instances to be effective [START_REF] Wang | Understanding contrastive representation learning through alignment and uniformity on the hypersphere[END_REF]. While most methods use a huge batch size [Chen et al., 2020b, Chen et al., 2020c, Chen et al., 2021a[START_REF] Tian | What makes for good views for contrastive learning?[END_REF], the popular MoCo [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF]Improved baselines with momentum contrastive learning[END_REF]] maintains a memory buffer of representations to keep a high number of negatives along with a small batch size. Hard negatives, i.e. negative samples that are difficult for the model to separate from positive ones, are the most important for contrastive learning [START_REF] Cai | Are all negatives created equal in contrastive instance discrimination?[END_REF], but they are also potentially harmful to the training because of the class collision problem [START_REF] Cai | Are all negatives created equal in contrastive instance discrimination?[END_REF], Wei et al., 2021a, Chuang et al., 2020], where semantically close instances can be used as negatives in the loss computation, which damages the quality of the representation learned. Several samplers of negatives have been proposed to alleviate this problem [START_REF] Dwibedi | With a little help from my friends: Nearest-neighbor contrastive learning of visual representations[END_REF], Wang et al., 2021a, Hu et al., 2021]. Other popular methods are based on a siamese architecture, i.e. using two copies of the model that are jointly updated, to perform contrastive learning without the use of negatives by ensuring consistency between the outputs of the two models [START_REF] Grill | Bootstrap your own latent-a new approach to self-supervised learning[END_REF], Caron et al., 2020, Caron et al., 2021, Zbontar et al., 2021, Bardes et al., 2022].

Recent works [START_REF] Zheng | Ressl: Relational self-supervised learning with weak augmentation[END_REF], Denize et al., 2023] have also tackled the class collision problem by introducing the relational aspect between instances.

Pretraining and Fine-tuning for Object Detection

Using pretrained models then finetuning is a popular choice to deal with scarce data settings in Object Detection, since the models can be difficult to train from a random initialization. Figure 3.16 represents a taxonomy of pretraining methods for Few-Shot Object Detection.

The use of pretraining in Object Detection

Backbone pretraining Object detectors trained in practice are rarely starting from a random initialization. The backbone of the models are usually pretrained on ImageNet [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF], before fine-tuning on the target dataset and detection [START_REF] Huang | A survey of self-supervised and few-shot object detection[END_REF].

task [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], He et al., 2019]. Until recently, the standard approach was to pretrain the backbone in fully supervised learning. While pretraining from a large general dataset allows for faster convergence during fine-tuning on a similar dataset, both in terms of number of samples and iterations, it is less helpful for the localization task [START_REF] He | Rethinking imagenet pretraining[END_REF]. Recent unsupervised approaches have started investigating pretraining tasks tailored for object detection by imposing local consistency, either at the pixel or region-level, with the goal of helping the model to locate. These methods respectively propose to match in the representation space the features corresponding to the same location in the input space [O Pinheiro et al., 2020, Xie et al., 2021[START_REF] Wang | Dense contrastive learning for self-supervised visual pre-training[END_REF], or ensure local consistency between features from regions in the image [START_REF] Roh | Spatially consistent representation learning[END_REF], Yang et al., 2021a, Xiao et al., 2021].

Pretraining the overall detection model Due to architecture mismatch between detection and classification models, only the backbone feature extractor can be initialized with pretraining while the detection-specific parts of the models must be learned from scratch. Few approaches in the literature have tackled this problem, by pretraining Figure 3.17. Illustration of SoCo [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF]. The pretraining is specialized to two-stage architectures and use a contrastive learning objective. [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF]. The detector must predict region proposals (f box (v) = b) given by an unsupervised algorithm [START_REF] Uijlings | Selective search for object recognition[END_REF], associated object embeddings (f emb (v) = ẑ) obtained by an existing pretrained self-supervised backbone [START_REF] Caron | Unsupervised learning of visual features by contrasting cluster assignments[END_REF], and object scores (f cat (v) = p), fixed to 1.

Figure 3.18. Illustration of DETReg

detection-specific parts along with the backbone [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF], or independently [START_REF] Dai | Up-detr: Unsupervised pre-training for object detection with transformers[END_REF], Bar et al., 2022]. SoCo [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF], illustrated in Figure 3.17, proposes a contrastive pretraining strategy inspired by BYOL [START_REF] Grill | Bootstrap your own latent-a new approach to self-supervised learning[END_REF], for the overall detection model but specialized to convolutional two-stage detectors. UP-DETR [START_REF] Dai | Up-detr: Unsupervised pre-training for object detection with transformers[END_REF] and DETReg [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF], the latter more recent method illustrated in Figure 3.18, use a fixed pretrained backbone to extract features from patches and pretrain the detector independently by locating and reconstructing the features. Initializing detectors from an overall pretraining leads to even faster convergence and stronger performance with limited labels [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF], Bar et al., 2022]. [Wang et al., 2020a]. Pretraining and fine-tuning detection models represents a strong baseline for FSOD, compared to more complex approaches presented in Section 3.2.2.

Fine-Tuning Detectors

Fine-tuning for Few-Shot Object Detection Similarly to FSC, recent work have shown that simply fine-tuning traditional object detectors with only minor architecture modifications achieves strong results on FSOD [Wang et al., 2020a]. This fine-tuning approach is illustrated in Figure 3.19. Following works have then further refined the fine-tuning stage for FSOD [Wu et al., 2020a, Fan et al., 2021].

Fine-tuning for Low-shot Object Detection Instead of having a distinction between base classes with many examples, and novel classes with few shots, Low-shot Object Detection (LSOD) assumes that the number of examples for all classes is limited. Data scarcity is generally simulated by considering a fraction of the labels of traditional object detection datasets. The most prominent benchmark is miniCOCO, which uses random subsets of 1%, 5% or 10% of the full COCO dataset. Several pretraining methods for object detection report numbers in this setting to compare performance, since comparison of transfer learning on the full COCO dataset may be of limited significance due to the large-scale supervision available [Yang et al., 2021a[START_REF] Wang | Dense contrastive learning for self-supervised visual pre-training[END_REF][START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF], Bar et al., 2022].

Unsupervised pretraining methods presented in this section have the advantage of leveraging large unlabeled data available by providing an initialization of the models more adapted to the target task [START_REF] Reed | Self-supervised pretraining improves self-supervised pretraining[END_REF]. One can further leverage these unlabeled data after fine-tuning, through semi-supervised learning.

Semi-supervised learning with few annotations

Semi-supervised learning is a learning paradigm aiming to train models by using both labeled and unlabeled data. It can improve performance by using additional unlabeled [Yang et al., 2021c].

samples compared to supervised learning algorithms which can only use labeled data, alleviating the need for numerous labels [START_REF] Oliver | Realistic evaluation of deep semi-supervised learning algorithms[END_REF]. One can easily extend fully supervised or unsupervised algorithms to obtain semi-supervised ones. In this section, we will focus on presenting few-annotation learning applications for image classification and object detection, and limit the discussion to consistency regularization and pseudo-labeling as they represent the most successful and promising approaches in semi-supervised learning. We refer the reader to [van Engelen andHoos, 2020, Yang et al., 2021c] for a general overview of the semi-supervised learning paradigm, with a taxonomy of deep learning-based methods given in Figure 3.20.

Few-Annotation Classification

In order to present and explain the key concepts in semi-supervised learning, we begin by the image classification problem. We are interested specifically in Few-Annotation Classification (FAC), in which the model has only access to few labeled examples along with a large number of unlabeled data.

Consistency Regularization

Consistency regularization methods are based on a prior assumption that realistic perturbations of data should not change the output of the model [START_REF] Oliver | Realistic evaluation of deep semi-supervised learning algorithms[END_REF]. These methods follow the general structure of Teacher-Student: the model learns as the student, and generates targets as the teacher to compute a consistency regularization term in the final loss function. The consistency constraint can be formally written as:

E x∈X [R (f θ (x), f θ (t(x)))] , (3.2)
where f θ (x) are the predictions of the student model for input x, f θ (t(x)) are the consistency target of the teacher model f with parameters θ for the same input transformed through t, and R measures the distance between the two vectors which is usually MSE or KL-divergence.

Consistency constraints can be considered at the input-level, by adding perturbations to input examples [START_REF] Rasmus | Semi-supervised learning with ladder networks[END_REF], Pezeshki et al., 2016, Sajjadi et al., 2016[START_REF] Miyato | Virtual adversarial training: a regularization method for supervised and semisupervised learning[END_REF], Xie et al., 2020a], or at the network-level, by averaging the weights [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semisupervised deep learning results[END_REF], Izmailov et al., 2018, Ke et al., 2019], the predictions [START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF] or even by dropping connections [Park et al., 2018, Zhang and[START_REF] Zhang | [END_REF].

Self-Training

Self-Training differs from consistency regularization in that the former relies on high confident predictions and the latter on consistency between rich transformations. Self-Training methods can be based on disagreement [START_REF] Zhou | Semi-supervised learning by disagreement[END_REF] of predictions on unlabeled instances, from different views of the data [START_REF] Qiao | Deep co-training for semi-supervised image recognition[END_REF] or between different models [START_REF] Chen | Tri-net for semi-supervised deep learning[END_REF], or based on Pseudo-labeling by leveraging the model's predictions on unlabeled data [START_REF] Grandvalet | Semi-supervised learning by entropy minimization[END_REF], Lee et al., 2013[START_REF] Xie | Selftraining with noisy student improves imagenet classification[END_REF], Beyer et al., 2019, Chen et al., 2020c, Wang et al., 2021c, Pham et al., 2021]. Self-training has recently been at the heart of a lot of improvements in unsupervised training with the advent of large-scale training and architectures [Chen et al., 2020c, Caron et al., 2021].

Hybrid Methods

Hybrid methods [START_REF] Verma | Interpolation consistency training for semi-supervised learning[END_REF], Berthelot et al., 2019, Li et al., 2020, Berthelot et al., 2020, Sohn et al., 2020a] combine ideas from both self-training and consistency regularization above-mentioned methods. They use input and network transformations along with entropy minimization [START_REF] Berthelot | Mixmatch: A holistic approach to semi-supervised learning[END_REF], distribution alignment [START_REF] Li | Dividemix: Learning with noisy labels as semi-supervised learning[END_REF], Berthelot et al., 2020, Verma et al., 2019] or pseudo-labeling [Sohn et al., 2020a]. These methods achieve currently state-of-the-art performance on various benchmarks in FAC, using as low as a single labeled example for each class with FixMatch [Sohn et al., 2020a] for which an illustration is given in Figure 3.21.

Semi-Supervised Object Detection for Few-Annotation Learning

First Semi-Supervised Object Detection (SSOD) methods were mostly based on selftraining [START_REF] Tang | Large scale semi-supervised object detection using visual and semantic knowledge transfer[END_REF], Wang et al., 2018a, Gao et al., 2019], or consistency regularization [START_REF] Jeong | Consistencybased semi-supervised learning for object detection[END_REF], before combining both. More recent methods

-54 -

.21. Illustration from FixMatch [Sohn et al., 2020a], a popular hybrid method taking inspiration from both pseudo-labeling and consistency regularization. Two differently augmented views are created for each unlabeled example, with one view going to the model as the teacher to create a pseudo-label that will be used to supervise the model as the student with the other view. The teacher model is an independent copy of the student model, updated throughout training by the Exponential Moving Average (EMA) [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semisupervised deep learning results[END_REF] of the student's weights.

[ [START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF], Liu et al., 2021, Tang et al., 2021, Xu et al., 2021] are mainly taking inspiration from hybrid methods presented above, and focusing on two-stage convolutional detectors, presented in Section 2.3.2. They achieve strong performance in FAL for object detection by leveraging additional unlabeled examples along with LSOD training data. A popular recent approach is illustrated in Figure 3.22.

Towards the contributions of this thesis

As can be seen throughout this chapter, the problem of learning with limited labels in Deep Learning spans various topics, and can be tackled from different ways. The most obvious option is to work on the data itself to artificially increase its diversity, but it has its limits. The main research direction is to refine training schemes to deal with limited data or labels. Meta-learning with episodic training, is a popular framework for FSL and originally designed for this setting. Besides, it has found applications for other problems such as Reinforcement Learning [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF] or Continual Learning [START_REF] Gupta | Look-ahead meta learning for continual learning[END_REF]. However, a strong theoretical justification of meta-learning algorithms matching with empirical results is still lacking. Inspired by recent theoretical results in multitask representation learning [START_REF] Du | Few-shot learning via learning the representation, provably[END_REF], Tripuraneni et al., 2020, Wang et al., 2021b] published at the beginning of this thesis, our first research direction investigate the link between meta-learning and multitask representation learning algorithms to leverage strong learning bounds in this setting. Then as novel works challenging meta-learning in favor of fine-tuning approaches was published [START_REF] Chen | A closer look at few-shot classification[END_REF], Raghu et al., 2020, Wang et al., 2020a], we focus on leveraging unlabeled data along with few annotated data in object detection through pretraining and semi-supervised learning to develop the other contributions of this thesis.

-56 -

Introduction

As discussed in the previous chapter, meta-learning is an appealing solution for learning efficient models in the scarce data regime. Indeed, meta-learning aims at producing a model on data coming from a set of (meta-train) source tasks to use it as a starting point for learning successfully a new previously unseen (meta-test) target task. Each of these tasks can be composed of small sets of data samples.

The success of many meta-learning approaches is directly related to their capacity of learning a good representation [START_REF] Raghu | Rapid learning or feature reuse? Towards understanding the effectiveness of MAML[END_REF] from a set of tasks making it closely related to Multi-Task Representation learning (MTR). For this latter, several theoretical studies [Baxter, 2000, Pentina and Lampert, 2014, Maurer et al., 2016, Amit and Meir, 2018, Yin et al., 2020] 1 provided probabilistic learning bounds that require the amount of data in the meta-train source task and the number of meta-train tasks to tend to infinity for it to be efficient. While capturing the underlying general intuition, these bounds do not suggest that all the source data is useful in such learning setup due to the additive relationship between the two terms mentioned above and thus, for instance, cannot explain the empirical success of MTR in few-shot classification (FSC) task. To tackle this drawback, two very recent studies [START_REF] Du | Few-shot learning via learning the representation, provably[END_REF], Tripuraneni et al., 2020] aimed at finding deterministic assumptions that lead to faster learning rates allowing MTR algorithms to benefit from all the source data. Contrary to probabilistic bounds that have been used to derive novel learning strategies for meta-learning algorithms [START_REF] Amit | [END_REF]Meir, 2018, Yin et al., 2020], there has been no attempt to verify the validity of the assumptions leading to the fastest known learning rates in practice or to enforce them through an appropriate optimization procedure.

Violated assumptions

Satisfied assumptions

𝜅(𝐖) ≫ 1 𝜎 𝑚𝑖𝑛 𝜎 𝑚𝑎𝑥

Source tasks

Target tasks learn their optimal feature representations. We provide elements of theoretical analysis that explain this behavior and explain the implications of it in practice. Our work is thus complementary to Wang et al. [START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF] and connects MTR, FSC and Meta-Learning from both theoretical and empirical points of view.

𝐖 = [𝐰 1 , 𝐰 2 , 𝐰 3 ]
2. We show that theoretical assumptions mentioned above can be forced during the training of meta-learning algorithms for both families of considered methods, and that enforcing them leads to better generalization of the considered algorithms for FSC baselines. We further show that our analysis can be extended to a Multi-Task Learning algorithm (MTL [START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF]), and that enforcing the same theoretical assumptions leads to improved results as well.

Before presenting our contributions, we give in the next section some necessary knowledge to introduce the concepts that will be presented in this chapter.

Preliminary Knowledge

We start by recalling the framework of Multi-Task Representation Learning (MTR). Then we provide review of the existing learning bounds and assumptions allowing one to obtain guarantees in this setting. Finally, we describe the meta-learning frameworks considered in the context of this contribution.

Multi-Task Representation Learning Setup

Given a set of T source tasks observed through finite size samples of size n 1 grouped into matrices X t = (x t,1 , . . . , x t,n 1 ) ∈ R n 1 ×d and vectors of outputs y t = (y t,1 , . . . , y t,n 1 ) ∈ R n 1 , ∀t ∈ [[T ]] := {1, . . . , T } generated by their respective distributions µ t , the goal of Multi-Task Representation Learning (MTR) is to learn a shared representation φ belonging to a certain class of functions

Φ := {φ | φ : X → V, X ⊆ R d , V ⊆ R k },
φ, W = arg min φ∈Φ,W∈R T ×k 1 T n 1 T t=1 n 1 i=1 (y t,i , w t , φ(x t,i ) ), (4.1)
where

: Y × Y → R + , with Y ⊆ R, is a loss function.
Once such a representation φ is learned, we want to apply it to a new previously unseen target task observed through a pair (X T +1 ∈ R n 2 ×d , y T +1 ∈ R n 2 ) containing n 2 samples generated by the distribution µ T +1 . We expect that a linear classifier w learned on top of the obtained representation leads to a low true risk over the whole distribution µ T +1 . For this, we first use φ to solve the following problem:

ŵT+1 = arg min w∈R k 1 n 2 n 2 i=1 (y T +1,i , w, φ(x T +1,i ) ). (4.2)
Then, we define the true target risk of the learned linear classifier ŵT+1 as: L( φ, ŵT+1 ) = E(x,y)∼µ T +1 [ (y, ŵT+1 , φ(x) )] and want it to be as close as possible to the ideal true risk L(φ * , w * T +1 ) where w * T +1 and φ * satisfy:

∀t ∈ [[T + 1]] and (x, y) ∼ µ t , y = w * t , φ * (x) + ε, ε ∼ N (0, σ 2 ). (4.3)
Equivalently, most of the works found in the literature seek to upper-bound the excess risk defined as ER( φ, ŵT+1 ) := L( φ, ŵT+1 ) -L(φ * , w * T +1 ).

Learning Bounds and Assumptions

First studies in the context of MTR relied on the probabilistic assumption [Baxter, 2000, Pentina and Lampert, 2014, Maurer et al., 2016, Amit and Meir, 2018, Yin et al., 2020] stating that meta-train and meta-test tasks distributions are all sampled i.i.d. from the same random distribution. This assumption, however, is considered unrealistic as in many learning settings, such as FSC, source and target tasks' data are often given by different draws (without replacement) from the same dataset. In this setup, the above-mentioned works obtained the bounds having the following form:

ER( φ, ŵT+1 ) ≤ O 1 √ n 1 + 1 √ T .
Such a guarantee implies that even with the increasing number of source data, one would still have to increase the number of tasks as well, in order to draw the second term to 0. A natural improvement to this bound was then proposed by [START_REF] Du | Few-shot learning via learning the representation, provably[END_REF] and [START_REF] Tripuraneni | Provable meta-learning of linear representations[END_REF]] that obtained the bounds on the excess risk behaving as

ER( φ, ŵT+1 ) ≤ O C(Φ) n 1 T + k n 2 , (4.4)
where C(Φ) is a measure of the complexity of Φ. Both these results show that all the source and target samples are useful in minimizing the excess risk. Thus, in the FSC regime where target data is scarce, all source data helps to learn well. From a set of assumptions made by the authors in both of these works, we note the following two:

Assumption 1: Diversity of the source tasks The matrix of optimal predictors W * should cover all the directions in R k evenly. More formally, this can be stated as 1), (

κ(W * ) := σ 1 (W * ) σ k (W * ) = O(
where σ i (•) denotes the i th singular value of W * . As pointed out by the authors, such an assumption can be seen as a measure of diversity between the source tasks that are expected to be complementary to each other to provide a useful representation for a previously unseen target task. In the following, we will refer to κ(W) as the condition number for matrix W.

Assumption 2: Consistency of the classification margin The norm of the optimal predictors w * should not increase with the number of tasks seen during metatraining2 . This assumption says that the classification margin of linear predictors should remain constant thus avoiding over-or under-specialization to the seen tasks.

While being highly insightful, the authors did not provide any experimental evidence suggesting that verifying these assumptions in practice helps to learn more efficiently in the considered learning setting. An intuition behind these assumptions can be seen in Figure 4.2. When the assumptions do not hold, the linear predictors can be biased towards a single part of the space and over-specialized to the tasks. The representation learned will not generalize well to unseen tasks. If the assumptions are respected, the linear predictors are complementary and will not under-or over-specialize to the tasks seen. The representation learned can more easily adapt to the target tasks and achieve better generalization.

Meta-Learning Algorithms

We precise now the meta-learning frameworks that are considered in this chapter. Meta-learning algorithms considered below learn an optimal representation sequentially via the so-called episodic training strategy introduced by [START_REF] Vinyals | Matching networks for one shot learning[END_REF], instead of jointly minimizing the training error on a set of source tasks as done in MTR. As introduced in Chapter 3, episodic training mimics the training process at the task scale with each task data being decomposed into a training set (support set S) and a testing set (query set Q). Recently, [Chen et al., 2020a] showed that the episodic training setup used in meta-learning leads to a generalization bounds of O( 1√ T ). This bound is independent of the task sample size n 1 , which could explain the success of this training strategy for FSC in the asymptotic limit. However, unlike the results obtained by [START_REF] Du | Few-shot learning via learning the representation, provably[END_REF] studied in this chapter, the lack of dependence on n 1 makes such a result uninsightful in practice as we are in a finite-sample size setting. This bound does not give information on other parameters to leverage when the task number cannot increase. We now recall two major families of meta-learning approaches below.

Metric-based methods

These methods learn an embedding space in which feature vectors can be compared using a similarity function (usually a L 2 distance or cosine similarity) [START_REF] Koch | Siamese neural networks for one-shot image recognition[END_REF], Vinyals et al., 2016, Snell et al., 2017, Sung et al., 2018, Allen et al., 2019, Li et al., 2019, Simon et al., 2020]. They typically use a form of contrastive loss as their objective function, similarly to Neighborhood Component Analysis (NCA) [START_REF] Goldberger | Neighbourhood components analysis[END_REF] or Triplet Loss [START_REF] Hoffer | Deep metric learning using triplet network[END_REF]. In this chapter, we focus our analysis on the popular Prototypical Networks [START_REF] Snell | Prototypical networks for few-shot learning[END_REF] (ProtoNet) that computes prototypes as the mean vector of support points belonging to the same class:

c i = 1 |S i | s∈S i φ(s)
, with S i the subset of support points belonging to class i.

ProtoNet minimizes the negative log-probability of the true class i computed as the softmax over distances to prototypes c i :

L proto (S, Q, φ) := E q∼Q -log exp(-d(φ(q), c i )) j exp (-d(φ(q), c j )) (4.6)
with d being a distance function used to measure similarity between points in the embedding space.

Gradient-based methods These methods learn through end-to-end or two-step optimization [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF], Nichol et al., 2018, Li et al., 2017, Oreshkin et al., 2018, Lee et al., 2019, Bertinetto et al., 2019, Park and Oliva, 2019, Chen et al., 2020e, Raghu et al., 2020] where given a new task, the goal is to learn a model from the task's training data specifically adapted for this task. MAML [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF] updates its parameters θ using an end-to-end optimization process to find the best initialization such that a new task can be learned quickly, i.e. with few examples.

More formally, given the loss t for each task t ∈ [[T ]], MAML minimizes the expected task loss after an inner loop or adaptation phase, computed by a few steps of gradient descent initialized at the model's current parameters:

L M AM L (θ) := E t∼η [ t (θ -α∇ t (θ))], (4.7)
with η the distribution of the meta-training tasks and α the learning rate for the adaptation phase. For simplicity, we take a single step of gradient update in this equation.

In what follows, we establish our theoretical analysis for the popular methods Pro-toNet and MAML. We add their improved variations respectively called Infinite Mixture Prototypes [START_REF] Allen | Infinite mixture prototypes for few-shot learning[END_REF] (IMP) and Meta-Curvature [START_REF] Park | Meta-curvature[END_REF] (MC) in the experiments to validate our findings. The interested reader can find an introduction to these two more advanced algorithms in Appendix A.2.

Understanding Meta-learning Algorithms through MTR Theory

In this section, we study the behavior of gradient-and metric-based meta-learning algorithms with respect to the theoretical insights from MTR theory.

Link between MTR and Meta-learning

Recently, [START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF] has shown that meta-learning algorithms that only optimize the last layer in the inner-loop, solve the same underlying optimization procedure as multi-task learning. In particular, their contributions have the following implications:

1. For Metric-based algorithms, the majority of methods can be seen as MTR problems. This is true, in particular, for ProtoNet and IMP algorithms considered in this work.

2. In the case of Gradient-based algorithms, such methods as ANIL [START_REF] Raghu | Rapid learning or feature reuse? Towards understanding the effectiveness of MAML[END_REF] and MetaOptNet [START_REF] Lee | Metalearning with differentiable convex optimization[END_REF] that do not update the embeddings during the inner-loop, can be also seen as multi-task learning. However, MAML and MC in principal do update the embeddings even though there exists strong evidence suggesting that the changes in the weights during their inner-loop are mainly affecting the last layer [START_REF] Raghu | Rapid learning or feature reuse? Towards understanding the effectiveness of MAML[END_REF]. Consequently, we follow [START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF] and use this assumption to analyze MAML and MC in MTR framework as well.

3. In practice, [START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF] showed that the mismatch between the multitask and the actual episodic training setup leads to a negligible difference.

In the following section, we start by empirically verifying that the behavior of metalearning methods reveals very distinct features when looked at through the prism of the considered MTR theoretical assumptions. We then set on a quest of explaining the differences in their behavior, leading to novel insights into meta-learning algorithms and interesting open problems for future research.

What happens in practice?

To verify whether theoretical results from MTR setting are also insightful for episodic training used by popular meta-learning algorithms, we first investigate the natural behavior of MAML and ProtoNet when solving FSC tasks on the popular miniIma-geNet [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF], tieredImageNet [START_REF] Ren | Meta-learning for semisupervised few-shot classification[END_REF] and Omniglot [START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF] datasets. The full experimental setup is detailed in Section 4.4.1.

To verify Assumption 1 from MTR theory, we want to compute singular values of W during the meta-training stage and to follow their evolution. In practice, as T is typically quite large, we propose a more computationally efficient solution that is to calculate the condition number only for the last batch of N predictors (with N T ) grouped in the matrix W N ∈ R N ×k that capture the latest dynamics in the learning process. We further note that

σ i (W N W N ) = σ 2 i (W N ), ∀i ∈ [[N ]
] implying that we can calculate the SVD of W N W N (or W N W N for k ≤ N ) and retrieve the singular values from it afterwards. We now want to verify whether w t cover all directions in the embedding space and track the evolution of the ratio of singular values κ(W N ) during training.

For the first assumption to be satisfied, we expect κ(W N ) to decrease gradually during the training thus improving the generalization capacity of the learned predictors and preparing them for the target task. To verify the second assumption, the norm of the linear predictors should not increase with the number of tasks seen during training, i.e., w 2 = O(1) or, equivalently, W 2 F = O(T ) and W N F = O(1). For gradient-based methods, linear predictors are directly the weights of the last layer of the model. Indeed, for each task, the model learns a batch of linear predictors and we can directly take the weights for W N . Meanwhile, metric-based methods do not use linear predictors but compute a similarity between features. In the case of ProtoNet, the similarity is computed with respect to class prototypes that are mean features of the instances of each class. For the Euclidean distance, this is equivalent to a linear model with the prototype of a class acting as the linear predictor of this class [START_REF] Snell | Prototypical networks for few-shot learning[END_REF]. This means that we can apply our analysis directly to the -65 - prototypes computed by ProtoNet. In this case, the matrix W * will be the matrix of the optimal prototypes and we can then take the prototypes computed for each task as our matrix W N .

From Figure 4.3, we can see that for MAML (blue), both W N F (left) and κ(W N ) (middle) increase with the number of tasks seen during training, whereas ProtoNet (red) naturally learns prototypes with a good coverage of the embedding space, and minimizes their norm. Since we compute the singular values of the last N predictors in κ(W N ), we can only compare the overall behavior throughout training between methods. For the sake of completeness, we also compute κ(W) at different stages in the training, provided in Figure 4.4. To do so, we fix the encoder φ T learned after T episodes and recalculate the linear predictors of the T past training episodes with this fixed encoder. We can see that κ(W) of ProtoNet also decreases during training and reach a lower final value than κ(W) of MAML. This confirms that the dynamics of κ(W N ) and κ(W) are similar whereas the values κ(W N ) between methods should not be directly compared. The behavior of κ(W) also validate our finding that ProtoNet learns to cover the embedding space with prototypes. This behavior is rather peculiar as neither of the two methods explicitly controls the theoretical quantities of interest, and still, ProtoNet manages to do it implicitly.

Before confirming this claim through extensive empirical evaluations involving more baseline methods and benchmark datasets, we first prove several results that provide an explanation to the difference of behavior of these two families of methods.

The case of Meta-Learning algorithms

The differences observed above for the two methods call for a deeper analysis of their behavior. To this end, we provide an explanation of why ProtoNet naturally leads to small condition number of the obtained predictors and a consistent behavior of their norm, while for MAML we consider a common simplified learning model leaving the general result as an open problem for future works.

ProtoNet We start by first explaining why ProtoNet learns prototypes that cover the embedding space efficiently. This result is given by the following theorem (cf. Appendix A.3 for the full proof).

Theorem 4.3.1. (Normalized ProtoNet) If ∀i c i = 1, then ∀ φ ∈ arg min φ L proto (S, Q, φ), the matrix of the optimal prototypes W * is well-conditioned, i.e. κ(W * ) = O(1).
Proof sketch. We prove this result by recalling that, similarly to other metric-based methods, L proto is a form of contrastive loss studied in [START_REF] Wang | Understanding contrastive representation learning through alignment and uniformity on the hypersphere[END_REF]. By rewriting L proto and using a part of their Theorem 1 proof on the convergence of the loss, we get the desired result. This theorem explains the empirical behavior of ProtoNet in FSC task: the minimization of its objective function naturally minimizes the condition number when the norm of the prototypes is low. In particular, it implies that norm minimization seems to initiate the minimization of the condition number seen afterwards due to the contrastive nature of the loss function minimized by ProtoNet. We confirm this latter implication through experiments in Section 4.4 showing that norm minimization is enough for considered metric-based methods to obtain the well-behaved condition number and that minimizing both seems redundant.

MAML Unfortunately, the analysis of MAML in the most general case is notoriously harder, as even expressing its loss function and gradients in the case of an overparametrized linear regression model with only 2 parameters requires using a symbolic toolbox for derivations [START_REF] Arnold | When MAML can adapt fast and how to assist when it cannot[END_REF].

To this end, we resort to the linear regression model considered in this latter paper and defined as follows. We assume for all t ∈ [[T ]] that the task parameters θ t are normally distributed with θ t ∼ N (0 d , I d ), the inputs x t ∼ N (0 d , I d ) and the output y t ∼ N ( θ t , x t , 1). For each t, we consider the following learning model and its associated square loss:

ŷt = w t , x t , t = E p(xt,yt|θt) (y t -w t , x t ) 2 . (4.8)
We can now state the following result (the full proof can be found in Appendix A.3). [START_REF] Gladwell | Outliers. Back Bay Books[END_REF]. Consider the learning model from Equation (4.8), let Θ i := [θ i , θ i+1 ] T , and denote by W i 2 the matrix of last two predictors learned by MAML at iteration i starting from w 0 = 0 d . Then, we have that:

Proposition 4.3.1. Let ∀t ∈ [[T ]], θ t ∼ N (0 d , I d ), x t ∼ N (0 d , I d ) and y t ∼ N ( θ t , x t ,
∀i, κ( W i+1 2 ) ≥ κ( W i 2 ), if σ min (Θ i ) = 0. (4.9)
Proof sketch. We use the analytical expression for the t th task gradient given in [START_REF] Arnold | When MAML can adapt fast and how to assist when it cannot[END_REF] for this case. We then show that when starting the optimization process from w 0 = 0 d , the sequence of the obtained linear predictors are interdependent such that W i+1 2 ∝ W i 2 + Θ i . We then use upper and lower bounds for singular values of a sum of matrices to derive the desired result.

This proposition provides an explanation of why MAML may tend to increase the ratio of singular values during the iterations. Indeed, the condition when this happens indicates that the optimal predictors forming matrix Θ i are linearly dependent implying that its smallest singular values becomes equal to 0. While this is not expected to be the case for all iterations, we note, however, that in FSC task the draws from the dataset are in general not i.i.d. and thus may correspond to co-linear optimal predictors. In every such case, the condition number is expected to remain non-decreasing, as illustrated in Figure 4. 3 (left) where for MAML, contrary to ProtoNet, κ(W N ) exhibits plateaus but also intervals where it is increasing. This highlights a major difference between the two approaches: MAML does not specifically seek to diversify the learned predictors, while ProtoNet does. Even though the condition number and the classifier norm both increase for MAML, it still achieves good performance. This means that both assumptions are not necessary to meta-learning. However, verifying them leads to a more favorable framework that allows for better performance as confirmed by our experimental evaluation below. Proposition 4.3.1 gives a hint to the essential difference between the methods studied. On the one hand, ProtoNet constructs its classifiers directly from the data of the current task and they are independent of the other tasks. On the other hand, for MAML, the weights of the classifiers are reused between tasks and only slightly adapted to be specific to each task. This limits the generalization capabilities of the linear predictors learned by MAML since they are based on predictors from previous tasks.

In the following section, we advance this insight further by showing that for some data generating distributions one may learn two data representations that achieve a perfect fidelity to the data generating process but differ drastically in terms of the coverage of the embedding space provided by their linear predictors.

Enforcing the assumptions

So far, we have gathered evidence for the fact that MTR theory seems to be insightful for meta-learning algorithms as well. As satisfying the assumptions from MTR theory is expected to come in hand with better generalization performance, we now study what impact forcing these assumptions may have on the learning process when the optimal predictors involved in the data generating process do not naturally satisfy them.

Why should we force the assumptions ? From the results obtain by [START_REF] Du | Few-shot learning via learning the representation, provably[END_REF], and with the same assumptions, we can easily make appear κ(W * ) to obtain a more explicit bound: Can we force the assumptions ? According to the previous result, satisfying the assumptions from MTR theory is expected to come in hand with better performance. However all the terms involved refer to optimal predictors, that we cannot act upon. Thus, we aim to answer the following question:

Proposition 4.3.2. If ∀t ∈ [[T ]], w * t = O(1) and κ(W * ) = O(1), and w T +1 follows a distribution ν such that E w∼ν [ww ] ≤ O 1 k , then ER( φ, ŵT+1 ) ≤ O C(Φ) n 1 T • κ(W * ) + k n 2 . ( 4
Given W * such that κ(W * ) 1, can we learn W with κ( W) ≈ 1 while solving the underlying classification problems equally well?

While obtaining such a result for any distribution seems to be very hard in the considered learning setup, we provide a constructive proof for the existence of a distribution for which the answer to the above-mentioned question is positive in the case of two tasks. The latter restriction comes out of the necessity to analytically calculate the singular values of W but we expect our example to generalize to more general setups and a larger number of tasks as well. Proof sketch. The construction used in this proof is illustrated in Figure 4.5. We define two uniform distributions µ 1 and µ 2 parametrized by a scalar ε > 0 and find a tuple (W * , φ * ) that satisfies the data generating process from Equation (4.3). To this end, φ * is picked as a linear operator that projects the data generated by µ i to a two-dimensional space by discarding its last d -2 dimensions and ensuring that κ(W * ) = 1 ε . The dependence of the ratio on ε allows us to make it arbitrary large. Then, we construct another operator φ that projects the data generated by µ i to a two-dimensional space by discarding the first and last d -1 dimensions. We then show that in this space the matrix of optimal predictors W has a ratio κ( W) ε→0 --→ 1.

The established results show that in some cases even when W * does not satisfy Assumptions 1-2 in the φ * space, it may still be possible to learn a new representation φ such that the optimal predictors in this space do satisfy them. The full proof of this result can be found in Appendix A.3.

w * 1 κ(W * ) ε→0 --→ +∞ w * 2 κ(W * ) ε→0 --→ +∞ Source task 1 in Φ * space Source task 2 in Φ * space ŵ1 κ( W) ε→0 --→ 1 ŵ2 κ( W) ε→0 --→ 1
Source task 1 in Φ space Source task 2 in Φ space . It shows that we can search for a representation φ such that optimal predictors in this space are fulfilling the assumptions, while solving the underlying problem equally well.

How to force the assumptions ? This can be done either by considering the constrained problem: (4.11) or by using a more common strategy that consists in adding κ(W) and W 2 F directly as regularization terms:

φ, W = arg min φ∈Φ,W∈R T ×k 1 T n 1 T t=1 n 1 i=1 (y t,i , w t , φ(x t,i ) ) s.t. κ(W) = O(1), w t = 1,
φ, W = arg min φ∈Φ,W∈R T ×k 1 T n 1 T t=1 n 1 i=1 (y t,i , w t , φ(x t,i ) ) + λ 1 κ(W) + λ 2 W 2 F . (4.12)
By adding W 2 F in the loss, we force the model to have a low norm on the weights. Since it cannot be put to 0 or below, the model will keep the norm relatively constant instead of increasing it. The second regularizer term is a softer way to apply the constraint on the norm rather than normalizing the weights as in Equation (4.11).

According to Theorem 7.1 from [START_REF] Lewis | Nonsmooth Analysis of Singular Values. Part I: Theory[END_REF], subgradients of singular values function are well-defined for absolutely symmetric functions. In our case, we are computing the squared singular values σ 2 (W) and we retrieve the singular values by taking the square root. This means that effectively, we are computing κ(W) = max(|σ(W)|)/ min(|σ(W)|), which is an absolutely symmetric function. Consequently, subgradients of the spectral regularization term κ(W) are well-defined and can be optimized efficiently when used in the objective function.

To the best of our knowledge, such regularization terms based on insights from the advances in MTR theory have never been used in the literature before. We also further use the basic quantities involved in the proposed regularization terms as indicators of whether a given meta-learning algorithm naturally satisfies the assumptions ensuring an efficient meta-learning in practice or not.

Positioning with respect to Previous Work

Below, we discuss several related studies aiming at improving the general understanding of meta-learning, and mention other regularization terms specifically designed for meta-learning.

Understanding meta-learning While a complete theory for meta-learning is still lacking, several recent works aim to shed light on phenomena commonly observed in meta-learning by evaluating different intuitive heuristics. For instance, [START_REF] Raghu | Rapid learning or feature reuse? Towards understanding the effectiveness of MAML[END_REF] investigate whether MAML works well due to rapid learning with significant changes in the representations when deployed on target task, or due to feature reuse where the learned representation remains almost intact. They establish that the latter factor is dominant and propose a new variation of MAML that freezes all but task-specific layers of the neural network when learning new tasks. In [START_REF] Goldblum | Unraveling meta-learning: Understanding feature representations for few-shot tasks[END_REF], the authors explain the success of meta-learning approaches by their capability to either cluster classes more tightly in feature space (task-specific adaptation approach), or to search for meta-parameters that lie close in weight space to many task-specific minima (full fine-tuning approach). Finally, the effect of the number of shots on the FSC accuracy was studied in [START_REF] Cao | A theoretical analysis of the number of shots in few-shot learning[END_REF] for ProtoNet. More recently, [START_REF] Ye | How to Train Your MAML to Excel in Few-Shot Classification[END_REF] studied the impact of the permutation of labels when training MAML. They show that this key part of meta-training, severely impacts MAML as it does not enjoy permutation invariance. Our work investigates a new aspect of meta-learning that has never been studied before and, unlike [START_REF] Raghu | Rapid learning or feature reuse? Towards understanding the effectiveness of MAML[END_REF], [START_REF] Cao | A theoretical analysis of the number of shots in few-shot learning[END_REF], [START_REF] Goldblum | Unraveling meta-learning: Understanding feature representations for few-shot tasks[END_REF] and [START_REF] Ye | How to Train Your MAML to Excel in Few-Shot Classification[END_REF], provides a more complete experimental evaluation with the two different approaches of meta-learning. Normalization Multiple methods in the literature introduce a normalization of their features either to measure cosine similarity instead of Euclidean distance [Gidaris and Komodakis, 2018, Qi et al., 2018, Chen et al., 2019] or because of the noticed improvement in their performance [START_REF] Wang | SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning[END_REF], Tian et al., 2020c, Wang et al., 2021b]. In this chapter, we proved in Section 4.3.3 above that for ProtoNet prototypes normalization is enough to achieve a good coverage of the embedding space, and we empirically show in Section 4.4 below that it leads to better performance. Since we only normalize the prototypes and not all the features, we do not measure cosine similarity. Moreover, with our Theorem 4.3.1, we give explanations through MTR theory regarding the link between the coverage of the representation space and performance. Common regularization strategies In general, we note that regularization in metalearning (i) is applied to either the weights of the whole neural network [START_REF] Balaji | MetaReg: Towards domain generalization using meta-regularization[END_REF], Yin et al., 2020], or (ii) the predictions [START_REF] Jamal | [END_REF]Qi, 2019, Goldblum et al., 2020] or (iii) is introduced via a prior hypothesis biased regularized empirical risk minimization [START_REF] Pentina | A PAC-Bayesian bound for lifelong learning[END_REF], Kuzborskij and Orabona, 2017, Denevi et al., 2018a[START_REF] Denevi | Learning to learn around a common mean[END_REF], Denevi et al., 2019]. Contrary to the first group of methods and to the weight decay approach [START_REF] Krogh | A simple weight decay can improve generalization[END_REF], we do not regularize the whole weight matrix learned by the neural network but the linear predictors of its last layer. While weight decay is used to avoid overfitting by penalizing large magnitudes of weights, our goal is to keep the classification margin unchanged during the training to avoid over-/under-specialization to source tasks. Similarly, spectral normalization proposed by [Miyato et al., 2018a] does not affect the condition number κ. Second, we regularize the singular values of the matrix of linear predictors obtained in the last batch of tasks instead of the predictions used by the methods of the second group (e.g., using the theoretic-information quantities in [START_REF] Jamal | Task agnostic meta-learning for few-shot learning[END_REF]). Finally, the works of the last group are related to the online setting with convex loss functions only and do not specifically target the spectral properties of the learned predictors.

Impact of enforcing theoretical assumptions

In this section, we investigate the impact of enforcing the aforementioned theoretical assumptions for meta-learning algorithms in practice.

Experimental Setup

We consider the FSC problem on the three benchmarks introduced in Chapter 3 that we recall briefly here:

1. Omniglot [START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF] rotations by multiples of 90 degrees.

2. miniImageNet [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF]] is a dataset made from randomly chosen classes and images taken from the ILSVRC-12 dataset [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF]. The dataset consists of 100 classes and 600 images for each class. The images are resized to 84 × 84 pixels and normalized.

3. tieredImageNet [START_REF] Ren | Meta-learning for semisupervised few-shot classification[END_REF] is also a subset of ILSVRC-12 dataset. However, unlike miniImageNet, training classes are semantically unrelated to testing classes. The dataset consists of 779, 165 images divided into 608 classes.

Here again, the images are resized to 84 × 84 pixels and normalized.

For each dataset, we follow a common experimental protocol used in [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF], Chen et al., 2019]. More specifically, we use a four-layer convolution backbone with 64 filters (C64) as done by [START_REF] Chen | A closer look at few-shot classification[END_REF] optimized with Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]] and a learning rate of 0.001. On miniImageNet and tieredImageNet, models are trained on 60000 5-way 1-shot or 5-shot episodes and on 30000 20-way 1-shot or 5-shot episodes for Omniglot. We use a batch size of 4 and evaluate on the validation set every 1000 episodes. We keep the best performing model on the validation set to evaluate on the test set. The seeds used for all experiments are 1, 10, 100 and 1000. For MAML and MC, we use an inner learning rate of 0.01 for miniImageNet and tieredImageNet, and 0.1 for Omniglot. During training, we perform 5 inner gradient step and 10 step during testing. For all FSC experiments, unless explicitly stated, we use the regularization parameters λ 1 = λ 2 = 1 in Equation (4.12). We measure the performance using the top-1 accuracy with 95% confidence intervals, reproduce the experiments with 4 different random seeds using a single NVIDIA V100 GPU, and average the results over 2400 test tasks.

We also provide experiments with the ResNet-12 architecture backbone [START_REF] Lee | Metalearning with differentiable convex optimization[END_REF]. In this case, we follow the recent practice and initialize the models with the weights pretrained on the entire meta-training set [START_REF] Ye | Few-shot learning via embedding adaptation with set-to-set functions[END_REF], Rusu et al., 2018, Qiao et al., 2018a]. Like in their protocol, this initialization is updated by meta-training with ProtoNet or MAML on at most 20000 episodes, grouping every 100 episodes into an epoch. Then, the best performing model on the validation set, evaluated at every epoch, is kept and the performance on 10000 test tasks is measured. For all experiments with the ResNet-12 architecture, the SGD optimizer with a weight decay of 0.0005 and momentum of 0.9 and a batch of episodes of size 1 are used. For ProtoNet, following the protocol of Ye et al. [START_REF] Ye | Few-shot learning via embedding adaptation with set-to-set functions[END_REF], an initial learning rate of 0.0002, decayed by a factor 0.5 every 40 epochs, is used. For MAML, following Ye et Chao [START_REF] Ye | How to Train Your MAML to Excel in Few-Shot Classification[END_REF], the initial learning rate is set to 0.001, decayed by a factor 0.1 every 20 epochs. The number of inner loop updates are respectively set to 15 and 20 with a step size of 0.05 and 0.1 for 1-shot and 5-shot episodes on the miniImageNet dataset, and respectively 20 and 15 with a step size of 0.001 and 0.05 on the tieredImageNet dataset.

Metric-based Methods

Using normalized class prototypes

Theorem 4.3.1 tells us that with normalized class prototypes that act as linear predictors, ProtoNet naturally decreases the condition number of their matrix. Furthermore, since the prototypes are directly the image features, adding a regularization term on the norm of the prototypes makes the model collapse to the trivial solution which maps all images to 0. To this end, we choose to ensure the theoretical assumptions for metric-based methods (ProtoNet and IMP) only with the prototype normalization:

φ, W = arg min φ∈Φ,W∈R T ×k 1 T n 1 T t=1 n 1 i=1 (y t,i , wt , φ(x t,i ) ), (4.13)
where w = w w are the normalized prototypes. According to Theorem 4.3.1, the normalization of the prototypes makes the problem similar to the constrained problem given in Equation (4.11).

As can be seen in Figures 4.6 and 4.7, the normalization of the prototypes has the intended effect on the condition number of the matrix of predictors. Indeed, κ(W N ) (left) stay constant and low during training, and we achieve a much lower κ(W) (right) than without normalization.

From Figure 4.8, we note that normalizing the prototypes from the very beginning of the training process has an overall positive effect on the obtained performance, and this gain is statistically significant in most of the cases according to the Wilcoxon signed-rank test (p < 0.05) [Wilcoxon, 1945, David and[START_REF] David | [END_REF]. In Table 4.1, we compare the performance obtained against state-of-the-art algorithms behaving similarly to Instance Embedding algorithms [START_REF] Ye | Few-shot learning via embedding adaptation with set-to-set functions[END_REF] such as ProtoNet, depending on the architecture used. Even with a ResNet-12 architecture, the proposed normalization still improves the performance to reach competitive results with the state-of-the-art. On the miniImageNet 5-way 5-shot benchmark, our normalized Pro-toNet achieves 80.95%, better than DSN (78.83%), CTM (78.63%) and SimpleShot (80.02%).

Further enforcing a low condition number on Metric-based methods

To guide the model into learning an encoder with the lowest condition number, we consider adding κ(W N ) as a regularization term in Equation (4.13). In addition to the normalization of the prototypes, this should further enforce the assumption on the condition number. Unfortunately, this latter strategy hinders the convergence of the network and leads to numerical instabilities. It is most likely explained by prototypes being computed from image features which suffer from rapid changes across batches, making the smallest singular value σ N (W N ) close to 0. Consequently, we propose to replace the condition number as a regularization term by the negative entropy of the vector of singular values as follows:

H σ (W N ) := N i=1 softmax(σ(W N )) i • log softmax(σ(W N )) i , (4.14)
where softmax(•) i is the i th output of the softmax function. Since uniform distribution has the highest entropy, regularizing with κ(W N ) or H σ (W N ) leads to a better coverage of R k by ensuring a nearly identical importance regardless of the direction.

We obtain the following regularized optimization problem:

φ, W = arg min φ∈Φ,W∈R T ×k 1 T n 1 T t=1 n 1 i=1 (y t,i , wt , φ(x t,i ) ) + λ 1 H σ (W), (4.15)
where w = w w are the normalized prototypes. In Table 4.2, we report the performance of ProtoNet without normalization, with normalization and with both normalization and regularization on the entropy. Finally, we can see that further enforcing a regularization on the singular values through the entropy does not help the training since ProtoNet naturally learns to minimize the singular values of the prototypes. In Table 4.3, we show that reducing the strength of the regularization with the entropy can help to improve the performance.

Gradient-based Methods

Gradient-based methods learn a batch of linear predictors for each task, and we can directly take them as W N to compute its SVD. In the following experiments, we consider the regularized problem of Equation (4.12) for MAML as well as Meta-Curvature (MC). As expected, the dynamics of W N F and κ(W N ) during the training of the regularized methods remain bounded and the effect of the regularization is confirmed with the lower value of κ(W) achieved (cf. Figure 4.6).

The impact of our regularization on the results is quantified in Figure 4.8 where a statistically significant accuracy gain is achieved in most cases, according to the Wilcoxon signed-rank test (p < 0.05) [Wilcoxon, 1945, David and[START_REF] David | [END_REF]. In Table 4.1, we compare the performance obtained to state-of-the-art gradient-based algorithms. We can see that our proposed regularization is globally improving the results, even with a bigger architecture such as ResNet-12 and with an additional pretraining. On the miniImageNet 5-way 5-shot benchmark, with our regularization MAML achieves 82.45%, better than TADAM (76.70%), MetaOptNet (78.63%) and MATE with MetaOptNet (78.64%). In Table 4.4, we compare the performance of MAML without regularization (λ 1 = λ 2 = 0), with a regularization on the condition number κ(W N ) (λ 1 = 1 and λ 2 = 0), on the norm of the linear predictors (λ 1 = 0 and λ 2 = 1), and with both regularization terms (λ 1 = λ 2 = 1) on Omniglot and miniImageNet. We can see that both regularization terms are important in the training and that using only a single term can be detrimental to the performance.

Multi-Task learning Methods

We implement our regularization on a recent Multi-Task Learning (MTL) method [START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF], following the same experimental protocol. The objective is to empirically validate our analysis on a method using the MTR framework. As mentioned in Section 4.3.5, the authors introduce feature normalization in their method, speculating that it improves coverage of the representation space [START_REF] Wang | Understanding contrastive representation learning through alignment and uniformity on the hypersphere[END_REF]. Using their code, we reproduce their experiments on three different settings compared in Table 4.5: the vanilla MTL, the MTL with feature normalization, and MTL with our proposed regularization on the condition number and the norm of the linear predictors. We use λ 1 = 1 in all the settings, and λ 2 = 1 in the 1-shot setting and λ 2 = 0.01 in the 5-shot settings. Our regularization, as well as the normalization, globally improve the performance over the non-normalized models. Notably, our regularization is the most effective when there is the less data which is well-aligned with the MTR theory in few-shot setting. We can also note that in most of the cases, the normalized models and the regularized ones achieve similar results, hinting that they may have a similar effect. Table 4.6 presents the effect of varying independently either parameter λ 1 or λ 2 in the regularization, the other being fixed to 1. From these results, we can see that performance is much more impacted by the condition number regularization (parameter λ 1 ) than by the normalization (parameter λ 2 ). Indeed, varying the regularization weight can lead from the lowest accuracy (74.64%, for λ 1 = 0) to one of the highest accuracies (76.15% for λ 1 = 0.2).

All of these results show that our analysis and our proposed regularization are also valid in the MTL framework.

Out-of-domain Analysis

In the theoretical MTR framework, one additional critical assumption made is that the task data marginal distributions are similar dataset. In this setting, we do not have the same guarantees that our regularization or normalization schemes will be as effective as in same-domain. To verify this, we measure out-of-domain performance on the CropDiseases dataset [START_REF] Mohanty | Using deep learning for image-based plant disease detection[END_REF] adopted by [START_REF] Guo | A broader study of cross-domain few-shot learning[END_REF]. Following their protocol, this dataset is used only for testing purposes. In this specific experiment, evaluated models are trained on miniImageNet.

On the one hand, for metric-based methods, the improvement in the same-domain setting does not translate to the cross-domain setting. From Figure 4.9, we can see that even though the low condition number in the beginning of training leads to improved early generalization capabilities of ProtoNet, this is not the case for IMP. We attribute this discrepancy between ProtoNet and IMP to a difference in cluster radius parameters of IMP and normalized IMP, making the encoder less adapted to out-ofdomain features. On the other hand, we found that gradient-based models keep their accuracy gains when evaluated in cross-domain setting with improved generalization capabilities due to our regularization. This can be seen on Figure 4.9, where we achieve an improvement of about 2 percentage points (p.p.) for both MAML and MC models on both 1-shot and 5-shot settings.

These results confirm that minimizing the norm and condition number of the linear predictors learned improves the generalization capabilities of meta-learning models. As opposed to metric-based methods which are already implicitly doing so, the addition of the regularization terms for gradient-based methods leads to a more significant improvement of performance in cross-domain.

Conclusion

In this chapter, we studied the validity of the theoretical assumptions made in recent papers of Multi-Task Representation Learning theory when applied to popular metric-and gradient-based meta-learning algorithms. We found a striking difference in their behavior and provided both theoretical and experimental arguments explaining that metric-based methods satisfy the considered assumptions, while gradient-based don't. We further used this as a starting point to implement a regularization strategy ensuring these assumptions and observed that it leads to faster learning and better generalization.

While this work proposes an initial approach to bridging the gap between theory and practice for Meta-Learning, some questions remain open on the inner workings of these algorithms. Considering settings different from the episodic one represents another important line of research for the understanding of few-shot learning [Tian et al., 2020c[START_REF]Free lunch for few-shot learning: Distribution calibration[END_REF]]. In the remaining chapters, we are interested in the more challenging Object Detection task that requires to take into account the location of objects along with their classification. For object detectors in FSL, recent work have found that pretraining and fine-tuning are more effective than episodic approaches [Wang et al., 2020a, Bar et al., 2022], which motivated the development presented in the subsequent chapters. [START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF] when varying either λ 1 or λ 2 , the other being fixed to 1, on the miniImageNet 5-way 5-shot benchmark. All accuracy results (in %) are averaged over 2000 test episodes on a single random seed.

-85 - The present chapter investigates unsupervised pretraining specifically designed for Object Detection with Transformers. We recall the positioning in the general training pipeline in Figure 5.1. The objective is to propose a strong pretraining strategy for a more efficient fine-tuning of these detectors with few labeled data. The chapter is organized as follows. In Section 5.2, we review previous work on unsupervised pretraining with a focus on application for object detection, and discuss the positioning of our method. Then, in Section 5.3, we introduce our proposed ProSeCo, and detail the localization-aware contrastive loss used in Section 5. 3.3. In Section 5.4, we present results on different benchmarks with a focus on learning with fewer data with several ablation studies. We conclude this chapter in Section 5.5.

Unsupervised Pretraining for Object Detection with Fewer Annotation

-87 -

Introduction

In this chapter, we address the object detection task in the scarce data regime. As shown in the recent literature, unsupervised pretraining which consists in initializing a model with a pretrained backbone, helps train big architectures more efficiently [Chen et al., 2020c, Caron et al., 2020, He et al., 2020]. While gathering data is not difficult in most cases, its labeling is always time-consuming and costly. Pretraining leverages huge amounts of unlabeled data and helps achieve better performance with fewer data and fewer iterations, when finetuning the pretrained models afterwards.

The design of pretraining tasks for dense problems such as Object Detection has to take into account the fine-grained information in the image. Furthermore, complex object detectors contain different specific parts that can be either pretrained independently [START_REF] Xiao | Region similarity representation learning[END_REF], Xie et al., 2021[START_REF] Wang | Dense contrastive learning for self-supervised visual pre-training[END_REF], Hénaff et al., 2021[START_REF] Dai | Up-detr: Unsupervised pre-training for object detection with transformers[END_REF], Bar et al., 2022] or jointly [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF]. The different pretraining possibilities for Object Detection in the literature are illustrated in Figure 5.2. A pretraining of the backbone tailored to dense tasks has been the subject of many recent efforts [START_REF] Xiao | Region similarity representation learning[END_REF], Xie et al., 2021[START_REF] Wang | Dense contrastive learning for self-supervised visual pre-training[END_REF], Hénaff et al., 2021] (Backbone Pretraining), but few have been interested in incorporating the detectionspecific components of the architectures during pretraining [START_REF] Dai | Up-detr: Unsupervised pre-training for object detection with transformers[END_REF], Bar et al., 2022[START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF] (Overall Pretraining). Among them, SoCo [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF] focuses on convolutional architectures and pretrains the whole detector, i.e. the backbone along with the detection heads (approach e. in Figure 5.2), whereas UP-DETR [START_REF] Dai | Up-detr: Unsupervised pre-training for object detection with transformers[END_REF] and DETReg [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF] pretrain only the transformers [START_REF] Vaswani | Attention is all you need[END_REF] in transformer-based object detectors [Carion et al., -88 -2020, Zhu et al., 2021] and keep the backbone fixed (approach c. in Figure 5.2).

Due to the numerous parameters that must be learned and the huge number of iterations needed because of random initialization, pretraining the entire detection model is expensive (Figure 5.2, e.). On the other hand, pretraining only the detection-specific parts with a fixed backbone leads to fewer parameters and allows leveraging strong pretrained backbones already available. However, fully relying on aligning embeddings given by the fixed backbone during pretraining and those given by the detection head, as done in DETReg or UP-DETR, introduces a discrepancy in the information contained in the features (Figure 5.2, c.). Indeed, while the pretrained backbone has been trained to learn image-level features, the object detector must understand object-level information in the image. Aligning inconsistent features hinders the pretraining quality.

In this chapter, we propose our second contribution, Proposal Selection Contrast (ProSeCo), an unsupervised pretraining method using transformer-based detectors with a fixed pretrained backbone. ProSeCo makes use of two models. The first one aims to alleviate the discrepancy in the features by maintaining a copy of the whole detection model. This model is referred to as a teacher in charge of the object proposals embeddings, and is updated through an Exponential Moving Average (EMA) of another student network making the object predictions and using a similar architecture. This latter network is trained by a contrastive learning approach that leverages the high number of object proposals that can be obtained from the detectors. This methodology, in addition to the absence of batch normalization in the architectures, reduces the need for a large batch size. We further adapt the contrastive loss commonly used in pretraining to take into account the locations of the object proposals in the image, which is crucial in object detection. In addition, the localization task is independently learned through region proposals generated by Selective Search [START_REF] Uijlings | Selective search for object recognition[END_REF]. Our contributions are summarized as:

• We propose Proposal Selection Contrast (ProSeCo), a contrastive learning method tailored for pretraining transformer-based object detectors.

• We introduce the information of the localization of object proposals for the selection of positive examples in the contrastive loss to improve its efficiency for pretraining.

• We show that our proposed ProSeCo outperforms previous pretraining methods for transformer-based object detectors on standard benchmarks as well as novel benchmarks.

Before detailing these contributions, we first provide a review of specific related work.

Related Work

Supervised Object Detection with transformer-based architectures As presented in Chapters 2 and 3, Object Detection is an important and extensively researched problem in computer vision [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF], Girshick, 2015, Ren et al., 2015, Redmon et al., 2016, Liu et al., 2016, Lin et al., 2017a, Tian et al., 2019]. It essentially combines object localization and classification tasks. We are particularly interested in this chapter and the following in the recent detector family based on an encoder-decoder architecture using Transformers [START_REF] Vaswani | Attention is all you need[END_REF], the DETR [START_REF] Carion | End-to-end object detection with transformers[END_REF] family of transformer-based architectures introduced in Chapter 2. The training complexity of this architecture was subsequently improved in Deformable DETR (Def. DETR) [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF], by changing the attention operations into deformable attention, resulting in a faster convergence speed. Several other follow-up works [Dai et al., 2021a, Meng et al., 2021, Wang et al., 2021e, Liu et al., 2022a, Yang et al., 2022, Li et al., 2022] have also focused on increasing the training efficiency of DETR. Transformer-based architectures now represent a strong alternative to traditional convolutional object detectors, reaching better performance for a similar training cost. Furthermore, recent work have shown strong results of transformer-based detectors in a data-scarce setting [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF], compared to convolutional architectures [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], which we also observe and discuss in Chapter 6.

Unsupervised Pretraining for detection models We introduced in Chapter 3 recent pretraining methods tailored for OD, but few of them have tackled the problem of pretraining detection-specific parts of the architectures. We discuss their limitations here. SoCo [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF] proposes a contrastive pretraining strategy for convolutional detectors inspired by BYOL [START_REF] Grill | Bootstrap your own latent-a new approach to self-supervised learning[END_REF]. Object locations are generated using Selective Search [START_REF] Uijlings | Selective search for object recognition[END_REF], and then object-level features are extracted and contrasted with each other using a teacher-student strategy. The small amount of object proposals and object features generated requires using a large batch size for the contrastive loss to be effective. Pretraining the backbone along with the detection modules this way makes the method difficult and costly to train due to the high amount of parameters to learn. For transformer-based architectures, UP-DETR [START_REF] Dai | Up-detr: Unsupervised pre-training for object detection with transformers[END_REF] and DETReg [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF] use a fixed pretrained backbone to extract features respectively from random patch, or from object locations given by Selective Search [START_REF] Uijlings | Selective search for object recognition[END_REF], then pretrain the detector by locating and reconstructing the features of the patch extracted from the input images. However, since the features to reconstruct are obtained by a backbone which was trained to encode image-level information, there is a discrepancy in the information between the features to match.

Our proposed ProSeCo is designed specifically for transformer-based detectors, and use a fixed backbone pretrained for local information. In this work, we leverage the high amount of object proposals generated by transformer-based detectors as instances for contrastive learning. Target object-level features and localizations are provided by a teacher detection model updated through EMA, inspired by recent advances in selfsupervised and semi-supervised learning [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], Grill et al., 2020, Denize et al., 2023[START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF]. The student detection model is pretrained by computing the contrastive loss between object proposals given by the student and teacher detectors. The large number of proposals generated by transformer-based detectors alleviates the need for a large batch size for the contrastive loss. The contrastive loss function used is further improved by introducing the location of objects for selecting positive proposals.

Overview of the approach

We present in this section our proposed unsupervised pretraining approach, illustrated in Figure 5.3, beginning with the data processing pipeline. Then, we detail the contrastive loss used with the localization-aware positive object proposal selection. The transformer-based detectors are built on a general architecture that consists of a back--91 - bone encoder (e.g. a ResNet-50), followed by several transformers encoder and decoder layers, and finally two prediction heads for the bounding boxes coordinates and class logits [START_REF] Carion | End-to-end object detection with transformers[END_REF], Zhu et al., 2021].

Data processing pipeline

Throughout the section, we assume to have sampled a batch of unlabeled images x = {x i } N b i=1 , with x i the i th image and N b the batch size. For each input image x i , we compute two different views with two asymmetric distributions of augmentations 1 T 1 and T 2 : a weakly augmented view x i = t 1 (x i ), with t 1 ∼ T 1 , and a strongly augmented view obtained from the weakly augmented one

x i = t 2 (t 1 (x i )), with t 2 ∼ T 2 .
To guide the model into discovering locations of objects in unlabeled images, we use 1 The exact sets of augmentations chosen are given in Table 5.1. -92 -bounding boxes obtained from the Selective Search algorithm [START_REF] Uijlings | Selective search for object recognition[END_REF], similarly to previous work [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF], Bar et al., 2022]. Since Selective Search is deterministic and the generated boxes are not ordered, we compute the boxes for all images offline and, at training time, randomly sample

K boxes b SS i = {b SS (i,j) ∈ R 4 } K j=1
for each image in the batch. Then, the two views and the corresponding boxes sampled are used as input for our method.

Pretraining method

As shown in Figure 5.3, our approach is composed of a student-teacher architecture [START_REF] Grill | Bootstrap your own latent-a new approach to self-supervised learning[END_REF], Denize et al., 2023[START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF]. With ProSeCo, we extend the student-teacher pretraining for transformer-based detectors to tackle the discrepancy in information-level when aligning features, and introduce a dual unsupervised bipartite matching presented below.

First, the backbone and the detection heads are respectively initialized from pretrained weights and randomly for both the student and teacher models. The teacher model is updated through an Exponential Moving Average (EMA) of the student's weights at every training iteration:

θ t ← αθ t + (1 -α)θ s .
(5.1)

where θ t and θ s are respectively the teacher's and student's weights, and α ∈ [0, 1] is the keep rate parameter. This way, the teacher model represents a slightly delayed but more stable version of the student. When α = 1, the teacher is not updated and remains constant. When α = 0, all weights are updated to be equal to those of the student. Naturally, there is a trade-off between a too high and too low keep rate parameter.

For both models, the classification heads in the detectors are replaced by an MLP, called projector, to obtain latent representations of the objects.

From the weakly augmented view x i , the teacher model provides object proposals

y i = {y (i,j) } N j=1 = {(z (i,j) , b (i,j) } N j=1
, with z (i,j) the latent embedding and b (i,j) the coordinates of the j th object found. The student model infers predictions ŷi = {ŷ (i,j) } N j=1 = {(ẑ (i,j) , b(i,j) } N j=1 from the corresponding strongly augmented view x i .

Then, we apply an unsupervised Hungarian algorithm [Munkres, 1957] for proposal matching to find from all the permutations of N elements S N , the optimal bipartite matching σprop i between the predictions ŷi of the student and the object proposals y i -93 -of the teacher:

σprop i = arg min σ∈S N N j=1
L prop match (y (i,j) , ŷ(i,σ(j)) ).

(5.2)

Therefore, for each image x i , the j th proposal y (i,j) found by the teacher is associated to the σprop i (j) th prediction of the student ŷ(i,σ prop i (j)) . Our matching cost L prop match for the Hungarian algorithm takes into account both features and bounding box predictions through a linear combination of features similarity L sim , the 1 loss of the box coordinates L coord , and the generalized IoU loss L giou from [START_REF] Rezatofighi | Generalized intersection over union: A metric and a loss for bounding box regression[END_REF]:

L sim (z (i,j) , ẑ(i,σ(j)) ) = z (i,j) , ẑ(i,σ(j)) z (i,j) 2 • ẑ(i,σ(j)) 2 (5.3) L coord (b (i,j) , b(i,σ i (j)) ) = b (i,j) -b(i,σ i (j)) 1
(5.4)

L prop match (y (i,j) , ŷ(i,σ(j)) ) = λ sim L sim z (i,j) , ẑ(i,σ(j)) + λ coord L coord b (i,j) , b(i,σ(j)) + λ giou L giou b (i,j) , b(i,σ(j)) .
(5.5)

Similarly, we also use an unsupervised Hungarian algorithm for box matching, to find the optimal bipartite matching σbox i ∈ S N between the predicted boxes bi of the student and the sampled boxes b SS i from Selective Search, using the matching cost L box match :

σbox i = arg min σ∈S N N j=1 L box match (y (i,j) , ŷ(i,σ(j)) ),
(5.6)

L box match (b SS (i,j) , b(i,σ(j)) ) = λ coord L coord b SS (i,j) , b(i,σ(j)) + λ giou L giou b SS (i,j) , b(i,σ(j)) .
(5.7)

Finally, the global unsupervised loss L u used for training is a combination of a loss function between the object latent embeddings of the teacher and student models, and between the object localization of the student predictions and Selective Search boxes.

-94 -More formally, it is computed as:

L u (x) = λ contrast L LocSCE (y, ŷ, σprop ) + 1 N b K N b i=1 K j=1 λ coord L coord b SS (i,j) , b(i,σ box i (j)) + K j=1 λ giou L giou b SS (i,j) , b(i,σ box i (j)) .
(5.8)

In the above equations, we define λ sim , λ coord , λ giou , λ contrast ∈ R + as the coefficients for the different losses. For the consistency in the latent embeddings of the objects, we introduce the object locations information in our contrastive loss L LocSCE . This loss is used to contrast the predictions ŷ = {ŷ i } N b i=1 of the student with object proposals y = {y i } N b i=1 found by the teacher, matched according to the proposal matching σprop =

{σ prop i } N b
i=1 over the batch. We detail the computations behind this loss in the next section.

Localization-aware contrastive loss

Inspired by advances in self-supervised learning, we propose a contrastive objective function [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF], Chen et al., 2020b, He et al., 2020] between object proposals that also maintains relations [START_REF] Zheng | Ressl: Relational self-supervised learning with weak augmentation[END_REF], Denize et al., 2023] among these proposals. This objective function extends the latest SCE [START_REF] Denize | Similarity contrastive estimation for self-supervised soft contrastive learning[END_REF], as well as the original InfoNCE [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF], for instance discrimination between object proposals and is illustrated in Figure 5.4. We compute the contrastive loss between all the object latent embeddings from a batch of image. The positive pair of objects in an image x i is given by the proposal matching σ prop i . First, we define the distributions of similarity between objects embeddings :

p (in,jm) = 1 i =n 1 j =m exp(z (i,j) • z (n,m) /τ t ) N b k=1 N l=1 1 i =k 1 j =l exp(z (i,j) • z (k,l) /τ t ) , (5.9) p (in,jm) = exp(z (i,j) • ẑ(n,m) /τ ) N b k=1 N l=1 exp(z (i,j) • ẑ(k,l) /τ ) .
(5.10)

The distribution p (in,jm) represents the relations between weakly augmented object embeddings scaled by the temperature τ t , and p (in,jm) the similarity between the strongly augmented embeddings and the weakly augmented ones, scaled by τ . Then, the target similarity distribution for the objective function is a weighted combination of one-hot label and the teacher's embeddings relations:

w (in,jm) = λ SCE • 1 i=n 1 j=m + (1 -λ SCE ) • p (in,jm) .
(5.11)

Strong view

Weak view To introduce the location information in our objective, we compute the pairwise Intersection over Union (IoU) between object proposals of the same image and consider overlapping objects as other positives when computing the target similarity distribution: (5.12) where IoU i (j, m) corresponds to the IoU between teacher proposals y (i,j) and y (i,m) found in the same image x i , and δ is the IoU threshold to consider the proposal as a positive example. Finally, we use this tailored target similarity distribution in our Localized SCE (LocSCE) loss:

w Loc (in,jm) = λ SCE • 1 i=n 1 IoU i (j,m)≥δ + (1 -λ SCE ) • p (in,jm) ,
L LocSCE (y, ŷ, σprop ) = - 1 N b N N b i=1 N b n=1 N j=1 N m=1 w Loc (in,jm) log(p (in,j σprop n (m)) ).
(5.13) Note that we do not match the proposals according to σprop in the target similarity, as we compare proposals obtained by the teacher model. We also require the full proposal as input of the loss to compute the pairwise IoU using the box coordinates.

We can also extend InfoNCE the same way:

L InfoNCE (z, ẑ, σprop ) = - 1 N b N N b i=1 N j=1 log exp(z (i,j) • ẑ(i,σ prop i (j)) /τ ) N b k=1 N l=1 exp(z (i,j) • ẑ(k,l) /τ ) . (5.14)
Similarly to our LocSCE, the localization of the objects can be introduced in the InfoNCE loss function to obtain the LocNCE objective as follows:

L LocNCE (y, ẑ, σprop ) = - 1 N b N N b i=1 N j=1 N m=1 log 1 IoU i (j,m)≥δ exp(z (i,j) • ẑ(i,σ prop i (m)) /τ ) N b k=1 N l=1 exp(z (i,j) • ẑ(k,l) /τ ) .
(5.15)

We recover the original formulations of both SCE and InfoNCE when δ = 1. These formulations lead to an effective batch size of N b •N . The localization-aware contrastive loss functions aim to pull together the objects embeddings that overlap subsequently with each others, as they should correspond to the same object in the image.

Experiments

In this section, we present a comparative study of the results of our proposed method on standard and novel benchmarks for learning with fewer data, as well as an ablative study on the most relevant parts. First, we recall the datasets introduced in Chapter 3 with the corresponding evaluation and training settings.

Implementation details

Datasets and evaluation We use different datasets throughout this work that we present here.

• For pretraining purposes, we use the standard ImageNet ILSVRC 2012 (IN) [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF] dataset, which contains 1.2M training images separated in 1000 class categories.

• We also use the MS-COCO (COCO) [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] dataset for pretraining in ablation studies, but mainly for finetuning and evaluation purposes. This dataset contains 80 classes of objects and about 118k training images (train2017 subset).

Performance is evaluated on the val2017 subset.

• For finetuning, we also use the PASCAL VOC 2007and 2012(VOC 07-12) [Everingham et al., 2010] datasets. This dataset contains 20 classes of objects, and we use the combination of the trainval subsets from both VOC2007 and VOC2012 for training, corresponding to about 16k training images in total. Performance is evaluated on the test subset from VOC2007.

• For a more complicated dataset, we use the Few-Shot Object Detection dataset [Fan et al., 2020a]. Since the dataset is designed as open-set, i.e. with different classes between training and testing, we separately use the train and test sets for benchmarking. We separate the test set into a training and testing subset, by randomly taking 80% of images for training and the 20% remaining for testing, and do the same for the train set. We make sure that all classes appears at least once in both training and testing subsets. The images selected for training and testing will be made available for reproducibility. This separation leads to about 11k training images and 3k testing images for the test set and about 42k training images and 10k testing images for the train set.

To evaluate the performance in learning with fewer data, following previous work [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF], Bar et al., 2022], we consider the Mini-COCO benchmarks, where we randomly sample 1%, 5% or 10% of the training data. Similarly, we also introduce the novel Mini-VOC benchmark, in which we randomly sample 5% or 10% of the training data. We also use the Few-Shot Object Detection (FSOD) dataset [Fan et al., 2020a] in novel FSOD-test and FSOD-train benchmarks. In all benchmarks, the image ids selected for training and testing will be made available for reproducibility.

Pretraining We initialize the backbone with the publicly available pretrained SCRL [START_REF] Roh | Spatially consistent representation learning[END_REF] checkpoint and pretrain ProSeCo for 10 epochs on IN. The hyperparameters are set as follows: the EMA keep rate parameter to 0.999, the IoU threshold δ = 0.5, a batch size of N b = 64 images over 8 A100 GPUs, and the coefficients in the different losses λ sim = λ contrast = 2 which is the same value used for the coefficient governing the class cross-entropy in the supervised loss. The projector is defined as a 2-layer MLP with a hidden layer of 4096 and a last layer of 256, without batch normalization. Following SCE [START_REF] Denize | Similarity contrastive estimation for self-supervised soft contrastive learning[END_REF], we set the temperatures τ = 0.1, τ t = 0.07 and the coefficient λ SCE = 0.5. Similarly to DETReg [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF], we sample K = 30 boxes from the outputs of Selective Search for each image at every iteration. Other training and architecture hyperparameters are defined as in Def. DETR [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF] with, specifically, the coefficients λ coord = 5 and λ giou = 2, the number of object proposals (queries) N = 300, and the learning rate is set to lr = 2 • 10 -4 . For weak augmentations T 1 , we use a random combination of flip, resize and crop, and for strong augmentations T 2 , we use a random combination of color jittering, grayscale and Gaussian blur. In T 1 , we resize images with the same range of scales as the supervised training protocol on COCO (Large-scale).

Augmentations used

We detail in Table 5.1 the distributions of augmentations used to create the weak view and the strong view. The Weak Augmentations follow standard supervised training for transformer-based detectors [START_REF] Carion | End-to-end object detection with transformers[END_REF], Zhu et al., 2021]. The Strong Augmentations follow typical contrastive learning augmentations [Chen et al., 2020b, Bar et al., 2022].

Finetuning protocols For finetuning the pretrained models, we follow the standard supervised learning hyperparameters of Def. DETR [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF]. In all experiments, we train the models with a batch size of 32 images over 8 A100 GPUs until the validation performance stops increasing, i.e. for Mini-COCO, up to 2000 epochs for 1%, 500 epochs for 5%, 400 epochs for 10%, for Mini-VOC, up to 2000 epochs for both 5% and 10%, up to 100 epochs for both FSOD-test and PASCAL VOC, and up to 50 epochs for FSOD-train. We always decay the learning rates by a factor of 0.1 after about 80% of the number of training epochs. To compare our method to DETReg [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF] on our novel benchmarks, we use their publicly available checkpoints from github.

Finetuning and transfer learning

Different datasets We evaluate the transfer learning ability of our pretrained model on several datasets. Tables 5.2a and 5.2b present the results obtained compared to previous methods in the literature when learning from fewer labeled data. We can see that our method outperforms state-of-the-art results in unsupervised pretraining on all benchmarks and datasets, and obtain even more strong results when training data is scarce. The improvement is even more significant as the overall performance with few training data is low. When using 5% of the COCO training data (i.e. Mini-COCO 5% in Table 5.2a), corresponding to about 5.9k images, ProSeCo achieves 28.8 mAP, which represents an improvement of +5.2 percentage point (p.p.) over the supervised pretraining baseline and +2 p.p. over both state-of-the-art overall pretraining methods.

Different models Then, we evaluate pretraining with other transformer-based detectors in different settings. In Table 5.3, we fine-tune with DAB-Def. DETR [Liu et al., 2022a] after pretraining with Def. DETR. We observe that ProSeCo still performs strongly when transferring the pretrained weights to other architectures. We can see that using a more consistent pretrained backbone is important for pretraining the overall detection model. In Table 5.4, we pretrain and fine-tune with Conditional DETR [START_REF] Meng | Conditional detr for fast training convergence[END_REF]. We can see that in the most scarce setting, pretraining the detection-specific parts with our ProSeCo is less effective than only pretraining the backbone with SCRL [START_REF] Roh | Spatially consistent representation learning[END_REF]. However, with more labels pretraining the overall detection models improves the performance. We can see that Conditional DETR has globally lower performance in FSL than Def. DETR, which might hinders overall pretraining in the scarcest setting.

Ablation Studies

In the following, we provide several ablation studies for our proposed approach. All experiments and results for the different ablations are compared on the Mini-COCO 5% benchmark with the pretrained SwAV backbone unless explicitly stated. Pretraining dataset and backbone In Table 5.5a, we show the effect of changing the pretraining dataset or the fixed backbone used in our pretraining method. We can see that using a backbone more adapted to dense tasks that learned local information (e.g. SCRL) helps the model by having consistent features (+1 p.p.), compared to a backbone pretrained for global features (e.g. SwAV). Furthermore, even with a less adapted backbone, our ProSeCo initialized with the SwAV backbone outperforms DE-TReg (+1.7 p.p.). [START_REF] Meng | Conditional detr for fast training convergence[END_REF] on different fractions of labeled data.

epochs. We obtain better results when pretraining the model on IN than using COCO thanks to the large number of different images in IN (about 10 times the number of images of COCO, leading to +0.4 p.p.), which is consistent with previous findings [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF]. The difference in the number of images might be slightly compensated in COCO by the closeness to the downstream task and the diversity of objects.

Location information in contrastive loss

In Table 5.5b, we show the effect of the location information in the contrastive loss SCE. We can see that when introducing multiple positive examples for each image based on the IoU threshold δ (i.e. ∀δ < 1), we achieve better results than with the original SCE loss (i.e. δ = 1). Notably, the best results are achieved with δ = 0.5 (+1.7 p.p.). We also compare the two different contrastive objectives considered for pretraining in Table 5.5b. We can see that using the InfoNCE loss leads to slightly better results (+0.3 p.p.). However, when introducing the location information, SCE benefits much more than InfoNCE (+1.7 p.p. compared to +0.6 p.p.). This might be that the selection of positives from the location helps to introduce easy positive examples, and thanks to this, the relational aspect of SCE can focus on the more difficult positives. In the end, LocSCE achieves stronger results than LocNCE (+0.8 p.p.).

Hyperparameters The Table 5.6 presents an ablation study on different important hyperparameters of our approach. We experimented first with the same batch size applied in [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF] (Abl. Batch), but found that using a smaller batch size (Base) leads to improved results (+0.2 p.p.). We evaluated different image scales as a parameter of the weak data augmentations distribution in Abl. Scales. Mid-scale corresponds to a resizing of the images such that the shortest edge is between 320 and 480 pixels, as used in previous work [START_REF] Dai | Up-detr: Unsupervised pre-training for object detection with transformers[END_REF], Bar et al., 2022], and Large-scale to a resize between 480 and 800 pixels, used for supervised learning on COCO. Exact values for these parameters can be found in Table 5.1. We found that increasing the size of the images during pretraining is important to have more meaningful information in the boxes, and a more precise localization of the boxes (+0.7 p.p.). Following the best results from [START_REF] Denize | Similarity contrastive estimation for self-supervised soft contrastive learning[END_REF], we evaluated the performance for τ t ∈ {0.05, 0.07}. We found that τ t = 0.07 leads to the best performance (+0.3 p.p.). We considered several EMA keep rate parameter values following previous work [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF], Denize et al., 2023], and found that 0.999 achieves the best results (+0.1 p.p.).

Increasing the number of queries In Table 5.7, we provide an ablation on the number of object proposals (queries) N in Def. DETR, when pretraining with ProSeCo and finetuning afterwards. A higher N leads to more parameters in the model and longer computing time, but we can see that the results of Def. DETR are relatively stable w.r.t. to the number of queries. On the other hand, ProSeCo benefits from increasing the number of queries since it means a higher effective batch size during contrastive learning. However, the default value of N = 300 leads to the best results both with and without pretraining.

Finetuning with a lot of data In Table 5.8, we present results when finetuning the models on the full COCO dataset (118k training images) under the 1× training schedule [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF], Li et al., 2022], i.e. 12 training epochs and decaying the learning rate in the last epoch. The improvements in the large-scale annotated data regime are limited, which can be observed also in previous work [START_REF] Dai | Up-detr: Unsupervised pre-training for object detection with transformers[END_REF], Bar et al., 2022]. As we can see, after pretraining on IN, our ProSeCo reaches similar results than DETReg [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF]. We believe that this limitation comes from the pretrained backbone that stays fixed during pretraining, and from the extensive

Conclusion

In this chapter, we aim to use large unlabeled datasets for an unsupervised pretraining of the overall detection model to improve performance when having access to fewer labeled data. In this end, we propose Proposal Selection Contrast (ProSeCo), a novel pretraining approach for Object Detection. Our method leverages the large number of object proposals generated by transformer-based detectors for contrastive learning, reducing the necessity of a large batch size, and introducing the location information of the objects for the selection of positive examples to contrast. We show from various experiments on standard and novel benchmarks in learning with few training data that ProSeCo outperforms previous pretraining methods. Throughout this chapter, we advocate for consistency in the level of information encoded in the features when pretraining. Indeed, learning object-level features during pretraining is more important than image-level when applied to a dense downstream task such as Object Detection.

Unsupervised pretraining allows to reduce the total amount of labels used. It improves the performance when fine-tuning on limited data by providing a better initialization of the weights before the fine-tuning phase. In this chapter, we focus on the post-finetuning stage. We recall the positioning in the general training pipeline in Figure 6.1. We want to improve object detectors based on Transformers and trained on few annotated data by leveraging unlabeled data through semi-supervision. The present chapter is organized as follows. In Section 6.2, we review previous work in the topic of object detection and semi-supervised learning and discuss their relations with our proposed method. Then, in Section 6.3, we introduce our proposed MT-DETR and the ideas it is built upon. In Section 6.4, we present the results on different FAL benchmarks and an ablation study of the effect of the different architecture choices. Finally, we conclude the chapter in Section 6.5.

-109 -

Pretraining Phase

Finetuning Phase Semi-Supervised Phase

Chapter 6 We are interested on Semi-Supervised learning for Object Detection.

Introduction

In the previous chapter, we studied how unsupervised pre-training can be beneficial for object detection task. In this new chapter, we consider a complementary setting where we can guide the learning using only a handful of labeled examples while simultaneously leveraging a large amount of unlabeled data. This corresponds to a particular case of semi-supervised learning (SSL) called few-annotation learning (FAL) hereafter.

For the task of Object Detection (OD), methods in the literature that tackle this setting [START_REF] Jeong | Consistencybased semi-supervised learning for object detection[END_REF][START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF], Liu et al., 2021, Xu et al., 2021, Zhou et al., 2021, Tang et al., 2021] have all considered object detectors based on traditional convolutional networks [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] with a set of specific post-processing heuristics required for them to work [START_REF] Hosang | Learning nonmaximum suppression[END_REF], Zhang et al., 2020]. More recent object detectors are based on an encoder-decoder architecture using transformers [START_REF] Vaswani | Attention is all you need[END_REF] that allows for end-to-end OD without depending on this hand-crafted pipeline [START_REF] Carion | End-to-end object detection with transformers[END_REF], Zhu et al., 2021]. However, they have not yet been tested in the SSL context.

The starting point of this contribution is the observation that current state-ofthe-art transformer-based architecture [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF] performs much better than traditional object detectors in a data-scarce fully supervised learning setting, also called Few-Shot Learning (FSL), for an equal number of parameters. However, when plugging it into a state-of-the-art Semi-Supervised Object Detection (SSOD) method [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], we observe that the model fails to converge, meaning that when used as is (Figure 6.2), applying SSL methods from literature to transformer-based object detectors does not guarantee good results. Thus, we propose a novel SSL method tailored for transformer-based architectures in order to take advantage of the effectiveness of transformers in FSL, and upscale these methods for FAL. Our proposed method achieves state-of-the-art in several FAL benchmarks.

More precisely, our contributions are summarized as:

• After showcasing the strong performance of transformer-based detectors using few labeled data, we propose Momentum Teaching DETR (MT-DETR), an approach for SSOD that leverages the specificities of transformer-based architectures and outperforms previous semi-supervised approaches in FAL settings.

• Contrary to convolution-based OD methods, our approach does not rely on heuristics and post-processing for constructing pseudo-labels. Thus, it eliminates sensitive hyperparameters.

Before presenting these contributions in details, we provide an overview of related work close to the setting considered in this chapter.

Related Work

Fully Supervised Object Detection

We have seen in previous Chapters 2 and 5 that OD combines the tasks of object localization and classification. The most popular OD models have been based on fully convolutional neural networks [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF], Ren et al., 2015, Redmon et al., 2016]. These methods can be separated into two-stage [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF], Girshick, 2015, Ren et al., 2015, Lin et al., 2017a] or one-stage [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF], Liu et al., 2016, Tian et al., 2019] detectors. The former methods make predictions of boxes and their class labels based on region proposals, e.g. from a Region Proposal Network (RPN) [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], while the latter make predictions w.r.t. to anchors [START_REF]Focal loss for dense object detection[END_REF] or a grid of possible reference points [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF], Zhou et al., 2019, Tian et al., 2019]. Both categories depend heavily on hand-designed heuristics, with the most prominent example being the Non-Maximal Suppression (NMS) postprocessing, widely used in state-of-the-art OD methods [START_REF] Hosang | Learning nonmaximum suppression[END_REF], Bodla et al., 2017]. Similarly to previous Chapter 5, we remain interested in this chapter to the recent transformer-based architectures. We found in our experiments that Def. DETR is a stronger baseline for FSL than the more popular Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] widely used in previous work, which motivated the focus on the transformer-based architectures. [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] with Feature Pyramid Network (FPN) [Lin et al., 2017a], a two-stage detector commonly used in SSOD methods, and Deformable DETR (Def. DETR) [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF], a state-of-the-art transformer-based object detector, with the same ResNet-50 backbone model. The performances are reported for different percentages (and the corresponding number of images) of COCO and VOC07 labeled training data. See Section 6.4.1 for more details on the experiments. Def. DETR performs better than FRCNN + FPN with fewer labeled data for a similar amount of parameters. † : Results from [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] if available, from our reproduction otherwise.

Semi-supervised Object Detection for Few-Annotation Learning

The goal of semi-supervised learning is to take advantage of unlabeled data along with labeled data during training. In the more specific case of FAL, it allows reducing the need of a large amount of labeled data by leveraging the use of unlabeled data. The problem of SSL in computer vision was historically tackled first for the image classification task as presented Chapter 3. Our work takes inspiration from recent hybrid methods [Sohn et al., 2020a, Chen et al., 2020c] adapted to OD, by training a student model to match the predicted probability distributions of proposals made by a teacher model.

Methods in the literature for SSOD are mainly relying on pseudo-labels provided by a teacher model after applying strong data augmentations on unlabeled data [START_REF] Jeong | Consistencybased semi-supervised learning for object detection[END_REF][START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF], Liu et al., 2021, Xu et al., 2021, Zhou et al., 2021, Tang et al., 2021]. The use of geometric transformations in these strong augmentations is particularly important for OD [START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF], due to the localization task intrinsic to the problem. The most recent and best performing ones [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], Xu et al., 2021, Tang et al., 2021] are also updating the teacher through Exponential Moving Average (EMA) [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] of the student's weights to continuously improve the teacher and, thus, the pseudo-labels given to the student. Although the use of EMA has improved the performance of the models, we propose in our work to stabilize the teacher, by applying an updating strategy throughout training, inspired by recent advances in self-supervised learning [START_REF] Grill | Bootstrap your own latent-a new approach to self-supervised learning[END_REF], Caron et al., 2021]. Pseudolabels are obtained, either by using a hard labeling [START_REF] Jeong | Consistencybased semi-supervised learning for object detection[END_REF][START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF], Liu et al., 2021, Xu et al., 2021, Zhou et al., 2021] approach, which consists in applying an arg max to the predictions, or a soft labeling [START_REF] Tang | Humble teachers teach better students for semi-supervised object detection[END_REF] approach, by fully using the predicted distribution. All the previous methods are relying on NMS and thresholding the confidence scores, i.e. the softmax of the predictions, given by the teacher model. However, the above-mentioned post-processing steps are sensitive to hyperparameters and introduce a bias into the model incentivizing it to be highly confident in its predictions, which may be suboptimal, particularly when few labeled data are available. Therefore, we aim to remove all these post-processing steps in this work. Furthermore, SSOD methods in the literature have been exclusively built and evaluated using two-stage OD architectures, and we found that they do not work as is for the more recent detection models based on transformers.

In this chapter, we investigate SSOD through the lens of FAL, and we focus our experiments in this setting, in contrast to previous work that address FAL with only a limited number of experiments.

A semi-supervised learning approach for transformer-based object detection

In this section, we first motivate our main idea to use a recent state-of-the-art transformerbased OD method in an SSL context by providing several results on both FSL and FAL settings. Then, we present Momentum Teaching DETR (MT-DETR), our transformerbased SSOD method more adapted to FAL and illustrated in Figure 6.3. More specifically, we describe the construction of the pseudo-labels for unlabeled data, and the update scheduling for the teacher model.

How do object detectors handle data scarcity ?

From the results presented in Table 6.1, we can see that Deformable DETR (Def. DETR) [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF], a recent state-of-the-art detection model based on transformers, achieves consistently better performance than the most popular two-stage method in FSL. We refer the reader to Section 6.4.1 for all the implementation details.

These results motivated us to implement Def. DETR in a state-of-the-art SSOD method to see how it performs in FAL settings. We opted for the recent Unbiased Teacher (UBT) [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], as its strong results in FAL were easily reproducible with the provided code. Surprisingly, we observed that with Def. DETR detector, the [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] with Feature Pyramid Network (FPN) [Lin et al., 2017a] and Deformable DETR (Def. DETR) [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF] in the Few-Shot and Few-Annotation Learning settings, using only 1% of labeled data on COCO (about 1180 images). See Section 6.4.1 for experimental details. In the fully supervised case, Def. DETR achieves better results than FRCNN. However, in the semi-supervised case implemented in Unbiased Teacher (UBT) [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], Def. DETR cannot converge.

model does not converge in all the FAL settings tested: 1% of COCO as labeled data (i.e. about 1180 labeled images), 5% and 10% of VOC 07 (i.e. 250 and 500 labeled images respectively). Even though it passes by an early best (about 17% mAP on 1% of COCO) at the beginning of training, the model collapses soon after. This diverging behavior is not satisfying in practice, even more so that the same method used with a Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] architecture converges (it achieves about 20% final mAP on 1% of COCO) in similar settings (c.f. Figure 6.2). All of this shows that current state-of-the-art SSOD methods are not adapted to more recent transformerbased architectures.

Inspired by these results, we propose an SSL method tailored for transformer-based OD called Momentum Teaching DETR (MT-DETR).

Overview of our approach

As shown in Figure 6.3, our approach is composed of a student-teacher architecture, which is common for semi-supervised learning [START_REF] Tarvainen | [END_REF]Valpola, 2017, Sohn et al., 2020a]. Even though the general structure of the approach bears similarities -114 - In the supervised branch ( in dotted and green), the supervised loss L s is computed with the predictions of the student on the labeled images. In the unsupervised branch ( in straight and red), the raw, i.e. unprocessed, outputs of the teacher model for the weakly augmented unlabeled images are used as soft pseudo-labels without applying any heuristic like NMS or confidence thresholding. After finding the best corresponding detection proposals with bipartite matching, the student model learns from the strongly augmented images to match the distribution of class probabilities and the bounding boxes in these pseudo-labels through the unsupervised loss L u .

with our ProSeCo presented in the previous Chapter 5, such as the student-teacher architecture, the strong and weak augmentations, and the dual bipartite matching, there are some important differences coming from the practical data conditions considered.

With ProSeCo, the input data being fully unlabeled, both Hungarian algorithms are unsupervised, and the localization task has to be learned partly from region proposals given by an unsupervised algorithm [START_REF] Uijlings | Selective search for object recognition[END_REF]. This is important to avoid a collapse of the model if the student relies only on the teacher. In the FAL settings that we consider here, the supervision from the labels, even with few of them, stabilizes the training and avoids a collapse of the model. Furthermore, the model has also access to class information from the supervision, such as the total number of class and also reusing classification layers for pseudo-labeling. We describe in this section the full semi-supervised training process.

Construction of the pseudo-labels

As mentioned above, the unsupervised loss L u takes into account the class predictions through a cross-entropy between the outputs of the student model and the matched outputs of the teacher model. We use the softmax of the outputs of the teacher model as soft pseudo-labels for the cross-entropy, as opposed to hard pseudo-labels obtained after taking the arg max.

Following the DETR philosophy [START_REF] Carion | End-to-end object detection with transformers[END_REF], we give to the students the raw soft pseudo-labels obtained from the teacher, i.e. we remove all handmade heuristics to process the teacher outputs, namely, the NMS and confidence thresholding. Both of these post-processing steps are sensitive to hyperparameters and restrict the diversity in the pseudo-labels. By introducing a bias to keep the most confident proposals, they have the unwanted effect of encouraging the models to always be highly confident in their predictions. In the case of FAL, where we have access to only a few labeled examples for each class, the model might not be confident for some classes, leading them to be discarded early by the post-processing. Relying on the model's confidence in certain predictions can be tricky. Using the full distributions makes the model less prone to focus on being highly confident in their predictions, and forces the model to take into account the relations between classes. Furthermore, the Hungarian algorithm used in transformer-based OD methods leverages the diversity of proposals given by the model and benefits from the fact that the model is not overconfident on a single class thanks to the matching loss. Indeed, the bipartite matching can favor proposals with better localizations even if the model is less confident in its class predictions, making the use of raw soft pseudo-labels more suitable for transformer-based detectors.

To obtain strong and insightful pseudo-labels helping the student, the teacher must be updated throughout training. We describe the update process in the following section.

Updating the Teacher model

To avoid a poor supervision from the teacher, its weights θ t are updated by an Exponential Moving Average (EMA) from the student's weights θ s using a keep rate α ∈ [0, 1], which we recall here from Chapter 5:

θ t ← αθ t + (1 -α)θ s .
(6.7)

For α = 1, the teacher is constant and for α = 0 its weights are equal to the student's. Therefore, there is a trade-off between a too high and too low keep rate parameter.

Inspired by the Self-supervised learning literature [START_REF] Grill | Bootstrap your own latent-a new approach to self-supervised learning[END_REF], Caron et al., 2021], we update α following a cosine scheduling from α start to α end : [START_REF] Carion | End-to-end object detection with transformers[END_REF], Zhu et al., 2021]. The Color Jitter, Grayscale, Gaussian Blur and CutOut augmentations follow Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] training, and the geometric augmentations (Rotate, Shear, Rescale, Pad and Translation) follow Soft Teacher [START_REF] Xu | End-to-end semi-supervised object detection with soft teacher[END_REF] training.

α α end -(α end -α start ) • (cos(πk/K) +
with k the current epoch and K the maximum number of epochs. This scheduling stabilizes the teacher model, especially in the last training iterations, to make it converge at the end of training.

Experimental Results

In this section, we present a comparative study of the results of our method to the state-of-the-art on FAL benchmarks, as well as an ablative study on the most relevant parts. Before that, we detail the datasets, the evaluation and training settings used for the different experiments.

with less than 100% of labeled data are reproduced on 3 different random subsets2 .

The training hyperparameters, are defined as in Def. DETR [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF]. The coefficients for the losses are set as λ class = 2, λ 1 = 5, λ giou = 2, and λ u = 4. Following the training schedule of Def. DETR, we always decay the learning rates by a factor of 0.1 after about 80% of training. The keep rate parameter α follows a cosine scheduling from α start = 0.9996 to α end = 1, with the value of α start chosen according to previous work [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF]. When using Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], we follow the official implementation3 and the hyperparameters provided.

Augmentations For strong and weak data augmentations, we follow the common data augmentations used in previous works [START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF], Liu et al., 2021, Xu et al., 2021]. We apply a random resizing and random horizontal flip for weak augmentations. We randomly add color jittering, grayscale, Gaussian blur, CutOut patches for strong augmentations and also randomly add rescaling, translation with padding, shearing and rotating as geometric transformations [START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF] in strong augmentations. In the supervised branch, images are also randomly augmented using weak and strong augmentations without any geometric transformations following Soft Teacher [START_REF] Xu | End-to-end semi-supervised object detection with soft teacher[END_REF] practices. It helps the student model to be augmentationagnostic, to better predict pseudo-labels coming from non-augmented images in the unsupervised branch. We remove the CutOut augmentation in the supervised branch in the most difficult settings of FAL-COCO 0.5% and 1%, since it can cover the only labeled small boxes available and is counterproductive. All the parameters for the different augmentations can be found in Table 6.2.

Results of FAL on COCO and VOC

Tables 6.3 and 6.4 present the results (mAP in %) obtained by our method compared to previous methods in the literature on the FAL-COCO and FAL-VOC 07-12 benchmarks. As can be seen in both tables, our approach is the only one to consider a transformer-based OD architecture (Def. DETR), as opposed to the commonly used two-stage architecture (FRCNN + FPN). When we implemented Def. DETR into Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] (UBT), we found that the model cannot converge in FAL settings (c.f. Figure 6.2).

First, we can see from both tables that our method always improve performance over the corresponding fully supervised FSL baseline (c.f. Table 6.1). With our method, we outperform state-of-the-art results on all labeled fractions of the dataset, and obtain even more strong results specifically when the annotations are scarce: globally about +1 performance point (p.p.) when using 1k or less labeled images, which is even more significant when the overall performance is low. For FAL-COCO with 1% of labeled images, our method achieves a mean of 22.03 mAP, which is about 1.2 p.p., or 6% of improvement over the state-of-the-art, UBT. Notably, on FAL-VOC with 10% of labeled images, we obtain mean performance of 43.15 mAP, corresponding to 2.81 p.p. or 7% of improvement over UBT. We note that our method also outperform the state-of-the-art when using more labeled data, such as with the 100% labeled VOC07 setting, where we improve of about 1.5 p.p. over UBT.

Ablation studies

In Table 6.5, we present an ablation study on the main parts of our approach. We review each ablation below.

EMA scheduling

The effect of the EMA scheduling is compared between the Best and Abl. Sched. rows. We can see that using a cosine scheduling to gradually reduce the EMA keep rate parameter α leads to an improvement of about 0.7 p.p., as opposed to using a constant value for α as done in other SSL approaches [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], Xu et al., 2021, Tang et al., 2021].

Initialization

In this ablation, we study the effect of end-to-end semi-supervised learning [START_REF] Xu | End-to-end semi-supervised object detection with soft teacher[END_REF] in the row Abl. Init. which consists in starting the semi-supervised training from scratch compared to an initialization after Fine-Tuning (FT) in the row Best, in which we initialize both student and teacher models from the weights of the fine-tuned model on the few labeled data. As can be seen in Table 6.5 and contrary to Soft Teacher [START_REF] Xu | End-to-end semi-supervised object detection with soft teacher[END_REF], starting the semi-supervised training from fine-tuned weights is much more effective (about 5.7 p.p. better) than starting from randomly initialized weights, since the teacher model will give useful pseudo-labels to the student from the start of training.

NMS

The importance of removing NMS to avoid filtering interesting pseudo-labels and introducing bias is showcased between the rows Best and Abl. NMS. We can see that, contrary to the common practice when using other detectors [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], Xu et al., 2021, Tang et al., 2021], the introduction of NMS leads to a performance drop of about 2.5 p.p. This is why we used raw pseudo labels, i.e. without any post-processing.

Confidence Thresholding

The effect of introducing a threshold to filter out the pseudo-labels given by the teacher with poor confidence is shown in the rows Best and Abl. Thresh.. We test the results using several common values in the literature (0.5, 0.7 and 0.9) [Sohn et al., 2020a, Liu et al., 2021, Xu et al., 2021]. A value of 0.7 seems to give the best final results (17.34 mAP) between the thresholding variants, but we can see that choosing the best threshold to apply is extremely sensitive. Similarly to Humble Teacher [START_REF] Tang | Humble teachers teach better students for semi-supervised object detection[END_REF], we also found that removing the confidence threshold to use all the soft pseudo-labels, which corresponds to the column with ø, leads to stronger results (22.24 mAP), less sensitivity and fewer hyperparameters.

Data augmentation

In this ablation study, we are interested in the effect of the different sets of data augmentation to generate the strong view in the unsupervised branch. In Table 6.6a, we group similar augmentations together and then study the effects of each group in Table 6.6b. First, we can see that the model does not converge when using the same setting as Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], confirming our observation in Figure 6.2, but achieves strong result when using raw soft pseudo-labels. Furthermore, using more augmentations to generate the strong view leads to better results.

Self-supervised Backbone

In Table 6.3, we also report results when initializing with a Self-supervised pretraining instead of a supervised one before the fine-tuning phase. As is common practice in Object Detection, detectors are usually initialized with the weights of the backbone obtained after a supervised training phase on ImageNet [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], He et al., 2019]. In these experiments, we use instead weights from our fully unsupervised ProS-eCo pretraining on ImageNet, presented in Chapter 5. This reduces even more the total amount of labels used in the overall training.

We can see first that initializing with ProSeCo improves the fine-tuning performance for all labeled fractions of COCO considered, confirming the results from Chapter 5. Then, with semi-supervision with MT-DETR, we observe that the unsupervised pretraining improves over the supervised one on the 5% and 10% fractions, but is less effective on the scarcer 0.5% and 1% fractions. This might be due to an overlapping between the classes contained in both datasets. The model benefits from having seen more examples of some class of object during the supervised pretraining for the most label-scarce benchmarks, since they contain very few examples for each class.

Conclusion

In this work, we experimented in different data scarce settings with the state-of-the-art transformer-based object detector Def. DETR [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF] and showed that it performs much better than the most popular two-stage detector Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] with FPN [Lin et al., 2017a]. Surprisingly, we found that Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], a state-of-the-art SSOD method, did not converge when applied with Def. DETR.

To address this issue, we propose Momentum Teaching DETR (MT-DETR), an SSL approach tailored for OD based on transformers, in order to leverage their good results with few labeled data. Our method is based on a student-teacher architecture and, contrary to common practice, discards all previously used handcrafted heuristics to process pseudo-labels generated by the teacher. These processing steps are sensitive to hyperparameters, and introduce biases with the unwanted effect of forcing the models to be overconfident in their predictions. We show that our proposed MT-DETR outperforms state-of-the-art methods, especially in FAL settings. Then, we combined our unsupervised pretraining with our semi-supervised method to reduce even more the number of labels used. This concludes our contributions in this thesis. In the last chapter we give a summary of the different contributions presented and outline future perspectives for learning with limited labels. [START_REF] Carion | End-to-end object detection with transformers[END_REF], Zhu et al., 2021]. The Photometric augmentations are used in Self-supervised learning and in Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] along with CutOut augmentations. The Geometric augmentations follow Soft Teacher [START_REF] Xu | End-to-end semi-supervised object detection with soft teacher[END_REF] training. (b) Performance comparison of different combination of data augmentations on FAL-COCO 1%. Using all data augmentations leads to the best results (in bold). Furthermore, we can see that the model do not converge when using NMS and confidence thresholding along with Basic, Photometric and CutOut augmentations, which corresponds to the same setting as Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] (in red). Using raw soft pseudo-labels (PL) stabilizes the training and makes it converge to a strong solution (in green).

-126 -

Conclusions and Perspectives

Summary of the contributions

Although Machine Learning (ML) and Deep Learning (DL) systems are becoming increasingly used in research and industry, they require a huge amount of labeled data to be effective. Major breakthroughs have been possible thanks to large-scale labeled datasets such as ImageNet [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF], containing 14M labeled images divided into 21k classes, or MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], with 2.5M labeled instances of 91 classes in 328k images. While images in these datasets represent common objects in natural scenes and considerable performance improvements have been achieved by iterating on them, they can be really semantically far from more specialized tasks which would require specific annotations. Annotating such large-scale datasets are incredibly time-consuming and costly, which represents a high entry cost to apply DL algorithms in real-world settings where there may not be a large amount of labeled data available. This explains why in this thesis, we are working towards Few-Annotation Learning (FAL), i.e. learning efficient models with limited labels while having access to unlabeled data, such that they can still perform well on new data, even if they have only seen a few labeled examples of that data during training. Our contributions are organized in three parts, each detailed in the previous chapters that we summarize below.

In Chapter 4, we investigate the Meta-Learning paradigm that is increasingly popular for Few-Shot Learning (FSL). We are specifically interested in theoretical justifications of the good empirical results, observed through the lens of Multi-task Representation Learning. We provide proofs and a better understanding of the behavior of metalearning algorithms as well as practical ways to force them to follow the best learning settings. To do so, we propose spectral-based regularization terms and normalization schemes to enforce important theoretical assumptions. This leads to more efficient meta-learning as supported by better empirical results.

Then, in Chapter 5, we develop a novel approach for pretraining Transformer-based Object Detectors, Proposal Selection Contrast (ProSeCo), such that every part of the detector can be properly initialized. The pretraining is based on a Proposal-Contrastive Learning framework. Object proposals given by a teacher model are contrasted with predictions of a student model. To introduce the location information that is important in Object Detection, we compute the Intersection over Union between proposals and consider the overlapping ones as positives during contrastive learning. This allows for more sample-efficient learning, as pretrained detectors achieve better performance for fewer annotated samples.

Finally, in Chapter 6, we focus on Semi-Supervised Learning for Object Detection to leverage large-scale unlabeled data along with few annotated samples. We propose the first method specifically designed for Transformer-based detectors, since training with previous approaches does not converge when using few annotated data. We observe that previous semi-supervised methods do not converge in FAL when applied with these detectors. To fix this converging issue, we investigate into different parts of the model and improve methodological aspects such as data augmentation, soft pseudolabels without counterproductive post-processing. When comparing performance on different few annotation learning benchmarks, our proposed method outperforms state of the art with a more significant gap when labels are scarce.

Perspectives and discussions

As mentioned in Chapter 3, research in learning with limited labels can be tackled in various ways. There is still a large room for improvements, and open problems for future works. We discuss potential research directions in this section.

Meta-Learning

While the work presented in Chapter 4 proposes an initial approach to bridging the gap between theory and practice for Meta-Learning and Multi-Task Representation Learning, some questions remain open on the inner workings of these algorithms. In particular, being able to take better advantage of the particularities of the training tasks during meta-training could help improve the effectiveness of these approaches. Further-more, the similarity between source and test tasks was not taken into account in this work, which is an additional assumption in the theory developed in [START_REF] Du | Few-shot learning via learning the representation, provably[END_REF]. Even though we provide a preliminary experimental study using different datasets between meta-training and meta-testing to foster future work on this topic, the theoretical learning bounds have to be updated to take into account the difference in the assumptions.

However, in light of recent works challenging the effectiveness of meta-learning compared to fine-tuning approaches [START_REF] Chen | A closer look at few-shot classification[END_REF], Tian et al., 2020c, Wang et al., 2020a], which require less computation, the benefit of meta-learning and episodic training for few-shot learning remains to be proven. While meta-learning can be interesting for specific problems such as Reinforcement Learning [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF] or Continual Learning [START_REF] Gupta | Look-ahead meta learning for continual learning[END_REF], it is still unsure if it is more advantageous for Few-Shot Learning.

Unsupervised pretraining

Future work for pretraining transformer-based object detectors, as presented in Chapter 5, could update the backbone during pretraining to further improve the consistency between the backbone and the detection heads. Even though it would likely be more costly to do so, it might help to improve performance in the large-scale annotated data settings. However, the benefits of pretrained backbone for downstream tasks are also contested. While they are common practice in object detection [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] and allows achieving good performance with lower supervised training iterations [START_REF] He | Rethinking imagenet pretraining[END_REF], similar performance can still be achieved with random initialization and more supervised training time [He et al., 2019, Newell and[START_REF] Newell | [END_REF]. This implies that pretrained backbone do not lead to intrinsically better representations. Moreover, the best backbones are currently selected according to ImageNet classification performance, which might not be suited for all downstream problems [START_REF] Ericsson | How well do self-supervised models transfer[END_REF].

Even though unsupervised pretraining does not require labeled data, it still needs large-scale datasets to be effective. Pretraining on ImageNet currently remains better than pretraining on the target dataset, thanks to the huge amount of diverse data available in ImageNet [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF]. However, pretraining on different datasets one after the other can help for target tasks with smaller-scale data [START_REF] Reed | Self-supervised pretraining improves self-supervised pretraining[END_REF].

Semi-Supervised Object Detection

In SSOD, future works could push the data scarcity in OD even further to consider very few labeled examples for each class, and better understand how to match the performance of SSL methods for image classification in this setting [Sohn et al., 2020a].

Better taking into account the uncertainty in the localization part might be an interesting step forward [START_REF] Liu | Unbiased teacher v2: Semi-supervised object detection for anchor-free and anchor-based detectors[END_REF]. Regarding transformer-based detectors, we found that improvements from semi-supervision are less significant than with convolutional methods, which would require more understanding. For instance in Table 6.3, we achieve an improvement of about 9 p.p. with our MT-DETR compared to the supervised baseline in the FAL-COCO 1% setting, whereas the best convolutional methods achieve 11 p.p. of improvements. Furthermore, we also found in Table 6.3 that the fewer the annotated data, the fewer the benefits of unsupervised pretraining in semisupervision with transformer-based detectors. More experiments are needed to fully understand how to leverage unsupervised pretraining for semi-supervised learning in FAL, as this can reduce the total amount of label used.

Broader Impacts

We propose to conclude this thesis by discussing the potential benefits and implications of research towards Few-Annotation Learning. Three themes are thus addressed in this section covering the accessibility of Deep Learning, the impact on computational costs and on environmental footprints which represent major issues nowadays.

Accessibility of Deep Learning

Improving the ability of DL methods to learn with fewer labeled data would make DL accessible to a wider audience, since it can be currently a technical barrier for applying DL methods to specialized problems such as medical or geological imaging. These tasks might not have enough data to begin with, since the gathering of data itself can be a challenge because of the cost of the procedures. Furthermore, annotating these data requires expertise and thus, qualified workers, which is expensive and timeconsuming. These practical problems have motivated research in learning with limited data and labels, but the means found to alleviate the necessity of labels comes with an additional computational cost which can also restrict accessibility of such techniques.

Computational costs

Unfortunately, few annotation learning does not imply few computations, and it's often the opposite. To compensate for the scarcity in labels, one must rely on costly unsupervised trainings, along or before supervised training. Compared to supervised pretraining, unsupervised pretraining require bigger batch size and longer training time [Chen et al., 2020c]. Whereas supervised training on the fully labeled ImageNet is done with a batch size of 256 and 90 epochs [START_REF] Akiba | Extremely large minibatch SGD: training resnet-50 on imagenet in 15 minutes[END_REF], recent self-supervised methods train for about 800 epochs with a batch size of 4096 [Chen et al., 2020c]. While self-supervised backbones achieve the best performance faster in terms of training time on the target dataset [He et al., 2019, Newell and[START_REF] Newell | [END_REF] compared to supervised backbones, the full training might not be globally faster if we also take into account the time of the pretraining phase. Thus, using off-the-shelf weights pretrained on large-scale common datasets can save a lot of training time, but in general, pretraining on novel datasets might not be interesting in practice. The information contained in the labels makes supervised training very efficient.

Even in few-shot learning, i.e. without access to large-scale unlabeled data, the popular Meta-learning framework, and in particular episodic training, requires to split the training dataset into small, independent, but overlapping tasks and to train on a large amount of such tasks [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF]. Recent methods [START_REF] Ye | Few-shot learning via embedding adaptation with set-to-set functions[END_REF][START_REF] Ye | [END_REF] even initialize the episodic training phase with fully supervised weights, making the full meta-learning training much more computationally expensive than simpler fine-tuning approaches. These computational perspectives have to be taken into account when considering FSL and FAL methods. However, with a high computational cost comes an environmental cost, but so does the process of annotating a dataset.

Environmental footprints

Looking through the lens of CO2 emissions, even though unsupervised trainings are obviously expensive because of hardware resources and computing time, the process of annotating a large-scale dataset is not green either. We generally only measure the cost of the training phase [START_REF] Luccioni | Estimating the carbon footprint of bloom, a 176b parameter language model[END_REF], but rarely look at the annotation part. All numerical details from the following discussion can be found in Appendix A.5.

Let us consider the ILSVRC-2012 [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF] dataset from ImageNet, having 1.2M labeled images and widely used for supervised pretraining of backbones. The annotation tasks were given to Amazon Mechanical Turk (AMT) to parallelize the process, using about 5 independent annotators to verify each image [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF] which results in about 2000 hours of annotation. From the carbon intensity of the resulting electrical consumption, detailed in Appendix A.5, we obtain an estimate of the total carbon footprint of this annotation of 286.4 kgCO2eq.

To put this footprint in perspective, it is equivalent to about 55 days of compute of a node with 8 GPUs A100 used at 100% in a cluster based in France, or 10 days of compute if the cluster is based in the USA (c.f. Appendix A.5 for more details). Considering that a full self-supervised pretraining of a regular sized CNN (e.g. a ResNet-50) on the same dataset takes about 2 days with the same computational budget, this might represent a lot of possible unsupervised experiments depending on the country. While these numbers might seem low, we considered here an image classification dataset. The annotation process is simple and can be done quickly since there is a single label per image. For more complicated annotations like in Object Detection tasks, each image can contain multiple instance of objects, and each one of them have to be segmented to provide precise location information. If we consider the COCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], the full annotation process took about 85 000 hours. This represents a total carbon footprint of about 12 tons of CO2eq, or about 12 transatlantic round trips by plane (from France to New-York), which is on another scale than ImageNet. This footprint is equivalent to 51 years of computation of a node with 8 A100 GPUs running at 100% of their capacity located in France, and about 9 years if the node is located in the USA. Once again, the numerical details can be found in Appendix A.5.

One might argue that the annotation process of these two popular large-scale datasets has since then been largely profitable both to the community and from an ecological point of view, considering the quantity of experiments using either these labels or pretrained weights resulting from them. Nevertheless, this might not be the case for private or single-use datasets, on which a small-scale annotation process along with Few-Annotation Learning methods would be more beneficial, both ecologically and economically. In the end, the amount of data to annotate must take into account the annotation cost and time, the complexity of the target task, the distance of the data to other already available datasets, and the number of experiments planned both in supervised and unsupervised settings.

Paper

Assumptions

Φ Bound [START_REF] Maurer | The benefit of multitask representation learning[END_REF] One may note that all the assumptions presented in this table can be roughly categorized into two groups. First one consists of the assumptions related to the data generating process (A1, A2.1, A2.4-7 and A3.1), technical assumptions required for the manipulated empirical quantities to be well-defined (A2.6) and assumptions specifying the learning setting (A3. [START_REF] Chapelle | Semisupervised learning[END_REF][4]. We put them together as they are not directly linked to the quantities that we optimize over in order to solve the meta-learning problem. The second group of assumptions includes A2.2 and A3.2: both defined as a measure of diversity between source tasks' predictors that are expected to cover all the directions of R k evenly. These assumptions are of primary interest as it involves the matrix of predictors optimized in Equation (4.1) as thus one can attempt to force it in order for W to have the desired properties.

] A1. ∀t ∈ [[T + 1]], µt ∼ η - O 1 √ n1 + 1 √ T [Du et al., 2020] A2.0. ∀t, w * t = Θ(1) A2.1. ∀t, x is ρ 2 -subgaussian A2.2. ∀t ∈ [[T ]], ∃c > 0 : Σt cΣT+1 A2.1-2.4, linear, k d O kd cn1T + k n2 A2.3. σ1(W * ) σk(W * ) = O(1) A2.3-2.5, general, k d O C(Φ) n1T + k n2 A2.4. w * T +1 ∼ µw : || Ew∼µw[ww T ]|| ≤ O( 1 k ) A2.
Finally, we note that assumption A2.2 related to the covariance dominance can be seen as being at the intersection between the two groups. On the one hand, this assumption is related to the population covariance and thus is related to the data generating process that is supposed to be fixed. On the other hand, we can think about a pre-processing step that precedes the meta-train step of the algorithm and transforms the source and target tasks' data so that their sample covariance matrices satisfy A2.2. While presenting a potentially interesting research direction, it is not clear how this can be done in practice especially under a constraint of the largest value of c required to minimize the bound. [START_REF] Du | Few-shot learning via learning the representation, provably[END_REF] circumvent this problem by adding A2.5, stating that the task data marginal distributions are similar.

where β is the meta-training update learning rate. Starting at w 0 = 0 d , we have that

w 1 = cθ 1 , w 2 = c((c -1)θ 1 + θ 2 ), . . . w n = c n i=1 θ i (c -1) n-i ,
where c := β(1 -α) 2 . We can now define matrices W i 2 as follows:

W 1 2 =    cθ 1 , c((c -1)θ 1 + θ 2 )    , W 2 2 =    c((c -1)θ 1 + θ 2 ), c((c -1) 2 θ 1 + (c -1)θ 2 + θ 3 )    , . . . W n 2 =    c n i=1 θ i (c -1) n-i , c n+1 i=1 θ i (c -1) n-i    .
We can note that for all i > 1:

W i+1 2 = (c -1) W i 2 + cΘ i .
Now, we can write:

κ( W i+1 2 ) = σ 1 ( W i+1 2 ) σ 2 ( W i+1 2 ) = σ 1 ((c -1) W i 2 + cΘ i ) σ 2 ((c -1) W i 2 + cΘ i ) ≥ σ 1 ((c -1) W i 2 ) -σ 2 (cΘ i ) σ 2 ((c -1) W i 2 + cΘ i ) ≥ σ 1 ((c -1) W i 2 ) -σ 2 (cΘ i ) σ 2 ((c -1) W i 2 ) + σ 2 (cΘ i ) ≥ κ( W i 2 ).
where the second and third lines follow from the inequalities for singular values σ

1 (A + B) ≤ σ 1 (A) + σ 2 (B) and σ i (A + B) ≥ σ i (A) -σ min (B)
[ww ] ≤ O 1 k , then ER( φ, ŵT+1 ) ≤ O C(Φ) n 1 T • κ(W * ) + k n 2 .
Proof. [START_REF] Du | Few-shot learning via learning the representation, provably[END_REF] assume that σ k (W * ) T k (Assumption 4.3 in their work). However, since we also have w *

t = O(1), it is equivalent to σ 1 (W * ) σ k (W * ) = O(1). We have σ 1 (W * ) σ k (W * ) T k and then σ 1 (W * ) T •σ k (W * ) = 1 T • κ(W * ) 1 k•σ k (W * ) which we use in their proof of Theorem 5.1 instead of 1 T 1 k•σ k (W * )
to obtain the desired result.

A.3.4 Proof of Proposition 4.3.3

Let us recall the data generating process and Proposition 4. Proof. Let us define two uniform distributions µ 1 and µ 2 parametrized by a scalar ε > 0 satisfying the data generating process from Equation (A.26):

1. µ 1 is uniform over {1 -kε, k, 1, . . .

d-3 } × {1} ∪ {1 + kε, k, -1, . . . d-3 } × {-1}; 2. µ 2 is uniform over {1 + kε, k, k-1 ε , . . . d-3 } × {1} ∪ {-1 + kε, k, 1+k ε , . . . d-3 } × {-1}.
where last d -3 coordinates of the generated instances are arbitrary numbers. We now define the optimal representation and two optimal predictors for each distribution -X -as the solution to the MTR problem over the two data generating distributions and A.27) One solution to this problem can be given as follows:

Φ = {φ| φ(x) = Φ T x, Φ ∈ R d×2 }: φ * , W * = arg min φ∈Φ,W∈R 2×2 2 i=1 E (x,y)∼µ i (y, w i , φ(x) ), (
Φ * =    1 0 0 . . . 0 0 1 0 . . . 0    T , W * =    1 ε 1 -ε    ,
where Φ * projects the data generated by µ i to a two-dimensional space by discarding its d -2 last dimensions and the linear predictors satisfy the data generating process from Eq. 4.3 with ε = 0. One can verify that in this case W * have singular values equal to √ 2 and √ 2ε, and κ(W * ) = 1 ε . When ε → 0, the optimal predictors make the ratio arbitrary large thus violating Assumption 1.

Let us now consider a different problem where we want to solve Eq. A.27 with constraints that force linear predictors to satisfy both assumptions:

φ, W = arg min φ∈Φ,W∈R 2×2 2 i=1 E (x,y)∼µ i (y, w i , φ(x) ), s.t. κ(W) ≈ 1 and ∀i, w i ≈ 1. (A.28)
Its solution is different and is given by

Φ =    0 1 0 . . . 0 0 0 1 . . . 0    T , W =    0 1 1 -ε    .
Similarly to Φ * , Φ projects to a two-dimensional space by discarding the first and last d -3 dimensions of the data generated by µ i . The learned predictors in this case also satisfy Eq. 4.3 with ε = 0, but contrary to

W * , κ( W) = 2+ε 2 +ε √ ε 2 +4 2+ε 2 -ε √ ε 2 +4 tends to 1 when ε → 0.

A.4 Detailed Performance Comparisons from

Section 4.4

A.5 Detailed computations of CO2 emissions from popular dataset labelization

We measure in this section the carbon footprint resulting from the labelization of two popular datasets widely used in the literature, namely ILSVRC-2012 [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF] and COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF].

The ILSVRC-2012 [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF] dataset, is composed of about 1.2M labeled images. The annotation tasks were given to Amazon Mechanical Turk (AMT) to parallelize the process, using about 5 independent annotators (turkers) to verify -XII - each image [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF], leading to about 6M total labels given. Since the average user can identify about 250 images in five minutes1 , this results in about 2000 worker hours. Turkers are using their own computers to complete the annotation tasks, and with an average computer requiring about 300 W, we obtain a total electrical consumption for the workers of about 600 kWh, without taking into account the Amazon servers running the services behind. Then, from the average demography of AMT [Ipeirotis, 2010] and the carbon intensity of the electricity grid for each country2 given in Table A.3, we can compute an estimate of the total electric consumption of about 1200 kWh and a resulting carbon footprint of the annotation process of about 286.4 kgCO2eq. To compare with an actual training experiment, we suppose that we have a node with 8 A100 GPUs, and we estimate that each GPU require 400 W3 . Thus, from Table A.3, the annotation process is equivalent to the emissions from 55 days of computation if the node is located in France, and about 10 days if it is located in the USA.

For the COCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], the authors report the time taken for each stage of the annotation process. The Category Labeling stage took about 20 000 worker hours, the Instance Spotting stage about 10 000 worker hours, and the Instance Segmentation stage about 55 000 worker hours, resulting in a total of 85 000 worker hours. Using the values given in Table A.3, we obtain a total electric consumption of about 25 500 kWh and a carbon footprint of about 12 tons of CO2eq. This footprint is equivalent to 51 years of computation of a node with 8 A100 GPUs running at 100% located in France, and about 9 years if the node is located in the USA.

List of Figures

1.2 Illustration of the annotation pipeline from the COCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]. The pipeline is split into 3 primary tasks: (a) labeling the categories (class of object) present in the image, (b) locating and marking all instances of the labeled categories, and (c) segmenting each object instance. . . . . [Krizhevsky, 2009]. Dimensions of the resulting features are written below each layers. The architecture is composed of 5 convolutional layers (conv) followed by 3 fullyconnected layers (fc). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Illustration of residual connection, or skip connection, inspired by [START_REF] He | Deep residual learning for image recognition[END_REF]. This is an important building block of the ResNet architecture. . . 2.8 The Transformer encoder-decoder architecture from [START_REF] Vaswani | Attention is all you need[END_REF] used for NLP tasks. The left and right halves respectively represent the Transformer encoder and Transformer decoder architectures. . . . . . . . 2.9 Illustration of the sequencing of (a) the Scaled Dot-product Attention and (b) the Multi-head Attention, from [START_REF] Vaswani | Attention is all you need[END_REF] [START_REF] Jiao | A survey of deep learning-based object detection[END_REF] of the basic architectures of (a) twostage detectors and (b) one-stage detectors, based on deep convolutional networks. The cls and loc outputs correspond respectively to the class and location predictions of objects. The detectors are composed of a backbone network (in yellow), with additional detection-specific layers (in blue). . . . 2.12 Illustration from [START_REF] Carion | End-to-end object detection with transformers[END_REF] 3.4 Example of random erasing on object detection task [START_REF] Zhong | Random erasing data augmentation[END_REF] .

-XLIV -3.5 Illustration of an augmentation learned in the feature space, from [START_REF] Tseng | Cross-domain few-shot classification via learned feature-wise transformation[END_REF] [START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF] of (a) the Optimization-based method MAML, (b) the corresponding pseudo-code. . . . . . . . . . . . . . . . . 3.10 Illustration of memory-augmented neural network from [START_REF] Wang | Generalizing from a few examples: A survey on few-shot learning[END_REF] 3.11 Illustrations from [START_REF] Snell | Prototypical networks for few-shot learning[END_REF] 3.13 The Few-shot object detection protocol proposed by [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF],

as illustrated by [START_REF] Huang | A survey of self-supervised and few-shot object detection[END_REF] 3.17 Illustration of SoCo [Wei et al., 2021b]. The pretraining is specialized to two-stage architectures and use a contrastive learning objective. . . . . . .

-XLV -3.18 Illustration of DETReg [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF]. The detector must predict region proposals (f box (v) = b) given by an unsupervised algorithm [START_REF] Uijlings | Selective search for object recognition[END_REF], associated object embeddings (f emb (v) = ẑ) obtained by an existing pretrained self-supervised backbone [START_REF] Caron | Unsupervised learning of visual features by contrasting cluster assignments[END_REF], and object scores (f cat (v) = p), fixed to 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.19 Fine-tuning for FSOD in two-stage, from [Wang et al., 2020a]. Pretraining and fine-tuning detection models represents a strong baseline for FSOD, compared to more complex approaches presented in Section 3.2.2. . . . . . 3.20 A general taxonomy of semi-supervised methods with deep learning, from [Yang et al., 2021c]

6.2

Comparison of mean final performance (mAP in %) between Faster R-CNN (FRCNN) [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] with Feature Pyramid Network (FPN) [Lin et al., 2017a] and Deformable DETR (Def. DETR) [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF] in the Few-Shot and Few-Annotation Learning settings, using only 1% of labeled data on COCO (about 1180 images). See Section 6.4.1 for experimental details. In the fully supervised case, Def. DETR achieves better results than FRCNN. However, in the semi-supervised case implemented in Unbiased Teacher (UBT) [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], Def. DETR cannot converge. . . . . .

-XLVIII -6.3 Overview of our Momentum Teaching DETR (MT-DETR) approach for SSOD. The method follows a student-teacher architecture, with the teacher updated through an Exponential Moving Average (EMA) of the student. The keep rate parameter for the EMA follows a cosine scheduling. In the supervised branch (in dotted and green), the supervised loss L s is computed with the predictions of the student on the labeled images. In the unsupervised branch (in straight and red), the raw, i.e. unprocessed, outputs of the teacher model for the weakly augmented unlabeled images are used as soft pseudo-labels without applying any heuristic like NMS or confidence thresholding. After finding the best corresponding detection proposals with bipartite matching, the student model learns from the strongly augmented images to match the distribution of class probabilities and the bounding boxes in these pseudo-labels through the unsupervised loss L u . . . . . . . 115

A.1 Illustration of the IMP methods [START_REF] Allen | Infinite mixture prototypes for few-shot learning[END_REF].

5.9

Comparison of pretraining cost between overall pretraining methods. We compare the number of pretraining epochs on IN, the total batch size, the total number of iterations, the total training time (in hours), the number of iterations per seconds (It. / sec.), and the hardware used (number and type of GPUs). We can see that our pretraining is globally less costly than SoCo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Performance (mAP in %) comparison between Faster R-CNN (FRCNN) [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] with Feature Pyramid Network (FPN) [Lin et al., 2017a], a two-stage detector commonly used in SSOD methods, and Deformable DETR (Def. DETR) [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF] [START_REF] Carion | End-to-end object detection with transformers[END_REF], Zhu et al., 2021]. The Color Jitter, Grayscale, Gaussian Blur and CutOut augmentations follow Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] training, and the geometric augmentations (Rotate, Shear, Rescale, Pad and Translation) follow Soft Teacher [START_REF] Xu | End-to-end semi-supervised object detection with soft teacher[END_REF] training.119 6.3 Performance (mAP in %) of our proposed approach on FAL-COCO, using different percentage of labeled data (with the corresponding number of images reported) and 100% of the dataset as unlabeled data. We also report the pretrained (Pretrain.) backbone used, the architecture (Arch.) of the underlying object detector which is either Two-Stage (2S) or Transformerbased (T). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 6.4 Performance (mAP in %) of our proposed approach on VOC with fully labeled VOC07 and unlabeled VOC12 to compare with previous work, and in the novel FAL-VOC 07-12 settings. Different percentage of VOC07 are used as labeled data (5%, 10% or 100%, with the corresponding number of images reported), and the full VOC12 dataset is used as unlabeled data.

We also report the architecture (Arch.) of the underlying object detector which is either Two-Stage (2S) or Transformer-based (T). * indicates our implementation of Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] in this novel setting to compare with our approach.

[github] : updated results after publication [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] taken from their official code released. [START_REF] Carion | End-to-end object detection with transformers[END_REF], Zhu et al., 2021]. The Photometric augmentations are used in Self-supervised learning and in Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] along with CutOut augmentations. The Geometric augmentations follow Soft Teacher [START_REF] Xu | End-to-end semi-supervised object detection with soft teacher[END_REF] training. (b) Performance comparison of different combination of data augmentations on FAL-COCO 1%. Using all data augmentations leads to the best results (in bold). Furthermore, we can see that the model do not converge when using NMS and confidence thresholding along with Basic, Photometric and CutOut augmentations, which corresponds to the same setting as Unbiased Teacher [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF]

Figure 1 . 1 .

 11 Figure 1.1. Who is the painter ? Illustration of the human capacity to learn from limited examples. The top left painting is the Mona Lisa by Leonardo Da Vinci and the top right one is the Pope Leo X after Raphael by Fernando Botero. The bottom painting is Mona Lisa, Age Twelve, guessing the painter is left as an exercise to the reader.

Figure 1 . 3 .

 13 Figure 1.3. Illustration of the different possible training data conditions encountered in practice, depending on the quantity of target data (the size of the cylinder) and labels (in green) available.

Contents 2 . 1

 21 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Learning Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3 Risk Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.4 Learning Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.5 Generalization Problem . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.6 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.7 Types of Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Simple Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Gradient-Based Learning . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Deeper Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Deep Computer Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Image Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Object Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Figure 2.1.We illustrate our problem using images of famous paintings, and groundtruth labels of the corresponding painters (left). We want to find a model (right, the green line) that can correctly predict the painters, which can be easily done by a human.

Figure 2 . 4 .

 24 Figure 2.4. Illustration of an MLP with two hidden layers such that input and output dimensions are respectively d 1 = 3 and d 2 = 2.

Figure 2 . 6 .

 26 Figure 2.6. Illustration of the AlexNet architecture, from[Krizhevsky, 2009]. Dimensions of the resulting features are written below each layers. The architecture is composed of 5 convolutional layers (conv) followed by 3 fully-connected layers (fc).

Figure 2 . 7 .

 27 Figure 2.7. Illustration of residual connection, or skip connection, inspired by[START_REF] He | Deep residual learning for image recognition[END_REF]. This is an important building block of the ResNet architecture.

Figure 2 . 9 .

 29 Figure 2.9. Illustration of the sequencing of (a) the Scaled Dot-product Attention and (b) the Multi-head Attention, from[START_REF] Vaswani | Attention is all you need[END_REF]. These operations are computed between the keys K, queries Q and values V matrices.

  Figure 2.10. An illustration of the difference between (a) Image Classification and (b) Object Detection tasks. In Image Classification, the most probable class among a set of predefined class is predicted for the whole image, ignoring the fact that the image could represent multiple different objects. In Object Detection, all objects among all known class of objects appearing in an image must be identified and located through a bounding box.

3. 3 . 3

 33 Pretraining and Fine-tuning for Object Detection . . . . . . . . . . . . 3.4 Semi-supervised learning with few annotations . . . . . . . . . . . . . . . . . 3.4.1 Few-Annotation Classification . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Semi-Supervised Object Detection for Few-Annotation Learning . . . . 3.5 Towards the contributions of this thesis . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 4 .Figure 3 . 5 .

 3435 Figure 3.4. Example of random erasing on object detection task[START_REF] Zhong | Random erasing data augmentation[END_REF]

Figure 3 . 6 .Figure 3 . 7 .

 3637 Figure 3.6. In this example, augmenting the original image of a cat with only color transformations overly biases towards spatial characteristics of the cat. Learning a deep model on limited data with these color transformations will result in more overfitting than learning without augmentations.

Figure 3 .

 3 Figure 3.10. Illustration of memory-augmented neural network from[START_REF] Wang | Generalizing from a few examples: A survey on few-shot learning[END_REF]

Figure 3 . 11 .

 311 Figure 3.11. Illustrations from[START_REF] Snell | Prototypical networks for few-shot learning[END_REF] of (a) the Metric-based method ProtoNet, (b) the corresponding pseudo-code.

Figure 3 .

 3 Figure 3.12. Training examples taken from the various datasets forming Meta-Dataset from[START_REF] Triantafillou | Meta-dataset: A dataset of datasets for learning to learn from few examples[END_REF].

Figure 3 .

 3 Figure 3.14. One-shot Object Detection example from [Karlinsky et al., 2019]. Support examples representing novel classes to detect are represented in the surrounding images. The query image is in the center, with detection results for the partridge category, one of the novel class.

Figure 3 .

 3 Figure 3.15. Illustration of the general process of Transfer Learning from [Chen et al., 2019]. The training can be separated in pretraining and fine-tuning stages, in combination with other techniques.

Figure 3 . 16 .

 316 Figure 3.16. A taxonomy of Few-shot Object Detection methods through the lens of pretraining, with Self-supervised learning, Meta-learning and Transfer learning methods[START_REF] Huang | A survey of self-supervised and few-shot object detection[END_REF].

Figure 3 . 19 .

 319 Figure 3.19.Fine-tuning for FSOD in two-stage, from [Wang et al., 2020a]. Pretraining and fine-tuning detection models represents a strong baseline for FSOD, compared to more complex approaches presented in Section 3.2.2.

Figure 3 . 20 .

 320 Figure 3.20. A general taxonomy of semi-supervised methods with deep learning, from[Yang et al., 2021c].

Figure 3 . 22 .

 322 Figure 3.22. Illustration of Unbiased Teacher [Liu et al., 2021], a popular approach in SSOD inspired by hybrid methods. Semi-supervised learning is initialized after a fully supervised training ( Burn-In stage). The model is based on a Teacher-Student architecture, using two differently augmented views of the data, and pseudo-labeling. The teacher model is an independent copy of the student model, updated throughout training by the Exponential Moving Average (EMA)[START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semisupervised deep learning results[END_REF] of the student's weights.

  Figure 4.2. Illustration of the intuition behind the assumptions derived from the MTR learning theory. (left) Lack of diversity and increasing norm of the linear predictors restrict them from being useful on the target task. (right) When the assumptions are satisfied, the linear predictors cover the embedding space evenly and their norm remains roughly constant on source tasks making them useful for a previously unseen task.

Figure 4 . 3 .

 43 Figure 4.3. Evolution of κ(W N ) (left), W N F (middle) and accuracy (right) during the training of ProtoNet ( red, left axes) and MAML ( blue, right axes) on miniImageNet ( mini, solid lines) and tieredImageNet ( tiered, dashed lines) with 5-way 1-shot episodes ( top), and on the Omniglot dataset with 20-way 1-shot episodes ( bottom).

Figure 4 . 4 .

 44 Figure 4.4. Evolution of κ(W) (in log scale) during the training of ProtoNet (in red) and MAML (in blue) on miniImageNet 5-way 1-shot episodes.

  that the terms w * t and κ(W * ) underlying the assump-tions directly impact the tightness of the established bound on the excess risk. The full proof can be found in Appendix A.3.

Proposition 4. 3 . 3 .

 33 Let T = 2, X ⊆ R d be the input space and Y = {-1, 1} be the output space. Then, there exist distributions µ 1 and µ 2 over X × Y, representations φ = φ * and matrices of predictors W = W * that satisfy the data generating model (Equation (4.3)) with κ( W) ≈ 1 and κ(W * ) 1.

Figure 4 . 5 .

 45 Figure 4.5. Visualization of the distributions used in the constructive example for the proof of Proposition 4.3.3, with = 0.02. In this example, κ( W) is closer to 1 than κ(W * ). It shows that we can search for a representation φ such that optimal predictors in this space are fulfilling the assumptions, while solving the underlying problem equally well.

Figure 4 . 6 .

 46 Figure 4.6. Evolution of κ(W N ) ( left), W N F ( middle) and the accuracy ( right) on 5-way 1-shot episodes from miniImageNet, for the metric-based methods ProtoNet ( first row) and IMP ( second row), as well as the gradient-based methods MAML ( third row) and MC ( last row) and their regularized or normalized counterparts.

  is a dataset of 20 instances of 1623 characters from 50 different alphabets. Each image was hand-drawn by different people. The images are resized to 28 × 28 pixels and the classes are augmented with

Figure 4 . 7 .

 47 Figure 4.7. Evolution of κ(W) on 5-way 1-shot episodes from miniImageNet, for ProtoNet ( red, straight lines), MAML ( blue, straight lines) along with their regularized and normalized counterparts ( respectively the same colors, in dashed lines).

Figure 4 . 8 .

 48 Figure 4.8. Accuracy gap (in p.p.) when adding the normalization of prototypes for Pro-toNet ( red) and IMP ( green), and both spectral and norm regularization for MAML ( blue) and MC ( purple) enforcing the theoretical assumptions on Omniglot ( left), miniImagenet ( middle) and tieredImagenet ( right) datasets. Exact performance values are reported in Appendix A.4.

Contents 5 . 1

 51 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Overview of the approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Data processing pipeline . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Pretraining method . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 Localization-aware contrastive loss . . . . . . . . . . . . . . . . . . . 5.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Finetuning and transfer learning . . . . . . . . . . . . . . . . . . . . . 5.4.3 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.4 Pretraining cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Figure 5 . 1 .

 51 Figure 5.1. Illustration of the position of our contributions from this chapter in the full training pipeline presented in Chapter 1. We are interested on Unsupervised Pretraining for Object Detection.

Figure 5 . 2 .

 52 Figure 5.2. Illustration of the different pretraining possibilities for Object Detection. The pretraining can be either limited to the backbone ( left), or overall including the detection heads ( right). The few previous overall approaches either suffer from a discrepancy in the features between the backbone, that is trained at the image-level ( global), and the detection heads, trained to encode object-level ( local) information ( c.), or from the cost of training lots of parameters with a large batch size ( e.).

Figure 5 . 3 .

 53 Figure 5.3. Overview of our proposed ProSeCo for unsupervised pretraining. The method follows a student-teacher architecture, with the teacher updated through an Exponential Moving Average (EMA) of the student. For each input image, K random boxes are computed using the Selective Search algorithm, and two different views are generated through an asymmetric set of weak augmentations T 1 and strong augmentations T 2 . Then, object predictions are obtained from the student model for the strongly augmented view, and object proposals from the teacher model with the weakly augmented view. Finally, the boxes predicted by the student are matched with the boxes sampled from Selective Search to compute the localization losses L coord and L giou , and the full predictions are matched with the object proposals to compute our novel contrastive loss L LocSCE between them.

Figure 5 . 4 .

 54 Figure 5.4. Illustration of the effect of the localized contrastive loss used. Predictions of the student and teacher models are contrasted with each other to leverage the large number of object proposals obtained from transformer-based detectors. To introduce the object locations information, overlapping proposals (in green) in each weak view, according to an IoU threshold δ, are also considered as positive along with the matched proposal. Proposals that neither match nor overlap the matched proposal, are considered as negative (in red) in the contrastive loss.

  : short edge = range(320,481,[START_REF] Bartlett | Rademacher and gaussian complexities: Risk bounds and structural results[END_REF] Large-scale: short edge = range(480,801,32)

6. 3 . 3

 33 Construction of the pseudo-labels . . . . . . . . . . . . . . . . . . . . 6.3.4 Updating the Teacher model . . . . . . . . . . . . . . . . . . . . . . 6.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Datasets, evaluation and training details . . . . . . . . . . . . . . . . 6.4.2 Results of FAL on COCO and VOC . . . . . . . . . . . . . . . . . . . 6.4.3 Ablation studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.4 Self-supervised Backbone . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 6 . 1 .

 61 Figure 6.1. Illustration of the position of our contributions from this chapter in the full training pipeline presented in Chapter 1. We are interested on Semi-Supervised learning for Object Detection.

Figure 6 . 3 .

 63 Figure 6.3. Overview of our Momentum Teaching DETR (MT-DETR) approach for SSOD.The method follows a student-teacher architecture, with the teacher updated through an Exponential Moving Average (EMA) of the student. The keep rate parameter for the EMA follows a cosine scheduling. In the supervised branch ( in dotted and green), the supervised loss L s is computed with the predictions of the student on the labeled images. In the unsupervised branch ( in straight and red), the raw, i.e. unprocessed, outputs of the teacher model for the weakly augmented unlabeled images are used as soft pseudo-labels without applying any heuristic like NMS or confidence thresholding. After finding the best corresponding detection proposals with bipartite matching, the student model learns from the strongly augmented images to match the distribution of class probabilities and the bounding boxes in these pseudo-labels through the unsupervised loss L u .

Contents 7 . 1

 71 Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Perspectives and discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.1 Meta-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 7.2.2 Unsupervised pretraining . . . . . . . . . . . . . . . . . . . . . . . . . 129 7.2.3 Semi-Supervised Object Detection . . . . . . . . . . . . . . . . . . . 129 7.3 Broader Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Accessibility of Deep Learning . . . . . . . . . . . . . . . . . . . . . . 130 7.3.2 Computational costs . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 7.3.3 Environmental footprints . . . . . . . . . . . . . . . . . . . . . . . . . 131

3 . 3 :

 33 ∀t ∈ [[T+ 1]] and (x, y) ∼ µ t , y = w * t , φ * (x) + ε, ε ∼ N (0, σ 2 ). (A.26) Proposition (4.3.3). Let T = 2, X ⊆ R d be the input space and Y = {-1, 1} be the output space. Then, there exist distributions µ 1 and µ 2 over X × Y, representations φ = φ * and matrices of predictors W = W * that satisfy the data generating model (Equation (A.26)) with κ( W) ≈ 1 and κ(W * ) 1.

  of DETR, the original architecture based on Transformers for end-to-end object detection. The model is also composed of a backbone network (top left), followed by detection-specific transformer encoders and decoders. . . . . . . . . . . . . . . . . . . . . . 3.1 A general taxonomy of data augmentations techniques from [Shorten and Khoshgoftaar, 2019]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Illustration from [Chen et al., 2020b] of common hand-crafted augmentations. 3.3 Different ways of mixing images. (left) Non-linear vs linear mixing from [Summers and Dinneen, 2019]. (right) Mixing with random patch and cropping [Takahashi et al., 2019]. . . . . . . . . . . . . . . . . . . . . . .

  of (a) the Metric-based method Pro-toNet, (b) the corresponding pseudo-code. . . . . . . . . . . . . . . . . . 3.12 Training examples taken from the various datasets forming Meta-Dataset from [Triantafillou et al., 2020]. . . . . . . . . . . . . . . . . . . . . . . .

  . The model is trained on base classes, then finetuned on a combination of base and novel classes from a limited support set. During testing, the model is evaluated on base and novel class detection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.14 One-shot Object Detection example from [Karlinsky et al., 2019]. Support examples representing novel classes to detect are represented in the surrounding images. The query image is in the center, with detection results for the partridge category, one of the novel class. . . . . . . . . . . . . . . 3.15 Illustration of the general process of Transfer Learning from [Chen et al., 2019]. The training can be separated in pretraining and fine-tuning stages, in combination with other techniques. . . . . . . . . . . . . . . . . . . . . 3.16 A taxonomy of Few-shot Object Detection methods through the lens of pretraining, with Self-supervised learning, Meta-learning and Transfer learning methods [Huang et al., 2021]. . . . . . . . . . . . . . . . . . . . . . . . .

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.21 Illustration from FixMatch[Sohn et al., 2020a], a popular hybrid method taking inspiration from both pseudo-labeling and consistency regularization. Two differently augmented views are created for each unlabeled example, with one view going to the model as the teacher to create a pseudo-label that will be used to supervise the model as the student with the other view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.22 Illustration of Unbiased Teacher[START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF], a popular approach in SSOD inspired by hybrid methods. Semi-supervised learning is initialized after a fully supervised training (Burn-In stage). The model is based on a Teacher-Student architecture, using two differently augmented views of the data, and pseudo-labeling. The teacher model is an independent copy of the student model, updated throughout training by the Exponential Moving Average (EMA)[START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semisupervised deep learning results[END_REF] of the student's weights. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Illustration of the position of our contributions from this chapter in the full training pipeline presented in Chapter 1. We are interested on Meta-Learning for Pretraining then Finetuning. . . . . . . . . . . . . . . . . . . 4.2 Illustration of the intuition behind the assumptions derived from the MTR learning theory. (left) Lack of diversity and increasing norm of the linear predictors restrict them from being useful on the target task. (right) When the assumptions are satisfied, the linear predictors cover the embedding space evenly and their norm remains roughly constant on source tasks making them useful for a previously unseen task. . . . . . . . . . . . . . . 4.3 Evolution of κ(W N ) (left), W N F (middle) and accuracy (right) during the training of ProtoNet (red, left axes) and MAML (blue, right axes) on miniImageNet (mini, solid lines) and tieredImageNet (tiered, dashed lines) with 5-way 1-shot episodes (top), and on the Omniglot dataset with 20-way 1-shot episodes (bottom). . . . . . . . . . . . . . . . . . . . . . 4.4 Evolution of κ(W) (in log scale) during the training of ProtoNet (in red) and MAML (in blue) on miniImageNet 5-way 1-shot episodes. . . . . . . . -XLVI -4.5 Visualization of the distributions used in the constructive example for the proof of Proposition 4.3.3, with = 0.02. In this example, κ( W) is closer to 1 than κ(W * ). It shows that we can search for a representation φ such that optimal predictors in this space are fulfilling the assumptions, while solving the underlying problem equally well. . . . . . . . . . . . . . . . . . 4.6 Evolution of κ(W N ) (left), W N F (middle) and the accuracy (right) on 5-way 1-shot episodes from miniImageNet, for the metric-based methods ProtoNet (first row ) and IMP (second row ), as well as the gradient-based methods MAML (third row ) and MC (last row ) and their regularized or normalized counterparts. . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Evolution of κ(W) on 5-way 1-shot episodes from miniImageNet, for Pro-toNet (red, straight lines), MAML (blue, straight lines) along with their regularized and normalized counterparts (respectively the same colors, in dashed lines). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 Accuracy gap (in p.p.) when adding the normalization of prototypes for Pro-toNet (red) and IMP (green), and both spectral and norm regularization for MAML (blue) and MC (purple) enforcing the theoretical assumptions on Omniglot (left), miniImagenet (middle) and tieredImagenet (right) datasets. Exact performance values are reported in Appendix A.4. . . . . . . . . . . . . . . . 4.9 Evolution of accuracy of 5-way 1-shot (top, resp. 5-way 5-shot, bottom) meta-test episodes from CropDisease during meta-training on 5-way 1shot (top, resp. 5-way 5-shot, bottom) episodes from miniImageNet, for ProtoNet, IMP, MAML, MC (from left to right) and their regularized or normalized counterparts (in red, green, blue and purple, respectively ). All results are averaged over 4 different random seeds. The shaded areas show 95% confidence intervals. . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Illustration of the position of our contributions from this chapter in the full training pipeline presented in Chapter 1. We are interested on Unsupervised Pretraining for Object Detection. . . . . . . . . . . . . . . . . . . . . . . 5.2 Illustration of the different pretraining possibilities for Object Detection. The pretraining can be either limited to the backbone (left), or overall including the detection heads (right). The few previous overall approaches either suffer from a discrepancy in the features between the backbone, that is trained at the image-level (global), and the detection heads, trained to encode object-level (local) information (c.), or from the cost of training lots of parameters with a large batch size (e.). . . . . . . . . . . . . . . . -XLVII -5.3 Overview of our proposed ProSeCo for unsupervised pretraining. The method follows a student-teacher architecture, with the teacher updated through an Exponential Moving Average (EMA) of the student. For each input image, K random boxes are computed using the Selective Search algorithm, and two different views are generated through an asymmetric set of weak augmentations T 1 and strong augmentations T 2 . Then, object predictions are obtained from the student model for the strongly augmented view, and object proposals from the teacher model with the weakly augmented view. Finally, the boxes predicted by the student are matched with the boxes sampled from Selective Search to compute the localization losses L coord and L giou , and the full predictions are matched with the object proposals to compute our novel contrastive loss L LocSCE between them. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Illustration of the effect of the localized contrastive loss used. Predictions of the student and teacher models are contrasted with each other to leverage the large number of object proposals obtained from transformer-based detectors. To introduce the object locations information, overlapping proposals (in green) in each weak view, according to an IoU threshold δ, are also considered as positive along with the matched proposal. Proposals that neither match nor overlap the matched proposal, are considered as negative (in red) in the contrastive loss. . . . . . . . . . . . . . . . . . . 6.1 Illustration of the position of our contributions from this chapter in the full training pipeline presented in Chapter 1. We are interested on Semi-Supervised learning for Object Detection. . . . . . . . . . . . . . . . . . .

3

 3 IMP represents each class by a set of clusters, and infers the number of clusters from the data to adjust its modeling capacity. . . . . . . . . . . . . . . . . . . . . . . . III A.2 Pseudocode algorithm of IMP from[START_REF] Allen | Infinite mixture prototypes for few-shot learning[END_REF]. . . . . . . . . . . III A.3 Pseudocode algorithm of MC from[START_REF] Park | Meta-curvature[END_REF]. The model considered here has only one layer, it is straightforward to extend to multiple layers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV A.4 Illustration from [Park and Oliva, 2019] of meta-curvature computation with G ∈ R 2×3×d . (Top) Tensor algebra view. (Bottom) Matrix-vector product view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V A.5 Performance gap (in p.p.) when applying regularization for gradient-based and normalization for metric-based methods throughout the training process on 5-way 1-shot and 5-shot episodes on miniImageNet (better viewed in color). Each data point is averaged over 2400 validation episodes and 4 different seeds and shaded areas report 95% confidence interval. We can see that the gap is globally positive throughout training and generally higher at the beginning of training. The increase in the gap at the end of training is linked to a lower overfitting. . . . . . . . . . . . . . . . . . . . XIV -XLIX -List of Tables 3.1 FSOD-dataset summary from [Fan et al., 2020a]. . . . . . . . . . . . . . 4.1 Performance comparison of FSC models on FSC benchmarks. For a given architecture, bold values are the highest accuracy and underlined values are near-highest accuracies (at less than 1-point lower than the highest one). 4.2 Performance of ProtoNet with and without our regularization on the entropy and/or normalization. All accuracy results (in %) are averaged over 2400 test episodes and 4 different random seeds and are reported with 95% confidence interval. Further enforcing regularization on the singular values can be detrimental to performance. . . . . . . . . . . . . . . . . . . . . . 4.3 Ablative study on the strength of the regularization with normalized Pro-toNet. All accuracy results (in %) are averaged over 2400 test episodes and 4 random seeds and are reported with 95% confidence interval. . . . . 4.4 Ablative study of the regularization parameter for MAML, on Omniglot (a) with 20-way 1-shot (top values) and 20-way 5-shot (bottom values) episodes, and miniImageNet (b) with 5-way 1-shot (top values) and 5-way 5-shot (bottom values) episodes. All accuracy results (in %) are averaged over 2400 test episodes and 4 different random seeds and are reported with 95% confidence interval. We can see that in all cases, using both regularization terms is important. . . . . . . . . . . . . . . . . . . . . . . 4.5 Performance comparison of MTL models [Wang et al., 2021b] on FSC benchmarks with the ResNet-12 backbone architecture. Bold values are the highest accuracy and underlined values are near-highest accuracies (at less than 1-point lower than the highest one). . . . . . . . . . . . . . . . 4.6 Performance of MTL [Wang et al., 2021b] when varying either λ 1 or λ 2 , the other being fixed to 1, on the miniImageNet 5-way 5-shot benchmark. All accuracy results (in %) are averaged over 2000 test episodes on a single random seed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 The different sets of augmentations used for each branch (weak or strong). Probability indicates the probability of applying the corresponding augmentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Performance (mAP in %) of our proposed pretraining approach after finetuning using different percentage of training data (with the corresponding number of images reported). We show that our ProSeCo outperforms previous pretraining methods in all benchmarks. . . . . . . . . . . . . . . Performance comparison when transferring pretrained weights to DAB-Deformable DETR [Liu et al., 2022a], after fine-tuning on different fractions of labeled data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Performance comparison when pretraining and fine-tuning with Conditional DETR [Meng et al., 2021] on different fractions of labeled data. . . . . . . 5.5 (a) Comparison after finetuning when using different pretrained backbone and/or pretraining datasets for ProSeCo. (b) Comparison of the effect of the location information using different IoU threshold δ and for the different constrastive loss considered. All performance (mAP in %) are measured on Mini-COCO 5%. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Ablation studies on different hyperparameters for the proposed method. The performance (mAP in %) are measured on Mini-COCO 5%. Green and bold columns names indicate a positive effect on the performance, and red columns a negative effect. . . . . . . . . . . . . . . . . . . . . . 5.7 Performance (mAP in %) comparison on Mini-COCO 5% when changing the number of object proposals in Def. DETR. . . . . . . . . . . . . . . . 5.8 Performance (mAP, AP 50 and AP 75 in %) comparison on the full COCO dataset with the 1× training schedule. . . . . . . . . . . . . . . . . . . .

  studies of the different parts of our method. Green and bold columns names indicate a positive effect on the performance and red columns a negative effect. The use of cosine scheduling, an initialization after fine-tuning (FT) and raw soft pseudo labels corresponds to the best combination found. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.6 (a) The different sets of augmentations considered during SSL. The Basic augmentations follow standard supervised training

  (in red). Using raw soft pseudo-labels (PL) stabilizes the training and makes it converge to a strong solution (in green). . . . . . . . . . . . . . . . . . . . . . . . . . 126 -LII -A.1 Overview of main theoretical contributions related to MTR learning with their assumptions, considered classes of representations and the obtained bounds on the excess risk. Here Õ(•) hides logarithmic factors. . . . . . . II A.2 Exact performance of the meta-learning setting considered in Section 4.4, without and with our regularization (or normalization in the case of Pro-toNet and IMP) to enforce the theoretical assumptions. All accuracy results (in %) are averaged over 2400 test episodes and 4 different seeds and are reported with 95% confidence interval. Episodes are 20-way classification for Omniglot and 5-way classification for miniImageNet and tieredImageNet.XIII A.3 General demography of AMT (in % of total AMT users) with the corresponding carbon intensity of the electricity grid (in gCO2eq/kWh) of each country. The carbon intensity of the other category is taken as the world average, and the value for France is given for reference. . . . . . . . . . . XV -LIII -

  Context and Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.3 Learning frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Challenges and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	Introduction
	AP	Average Precision metric
	AP 50	AP calculated at IoU threshold of 0.5
	AP 50-95 AP calculated for IoU threshold from 0.5 to 0.95 with step size of 0.05
	mAP Contents	mean Average Precision metric, mean over all the class APs
	1.1

4 1.2.1 The Problem of Labeling Data . . . . . . . . . . . . . . . . . . . . . 4 1.2.2 Training data conditions in practice . . . . . . . . . . . . . . . . . . . 5 1.2.3 Issues addressed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3 Outline and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.1 Structure of the manuscript . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.2 Scientific contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 11

  Data Augmentation with few images . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Hand-crafted augmentations . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Learning augmentations . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Limitations of augmentation procedures . . . . . . . . . . . . . . . . 3.2 Meta-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transfer Learning by Pretraining . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Pretraining and Fine-tuning for Few-Shot Learning . . . . . . . . . . . 3.3.2 Self-supervised Pretraining . . . . . . . . . . . . . . . . . . . . . . . .

	Learning with few labels: an
	Overview
	Contents
	3.1 3.2.1 Meta-Learning for Few-Shot Image Recognition . . . . . . . . . . . .
	3.2.2 Meta-Learning for Few-Shot Object Detection . . . . . . . . . . . . .
	3.3

Table 3 .

 3

1. FSOD-dataset summary from

  generally represented as (deep) neural networks, and linear predictorsw t ∈ R k , ∀t ∈ [[T ]]grouped in a matrix W ∈ R T ×k . More formally, this is done by solving the following joint optimization problem:

  3 , which does not hold in a crossdomain setting, where we evaluate a model on a dataset different from the training

			ProtoNet		ProtoNet + norm		IMP			IMP + norm		MAML			MAML + reg		MC				MC + reg
		0.58																					0.60
		0.56							0.58							0.58
	Accuracy	0.48 0.50 0.52 0.54						Accuracy	0.52 0.54 0.56						Accuracy	0.50 0.52 0.54 0.56						Accuracy	0.55 0.45 0.50
		0.44 0.46							0.50							0.48							0.40
		0	10k	20k	30k	40k	50k	60k	0	10k	20k	30k	40k	50k	60k	0	10k	20k	30k	40k	50k	60k	0	10k	20k	30k	40k	50k	60k
				Training iteration				Training iteration				Training iteration					Training iteration
			ProtoNet		ProtoNet + norm		IMP			IMP + norm		MAML			MAML + reg		MC				MC + reg
		0.80 0.81							0.83 0.82							0.82							0.80
	Accuracy	0.73 0.74 0.75 0.76 0.77 0.78 0.79						Accuracy	0.76 0.77 0.78 0.79 0.80 0.81						Accuracy	0.68 0.70 0.80 0.72 0.74 0.76 0.78						Accuracy	0.50 0.75 0.55 0.60 0.65 0.70
		0	10k	20k	30k	40k	50k	60k	0	10k	20k	30k	40k	50k	60k	0	10k	20k	30k	40k	50k	60k	0	10k	20k	30k	40k	50k	60k
				Training iteration				Training iteration				Training iteration					Training iteration
	Figure 4.9. Evolution of accuracy of 5-way 1-shot ( top, resp. 5-way 5-shot, bottom)
	meta-test episodes from CropDisease during meta-training on 5-way 1-shot ( top, resp.

5-way 5-shot, bottom) episodes from miniImageNet, for ProtoNet, IMP, MAML, MC ( from left to right) and their regularized or normalized counterparts ( in red, green, blue and purple, respectively). All results are averaged over 4 different random seeds. The shaded areas show 95% confidence intervals.

Table 4 .

 4

	1. Performance comparison of FSC models on FSC benchmarks. For a
	given architecture, bold values are the highest accuracy and underlined values are
	near-highest accuracies (at less than 1-point lower than the highest one).

Table 4 .

 4

	Dataset	Episodes	Original	λ1 = 0	λ1 = 1	λ1 = 0.1	λ1 = 0.01	λ1 = 0.001	λ1 = 0.0001
	miniImageNet	5-way 1-shot 49.53 ± 0.41% 50.29 ± 0.41% 49.43 ± 0.40% 50

2. Performance of ProtoNet with and without our regularization on the entropy and/or normalization. All accuracy results (in %) are averaged over 2400 test episodes and 4 different random seeds and are reported with 95% confidence interval. Further enforcing regularization on the singular values can be detrimental to performance.

.19 ± 0.41% 50.44 ± 0.42% 50.46 ± 0.42% 50.45 ± 0.42%

		5-way 5-shot 65.10 ± 0.35% 67.13 ± 0.34% 65.71 ± 0.35% 66.69 ± 0.36%	66.69 ± 0.34%	67.2 ± 0.35%	67.12 ± 0.35%
	Omniglot	20-way 1-shot 95.56 ± 0.10% 95.89 ± 0.10% 91.90 ± 0.14% 94.38 ± 0.12%	95.60 ± 0.10%	95.7 ± 0.10%	95.77 ± 0.10%
		20-way 5-shot 98.80 ± 0.04% 98.80 ± 0.04%	96.40 ± 0.07% 97.93 ± 0.05%	98.62 ± 0.04%	98.76 ± 0.04%	98.91 ± 0.03%

Table 4 .

 4

	3. Ablative study on the strength of the regularization with normalized
	ProtoNet. All accuracy results (in %) are averaged over 2400 test episodes and 4
	random seeds and are reported with 95% confidence interval.

Table 4 .

 4

			miniImageNet 5-way	tieredImageNet 5-way
	Model	Arch.	1-shot	5-shot	1-shot	5-shot
	MTL	R12	55.73 ± 0.18 76.27 ± 0.13 62.49 ± 0.21	81.31 ± 0.15
	MTL + norm.	R12	59.49 ± 0.18

4. Ablative study of the regularization parameter for MAML, on Omniglot (a) with 20-way 1-shot ( top values) and 20-way 5-shot ( bottom values) episodes, and miniImageNet (b) with 5-way 1-shot ( top values) and 5-way 5-shot ( bottom values)

episodes. All accuracy results (in %) are averaged over 2400 test episodes and 4 different random seeds and are reported with 95% confidence interval. We can see that in all cases, using both regularization terms is important.

77.3 ± 0.13 66.66 ± 0.21 83.59 ± 0.14

	λ 1	1	0.8	0.6	0.4	0.2	0.1	0.05 0.01	0
	Accuracy	75.84 75.85 76.02 76.11 76.15 75.99 75.65 75.08 74.64
	(λ 2 = 1)
	λ 2	1	0.8	0.6	0.4	0.2	0.1	0.05 0.01	0
	Accuracy	75.84 76.09 75.81 76.28 76.23 76.1 76.25 76.42 76.06
	(λ 1 = 1)
		MTL + reg. (Ours) R12	61.12 ± 0.19 76.62 ± 0.13 66.28 ± 0.22	81.68 ± 0.15
	Table 4.5. Performance comparison of MTL models [Wang et al., 2021b] on FSC
	benchmarks with the ResNet-12 backbone architecture. Bold values are the highest
	accuracy and underlined values are near-highest accuracies (at less than 1-point lower
	than the highest one).

Table 4 .6. Performance of MTL

 4

Table 5 .

 5

1. The different sets of augmentations used for each branch ( weak or strong).

Probability indicates the probability of applying the corresponding augmentation.

Table 5 .

 5

2. Performance (mAP in %) of our proposed pretraining approach after fine-tuning using different percentage of training data (with the corresponding number of images reported). We show that our ProSeCo outperforms previous pretraining methods in all benchmarks.

Table 5 .

 5 To compare with IN, we also pretrain ProSeCo on COCO for 120

	Pretraining		Mini-COCO
		1% (1.2k) 5% (5.9k) 10% (11.8k)
	Supervised	16.7	27.0	31.0
	DETReg	17.7	28.6	33.0
	ProSeCo w/ SwAV	16.2	28.7	33.9
	ProSeCo w/ SCRL	19.4	29.9	34.1
	Table 5.3. Performance comparison when transferring pretrained weights to DAB-
	Deformable DETR [Liu et al., 2022a], after fine-tuning on different fractions of labeled
	data.
	Pretraining		Mini-COCO
	1% (1.2k) 5% (5.9k) 10% (11.8k)
	Supervised	8.7	19.3	24.6
	SCRL	11.7	20.9	26.2
	ProSeCo	9.6	22.4	27.5

4. Performance comparison when pretraining and fine-tuning with Conditional DETR

Table 5 .

 5

	Batch size Images scale Temperature τ t	EMA	mAP
	192 64 Mid Large 0.05	0.07	0.99 0.996 0.999
	Base			26.7
	Abl. Batch			26.5
	Abl. Scale			27.4
	Abl. Temp.			27
	Abl. EMA			26.3
				26.8
	Best			27.8

6. Ablation studies on different hyperparameters for the proposed method.

The performance (mAP in %) are measured on Mini-COCO 5%. Green and bold columns names indicate a positive effect on the performance, and red columns a negative effect.

Table 5 .

 5 In the following chapter, we 7. Performance (mAP in %) comparison on Mini-COCO 5% when changing the number of object proposals in Def. DETR.

	Method	N Performance
		100	23.1
	Supervised	200	23.0
		300	23.6
		500	23.3
		100	25.7
	ProSeCo (Ours)	200	26.5
		300	27.8
		500	27.2
	Method		COCO
			mAP AP 50 AP 75
	Supervised		37.4 55.5 40.5
	DETReg [Bar et al., 2022] 38.9 56.6 42.3
	ProSeCo (Ours)		38.9 56.2 42.4

Table 5 .8. Performance

 5 (mAP, AP 50 and AP 75 in %) comparison on the full COCO dataset with the 1× training schedule. are interested in improving the performance after the fine-tuning phase through semisupervision. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Fully Supervised Object Detection . . . . . . . . . . . . . . . . . . . . 6.2.2 Semi-supervised Object Detection for Few-Annotation Learning . . . . 6.3 A semi-supervised learning approach for transformer-based object detection . . 6.3.1 How do object detectors handle data scarcity ? . . . . . . . . . . . . .

	Few Annotation Learning for
	Semi-Supervised Object
	Detection
	Contents
	6.1

6.3.2 Overview of our approach . . . . . . . . . . . . . . . . . . . . . . . .

Table 6 . 2 .

 62 The different sets of augmentations used during SSL for each branch. The Horizontal Flip and Resize augmentations follow standard supervised training

Table 6 . 5 .

 65 Ablation studies of the different parts of our method. Green and bold columns names indicate a positive effect on the performance and red columns a negative effect. The use of cosine scheduling, an initialization after fine-tuning (FT) and raw soft pseudo labels corresponds to the best combination found.

Table 6 .

 6 6. (a) The different sets of augmentations considered during SSL. The Basic augmentations follow standard supervised training

Table A . 1 .

 A1 Overview of main theoretical contributions related to MTR learning with their assumptions, considered classes of representations and the obtained bounds on the excess risk. Here Õ(•) hides logarithmic factors.

  and the desired result is obtained by setting σ min (θ i ) = 0.

	A.3.3 Proof of Proposition 4.3.2
	Let us first recall Proposition 4.3.2:
	Proposition (4.3.2). If ∀t ∈ [[T ]], w * t = O(1) and κ(W * ) = O(1), and w T +1
	follows a distribution ν such that E w∼ν
	-IX -

  metric-based methods in TableA.2a. Figure A.5 shows the performance gap throughout training for all methods on miniImageNet. We can see on both TableA.2 and Figure A.5 that the gap is globally positive, which shows the increased generalization capabilities of enforcing the assumptions.

	Method	Dataset	Episodes without Reg./Norm. with Reg./Norm.
		Omniglot	1-shot	95.56 ± 0.10%	95.89 ± 0.10%
			5-shot	98.80 ± 0.04%	98.80 ± 0.04%
	ProtoNet	miniImageNet	1-shot 5-shot	49.53 ± 0.41% 65.10 ± 0.35%	50.29 ± 0.41% 67.13 ± 0.34%
		tieredImageNet	1-shot	51.95 ± 0.45%	54.05 ± 0.45%
			5-shot	71.61 ± 0.38%	71.84 ± 0.38%
		Omniglot	1-shot	95.77 ± 0.20%	95.85 ± 0.20%
			5-shot	98.77 ± 0.08%	98.83 ± 0.07%
	IMP	miniImageNet	1-shot 5-shot	48.85 ± 0.81% 66.43 ± 0.71%	50.69 ± 0.80% 67.29 ± 0.68%
		tieredImageNet	1-shot	52.16 ± 0.89%	53.46 ± 0.89%
			5-shot	71.79 ± 0.75%	72.38 ± 0.75%
	Table A.2 provides the detailed performance of our reproduced methods with and with-
	out our regularization for gradient-based methods in Table A.2b, or normalization for
			-XI -

(a)

Metric-based methods

Table A . 3 .

 A3 General demography of AMT (in % of total AMT users) with the corresponding carbon intensity of the electricity grid (in gCO2eq/kWh) of each country. The carbon intensity of the other category is taken as the world average, and the value for France is given for reference.

		Country AMT demography Carbon intensity of electricity
		USA	47%	379
		India	34%	633
	Method Dataset Omniglot Other France	Episodes without Reg./Norm. with Reg./Norm. 1-shot 91.72 ± 0.29% 95.67 ± 0.20% 5-shot 97.07 ± 0.14% 98.24 ± 0.10% MC MAML ProtoNet IMP 19% 442 Episode 1shot 5shot -68
	MAML MC	miniImageNet tieredImageNet Omniglot miniImageNet tieredImageNet 0 10k 20k 30k 40k 50k 60k 1-shot 47.93 ± 0.83% 49.16 ± 0.85% 5-shot 64.47 ± 0.69% 66.43 ± 0.69% 1-shot 50.08 ± 0.91% 51.5 ± 0.90% 5-shot 67.5 ± 0.79% 70.16 ± 0.76% 1-shot 96.56 ± 0.18% 95.95 ± 0.20% 5-shot 98.88 ± 0.08% 98.78 ± 0.08% 1-shot 49.28 ± 0.83% 49.64 ± 0.83% 5-shot 63.74 ± 0.69% 65.67 ± 0.70% 1-shot 55.16 ± 0.94% 55.85 ± 0.94% 5-shot 71.95 ± 0.77% 73.34 ± 0.76% Training iteration 15 10 5 0 5 10 Accuracy gap (p.p.)
	(b) Gradient-based methods Figure A.5. Performance gap (in p.p.) when applying regularization for gradient-
	Table A.2. Exact performance of the meta-learning setting considered in Section 4.4, based and normalization for metric-based methods throughout the training process on
	without and with our regularization (or normalization in the case of ProtoNet and 5-way 1-shot and 5-shot episodes on miniImageNet ( better viewed in color). Each
	IMP) to enforce the theoretical assumptions. All accuracy results (in %) are averaged data point is averaged over 2400 validation episodes and 4 different seeds and shaded
	over 2400 test episodes and 4 different seeds and are reported with 95% confidence areas report 95% confidence interval. We can see that the gap is globally positive
	interval. Episodes are 20-way classification for Omniglot and 5-way classification for throughout training and generally higher at the beginning of training. The increase in
	miniImageNet and tieredImageNet. the gap at the end of training is linked to a lower overfitting.
			-XIII --XIV -

  1.3 Illustration of the different possible training data conditions encountered in practice, depending on the quantity of target data (the size of the cylinder) and labels (in green) available. . . . . . . . . . . . . . . . . . . . . . . . Auxiliary (Aux.) labeled data is represented in orange. . . . . . . . . 1.5 A tentative general pipeline encompassing approaches of FSL and FAL, coping with limited target labels. The semi-supervised phase (dotted lines) is possible if unlabeled target data are also available (FAL). . . . . . . . . Illustration of the relationship between capacity and error from [Goodfellow et al., 2016]. Since we only have access to training data, training and test error behave differently. At the left end of the graph, training error and generalization error are both high. This is the underfitting regime. As we increase capacity, training error decreases, but the gap between training and generalization error increases. Eventually, the size of this gap outweighs the decrease in training error, and we enter the overfitting regime, where capacity is too large, above the optimal capacity. . . . . . . . . . . . . . . 2.3 Although there is a non-zero probability to generate a real photo by randomly sampling values for pixels, we never actually observe this in practice. This suggests that natural images lie in specific and narrow regions of the image space, which can be difficult to represent with a simple model. . . . 2.4 Illustration of an MLP with two hidden layers such that input and output dimensions are respectively d 1 = 3 and d 2 = 2. . . . . . . . . . . . . . . . An illustration of the 2D convolution operation used in CNNs from [Goodfellow et al., 2016]. The input and output are also called feature maps. A kernel matrix slides across the input feature map to compute the output feature map. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Illustration of the AlexNet architecture, from

	2.5
	-XLIII -

1.4 Illustration of commonly used approaches for (a) FSL and (b) FAL problems. 1.6 Illustration of the position of our contributions presented in (a) Chapter 4, (b) Chapter 5 and (c) Chapter 6, in the general training pipeline from Figure 1.5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.1

We illustrate our problem using images of famous paintings, and groundtruth labels of the corresponding painters (left). We want to find a model (right, the green line) that can correctly predict the painters, which can be easily done by a human. . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2

  . These operations are computed between the keys K, queries Q and values V matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.10 An illustration of the difference between (a) Image Classification and (b) Object Detection tasks. In Image Classification, the most probable class among a set of predefined class is predicted for the whole image, ignoring the fact that the image could represent multiple different objects. In Object Detection, all objects among all known class of objects appearing in an image must be identified and located through a bounding box. . . . . . . . 2.11 Illustration from

  . (a) Learned feature-wise transformation layer. (b) The layers are inserted and learned during training, to simulate feature distributions extracted from tasks in various domains. . . . . . . . . . . . . . . . . . . 3.6 In this example, augmenting the original image of a cat with only color transformations overly biases towards spatial characteristics of the cat. Learning a deep model on limited data with these color transformations will result in more overfitting than learning without augmentations. . . . . 3.7 A general illustration of meta-learning for FSC. The training dataset is separated into episodes, i.e. small learning tasks. In FSC, an episode is an N -way K-shot classification problem. We illustrate here 3-way 2-shot episodes. The meta-learning algorithm learns each task at the inner-level and between tasks at the outer-level. . . . . . . . . . . . . . . . . . . . . 3.8 A taxonomy of Meta-Learning and its applications going beyond FSL, from [Hospedales et al., 2021]. . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Illustrations from

  , a state-of-the-art transformerbased object detector, with the same ResNet-50 backbone model. The performances are reported for different percentages (and the corresponding number of images) of COCO and VOC07 labeled training data. See Section 6.4.1 for more details on the experiments. Def. DETR performs better than FRCNN + FPN with fewer labeled data for a similar amount of parameters. † : Results from[START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] if available, from our reproduction otherwise. . . . . . . . . . . . . . . . . . . . . . . . . . . . The different sets of augmentations used during SSL for each branch. The Horizontal Flip and Resize augmentations follow standard supervised training

	6.2
	-LI -

https://www.mturk.com/

https://www.nytimes.com/2012/11/20/science/for-web-images-creating-new-tec hnology-to-seek-and-find.html

https://pytorch.org/

https://www.tensorflow.org/

In this chapter, we aim to use the recent advances in MTR theory[START_REF] Tripuraneni | Provable meta-learning of linear representations[END_REF], Du et al., 2020] to explore the inner workings of these popular metalearning methods. Our rationale for such an approach stems from a recent work[START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF] proving that the optimization problem behind the majority of meta-learning algorithms can be written as an MTR problem. Thus, we believe that looking at meta-learning algorithms through the recent MTR theory lens, could lead to better understanding the capacity to work well in few-shot regime. In particular, we take a closer look at two families of meta-learning algorithms, notably: gradient-based algorithms[START_REF] Ravi | Optimization as a model for few-shot learning[END_REF], Nichol et al., 2018, Li et al., 2017, Oreshkin et al., 2018, Lee et al., 2019, Bertinetto et al., 2019, Park and Oliva, 2019, Chen et al., 2020e, Raghu et al., 2020] including MAML[START_REF] Finn | Model-agnostic metalearning for fast adaptation of deep networks[END_REF] and metricbased algorithms[START_REF] Koch | Siamese neural networks for one-shot image recognition[END_REF], Vinyals et al., 2016, Snell et al., 2017, Sung et al., 2018, Allen et al., 2019, Li et al., 2019, Simon et al., 2020] with its most prominent example given by ProtoNet[START_REF] Snell | Prototypical networks for few-shot learning[END_REF], which we presented in the previous chapter.Our main contributions in this chapter are then two-fold:1. We empirically show that tracking the validity of assumptions on optimal predictors used in[START_REF] Tripuraneni | Provable meta-learning of linear representations[END_REF], Du et al., 2020] reveals a striking difference between the behavior of gradient-based and metric-based methods in how they[START_REF] Gladwell | Outliers. Back Bay Books[END_REF] We omit the results on meta-learning in the context of online convex optimization[START_REF] Finn | Online meta-learning[END_REF], Balcan et al., 2019, Khodak et al., 2019, Denevi et al., 2019] as they concern a different learning setup.

While not stated separately, this assumption is used in[START_REF] Du | Few-shot learning via learning the representation, provably[END_REF] to derive the final result on p.5 after the discussion of Assumption 4.3.

We refer the reader to Appendix A.1 for more information on this assumption

The weak and strong augmentations are described in Section 6.4.1.

https://github.com/CEA-LIST/MT-DETR

https://github.com/facebookresearch/unbiased-teacher

https://www.nytimes.com/2012/11/20/science/for-web-images-creating-new-tec hnology-to-seek-and-find.html

https://ourworldindata.org/grapher/carbon-intensity-electricity

https://www.nvidia.com/en-us/data-center/a100/ -XV -

Remerciements

(

supervision during fine-tuning. However, as we can see in Table 5.2b, we outperform DETReg on our FSOD-train benchmark, which represents a setting with mid-scale annotated data (42k training images).

Pretraining cost

We compare the cost of pretraining in terms of memory and hardware used to SoCo [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF] in Table 5.9 since it is the closest in terms of pretraining pipeline.

The information are derived from their paper and official github repository.

Even though A100 GPUs are faster than V100 GPUs, we are training much faster which is partly explained by the fact that they learn the backbone along with the detection heads during pretraining, leading to more parameters to learn and more computations. Furthermore, our ProSeCo requires a smaller batch size leading to less memory and thus less GPUs needed. -107 -Both student and teacher models are initialized from a fully supervised model trained on the few labeled data available. Then, during the semi-supervised training, the method takes as inputs a batch of labeled images B l = {(x l i , y l i )} N l i=1 and a batch of unlabeled images B u = {x u i } N u i=1 . We define x l i and x u i as the i th labeled and unlabeled image respectively,

as the corresponding k i ground truth class labels and box coordinates, and finally, N l and N u are respectively the labeled and unlabeled batch sizes. The student model is updated by a weighted combination of a supervised loss L s and an unsupervised loss L u with weight λ u ∈ R + :

Below, we first describe the supervised branch, which computes the supervised loss using the batch of labeled data B l . Then, we detail the unsupervised branch, which computes the unsupervised loss with the batch of unlabeled data B u .

Supervised branch

To compute the supervised loss, the supervised branch follows the supervised learning of Def. DETR [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF], which is an improved version of DETR [START_REF] Carion | End-to-end object detection with transformers[END_REF]. For each image x l i , the student model infers N predictions

and their associated predicted labels logits ĉl (i,j) ∈ R C+1 , with the (C + 1) th logit representing the no object (∅) class. Then, the Hungarian algorithm [Munkres, 1957] finds from all the permutations of N elements S N , the optimal bipartite matching σl i between the predictions ŷl i of the student model and the ground truth labels y l i : σl i = arg min σ∈S N N j L match (y l (i,j) , ŷl (i,σ(j)) ). Thus, for each labeled image x l i , the j th ground truth y l (i,j) is associated to σl i (j). Similarly to the loss used in object detectors, the matching cost L match used in the Hungarian algorithm takes into account both class and bounding box predictions through a linear combination of the Focal loss [START_REF]Focal loss for dense object detection[END_REF]] L focal , the 1 loss of the box coordinates, and the generalized IoU loss [START_REF] Rezatofighi | Generalized intersection over union: A metric and a loss for bounding box regression[END_REF] L giou , respectively. These loss functions are then used to compute the supervised loss L s as well:

(6.3)

-116 -

In the above equations, we define λ class , λ 1 , λ giou ∈ R + as the coefficients in the matching cost and 1 X the indicator function, such that ∀x, 1 X (x) = 1 iff x ∈ X .

Unsupervised branch Our main contribution is the unsupervised loss for transformerbased OD. In the unsupervised branch, we produce two different views for each unlabeled image x u i : a weakly augmented view x u i and a strongly augmented view x u i 1 . Then, the teacher model provides soft pseudo-labels

, with c u (i,j) the predicted logits, for each weakly augmented unlabeled image x u i , and the student model infers predictions

from the corresponding strongly augmented unlabeled view x u i . We apply the same Hungarian algorithm with the same matching cost L match to obtain the best permutation σu

), that matches the predictions of the student with the closest pseudo-label. In the unsupervised loss L u , we follow the consistency regularization paradigm [START_REF] Buciluǎ | Model compression[END_REF], Chen et al., 2020c, Caron et al., 2021]. We train the student network to match the probability distributions of the classes predicted by the student with the soft pseudo-labels proposed by the teacher. We learn to match these distributions by minimizing the cross-entropy between the two class distribution outputs normalized by a softmax function. We define respectively: ) , the student and teacher class distribution outputs, where c (k) is the k th element of c, ∀c ∈ R C+1 . Then the cross-entropy loss is defined as:

and finally, we compute the unsupervised loss L u as:

(6.6)

For FAL, we have little information from the labeled data. Therefore, the quality of the pseudo-labels and their contained information play an important part in the training.

Datasets, evaluation and training details

Datasets and evaluation protocol We briefly recall the datasets introduced in Chapter 3 that we used here. To evaluate our proposed method, we use the MS-COCO (COCO) [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] and PASCAL VOC (VOC) [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] datasets which are standard for object detection, following the settings of existing works [START_REF] Jeong | Consistencybased semi-supervised learning for object detection[END_REF][START_REF] Sohn | A simple semi-supervised learning framework for object detection[END_REF], Liu et al., 2021, Xu et al., 2021, Zhou et al., 2021, Tang et al., 2021]. COCO is a dataset with 80 classes, and VOC contains 20 classes. We are specifically interested in two Few Annotation Learning (FAL) settings: On FAL-COCO, we randomly sample 0.5, 1, 5 or 10% (respectively about 590, 1180, 5900 and 11800 images) of the training set (train2017 ) used as the labeled set and use the full training set for the unlabeled set (about 118k images). Performance is evaluated on val2017.

On FAL-VOC 07-12, we restrict the labeled training set (VOC07 trainval) to a random sample of 5 or 10% (respectively 250 and 500 labeled images), and use the full VOC12 trainval (about 11k images) as unlabeled training set. We introduce this novel setting to evaluate our approach in a FAL setting on VOC. We also compare the results with previous SSOD methods using the full VOC07 [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] in this novel setting to compare with our approach.

[github] : updated results after publication [START_REF] Liu | Unbiased teacher for semi-supervised object detection[END_REF] taken from their official code released. 3 images) and VOC12 trainval as unlabeled training set. Performance is evaluated on VOC07 test set.

In all settings, performance is reported and compared using the AP 50:95 (mAP, in %) evaluation metric using the official COCO and VOC evaluation codes, respectively.

Training For a fair comparison, a fully supervised ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] pretrained on ImageNet [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF] is used as a backbone for all the methods. For fine-tuning Def. DETR [START_REF] Zhu | Deformable detr: Deformable transformers for end-to-end object detection[END_REF] on the few labeled data, we train the model with a batch size of 32 images on 8 GPUs until the validation performance stops increasing, i.e. for COCO, up to 2000 epochs for 1%, 500 epochs for 5%, 400 epochs for 10%, and for VOC, up to 2000 epochs for both 5% and 10%. For semi-supervised learning, we train MT-DETR for 50 (respectively, 250) epochs of the unlabeled data on COCO (respectively, VOC) with a batch size of 48 labeled images and 48 unlabeled images (respectively, 24 and 24)

A.1 Review of Multi-task Representation Learning Theory

We formulate the main results of the three main theoretical analyses of Multi-task Representation (MTR) Learning Theory provided in [START_REF] Maurer | The benefit of multitask representation learning[END_REF], Du et al., 2020, Tripuraneni et al., 2020] in Table A.1 to give additional details for Sections 4.2.2 and 4.4.5. [START_REF] Allen | Infinite mixture prototypes for few-shot learning[END_REF].

A.2 Introduction to IMP and MC algorithms A.2.1 Infinite Mixture Prototypes

The Infinite Mixture Prototypes (IMP) [START_REF] Allen | Infinite mixture prototypes for few-shot learning[END_REF] algorithm represents each class by a set of clusters, instead of a single cluster like ProtoNet [START_REF] Snell | Prototypical networks for few-shot learning[END_REF].

By inferring the number of clusters, IMP interpolates between nearest neighbor and prototypical representations. IMP can adapt its capacity to avoid underfitting by learning the cluster variance and by multi-modal clustering.

Within an episode, the cluster are initialized with class-wise means. Then, during inference, the distance to all existing clusters are computed for all support points. If the distance exceeds a threshold λ, a new cluster is made with mean equal to that point. IMP then updates soft cluster assignments z i,c as the normalized Gaussian density for cluster membership. Finally, cluster means µ c are computed by the weighted mean of their members. Since each class can have multiple clusters, query points x are classified by the softmax over distances to the closes cluster in each class n:

with c * n = arg min c:lc=n d(h φ (x ), µ c ) indexing the clusters, where each cluster c has label l c , and d(•, •) is the distance function of choice. IMP optimizes the embedding function h φ with parameters φ and cluster variances σ by stochastic gradient descent across episodes.

A.2.2 Meta-Curvature

The Meta-Curvature (MC) [START_REF] Park | Meta-curvature[END_REF] method learns a curvature for better generalization and faster model adaptation in the meta-learning framework. The curvature is learned along with the model's initial parameters simultaneously via the -IV - meta-learning process. The goal is that the meta-learned curvature works collaboratively with the meta-learned model's initial parameters to produce good generalization performance on new tasks with fewer gradient steps. The meta-curvature is computed efficiently through tensor algebra.

If we consider convolutional neural networks as our model, we can formally define meta-curvature matrices M o ∈ R Cout×Cout , M i ∈ R C in ×C in , and M f ∈ R d×d , where C out , C in and d are the number of output channels, the number of input channels and the filter size respectively. Then the meta-curvature function for a multidimensional tensor G is computed through n-mode products and has all matrices as learnable parameters:

Then, we can expand the matrices as follows:

where ⊗ is the Kronecker product, I k is the k dimensional identity matrix, and the three expanded matrices are all the same size M o , M i , M f ∈ R CoutC in d×CoutC in d . Finally, the gradients can be transformed with the meta-curvature learned as:

where L proto (S, Q, φ) = E (q,i)∼Q -log exp (-d(φ(q), c i ))

j exp (-d(φ(q), c j )) = E (q,i)∼Q [d(φ(q), c i )]

(1)

with c i = 1 k s∈S i φ(s) the prototype for class i, S i ⊆ S being the subset containing instances of S labeled with class i.

Distance For ProtoNet, we consider the Euclidean distance between the representation of a query example φ(q) and the prototype of a class i c i :

Then, with respect to class i, the first term is constant and do not affect the softmax probabilities. The remaining terms are:

Now we can recall and prove Theorem 4.3.1: Q, φ), the matrix of the optimal prototypes W * is well-conditioned, i.e. κ(W * ) = O( 1).

-VI -Proof. We can rewrite the first term in L proto as

and the second term as

By dropping the constant part in the loss, we obtain: -VII -Then, we can write the second term as E q∼Q log E c∼C•φ -1 exp (2φ(c) φ(q)) (A.21) .22) where C is the distribution of prototypes of S, i.e.each data point in C is the mean of all the points in S that share the same label, and C • φ -1 is the probability measure of prototypes, i.e.the pushforward measure of C via φ.

We now consider the following problem: (A.23)

The unique minimizer of Eq. A.23 is the uniform distribution on S d , as shown in [START_REF] Wang | Understanding contrastive representation learning through alignment and uniformity on the hypersphere[END_REF]. This means that learning with L proto leads to prototypes uniformly distributed in the embedding space. By considering W * the matrix of the optimal prototypes for each task then W * is well-conditioned, i.e.κ(W * ) = O(1).

A. .24), let Θ i := [θ i , θ i+1 ] T , and denote by W i 2 the matrix of last two predictors learned by MAML at iteration i starting from w 0 = 0 d . Then, we have that: ∀i, κ( W i+1

2 ) ≥ κ( W i 2 ), if σ min (Θ i ) = 0. (A.25)

Proof. We follow [START_REF] Arnold | When MAML can adapt fast and how to assist when it cannot[END_REF] and note that in the considered setup the gradient of the loss for each task is given by

so that the meta-training update for a single gradient step becomes: