Armand Joulin Ingénieur

Piotr Bojanowski

Keywords: self-supervised learning, computer vision, unsupervised learning, deep learning, artificial intelligence. v vi apprentissage auto-supervisé, vision par ordinateur, apprentissage non-supervisé, apprentissage profond, intelligence artifielle

Humans and many animals can see the world and understand it effortlessly which gives hope that visual perception could be realized by computers and Artificial Intelligence. More importantly, living beings acquire such an understanding of the visual world autonomously, without the intervention of a supervisor explicitly telling them what, where or who is to be seen. This suggests that visual perception can be achieved without too much explicit human supervision but simply by letting systems observe large amounts of visual inputs. In particular, this manuscript tackles the problem of self-supervised learning which consists in training deep neural networks without using any human annotations. Typically, neural networks require large amounts of annotated data, which have limited their applications in fields where accessing annotations is expensive or difficult. Moreover, manual annotations are biased towards a specific task and towards the annotators own biases, which can result in noisy and unreliable signals. Training systems without annotations could lead to better, more generic and robust representations. In this manuscript, we present different contributions to the fast-growing field of self-supervised visual representation learning.

In particular, we start by extending a promising category of self-supervised approaches, namely deep clustering, which trains deep networks while simultaneously mining groups of visually consistent images in a data collection. We then identify the limits of deep clustering methods such as their difficulty to scale to very large datasets or the fact that they are prone to trivial solutions. As a result, we propose novel and improved self-supervised methods that outperform their supervised counterparts on several benchmarks and exhibit interesting properties. For example, the resulting self-supervised networks contain generic representations that transfer well to different datasets and tasks. We also find that they contain explicit information about the semantic segmentation of an image. Finally, we make an effort throughout this manuscript to assess our self-supervised models in the wild, by training them on hundreds of millions of random unlabeled images from the Internet.

Acknowledgments are usually keenly anticipated by the reader and they arguably constitute one of the most important part of a PhD manuscript. But writing them is a difficult exercise. Indeed, trying to express the debt of tremendous gratitude I owe to all who have helped making this journey so rewarding and exciting is pretty wild and the following few hundred words will never capture all I want them to.

Without further ado, I would like to thank my supervisors, Piotr, Julien and Armand, for being the best. I feel very proud to have worked and grown with you during these past almost-four years and I cannot thank you enough for everything you have brought to me. Our achievements would simply not have been possible without your guidance, advice, countless ideas, patience and great rigor. Armand, I especially admire your vision, leadership, determination and inclination for setting high expectations throughout our different projects. I also want to thank you for your support and for advocating for me on multiple occasions. Julien I feel very fortunate to have ended up in your team at Thoth, which I think is a great and thriving working environment thanks to the way you manage it. Piotr, I thank you for creating such a relationship of trust between us. I owe you everything and I think you know how crucial your support has been during my PhD... Finally, I thank all three of you for gathering such an incredible jury for my defense. I cannot say how proud and honored I am to defend in front of researchers whom I admire and whose works have influenced and sometimes even directly seeded the contributions presented in this manuscript. I thank the different jury members for dedicating of their precious time to review the manuscript and attend the defense.

Throughout my PhD, I have had the chance to collaborate with different colleagues at Facebook and I would like to thank them: Ari, Benjamin, Daniel, Ed, Edouard, Gabriel, Herve, Kanika, Lucas, Marc, Matthew, Mike, Nicolas Ballas, Nicolas Carion, Patrick, Sreya, Theo, Timothee and Vasil. At the very beginning of this journey was Matthijs, whom I thank for giving me the chance to intern at FAIR before the PhD while I did not know anything, and for his continued support throughout my PhD. I also want to thank and acknowledge ix x ACKNOWLEDGMENTS Priya and Ishan for merging our efforts into a fruitful and exciting collaboration. Cocktails and ski are long overdue ;)! I was also very lucky to collaborate on different projects with other fellow PhD students like Alaa who knows so many papers and always keeps up with the latest approaches; Gautier who patiently taught me the rudiments of NLP; Hugo whose work impresses me so much and who beats me in GPU consumption on the FAIR cluster and Mido who is so nice, inclusive, hardworking and talented.

Overall, I would like to thank all my colleagues at Facebook for creating such an amazing workplace. I have loved each of our conversations and I hope we will stay in touch. In particular, I thank Y-Lan, Joelle and Herve whom I admire a lot and consider as mentors; I thank Laurens, Ross and Antoine for taking the time to discuss my plans after the PhD; I thank Alexis and Camille for believing in me when I was a master student; I thank Alexandre for on-boarding me and answering all my stupid questions during my internship; I thank Jakob, Matthijs and Timothee for making FAIR Grenoble a real thing; I thank Francisco for helping me all the time with PyTorch; I thank Yann LeCun for featuring me in prime-time TF1; last but not least, I thank the entire PhD crew at FAIR Paris and in particular Leonard for his hilarious memes and Pierre, Louis and Othman for collecting the best and the worst from public datasets. Finally I am grateful to the CIFRE program and to Nathalie, Patricia and Pierre-Louis thanks to whom I have had this amazing opportunity to work within FAIR while being a PhD student.

I wish to thank particularly the people who have helped me preparing the after-PhD during these last few months. My special thanks go to Louis; I cannot express how valuable your support has been throughout the entire process. I want to thank Jeremy, Timothee and Yana for helping me reaching the bar in coding interviews, I think they remember that I was not anywhere near that point when we started! I also thank Ahmet, Angela, Alexandre, Alexis, Armand, Guillaume, Herve, Ishan, Mido and Piotr for their general advice, feedback and help during this long process.

Throughout these three years at Inria Grenoble, I was lucky to cross the path of amazing PhD students, engineers and researchers. I have really appreciated the friendly and open-minded atmosphere of the Thoth team and I will miss among others the long tables at "Le Champ de l'Etoile", the coffee breaks and the amazing retreats organized by Julien and Nathalie. It is very important to me to thank my day-to-day companions of the team at Inria: Andrei, Ð.Khuê, Daan, Ekaterina, Hubert, Heesung, Jocelyn, Jules, Juliette, Karteek, Lina, Matthieu, Nathalie, Nikita, Pauline, Pierre-Louis, Ricardo, Roman and Zhiqi. I would like to give special thanks to the (extended) "team Juju" founded back in 2018; to Alberto for being so unique and for admonishing raclette-based dermatology advice; to Alexandre and Minttu for being the best caving companions; to Bruno for being an excellent Dixit player xi and for singing diligently; to Bulent for enjoyable coffee meetings; to Dexiong for the best karaoke, tea, hotpot and jiaozi sessions and for reminding Valentin and I we couldn't speak Chinese if our lives depended on it; to Enrico for insightful meme explanations; to Florent for being a great office-mate; to Gedeon for seeking the sun; to Gregoire for fervently tagging our mark _|| ||_ everywhere and to ally against our mutual enemy; to Konst for introducing me to (rainy) hikes in Grenoble; to Margot for obviously favoring my defense over her husband's one; to Masha for being extraordinary and for providing the most delicious lemon pie ever; to Michael for always being nice and for fun conversations over baguettes; to Pia who has encouraged me to color my hair; to Pierre for daring to go outside of his comfort zone at the retreat; to Sasha for great wine tasting skills and matching sense of competition; to Timothee for amazing blind test performance; to Theo for short coffee breaks and for supposedly making my life better; to Thomas for exquisite vim configs; to Valentin for dedicating so much effort into optimizing my office ergonomics; to Vlad for Russian candies and political regime debates; and finally I choose not to thank Houssam since he has been trying to organize a ski event the same day as the defense ;). I did not know anyone in Grenoble when arriving three years ago and I realize how fortunate I am to have forged such bonds of friendship and solidarity within the team.

My friends and family have given me a lot of support throughout this journey and I would like to thank in particular Aurelie for always being present; Chloe for her words in difficult times, her constant motivation and the best double dates with Antoine; Corentin for listening to my incessant complaints; Marion for offering me the most welcomed breaks during the PhD; Sedrique for always backing me up and never judging me; Timothee and Lauriane for being awesome; and all my friends of the Kes with a particular shout-out to Gwendo and Fabien for making all the way to Grenoble to cheer on me. I also thank the friends and family members that have supported me throughtout this journey and that I have not mentioned above: Antoine, Clement, Laetitia, Marc, and all my cousins, uncles and aunts; Stephane, Catherine, Boris and the entire Lucas family; and also Dasha, Elodie, Jack, Johanna, Ludo, Mayra, Olivier and Zach. I thank and apologize to the people that I might have failed to mention in these ackowledgements... Perhaps the most extraordinary part of this PhD journey has been to encounter Thomas, with whom I have shared every single piece of it and who already knows all I would like to tell him. Finally, I thank my dear parents and my beloved sister Chloe a thousand times for their endless support and love throughout all my life.

Chapter 1 Introduction

Visual Representation Learning

Designing data representation is a major component of many Artificial Intelligence (AI) systems and has evolved significantly in the last decades. The performance of a system depends heavily on the quality of the data representation it is given in input. For example, as humans, arithmetic will be more easily and quickly performed if quantities are represented with Arabic numerals rather than binary representations or Roman numerals. Likewise, in Machine Learning (ML), input representations play a central role. For example when a simple machine learning system is tasked to identify the risk for prostate cancer [START_REF] Stamey | Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate. ii. radical prostatectomy treated patients[END_REF], it does not interact with the patient directly. Instead, it inputs a set of different variables such as clinical and demographic conditions gathered by a specialist. This set of variables constitutes the patient representation seen by the ML algorithm, which can then make prediction by learning how these different variables interact with each other. However, simple ML algorithms do not act on the representation itself. In this manuscript instead, we will consider more advanced AI systems, able to build their own internal representations from raw input data.

Visual representations. Although studying representations is important in all domains of AI, we focus primarily on computer vision applications in this manuscript and more particularly on image recognition. Generally speaking, our goal is to improve the visual perception of AI systems, with the hope that it matches, or even outperforms, the natural ability of humans to perceive the visual world. Indeed, most living beings can see the world and understand it effortlessly without requiring tremendous amounts of energy, which motivates that visual perception could be realized by a computer. Intuitively, we would like an AI to reproduce the mental representations that we spontaneously have when looking at a scene. These high-level mental representations allow us to reason and understand what/where/how/who is represented in front of our eyes.

In computer vision, we refer to the representations extracted from raw images by visual representations, features or descriptors. Examples of such features can be specific structures and patterns, like edges, points or objects. They can also be given by a feature extractor function (or "encoder") that maps raw pixel values to a vector of fixed dimension which plays the role of the image representation. Good image featurization is difficult because it is not a well defined problem: it is not clear what information exactly needs to be extracted nor how to extract it. For example, let us imagine we want to create an application that detects dogs in images. Then, it might be a good idea to have the presence of muzzle in the image as a feature. However, describing precisely a muzzle in an image is tedious: it has a well defined shape but muzzle realizations in pixels are infinite due to varying lighting conditions, shadows, occlusions, etc.

The first generation of methods tackling the difficult problem of visual representations includes involved algorithms such as SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] and HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] that were obtained by manual design. These handcrafted extraction pipelines produce features with desirable properties like invariance to scale, illumination or rotation.

Deep learning. Another solution is to learn the visual representations directly from the data. A major shift of paradigm from machine learning to deep learning is to not only learn how to make prediction given a representation but to learn this representation itself from raw inputs, in an end-to-end fashion. Such learned representations usually result in much better performance compared to the handcrafted ones. They are also inherently more dynamic: deep learning models can learn automatically the set of features useful for a new given task while it can take years for researchers to discover these descriptors by successive trials and errors. Actually, the ability to learn and extract representations is directly baked in the deep neural network architectures used in deep learning. For example, a multi-layer perception (MLP) maps input to output by composition of successive parametrized simple functions. As a result, the output of each function (or "layer") can be thought of as a new representation of the input, which is then fed to the following layer. This way, the AI system can discover for itself intermediate representations that are useful for solving the considered task as illustrated in Figure 1.1. For example, the "XOR" classification problem is a well-known task that cannot be solved by a linear model. Yet, a very simple deep network, namely a MLP, can solve this problem since it has the ability to learn how to transform the data into an intermediate hidden representation that is linearly separable. work performed in this thesis is part of a general effort within the research community to improve self-supervised representation learning.

Why Learning Without Labels?

The goal of self-supervised representation learning is to automatically uncover some structure and understanding from images without having to label them one by one with human annotators. This setting might seem artificial, after all there already are large and richly annotated datasets like ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. In this section, we argue that studying self-supervised learning is important for several theoretical and practical considerations.

Towards true intelligence. When observing a collection of images, as humans we naturally and autonomously remark that similar structures, concepts and patterns appear across images without the intervention of a supervisor telling us what is to be seen. For instance, when observing photographs of an unknown object, we will be able to recognize it in future photographs, though not to name it, without being explicitly told that these are images representing the same object. Intuitively, we hypothesize that a truly intelligent system should be able to learn and reason about the visual world without the intervention of explicit human supervision, but simply by letting the AI observe and perceive the world. This way, studying self-supervised approaches could be a step towards more intelligent and autonomous systems.

Data biases. Another consideration is the fact that annotating data is inherently an illdefined and ambiguous process. Indeed, there are usually several forms of valid annotations and the arbitrary decisions taken by the annotator in the process induce noise and biases in the dataset. A self-supervised approach should suffer less from this inevitable bias and unreliable signal introduced by collecting supervised data. However, using self-supervised methods can remove the bias in the annotations only and will not impact the biases present in the images themselves. Overall, we hypothesize that pre-training systems without any annotations could lead to more generic and robust representations. Labels are expensive. From a down-to-earth perspective we observe that getting annotations is almost always expensive, which encourages the development of methods that train without them. Self-supervised training allows to acquire representations from any specific domain without requiring the tedious design of a large fully-supervised dataset for the considered domain.

For example, in histopathology, we surprisingly observe that most works in the literature [START_REF] Courtiol | Classification and disease localization in histopathology using only global labels: A weakly-supervised approach[END_REF] have been relying on the visual representations learned on ImageNet, an object-centric dataset with an over-representation of dogs [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. A natural assumption would be to think that features learned on natural-looking images mostly representing dogs are unlikely to transfer well to the medical domain due to the evident discrepancy between the considered domains. However, in the absence of large annotated datasets of histopathology data and in the absence of mature self-supervised approaches capable of successfully leveraging unlabeled data, this has been the most commonly adopted solution so far [START_REF] Courtiol | Classification and disease localization in histopathology using only global labels: A weakly-supervised approach[END_REF]. Hence, we hypothesize that designing better self-supervised approaches could lead to major performance leaps in similar fields where accessing annotations is difficult.

In addition, self-supervised learning may be a more dynamic and flexible strategy compared to the supervised approach. Whenever we encounter a new domain or modality, relying on supervised learning means that we need to constitute a new large annotated dataset. Let us imagine a new image capture modality appears, for instance RGB + a weak sense of depth (this is what dual-lens cameras on mobile phones produce). It would be extremely tedious to select and re-annotate the million of ImageNet images for this modality. Overall, the need for large amounts of annotations have limited the applications of neural networks, which calls for improving methods that work without labels.

Tons of unlabeled data. Last but not least, it is worth noting that unlabeled, raw data already exist in tremendous quantities on the Internet: for example, millions of images are uploaded on social media platforms every single day. Leveraging this data with supervised learning would require an unrealistic amount of manual annotations, despite the expert knowledge in crowd-sourcing accumulated by the community over the years. This is not scalable. Another way of using this data is to replace manual annotations by raw metadata, like hashtags or localization coordinates prediction. For example using raw metadata as a supervisory signal has been shown to perform well [START_REF] Joulin | Learning visual features from large weakly supervised data[END_REF][START_REF] Sun | Revisiting unreasonable effectiveness of data in deep learning era[END_REF], even surpassing ImageNet pre-training when trained on billions of images [START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF]. However, metadata are not always available, and when they are, they do not necessarily cover the full extent of a dataset. All in all, this motivates the design of methods that can be trained on internet-scale datasets with no supervision at all.

Outline and Contributions

Let us outline the organization of the manuscript: after a detailed overview of the different methods for visual self-supervised representation learning in Chapter 2, we present the different contributions conducted during this PhD program to the fast-growing field of self-supervised learning.

Deep clustering for representation learning on uncurated data

As we will detail in Chapter 2, most of the research in self-supervised learning has primarily focused on the design of new pretext tasks [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF][START_REF] Dosovitskiy | Discriminative unsupervised feature learning with exemplar convolutional neural networks[END_REF][START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF], Zhang et al., 2016]. Several interesting observations and results have emerged from these works but they have not exhibited features competitive with supervised representations obtained from labels classification. This suggests that the task of labels classification is sufficient for pre-training networks, provided that suitable labels are available. We note that in a supervised dataset, the labels partition the images into different groups (i.e. the classes). Based on this observation, in Chapter 3 we build upon self-supervised methods that aim at discovering such groups automatically by means of clustering. More precisely, our work revisits the DeepCluster framework which was published prior to the beginning of this PhD program in [START_REF] Caron | Unsupervised representation learning with clustering in deep convolutional networks[END_REF], [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF]. DeepCluster is a simple clustering method that jointly learns the parameters of a neural network and the cluster assignments, or "pseudo-labels" of the resulting features. Promising performance was reported in [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF] but most experiments and explorations were conducted in a highly controlled setting by training on ImageNet, a curated dataset made of carefully selected images to form well-balanced and diversified classes [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. Simply discarding the labels does not undo this careful selection, as it only removes part of the human supervision. Because of that, previous works that have experimented with noncurated raw data report a degradation of the quality of features [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF], Doersch et al., 2015]. The first contribution of this manuscript is to explore visual representations learning from unlabeled and non-curated datasets with deep clustering. We focus on the YFCC100M dataset [START_REF] Thomee | Yfcc100m: The new data in multimedia research[END_REF] as a source of uncurated data which contains 99 million images from the Flickr photo-sharing website. This dataset is unbalanced, with a "long-tail" distribution of hashtags contrasting with the well-behaved label distribution of ImageNet. For example in YFCC100M, guenon and baseball correspond to labels with 1300 associated images in ImageNet, while there are respectively 226 and 256, 758 images associated with these hashtags in YFCC100M. Our goal in Chapter 3 is to understand if trading manually curated data for scale leads to an improvement in the feature quality.

Outline. We start Chapter 3 by giving some preliminaries on DeepCluster [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF], a method for self-supervised representation learning with clustering published just before this PhD program. Then, we extend DeepCluster to the large-scale training on uncurated data which results in the improved DeeperCluster model. Finally, we establish the limitations of deep clustering, which we will tackle in the following chapter.

Publication. Chapter 3 is largely based on the paper "Unsupervised pre-training of image features on non-curated data", Mathilde Caron, Piotr Bojanowski, Julien Mairal and Armand Joulin, International Conference on Computer Vision, ICCV 2019 (see [START_REF] Caron | Unsupervised pre-training of image features on non-curated data[END_REF]). The code to reproduce results presented in this chapter is publicly available at https://github.com/facebookresearch/deepcluster and at https: //github.com/facebookresearch/DeeperCluster.

Overcoming deep clustering limitations with SwAV

Models combining representation learning and clustering, i.e. Deep(er)Cluster, learn good representations that improve the performance of neural networks on downstream tasks compared to training from scratch as detailed in Chapter 3. However, deep clustering in its original formulation yields a number of major limitations. Among these is the lack of scalability. Indeed, before each epoch a new clustering is performed to provide pseudolabels to use during the epoch. This iterative process relies on the size of dataset and does not scale to very large-scale trainings where only one or two epochs are typically performed. A straightforward way to overcome this limitation would be to use an online version of kmeans [START_REF] Zhan | Online deep clustering for unsupervised representation learning[END_REF]. However, similarly to [START_REF] Asano | Self-labelling via simultaneous clustering and representation learning[END_REF] and Chen et al. [2020d], we have been questioning the very importance of k-means clustering into our model. Other limitations include the use of empirical tricks to avoid trivial parametrizations or the fact that dependence in data augmentation is crucial but only implicitly and not properly accounted for. We address these different limitations by designing a new model, SwAV, that learns features by directly comparing the descriptors of different crops of a same image. We impose an equipartition constraint of the descriptors thanks to Optimal Transport [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF] to prevent all images in a mini-batch to collapse to the same representation. Specifically, SwAV simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or "views") of the same image. We validate our model by improving the self-supervised state of the art on ImageNet, as well as surpassing supervised pre-training on a large benchmark of transfer tasks and datasets.

Outline. Chapter 4 introduces SwAV, a method for online self-supervised learning based on matching distorted views of a same image. First, we present the method and probe its performance on ImageNet before assessing SwAV in a more realistic scenario by training on 1 billion uncurated images from Instagram.

Publication. Chapter 4 is primarily based on the paper "Unsupervised learning of visual features by contrasting cluster assignments", Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski and Armand Joulin, Conference on Neural Information Processing Systems, NeurIPS 2020 (see [START_REF]1007 73.5 66[END_REF]). Code and models for SwAV can be found at https://github.com/facebookresearch/swav. This chapter also contains pieces of work publicly available in the technical report "Self-supervised Pretraining of Visual Features in the Wild", Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin and Piotr Bojanowski (see [START_REF] Goyal | Self-supervised pretraining of visual features in the wild[END_REF]). Note that we also conducted and published an extension of SwAV to semi-supervised learning in "Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples", Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas and Michael Rabbat, International Conference on Computer Vision, ICCV 2021 (see [START_REF] Assran | Semisupervised learning of visual features by non-parametrically predicting view assignments with support samples[END_REF]) which we do not present in this manuscript for conciseness.

Improving self-supervised learning with vision transformers

In Chapter 5, we take an orthogonal direction by exploring if we can use the progress in network architectures to improve self-supervised representations. Indeed, transformers [START_REF] Vaswani | Attention is all you need[END_REF] have recently emerged as an alternative to convolutional neural networks (convnets) for visual recognition [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF][START_REF] Zhao | Exploring self-attention for image recognition[END_REF]. Their adoption has been coupled with a training strategy inspired by natural language processing (NLP), that is, pre-training on large quantities of data and finetuning on the target dataset [START_REF] Devlin | Pre-training of deep bidirectional transformers for language understanding[END_REF], Radford et al., 2019]. The resulting Vision Transformers (ViT) [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] are competitive with convnets but, they currently do not have substantial benefits over convnets: they are computationally more demanding, require more training data, and there is no evidence of particular properties arising in their features.

In this final contribution in Chapter 5, we question if a reason for these setbacks has been the use of supervision when pre-training these networks on images and explore selfsupervised pre-training as an alternative. The motivation is that a core feature of BERT pre-training [START_REF] Devlin | Pre-training of deep bidirectional transformers for language understanding[END_REF] is the use of a self-supervised task where the network learns to predict masked words in a sentence. These words provide a richer signal to learn from than a single label per sentence. The problem is similar in the case of images since image-level supervision often reduces the diversity in an image to a single concept from a predefined set of a few thousand categories [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. However, as shown in [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF], directly using the BERT pretext task has limited success and may not be the optimal strategy for images. We address this problem by investigating if self-supervised methods originally designed for convnets provide additional benefits to the features produced by ViTs. During this process, we have identified interesting properties that do not emerge with supervised ViTs, nor with convnets:

-Self-supervised ViT features explicitly contain the scene layout and, in particular, object boundaries. This information is directly accessible in the self-attention modules of the last block.

-Self-supervised ViT features perform particularly well with a basic nearest neighbors classifier (k-NN) without any finetuning, linear classifier nor data augmentation, achieving 78.3% top-1 accuracy on ImageNet.

Outline. Chapter 5 starts by introducing a new method for self-supervised learning, DINO, specifically designed to work well with vision transformers. We then expose different interesting properties that emerge when training vision transformers in a selfsupervised setting. Overall we find that combining vision transformers and self-supervised learning bring considerable boosts of performance in the standard benchmark and reveals intriguing properties.

Publication. Chapter 5 is based on the paper "Emerging properties in self-supervised vision transformers", Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski and Armand Joulin, International Conference on Computer Vision, ICCV 2021 (see [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF]). The code is available at https://github. com/facebookresearch/dino.

Finally, to wrap up, we summarize in Chapter 6 our contributions and give a (subjective) overview of the current challenges in self-supervised learning.

Chapter 2

Related work

Self-supervised representation learning lives at the intersection of two broad domains of deep learning: unsupervised learning and representation learning. In the words of [START_REF] Goodfellow | Deep Learning[END_REF], the former refers to most attempts to extract information from a distribution that do not require human labor to annotate examples. On the other hand, the latter is the problem of learning a good feature extractor from the data. By "good", we mean a function that produces general-purpose visual representations useful for solving downstream tasks. In our work, we choose to instantiate this featurization function with a parametrized deep neural network and we task ourselves to learn its parameters (or weights) without using any manual annotations.

The prominent way of learning the weights of a neural network is to minimize a cost function using mini-batch stochastic gradient descent [START_REF] Bottou | Stochastic gradient descent tricks[END_REF] and backpropagation to compute the gradients of the cost with respect to the network weights [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]]. In the supervised case, this cost (or task, loss, objective) function is largely defined by the annotations and simply measures how well the output of the network matches the annotated ground truth. However, in the absence of annotations, defining a learning task is an open problem.

A solution is to use a surrogate (or pretext) task automatically generated from unlabeled data. When trained for this task, the network shall hopefully acquire an understanding of the structure of the scene and objects present in the images and, as a result, produce features that perform well on downstream tasks [START_REF] Ahmed | Training hierarchical feed-forward visual recognition models using transfer learning from pseudo-tasks[END_REF]. In this section, we give a non-exhaustive overview of the numerous methods developed in the image recognition literature to tackle the challenging problem of designing good label-free pretext tasks. 11

Generating Images

First of all, a classical unsupervised task is to ask a deep model to reconstruct its input data, and even to generate new probable inputs. We refer to this class of methods as generative approaches since they output in the image space and directly model the data generating function. Intuitively, the generative surrogate task of "predicting the data" should encourage the model to understand the data and build useful representations of it. For example, as humans, painting an existing, or imaginary, house requires some mental representation of what a house is and of its structure. Likewise, fitting the data generating distribution shall require a deep model to uncover some underlying structure of the data and to build rich, abstract representations of it. In this section, we briefly describe popular generative models like autoencoders or generative adversarial networks.

Autoencoders

A quintessential example of an unsupervised representation learning approach is the autoencoder model [START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF][START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF][START_REF] Huang | Unsupervised learning of invariant feature hierarchies with applications to object recognition[END_REF][START_REF] Masci | Stacked convolutional auto-encoders for hierarchical feature extraction[END_REF][START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF]. It is composed of two parts: an encoder that maps some given input to an intermediate representation, and a decoder that inputs this representation and outputs back into the input space. The two networks are trained together with a reconstruction objective. In other words, the encoded-then-decoded output has be as close as possible as the original input. It is important to constrain somehow the intermediate representation otherwise both networks could trivially learn the identity mapping. Examples of such constraints include dimension bottleneck [START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF], data corruption as in denoising autoencoders [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF] or information bottleneck as in variational autoencoders (VAE) [START_REF] Kingma | Auto-encoding variational bayes[END_REF].

Interestingly, VAEs belong to a class of generative approaches, namely the latent variable models, that propose to explain entirely the data through different and distinct factors of variations. Such factors may be the nature of the represented object, the lighting conditions or the angle of the camera for example. The factors of variations, or latent variables, can be directly viewed as representations of the data as they have the power of describing any image from the data generating function. Follows the conditional independence assumption which states that all pixels of an image can be generated at once (i.e. independently from each other) given the knowledge of the image's corresponding latent variables. This is in contrast with the autoregressive models which we mention later in Section 2.1.3 that generate each pixel sequentially given all the previously generated pixels.

Generative adversarial networks

Another latent variable based generative model is the popular generative adversarial network (GAN) proposed by [START_REF] Goodfellow | Generative adversarial nets[END_REF]. A GAN is made up of two neural networks, namely the generator and the discriminator, that compete against one another. The generator produces images given vectors of the latent space while the discriminator has to determine whether the images it sees are real (i.e. come from the dataset) or not (i.e. have been generated by the generator). This way, the generator is encouraged to produce realistic images in order to fool the discriminator. Interestingly, some experiments on GANs by [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF] have shown that their latent space captures semantic variations in the data distribution and that the representations learned by the generator and discriminator can be transferred to other visual tasks. However, the performance gains are (arguably) moderate. Moreover, it is unclear in a GAN what component of the model to re-use as a feature encoder (the generator maps from latent space to image space and the discriminator is trained to separate real from generated images). To overcome this difficulty, [START_REF] Donahue | Adversarial feature learning[END_REF] and [START_REF] Dumoulin | Adversarially learned inference[END_REF] concurrently propose to enhance the representation learning capacity of the original GAN by adding an auxiliary encoder, which acts as a feature encoder. The purpose of this encoder is to predict directly from images their corresponding semantic representations in the latent space. In other words, it learns the inverse mapping of the generator, which allows to produce competitive visual features. More recently, GAN implementation and training have improved drastically thanks to thorough and large-scale explorations [START_REF] Brock | Large scale gan training for high fidelity natural image synthesis[END_REF]. As expected, this has also led to improved performance for their internal representations [START_REF] Donahue | Large scale adversarial representation learning[END_REF].

Autoregressive models

Autoregressive methods drop the latent variables strategy and simply generate pixels sequentially based on the previously generated pixels. A successful implementation with convolutional neural networks is the Pixel-CNN of [START_REF] Oord | Conditional image generation with pixelcnn decoders[END_REF]. Recently, Chen et al. [2020a] have proposed to train a transformer [START_REF] Vaswani | Attention is all you need[END_REF] for the autoregressive next pixel prediction task. The resulting representations have been the most competitive within the generative representation learning family so far.

Overall, generative modeling is a promising and fruitful direction in unsupervised representation learning. However, fitting the data generating function is a very complex learning task with a number of considerable challenges (see [START_REF] Lucas | Deep generative models : over-generalisation and mode-dropping[END_REF] for a comprehensive overview). Moreover, we may wonder if being able to predict everything is necessary to learn useful visual representations. For example, the fact that in the supervised case, labels classification is enough to bring about good representations motivates the development of not need to be provided by annotators. Intuitively, performing well on the surrogate colorization task requires a semantic understanding of the extent of objects and scenes present in the images, which encourages the network to learn high level features. However, color does not always exist in images (in the medical domain for example) and consequently [START_REF] Zhang | Split-brain autoencoders: Unsupervised learning by cross-channel prediction[END_REF] have proposed to recast colorization to the more generic scenario of cross-channel prediction.

Spatial cues

Another landmark strategy in self-supervised learning is to manipulate and exploit the spatial layout of images to generate surrogate tasks [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF]. For instance, inspired by word2vec [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] in NLP, [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF] propose to train a network to predict the relative position of a pair of patches from the same image as illustrated in the middle panel of Figure 2.1. Intuitively, this difficult task should force the network to learn rich representations of object parts and their spacial arrangement. For example, being able to predict that a car wheel patch is usually located below a windscreen patch requires some understanding of the concept of vehicles. A number of follow-up works [START_REF] Kim | Learning image representations by completing damaged jigsaw puzzles[END_REF][START_REF] Lee | Unsupervised representation learning by sorting sequences[END_REF][START_REF] Mundhenk | Improvements to context based self-supervised learning[END_REF][START_REF] Cruz | Deeppermnet: Visual permutation learning[END_REF] have proposed several extensions and refinements along the seminal work of [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF]. Of particular interest, [START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF] extend the relative patch position prediction task to 9 patches, sampled on a 3 ◊ 3 grid in the image. The task of the network is to predict the permutation applied to the set of shuffled patches. [START_REF] Goyal | Scaling and benchmarking self-supervised visual representation learning[END_REF] show that making this task even more complex by increasing the number of possible patch permutations usually improves the quality of the resulting representations.

Another self-supervised approach exploiting spatial cues is the inpainting task (see right panel of Figure 2.1). [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF] propose to train a network to generate an image region from its surroundings. A network encodes the context of the image into a compact latent representation while a decoder inputs that latent representation and fills in the missing image patch pixels. The connection between these two networks is performed with a channel-wise fully connected layer that allows propagation of information across channels. The authors use a combination of two different losses: (i) a reconstruction loss that tends to give blurry patches since it is safer to predict the mean of the distribution, and (ii) an adversarial loss that encourages sharpest and more realistic results.

Overall, these works exploiting spatial layout in images have particularly inspired one of the contribution of this thesis, the multi-crop training, which we present in Chapter 4 (see Section 4.2.4). prediction, [START_REF] Bibliography | Learning to see by moving[END_REF] train a network to predict the camera transformation between pairs of images. Another example is the work of [START_REF] Jayaraman | Learning image representations tied to ego-motion[END_REF] that learn image embedding from unlabeled video. Starting from egocentric video together with observer ego-motion signals, they train a system on a "view prediction" task, to learn equivariant visual features that respond predictably to observer ego-motion. In this target equivariant feature space, pairs of images related by the same ego-motion are related by the same feature transformation too. Similarly, [START_REF] Xiong | Self-supervised representation learning from flow equivariance[END_REF] encourage the features to obey the same flow transformation as input image pairs.

Taking a different approach, [START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF] propose a task where patches from a same video should be close in feature space. More precisely, from an anchor patch in a video, they form a positive pair by tracking it along thirty consecutive frames and form a negative pair with a random (or more sophistically selected) patch. The task of the network is to bring the positive pair closer together and to pull away the negative pair. The work of [START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF] echoes the view-invariant features paradigm which we describe later in Section 2.3 and uses motion as a way of generating different "views" of a common concept.

A different approach is that of [START_REF] Pathak | Learning features by watching objects move[END_REF] who use motion to mine unsupervised segmentations from videos. Then, a deep network is trained to predict those "ground truth" segmentations from images. As shown in the corresponding paper and in Figure 2.2 (right), the unsupervised segmentations are noisy and of quite poor quality. However, because they do not exhibit consistent or systematic error patterns, they may be seen as perturbations around a good quality segmentation. The motivation of [START_REF] Pathak | Learning features by watching objects move[END_REF] is to hypothesize that a deep network will not fit this noise but output something closer to the underlying correct segmentation. This is qualitatively verified in Figure 2.2 (right) where we observe that the learned segmentations are of better quality that the noisy ground-truth segmentations used during training.

Though promising, a pitfall of these approaches relying on motion is the dependency on auxiliary information or offline algorithms, which might make them impractical in some cases. For example, if metadata about the camera is not de facto present in a visual collections, it might turn out to be as least as tedious to collect as the manual annotations used in supervised learning.

Other cues

Many other cues have been exploited in order to design self-supervised tasks, for example audio [START_REF] Owens | Ambient sound provides supervision for visual learning[END_REF], counting instances [START_REF] Noroozi | Representation learning by learning to count[END_REF] or physical interaction [START_REF] Pinto | The curious robot: Learning visual representations via physical interactions[END_REF]. Interestingly, [START_REF] Li | Unsupervised visual representation learning by graph-based consistent constraints[END_REF] leverage the first generation of handcrafted visual representations to learn representations with deep networks. Concretely, they build a graph of nearest neighbors on SIFT [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF] and Fisher Vector [START_REF] Sánchez | Image classification with the fisher vector: Theory and practice[END_REF] descriptors. From this graph, they infer positive image pairs from cycle consistency and negative image pairs from large geodesic distances (accumulated weights along the shortest path between the two considered nodes). The task of the network is to learn an embedding in which images semantically similar (positive pairs) are close while images semantically different (negative pairs) are distant.

Combining cues

The nature of what the features capture and focus on is likely to differ from a selfsupervised surrogate task to another. As a result, different strategies for combining multiple cues have been explored [Doersch andZisserman, 2017, Wang et al., 2017]. [START_REF] Wang | Transitive invariance for self-supervised visual representation learning[END_REF] propose to construct an affinity graph with two types of edges: inter-instance invariance (similar viewpoint) and intra-instance invariance (similar instance). Inter-instance edges are found with the relative patch prediction task of [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF] while intra-instance edges are mined through tracking a patch along thirty frames of a video like in [START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF]. From this affinity graph, they infer positive (resp. negative) pairs that need to be brought together (resp. pulled away) by the network. [START_REF] Doersch | Multi-task self-supervised visual learning[END_REF] also aim at leveraging the diversity of the learned representation by combining different pretext tasks. A novelty in the approach of [START_REF] Doersch | Multi-task self-supervised visual learning[END_REF] is to use a form of regularization that encourages the network to separate group of features useful for different tasks. In other words, the network determines which combination of layers to use for each of the pretext tasks.

Overall, we have described a number of diverse self-supervised tasks proposed in the literature in the past years. Lastly, we will describe in more details self-supervised approaches that encourage view-invariant representations [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF]. This paradigm has been the most successful lately [START_REF]1007 73.5 66[END_REF], Chen et al., 2020b, Grill et al., 2020[START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF] and is the seed to most of the contributions presented in this manuscript [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF][START_REF] Caron | Unsupervised pre-training of image features on non-curated data[END_REF][START_REF]1007 73.5 66[END_REF][START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF].

View-Invariant Features

The seminal work of [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF] propose to learn representations that model view-invariant factors. This is motivated by the fact that the way we view a scene does not usually affect its semantic content. For example, the meaning or purpose of an image of a dog is most of the time preserved regardless of the position of the camera 2.3.1 Original parametric implementation [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF][START_REF] Dosovitskiy | Discriminative unsupervised feature learning with exemplar convolutional neural networks[END_REF] implement the view-invariant paradigm as an instance classification approach. The problem is formulated as an "N -way" classification problem where N is the size of the dataset. In other words, each image acts as a distinct class and a deep network is trained to discriminate between them up to data augmentations. More concretely, let us denote by f θ the neural network mapping, where ◊ is the set of corresponding parameters. We refer to the vector obtained by applying this mapping to an image as feature or representation. Given a training set X = {x 1 ,x 2 ,...,x N } of N images, we recall that our goal is to find a parameter ◊ ú such that the mapping f θ ú produces good general-purpose features. In the context of supervised learning, each image x n is associated with a label y n in [0,k≠ 1] which represents the image's membership to one of k possible predefined classes. A parametrized classifier g W predicts the correct labels on top of the features f θ (x n ). The parameters W of the classifier and the parameter ◊ of the mapping are then jointly learned by minimizing the following per-example negative log likelihood loss on temperature-softmax outputs:

min θ,W ≠g W (f θ (x n )) yn /• + log k≠1 ÿ j=0 exp 1 g W (f θ (x n )) j /• 2 (2.1)
Note that we explicit here the per-example loss only and in practice we minimize its expectation over the training dataset. In the work of [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF], a random transformation t is applied to each original dataset image x n . We denote by T the set of all possible data transformations. Overall, the following per-example instance classification objective is minimized:

min θ,W E t≥T S U ≠g W (f θ (t(x n ))) n /• + log k≠1 ÿ j=0 exp 1 g W (f θ (t(x n ))) j /• 2 T V , (2.2)
where the label y n of each transformed image is simply the id n of the original image. Explicitly learning a classifier g W to discriminate between all images does not scale well with the number of images N . For example on ImageNet, W would be of size 1, 281, 167 ◊ d where d is the feature dimension. To overcome this major limitation, a number of approaches propose to use a noise contrastive estimator (NCE) [START_REF] Gutmann | Noise-contrastive estimation: A new estimation principle for unnormalized statistical models[END_REF] to directly compare instances instead of classifying them [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF]. These methods are collected under the umbrella of "contrastive" representation learning.

Contrastive implementations

Contrastive methods drop learning a parametrized instance-level classifier g W altogether. Instead, the task becomes to retrieve positive pairs of features out of a pool of negative pairs. Concretely, given an anchor feature representation f θ (t(x n )), a positive pair is formed by featurizing a different view of the same instance x n (denoted v + ). Negative pairs are formed by featurizing a set v ≠ of k different instances from the dataset. Contrastive implementations like that of [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF], [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] or Chen et al. [2020b] minimize a per-example objective of the following form:

min θ,W E t≥T S U ≠f θ (t(x n )) | v + /• + log ÿ voev ≠ exp (f θ (t(x n )) | v/•) T V , (2.3)
where the different feature vectors are normalized. This loss can be interpreted as the log loss of a "|v ≠ | +1"-way softmax-based classifier that tries to classify the anchor as v + . This contrastive implementation requires comparing features from a large number of images simultaneously and different strategies have been proposed to that end. For example, [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF] store positive element v + and negative elements v ≠ into a memory bank composed of computations from the previous epochs. As a result, this approach relies on the size of the dataset and does not scale to very large-scale trainings. [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] propose an online memory bank strategy as a workaround. Computations from the previous iterations are stored in a rolling memory bank. Positive and negatives features are computed with an auxiliary dedicated momentum network which is an exponential moving average version of the main network. This ensures consistency between the features of the memory bank. Another approach is to simply use the examples from the same mini-batch as positive and negatives [Chen et al., 2020b]. Contrastive loss functions can also be based on other forms, such as margin-based losses and variants of NCE losses [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF][START_REF] Henaff | Data-efficient image recognition with contrastive predictive coding[END_REF][START_REF] Hjelm | Learning deep representations by mutual information estimation and maximization[END_REF][START_REF] Oord | Representation learning with contrastive predictive coding[END_REF], Tian et al., 2020a[START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF].

Other implementations

A caveat of the contrastive approach is that it requires comparing features from a large number of images simultaneously to have good performance. Interestingly, a number of approaches have implemented the "view-invariant" principle without falling back on the contrastive formulation. Among these methods are the SwAV and DINO contributions presented respectively in Chapters 4 and 5.

At the root of this thesis is the work of [START_REF] Bojanowski | Unsupervised learning by predicting noise[END_REF] which implements the instance discrimination problem by means of discriminative clustering [START_REF] Bach | Diffrac: a discriminative and flexible framework for clustering[END_REF]. Discriminative clustering optimizes a grouping of the data into distinct clusters that, if used as pseudo-labels to learn a classifier, produces the maximum of separability (as measured by the loss value at an optimal classifier). Note that it is crucial to add constraints to avoid the trivial solution where all examples have the same pseudo-label. Different classifiers can be used, leading to different implementations of the discriminative clustering framework. For instance, [START_REF] Bach | Diffrac: a discriminative and flexible framework for clustering[END_REF] use a linear classifier with a ridge regression loss. [START_REF] Bojanowski | Unsupervised learning by predicting noise[END_REF] use a deep network that classifies into a set of N fixed targets uniformly arranged on the unit sphere in feature space. This way, the loss encourages representations to be uniformly spread out in the feature space. The approach iterates between training the network to classify the examples into their assigned targets and obtaining new target assignments for the representations. As different views are used for assigning and predicting the targets, the model trains implicitly for view-invariant representations. This work has influenced greatly the contributions presented in this manuscript (Chapters 3 and 4).

Interestingly, recent works have shown that we can instantiate the view-invariant learning paradigm without explicitly discriminating between images. [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF] propose a metric-learning formulation called BYOL, where features are trained by matching them to representations obtained with a momentum encoder. It has been shown that methods like BYOL work even without a momentum encoder, at the cost of a drop of performance [Chen andHe, 2020, Grill et al., 2020]. We note that the contribution presented in Chapter 5 completes the interpretation initiated in BYOL of self-supervised learning as a form of Mean Teacher self-distillation [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF] with no labels. Several other works echo this direction, showing that one can train features by using whitening [START_REF] Ermolov | Whitening for self-supervised representation learning[END_REF] or redundancy reduction of the features [START_REF] Zbontar | Barlow twins: Self-supervised learning via redundancy reduction[END_REF]. The latter work proposes a loss function that measure the cross-correlation matrix between the representations of two distorted versions of a sample, and makes it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.

Grouping variants

We have seen that most of the implementations of the view-invariant paradigm follow a per-instance discrimination framework. However, pulling representations from different instances apart can imper the learned embeddings if it turns out that these instances are actually semantically similar. This limitation was already identified in the analysis of [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF]. As a workaround, they propose to use clustering to group the instances with strong similarities into a single pseudo-class instead of having a pseudo-class per instance as in their original formulation. Their results show that this variant improves performance of the model and foresee that potentially, using even more data or performing clustering and network training within a unified framework could further improve the quality of the learned features. This analysis has influenced greatly the development of deep clustering approaches which we detail in Chapter 3.

Other works propose to model inter-examples similarities thanks to clustering. Some methods rely explicitly on a clustering loss [START_REF] Asano | Self-labelling via simultaneous clustering and representation learning[END_REF], Caron et al., 2018[START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF], Yang et al., 2016] to learn simultaneously image features and clusters. The approach of [START_REF] Yang | Joint unsupervised learning of deep representations and image clusters[END_REF] iterates over a clustering and a training stages. The former optimizes a grouping of the representations given by the network. The latter trains the network to classify the images into the pseudo-labels given by the group assignments. [START_REF] Yang | Joint unsupervised learning of deep representations and image clusters[END_REF] use agglomerative clustering, which starts with a large number of very small clusters and merges them progressively. However, [START_REF] Douze | An evaluation of large-scale methods for image instance and class discovery[END_REF] show that this choice of clustering method is not optimal when working with large-scale datasets.

Importance of spatial cues

An effective direction to improve performance of the view-invariant paradigm is to consider transformations targeting distortions of the spatial layout of an image (i.e. cropping and scaling for example). Indeed, many works have exploited spatial cues to induce desired invariants or properties in the representations [START_REF] Hjelm | Learning deep representations by mutual information estimation and maximization[END_REF][START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF][START_REF] Oord | Representation learning with contrastive predictive coding[END_REF]. For example, [START_REF] Hjelm | Learning deep representations by mutual information estimation and maximization[END_REF] and [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF] define a surrogate task where local representations of an image (i.e. representation of a small image portion) need to be predicted from a global image descriptor (i.e. representation of a large portion) obtained with another view of that image. Besides, [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF] and [START_REF] Hénaff | Data-efficient image recognition with contrastive predictive coding[END_REF] encourage matching of adjacent non-overlapping views of a same image. Interestingly, Chen et al. [2020b] highlight the importance of the random scaling and cropping data augmentation (torchvision.transforms.RandomResizedCrop in PyTorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF]) and show that using it to generate different matching views naturally encompasses the two learning tasks above (i.e. "global-to-local" and "adjacent" view predictions) and many more. Intuitively, encouraging "crop-invariant" features is an interesting idea. Concretely, it means that a crop of an image needs to be predicted from another crop of the same image. To do so, the network has to either output a constant representation for every possible input (which is prevented by different constraints or designs) or to actually output a representation about that image. Importantly, this representation needs to be about the image in general and not just about the particular considered crop since it then needs to be predicted from another completely independent crop of that same image. Based on that observation, one contribution of this thesis is to encourage holistic "local-to-global" matching of the views, which we show to be very effective for downstream classification tasks (see details of multi-crop in Section 4.2.4).

Other variants and refinements

Finally, a number of variants and refinements have been proposed to the original view-invariant self-supervised paradigm. For instance, some works [START_REF] Kalantidis | Hard negative mixing for contrastive learning[END_REF][START_REF] Zhuang | Local aggregation for unsupervised learning of visual embeddings[END_REF] propose different strategies to mine hard negatives in the contrastive learning implementation (Section 2.3.2). Other works ablate and analyze the different invariant injected through the data augmentation pipeline [START_REF] Purushwalkam | Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases[END_REF]Gupta, 2020, Tian et al., 2020b] or propose to match more elaborate representations, e.g., Bag-of-Words [Gidaris et al., 2020a,b]. Other related directions, beyond the scope of this manuscript, are the focus on designing dense local representations [START_REF] Jabri | Space-time correspondence as a contrastive random walk[END_REF][START_REF] Pinheiro | Unsupervised learning of dense visual representations[END_REF], or the use of different modalities as a way of generating the different "views" of a common concept [Alayrac et al., 2020, Wang andGupta, 2015].

Chapter 3 Deep Clustering

We have seen in Chapter 2 that research in self-supervised learning has primarily mainly focused on generative modeling [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF] or on designing surrogate tasks for which the labels can be computed automatically by manipulation of the input data [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF][START_REF] Dosovitskiy | Discriminative unsupervised feature learning with exemplar convolutional neural networks[END_REF][START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF], Zhang et al., 2016]. These works pioneered the field of visual self-supervision but, at the time of their publication, showed moderate results compared to that of pre-training with supervised ImageNet classification. This suggests that the task of classification is promising for representation learning, given that labels are provided to that end.

A trivial but important observation is to see that the labels of ImageNet partitions the dataset into distinct groups (i.e. the classes). The seed of this chapter is to ask whether we can find those groups automatically by means of clustering. As a matter of fact, clustering, a class of unsupervised learning methods, has been extensively applied and studied in computer vision. However, only little work has been done to adapt it to the end-to-end training of visual features on large scale datasets. In this chapter, we start by giving some preliminaries on deep clustering with DeepCluster, a clustering method presented at ECCV in Munich [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF] (prior to the PhD program) that jointly learns the parameters of a neural network and the cluster assignments of the resulting features.

Second, we study representation learning with deep clustering in an uncurated scenario. We train our model on 96 million images from YFCC100M [START_REF] Thomee | Yfcc100m: The new data in multimedia research[END_REF], achieving state-of-the-art results among unsupervised methods on standard benchmarks, which confirms the potential of self-supervised learning in a realistic scenario, when only uncurated raw data are available. This effort was presented at ICCV in Seoul [START_REF] Caron | Unsupervised pre-training of image features on non-curated data[END_REF] and is the base of this chapter. 25

Introduction

Self-supervised representation learning with clustering

Pre-trained convolutional neural networks, or convnets, have become the building blocks in most computer vision applications [START_REF] Carreira | Human pose estimation with iterative error feedback[END_REF][START_REF] Chen | Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[END_REF], Ren et al., 2015[START_REF] Weinzaepfel | Deepflow: Large displacement optical flow with deep matching[END_REF]. They produce excellent general-purpose features that can be used to improve the generalization of models learned on a limited amount of data [START_REF] Sharif Razavian | Cnn features off-the-shelf: an astounding baseline for recognition[END_REF]. The existence of ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF], a large fully-supervised dataset, has been fueling advances in pre-training of convnets. However, [START_REF] Stock | Convnets and imagenet beyond accuracy: Explanations, bias detection, adversarial examples and model criticism[END_REF] have presented empirical evidence that the performance of state-of-the-art classifiers on ImageNet is largely underestimated, and little error is left unresolved. This explains in part why the performance has been saturating despite the numerous novel architectures proposed in recent years [START_REF] Chen | Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[END_REF][START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] El-Nouby | Xcit: Cross-covariance image transformers[END_REF], He et al., 2015, Huang et al., 2016]. As a matter of fact, ImageNet is relatively small by today's standards; it "only" contains a million images that cover the specific domain of object classification. A natural way to move forward is to build a bigger and more diverse dataset, potentially consisting of billions of images. This, in turn, would require a tremendous amount of manual annotations, despite the expert knowledge in crowd-sourcing accumulated by the community over the years. Replacing labels by raw metadata leads to biases in the visual representations with unpredictable consequences [Misra et al., 2016b]. This calls for methods that can be trained on internet-scale datasets with no supervision.

Unsupervised learning has been widely studied in the Machine Learning community [START_REF] Friedman | The elements of statistical learning[END_REF], and algorithms for clustering, dimension reduction or density estimation are regularly used in computer vision applications [START_REF] Joulin | Discriminative clustering for image co-segmentation[END_REF][START_REF] Shi | Normalized cuts and image segmentation[END_REF][START_REF] Turk | Face recognition using eigenfaces[END_REF]. For example, the "bag of features" model uses clustering on handcrafted local descriptors to produce good image-level features [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF]. A key reason for their success is that they can be applied on any specific domain or dataset, like satellite or medical images, or on images captured with a new modality, like depth, where annotations are not always available in quantity. Several works have shown that it was possible to adapt unsupervised methods based on density estimation or dimension reduction to deep models [Goodfellow et al., 2014, Kingma and[START_REF] Kingma | Auto-encoding variational bayes[END_REF], leading to promising all-purpose visual features [Bojanowski andJoulin, 2017, Donahue et al., 2016]. Despite the primeval success of clustering approaches in image classification, very few works [START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF], Yang et al., 2016] have been proposed to adapt them to the end-to-end training of convnets, and never at scale. An issue is that clustering methods have been primarily designed for linear models on top of fixed features, Figure 3.1 -Illustration of DeepCluster. We iteratively cluster deep features and use the cluster assignments as pseudo-labels to learn the parameters of the convnet. and they scarcely work if the features have to be learned simultaneously. For example, learning a convnet with k-means would lead to a trivial solution where the features are zeroed, and the clusters are collapsed into a single entity.

In preliminaries, we review DeepCluster, a self-supervised approach that show that it is possible to obtain useful general-purpose visual features with a clustering framework. DeepCluster, summarized in Figure 3.1, consists in alternating between clustering of the image descriptors and updating the weights of the convnet by predicting the cluster assignments. For simplicity, we focus on k-means in this chapter, but other clustering approaches can be used, like Power Iteration Clustering (PIC) [START_REF] Lin | Power iteration clustering[END_REF]. The overall pipeline is sufficiently close to the standard supervised training of a convnet to reuse many common tricks [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF]. Despite its simplicity, DeepCluster achieves significantly higher performance than the state of the art among self-supervised methods on standard transfer tasks at the time of the publication (i.e. 2018).

Training on uncurated data

However, these first results are obtained by training DeepCluster on ImageNet (without labels). Designing large fully-annotated datasets like ImageNet has required a significant effort from the community, not only due to the cost of manual labeling but also due to the compilation and cleansing of the data. As a result, we discuss the use of ImageNet, a highly curated dataset, as a training set for unsupervised models. It contains properly cropped images of a pre-selected number of classes, and nothing else. As it is designed for a fine-grained classification task, this dataset provides well-balanced and diversified data. Discarding the labels does not undo this careful selection, and only removes part of the human supervision.

The challenge tackled is this chapter is to learn good general-purpose representations captures finer relations between images when the number of clusters scales with the dataset size. Clustering approaches infer target labels at the same time as features are learned. Thus, target labels evolve during training, making clustering-based approaches unstable. Furthermore, these methods are sensitive to data distribution as they rely directly on cluster structure in the underlying data. Explicitly dealing with unbalanced category distribution might be a solution but it assumes that we know the distribution of the latent classes. We design our method without this assumption. On the other hand, other methods use pretext task by predicting pseudo-labels automatically extracted from input signals [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF]. For instance, methods like RotNet [START_REF] Friedman | The elements of statistical learning[END_REF] (described in Section 2.2.3), leverage intra-image statistics to build supervision, which are often independent of the data distribution. However, the dataset size has little impact on the nature of the task and on the performance of the resulting features. A solution to leveraging larger datasets require manually increasing the difficulty of the self-supervision task [START_REF] Goyal | Scaling and benchmarking self-supervised visual representation learning[END_REF]. Our approach automatically increases complexity through the clustering strategy. Our approach, DeeperCluster, automatically generates targets by clustering the features of the entire dataset, under constraints derived from the rotation prediction pretext task. Due to the "long-tail" distribution of raw non-curated data, processing huge datasets and learning a large number of targets is necessary, making the problem challenging from a computational point of view. For this reason, we propose a hierarchical formulation that is suitable for distributed training. This enables the discovery of latent categories present in the "tail" of the image distribution.

Related work

Several approaches related to our work learn deep networks with no supervision and are detailed in Chapter 2. Here, we give a brief overview of methods that combine clustering and unsupervised learning with deep networks or approaches that train on large-scale uncurated data.

Clustering and deep unsupervised learning. [START_REF] Coates | Learning feature representations with k-means[END_REF] also use k-means to pre-train convnets, but learn each layer sequentially in a bottom-up fashion, while we do it in an end-to-end fashion. Other clustering losses [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF][START_REF] Liao | Learning deep parsimonious representations[END_REF][START_REF] Xie | Unsupervised deep embedding for clustering analysis[END_REF], Yang et al., 2016] have been considered to jointly learn convnet features and image clusters but they have never been tested on a scale to allow a thorough study on modern convnet architectures. [START_REF] Liao | Learning deep parsimonious representations[END_REF] combines the autoencoder reconstruction loss (detailed in Section 2.1.1) with a clustering regularization term. They benchmark different clustering strategies: Spatial clustering corresponds to the grouping of the pixels in a given layer over all localizations and examples; channel co-clustering allows to group the different feature maps (channels) in a given layer across examples; and sample clustering refers to the clustering of the set of representations given by a network over the whole dataset. Of particular interest, [START_REF] Yang | Joint unsupervised learning of deep representations and image clusters[END_REF] iteratively learn convnet features and clusters with a recurrent framework. Their model offers promising performance on small datasets but may be challenging to scale to the number of images required for convnets to be competitive. Closer to our work, [START_REF] Bojanowski | Unsupervised learning by predicting noise[END_REF] learn visual features on a large dataset with a loss that attempts to preserve the information flowing through the network [START_REF] Linsker | Towards an organizing principle for a layered perceptual network[END_REF]. Their approach discriminates between all images in a similar way as exemplar SVM [START_REF] Malisiewicz | Ensemble of exemplar-svms for object detection and beyond[END_REF], while we are simply clustering them.

Learning on non-curated datasets. Some methods [START_REF] Chen | Webly supervised learning of convolutional networks[END_REF], Gomez et al., 2017, Ni et al., 2015, Tian et al., 2021] aim at learning visual features from noncurated data streams. They typically use metadata such as hashtags [START_REF] Joulin | Learning visual features from large weakly supervised data[END_REF][START_REF] Sun | Revisiting unreasonable effectiveness of data in deep learning era[END_REF] or geolocalization [START_REF] Weyand | Planet-photo geolocation with convolutional neural networks[END_REF] as a source of noisy supervision. In particular, [START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF] train a network to classify billions of Instagram images into predefined and clean sets of hashtags. They show that with little human effort, it is possible to learn features that transfer well to ImageNet, even achieving state-of-the-art performance if finetuned. As opposed to our work, they use an extrinsic source of supervision that had to be cleaned beforehand.

Preliminaries: DeepCluster

We start this chapter by re-introducing the DeepCluster framework which is the seed to the contributions presented in the following of the chapter.

DeepCluster methodology

After a short introduction to the supervised learning of convnets, we describe the DeepCluster approach as well as the specificities of its optimization.

Preliminaries. Modern approaches to computer vision, based on statistical learning, require good image featurization. In this context, convnets are a popular choice for mapping raw images to a vector space of fixed dimension. When trained on enough data, they constantly achieve the best performance on standard classification benchmarks [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. We denote by f θ the convnet mapping, where ◊ is the set of corresponding parameters. We refer to the vector obtained by applying this mapping to an image as feature or representation. Given a training set X = {x 1 ,x 2 ,...,x N } of N images, we want to find a parameter ◊ ú such that the mapping f θ ú produces good general-purpose features.

These parameters are traditionally learned with supervision, i.e. each image x n is associated with a label y n in {0, 1} k . This label represents the image's membership to one of k possible predefined classes. A parametrized classifier g W predicts the correct labels on top of the features f θ (x n ). The parameters W of the classifier and the parameter ◊ of the mapping are then jointly learned by optimizing the following problem:

min θ,W 1 N N ÿ n=1 ¸(g W (f θ (x n )) ,y n ) , (3.1)
where ¸is the multinomial logistic loss, also known as the negative log-softmax function. This cost function is minimized using mini-batch stochastic gradient descent [START_REF] Bottou | Stochastic gradient descent tricks[END_REF] and backpropagation to compute the gradient [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]].

Unsupervised learning by clustering. When ◊ is sampled from a Gaussian distribution, without any learning, f θ does not produce good features. However the performance of such random features on standard transfer tasks, is far above the chance level. For example, a multi-layer perceptron classifier on top of the last convolutional layer of a random AlexNet achieves 12% in accuracy on ImageNet while the chance is at 0.1% [START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF]. The good performance of random convnets is intimately tied to their convolutional structure which gives a strong prior on the input signal. The idea of this work is to exploit this weak signal to bootstrap the discriminative power of a convnet. We cluster the output of the convnet and use the subsequent cluster assignments as "pseudo-labels" to optimize Eq. (3.1). This deep clustering (DeepCluster) approach iteratively learns the features and groups them.

Clustering has been widely studied and many approaches have been developed for a variety of circumstances. In the absence of points of comparisons, we focus on a standard clustering algorithm, k-means. Preliminary results with other clustering algorithms indicates that this choice is not crucial. k-means takes a set of vectors as input, in our case the features f θ (x n ) produced by the convnet, and clusters them into k distinct groups based on a geometric criterion. More precisely, it jointly learns a d ◊ k centroid matrix C and the cluster assignments y n of each image n by solving the following problem:

min CoeR d◊k 1 N N ÿ n=1 min ynoe{0,1} k Îf θ (x n ) ≠ Cy n Î 2 2 such that y € n 1 k =1. (3.2)
Solving this problem provides a set of optimal assignments (y ú n ) nAEN and a centroid matrix C ú . These assignments are then used as pseudo-labels; we make no use of the centroid matrix.

Overall, DeepCluster alternates between clustering the features to produce pseudo-labels using Eq. (3.2) and updating the parameters of the convnet by predicting these pseudo-labels using Eq. (3.1). This type of alternating procedure is prone to trivial solutions; we describe how to avoid such degenerate solutions in the next section.

Avoiding trivial solutions. The existence of trivial solutions is not specific to the unsupervised training of neural networks, but to any method that jointly learns a discriminative classifier and the labels. Discriminative clustering suffers from this issue even when applied to linear models [START_REF] Xu | Maximum margin clustering[END_REF]. Solutions are typically based on constraining or penalizing the minimal number of points per cluster [Bach andHarchaoui, 2008, Joulin and[START_REF] Joulin | A convex relaxation for weakly supervised classifiers[END_REF]. These terms are computed over the whole dataset, which is not applicable to the training of convnets on large scale datasets. In the following paragraphs, we briefly describe the causes of these trivial solutions and give simple and scalable workarounds.

A discriminative model learns decision boundaries between classes. An optimal decision boundary is to assign all of the inputs to a single cluster [START_REF] Xu | Maximum margin clustering[END_REF]. This issue is caused by the absence of mechanisms to prevent from empty clusters and arises in linear models as much as in convnets. A common trick used in feature quantization [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF] consists in automatically reassigning empty clusters during the k-means optimization. More precisely, when a cluster becomes empty, we randomly select a non-empty cluster and use its centroid with a small random perturbation as the new centroid for the empty cluster. We then reassign the points belonging to the non-empty cluster to the two resulting clusters.

If the vast majority of images is assigned to a few clusters, the parameters ◊ will exclusively discriminate between them. In the most dramatic scenario where all but one cluster are singleton, minimizing Eq. (3.1) leads to a trivial parametrization where the convnet will predict the same output regardless of the input. This issue also arises in supervised classification when the number of images per class is highly unbalanced. For example, metadata, like hashtags, exhibits a Zipf distribution, with a few labels dominating the whole distribution [START_REF] Joulin | Learning visual features from large weakly supervised data[END_REF]. A strategy to circumvent this issue is to sample images based on a uniform distribution over the classes, or pseudo-labels. This is equivalent to weight the contribution of an input to the loss function in Eq. (3.1) by the inverse of the size of its assigned cluster.

Implementation details. For comparison with previous works at the time of the publication of DeepCluster paper, we use a standard AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] architecture. It consists of five convolutional layers with 96, 256, 384, 384 and 256 filters; and of three fully connected layers. We remove the Local Response Normalization layers and use batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF]. We also consider a VGG-16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] architecture with batch normalization. Unsupervised methods often do not work directly on color and different strategies have been considered as alternatives [Doersch et al., 2015, Noroozi and[START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF]. We apply a fixed linear transformation based on Sobel filters to remove color and increase local contrast [Bojanowski andJoulin, 2017, Paulin et al., 2015]. We train DeepCluster on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] unless mentioned otherwise. It contains 1, 281, 167 images uniformly distributed into 1, 000 classes. We cluster the central cropped images features and perform data augmentation (random horizontal flips and crops of random sizes and aspect ratios) when training the network. This enforces invariance to data augmentation which turn out to be crucial for DeepCluster and relates to the view-invariant self-supervised paradigm [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF] detailed in Section 2.3. The network is trained with dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF], a constant step size, an ¸2 penalization of the weights ◊ and a momentum of 0.9. Each mini-batch contains 256 images. For the clustering, features are PCA-reduced to 256 dimensions, whitened and ¸2-normalized. We use the k-means implementation of [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF]. Note that running k-means takes a third of the time because a forward pass on the full dataset is needed. One could reassign the clusters every n epochs, but we found out that our setup on ImageNet (updating the clustering every epoch) was nearly optimal. On Flickr, the concept of epoch disappears: choosing the trade-off between the parameter updates and the cluster reassignments is more subtle. On ImageNet, we train the models for 500 epochs, which takes 12 days on a Pascal P100 GPU for AlexNet. We select hyperparameters on a down-stream task, i.e., object classification on the validation set of PASCAL VOC with no fine-tuning.

Probing DeepCluster on ImageNet

We compare our approach to previous state-of-the-art models on standard benchmarks on Pascal VOC dataset before studying the behavior of DeepCluster during training. Finally, we qualitatively assess the filters learned with DeepCluster. Note that in this section all models are trained on ImageNet without labels which is a highly curated dataset. While it provides a controlled setting, allows easier comparison with previous state of the art and helps understanding the impact of the labels on the performance of a network, one has to keep in mind the limitations of such a setting as described in Section 3.1.2. We study the uncurated scenario in the following section of this chapter (i.e. Section 3.3). (NMI), defined as:

Classification

NMI(A; B)= I(A; B) Ò H(A)H(B)
where I denotes the mutual information and H the entropy. This measure can be applied to any assignment coming from the clusters or the true labels. If the two assignments A and B are independent, the NMI is equal to 0. If one of them is deterministically predictable from the other, the NMI is equal to 1. Figure 3.3(b) shows the evolution of this measure during training. The NMI is increasing, meaning that there are less and less reassignments and the clusters are stabilizing over time. However, NMI saturates below 0.8, meaning that a significant fraction of images are regularly reassigned between epochs. In practice, this has no impact on the training and the models do not diverge.

Choosing the number of clusters. We measure the impact of the number k of clusters used in k-means on the quality of the model. We report the same down-stream task as in the hyperparameter selection process, i.e. mAP on the PASCAL VOC 2007 classification validation set. We vary k on a logarithmic scale, and report results after 300 epochs in Figure 3.3(c). The performance after the same number of epochs for every k may not be directly comparable, but it reflects the hyper-parameter selection process used in this work. The best performance is obtained with k =10, 000. Given that we train our model on ImageNet, one would expect k =1 0 0 0to yield the best results, but apparently some amount of over-segmentation is beneficial.

Second, we report in Table 3.2 the results for the different PASCAL VOC transfer tasks with a model trained with the PIC version of DeepCluster. For this set of transfer tasks, the models trained with k-means and PIC versions of DeepCluster perform in comparable ranges.

If v i is a local maximum (ie. 'j " = i, v j AE v i ), then no edge starts from it. The clusters are given by the connected components of G Õ . Each cluster has one local maximum. An advantage of PIC clustering is not to require the setting beforehand of the number of clusters. However, the parameter ‡ influences the number of clusters: when it is larger, the edges become more uniform and the number of clusters decreases, and the other way round when ‡ increased. In the following, we set ‡ =0.2.

First, we give an insight about the distribution of the images in the clusters. We show in Figure 3.4(a) the sizes of the clusters produced by the k-means and PIC versions of DeepCluster at the last epoch of training (this distribution is stable along the epochs). We observe that k-means produces more balanced clusters than PIC. Indeed, for PIC, almost one third of the clusters are of a size lower than 10 while the biggest cluster contains roughly 3000 examples. In this situation of very unbalanced clusters, it is important in our method to train the convnet by sampling images based on a uniform distribution over the clusters to prevent the biggest cluster from dominating the training.

Stopping criterion. We monitor how the features learned with DeepCluster evolve along the training epochs on a down-stream task: object classification on the validation set of PASCAL VOC with no fine-tuning. We use this measure to select the hyperparameters of our model as well as to check when the features stop improving. In Figure 3.4(b), we show the evolution of both the classification accuracy on this task and a measure of the clustering quality (NMI between the cluster assignments and the true labels) throughout the training. Unsurprisingly, we notice that the clustering and features qualities follow a similar dynamic.

Influence of data pre-processing with Sobel filtering In this section we experiment with our method on raw RGB inputs and evaluate the impact of the Sobel filtering. The difficulty of learning convnets on raw images has been noted before [START_REF] Bojanowski | Unsupervised learning by predicting noise[END_REF], Doersch et al., 2015[START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF][START_REF] Paulin | Local convolutional features with unsupervised training for image retrieval[END_REF]. We provide some insights into the reasons why Sobel filtering (illustrated in Figure 3.5) is crucial to obtain good performance with our method.

In our preliminary experiments we have observed that the performance of DeepCluster on raw RGB images degrades significantly. For example, training a MLP on top of the DeepCluster frozen pre-trained features on an AlexNet (following the experiment protocol of [START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF]) lead to 44.0% top-1 accuracy with Sobel filtering and 37.1% without (≠6.9%). In order to better understand why this happens, in Figure 3.6 we randomly select a subset of 3000 clusters and sort them by standard deviation to their mean color. If the standard deviation of a cluster to its mean color is low, it means that the images RGB Sobel Figure 3.5 -Illustration of Sobel filtering. Visualization of two images pre-processed with Sobel filter. Sobel gives a 2 channels output which at each point contain the vertical and horizontal derivative approximations.

of this cluster tend to have a similar colorization. Indeed, we show in Figure 3.6 (right) examples of these clusters that have a low standard deviation to the mean color. We observe in Figure 3.6 (left) that the clustering on features learned with DeepCluster focuses much more on color than the clustering performed on other self-supervised approach (namely RotNet [START_REF] Friedman | The elements of statistical learning[END_REF]). Indeed, clustering by color and low-level information produces balanced clusters that can easily be predicted by a convnet. As a result, clustering by color constitutes a solution to DeepCluster formulation. However, as we want to avoid an uninformative clustering essentially based on colors, we remove some part of the input information by feeding the network with the image gradients instead of the raw RGB image (see Figure 3.5). Interestingly, we have observed that using Sobel filter during RotNet training has almost no impact. Finally, note that more recent approaches replace Sobel filtering by strong color jittering augmentation [Chen et al., 2020b].

Visualizing first layer filters.

DeeperCluster methodology

In this section, we describe how we extend deep clustering in order to scale up to large numbers of uncurated images and targets. To that end, we propose to combine selfsupervised learning based on pretext tasks (for intra-image modeling) and on clustering (for inter-image modeling).

Combining pretext tasks and clustering. We assume that the inputs x 1 ,...,x N are rotated images, each associated with a target label r n encoding its rotation angle and a cluster assignment y n . The cluster assignment changes during training along with the visual representations. We denote by R the set of possible rotation angles and by Y, the set of possible cluster assignments. A way of combining rotation prediction with deep clustering is to add the corresponding losses. However, summing these losses implicitly assumes that classifying rotations and cluster memberships are two independent tasks, which may limit the signal that can be captured. Instead, we work with the Cartesian product space R◊Y, which can potentially capture richer interactions between the two tasks. We get the following optimization problem:

min θ,W 1 N N ÿ n=1 ¸(r n ¢ y n ,Wf θ (x n )), (3.3)
where ¸is a loss function. Note that any clustering or self-supervised approach with a multi-class objective can be combined with this formulation. For example, we could use a self-supervision task that captures information about tiles permutations [START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF] or frame ordering in a video [START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF]. However, this formulation does not scale in the number of combined targets, i.e., its complexity is O(|R||Y|). This limits the use of a large number of cluster or a pretext task with a large output space [Zhang Scaling up to large number of targets. Hierarchical losses are commonly used in language modeling where the goal is to predict a word out of a large vocabulary [START_REF] Brown | Class-based n-gram models of natural language[END_REF]. Instead of making one decision over the full vocabulary, these approaches split the process in a hierarchy of decisions, each with a smaller output space. For example, the vocabulary can be split into clusters of semantically similar words, and the hierarchical process would first select a cluster and then a word within this cluster. Following this line of work, we partition the target labels into a 2-level hierarchy where we first predict a super-class and then a sub-class among its associated target labels. The first Figure 3.9 -Visualizing filters from deeper layers with human attributes. Filter visualization by learning an input image that maximizes the response to a target filter [START_REF] Yosinski | Understanding neural networks through deep visualization[END_REF] in the last convolutional layer of a VGG-16 convnet trained with DeepCluster. Here, we manually select filters that seem to trigger on human characteristics (eyes, noses, faces, fingers, fringes, groups of people or arms).

level is a partition of the images into S super-classes and we denote by y n the super-class assignment vector in {0, 1} S of the image n and by y ns the s-th entry of y n . This super-class assignment is made with a linear classifier V on top of the features. The second-level of the hierarchy is obtained by partitioning within each super-class. We denote by z s n the vector in {0, 1} ks of the assignment into k s sub-classes for an image n belonging to super-class s. There are S sub-class classifiers W 1 ,...,W S , each predicting the sub-class memberships within a super-class s. The parameters of the linear classifiers (V, W 1 ,...,W S ) and ◊ are jointly learned by minimizing the following loss function:

1 N N ÿ n=1 C ¸1Vf θ (x n ),y n 2 + S ÿ s=1 y ns ¸(W s f θ (x n ),z s n ) D , (3.4)
where ¸is the negative log-softmax function. Note that an image that does not belong to the super-class s does not belong either to any of its k s sub-classes.

Choice of super-classes. A natural partition would be to define the super-classes based on the target labels from the self-supervised task and the sub-classes as the labels produced by clustering. However, this would mean that each image of the entire dataset would be W 1 ,...,W S are only shared within a communication group. Each communication group s deals only with the subset of images and the rotation angle associated with the super-class s.

Distributed k-means. Every T epochs, we recompute the super and sub-class assignments by running two consecutive k-means on the entire dataset. This is achieved by first randomly splitting the dataset across different GPUs. Each GPU is in charge of computing cluster assignments for its partition, whereas centroids are updated across GPUs. We reduce communication between GPUs by sharing only the number of assigned elements for each cluster and the sum of their features. The new centroids are then computed from these statistics. We observe empirically that k-means converges in 10 iterations. We cluster 96M features of dimension 4096 into m =4clusters using 64 GPUs (1 minute per iteration).

Then, we split this pool of GPUs into 4 groups of 16 GPUs. Each group clusters around 23M features into 80k clusters (4 minutes per iteration).

Implementation details. The loss in Eq. (3.4) is minimized with mini-batch stochastic gradient descent [START_REF] Bottou | Stochastic gradient descent tricks[END_REF]. Each mini-batch contains 3072 instances distributed across 64 GPUs, leading to 48 instances per GPU per mini-batch. We use dropout, weight decay, momentum and a constant learning rate of 0.1. We reassign clusters every 3 epochs. We use the Pascal VOC 2007 classification task without finetuning as a downstream task to select hyper-parameters. In order to speed up experiments, we initialize the network with RotNet trained on YFCC100M. Before clustering, we perform a whitening of the activations and ¸2-normalize each of them. We use standard data augmentations, i.e., cropping of random sizes and aspect ratios and horizontal flips [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]). We use the VGG-16 architecture [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] with batch normalization layers.

We pre-process images with a Sobel filtering. We train our models on the 96M images from YFCC100M [START_REF] Thomee | Yfcc100m: The new data in multimedia research[END_REF] that we managed to download. We use this publicly available dataset for research purposes only.

Results

In this set of experiments, we compare RotNet, DeepCluster and DeeperCluster performance when trained on curated and non-curated datasets. We also evaluate the impact of the size of the dataset and number of clusters in DeeperCluster.

Comparison with RotNet and DeepCluster. In Table 3 published numbers [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF] with a VGG-16 architecture. We do not perform any finetuning or layer selection. We average-pool the features of the last layer resulting in representations of 8192 dimensions. Our approach outperforms both RotNet and DeepCluster, even when they are trained on curated datasets (except for ImageNet classification task where DeepCluster trained on ImageNet yields the best performance). More interestingly, we see that the quality of the dataset or its scale has little impact on RotNet while it has on DeepCluster. This is confirming the intuition that pretext tasks methods based on data manipulation are more robust than clustering to a change of dataset distribution.

Influence of dataset size and number of clusters. To measure the influence of the number of images on features, we train models with 1M, 4M, 20M, and 96M images and report their accuracy on the validation set of the Pascal VOC 2007 classification task (FC68 setting). We also train models on 20M images with a number of clusters that varies from 10k to 160k. For the experiment with a total of 160k clusters, we choose m =2which results in 8 super-classes. In Figure 3.11, we observe that the quality of our features improves when scaling both in terms of images and clusters. Interestingly, between 4M and 20M of YFCC100M images are needed to meet the performance of our method on ImageNet. Augmenting the number of images has a bigger impact than the number of clusters. Yet, this improvement is significant since it corresponds to a reduction of more than 10% of the relative error w.r.t. the supervised model. Yet, in its current form, the proposed approach has some major limitations which we detail in the following.

Limited scalability. A first limitation of Deep(er)Cluster is its inefficiency and limited scalability. Indeed, as shown in Eq. (3.1), both a linear classifier g W and the network weights ◊ are trained to classify the images into their corresponding pseudo-label between two assignments. Intuitively, this classification layer represents prototypes for the different pseudo-classes. However, since there is no mapping between two consecutive cluster assignments, the prototypes learned for an assignment becomes irrelevant for the following one and hence need to be learned again from scratch. In practice, we indeed re-set the classification layer g W to random weights whenever we have a new set of pseudo-labels. This considerably disrupts the stability of the network training, makes it quite inefficient and, as a result, less scalable. Overall, the approach is not much scalable beyond the results that we have already shown with DeeperCluster. Indeed, we have seen that it is possible to train the models on large dataset such as YFCC100m but that is very costly and going beyond this scale would be quite tedious with the current model. The main problem is that our approach depends directly on the size of the dataset. The approach alternates between some epochs of training and a clustering phase that is very costly since it requires to perform a forward pass on the entire dataset to collect the descriptors to cluster. This iterative process relies on the notion of epoch and on the size of the dataset. Indeed, to have good performance, we need to iterate a consequent number of times in order to refine the pseudo-labels regularly. If the dataset is not too large, like when training on ImageNet, we can refine the pseudo-labels between each epoch: this happens hundreds of time in a typical training. Now let us imagine we have a huge dataset where we can only afford a couple of passes on the dataset. This means that the pseudo-labels will be refined only a handful of times. Worse, if we have an even bigger dataset where we can only afford two passes, then the pseudo-labels are basically never refined. During the whole training (which consists in only one epoch in this case), we train with the first pseudo-labels which have been obtained from a randomly initialed network. As these pseudo-labels are likely to be semantically very poor, the resulting features will surely be pretty bad as well. Overall, a natural way to overcome the limitation of limited scalability is to adapt the method to work with an online version of k-means as done by [START_REF] Zhan | Online deep clustering for unsupervised representation learning[END_REF] for example. However, given that we have observed in Table 3.2 that the choice of the clustering algorithm does not impact the performance much, similarly to [START_REF] Asano | Self-labelling via simultaneous clustering and representation learning[END_REF] and Chen et al. [2020d] we question the role of k-means and clustering altogether in DeepCluster.

Do we really need k-means?

We remark that k-means clustering provide two distinct outputs: (i) the cluster assignments that we use as pseudo-labels (i.e. y n oe{ 0, 1} k in Eq. (3.2)) and (ii) a set of centroids (i.e. C oe R d◊k in Eq. (3.2)). However, we make no use of the latter and only care about the former. This means that we are using only one component of k-means. Follows that we intuit that using k-means, an extremely simple clustering algorithm, might already be an overkill and we could find a simpler way to obtain the pseudo-labels that we use during training. We will see in the following chapter that we can directly use the output of the deep network to provide pseudo-labels, without resorting to an external clustering algorithm.

Empirical tricks to avoid trivial solution. As detailed previously in Section 3.2.1, our formulation is prone to trivial solution and to the "collapse" of the network to a constant output (thus all images have the same representation). We use heuristics to avoid this situation but it would be desirable to have a more principled formulation that prevents collapse by design. Implicit importance of data augmentation. Last but not least, we describe a final major limitation: we have presented data augmentation merely as an implementation detail in this chapter while it turns out to be a crucial component of the method. Indeed, let us conduct a final experiment where we ablate the different data transformations used during DeepCluster training. In Table 3.4, we experiment with three different flips: vertical, horizontal (both occurring with a probability of 0.5) and no flip. We also benchmark three different types of cropping: (i) (default cropping) a crop of random size (0.08 to 1.0 of the original size) and random aspect ratio (of 3/4 to 4/3 of the original aspect ratio), (ii) crop of fixed size with a random localization and (iii) no cropping, i.e. we take a central crop, as used for the clustering phase. We observe in Table 3.4 that the only configurations giving good features are that using random cropping and scaling (rows 1, 2 and 3). This transformation is thus crucial for DeepCluster. We also notice that flipping does not play a key role and that we could remove it. Using vertical flip degrades the performance which is consistent with the method of [START_REF] Friedman | The elements of statistical learning[END_REF] that trains the network to be discriminative to this transformation rather than invariant.

Random flips

Overall, this dependence in the data augmentation reinforces the interpretation of our framework as a variant of the view-invariant paradigm as described in Section 2.3.4. In the following chapter, we will make data cropping more central to the model which will allow us to have substantial boosts of performance.

Chapter 4

The Self-Supervised SwAV Approach

In this chapter, we present a new method for self-supervised representation learning, SwAV, that overcomes the difficulties and limitations of both deep clustering (as detailed in Section 3.4) and contrastive representation learning (as mentioned in Section 2.3.2). Compared to deep clustering, SwAV works online (i.e. at the mini-batch level), scales to unlimited amount of data, has guarantees that prevent mode collapse and does not rely on an external clustering algorithm. Compared to contrastive methods, SwAV does not rely on "negative samples" which makes it more practical since it works without requiring a large memory bank [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF] nor large mini-batches [Chen et al., 2020b]. SwAV yields excellent results when trained on ImageNet without labels, achieving 75.3% top-1 accuracy with a standard ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] on the ImageNet linear evaluation protocol [START_REF] Zhang | Colorful image colorization[END_REF]. This performance was more than 4 points above the state of the art at the time of the publication. Perhaps more importantly, SwAV features transfer very well to various downstream tasks, outperforming supervised features on classification and detection tasks on all the considered datasets. This work was presented at NeurIPS in 2020 [START_REF]1007 73.5 66[END_REF].

Similarly to our previous study on deep clustering (see Section 3.3), we assess SwAV at scale when trained on uncurated data. We consider billions of Instagram images and train a very deep network with billions of parameters. This effort was described in a technical report (see [START_REF] Goyal | Self-supervised pretraining of visual features in the wild[END_REF]).

Introduction

As detailed in the first parts of this manuscript (see Chapter 1 and Chapter 2), selfsupervised learning aims at obtaining features without using manual annotations and has rapidly been closing the performance gap with supervised pretraining in computer vi-53 sion [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF], Chen et al., 2020b, He et al., 2020[START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF]. Many successful methods build upon the instance discrimination task that considers each image of the dataset (or "instance") and its transformations as a separate class [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with exemplar convolutional neural networks[END_REF] (see a review in Section 2.3). This task yields representations that are able to discriminate between different images, while achieving some invariance to image transformations. Contrastive formulations of this view-invariant paradigm rely on a combination of two elements: (i) a contrastive loss [START_REF] Hadsell | Dimensionality reduction by learning an invariant mapping[END_REF]] and (ii) a set of image transformations. The contrastive loss removes the notion of instance classes by directly comparing image features while the image transformations define the invariances encoded in the features. Both elements are essential to the quality of the resulting networks [Chen et al., 2020b, Misra and[START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF] and our work improves upon both the objective function and the transformations.

The contrastive loss explicitly compares pairs of image representations to push away representations from different images while pulling together those from transformations, or views, of the same image. Since computing all the pairwise comparisons on a large dataset is not practical, most implementations approximate the loss by reducing the number of comparisons to random subsets of images during training [Chen et al., 2020b, He et al., 2020[START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF]]. An alternative to approximate the loss is to approximate the taskthat is to relax the instance discrimination problem [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF]. For example, clustering-based methods discriminate between groups of images with similar features instead of individual images [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF]. The objective in clustering is tractable, but has its own limitations as detailed in Section 3.4. For instance, it does not scale well with the dataset as it requires a pass over the entire dataset to get image cluster assignments that are used as pseudo-labels during training. In this chapter, we use a different paradigm and propose to compute pseudo-labels in a much simpler fashion: the pseudo-label used as target for a view is directly the normalized output of another view of the same image. This formulation encourages distorted views of a same image to match in feature space while not relying on explicit pairwise feature comparisons as in the contrastive implementations. Specifically, we propose a simple "swapped" prediction problem where we predict the pseudo-label (or "assignment") of a view from the representation of another view. We learn features by Swapping Assignments between multiple Views of the same image (SwAV). Unlike deep clustering, the features and the pseudo-labels are learned online, allowing our method to scale to potentially unlimited amounts of data. Compared to contrastive methods, SwAV works with small and large batch sizes, does not rely on direct feature comparisons (i.e. "positives" and "negatives") and hence does not need a large memory bank [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF] nor a momentum encoder [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF].

Besides this scalable implementation of the view-invariant paradigm, we also propose an improvement to the image transformations. Most methods are trained to align the representations of two independent and identically distributed views of an image. We hypothesize that this is sub-optimal for two reasons. First, there has been evidence that comparing more views than two during training improves the resulting model [START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF]. Second, matching views from different distributions (as opposed to "identically distributed") could be a way to enforce desired properties in the learned representations. We propose that some views have access to only a narrow, local crop of the original image while other views cover a global and large area of that image. Thus, SwAV learns to extrapolate and generalize from the local view to the global content of the image, hence encouraging holistic representations. This strategy, called multi-crop, is simple and boosts considerably the performance of different self-supervised methods.

We validate our contributions by evaluating our method on several standard selfsupervised benchmarks. Overall, in this chapter, we present the following contributions:

-We propose a new and scalable implementation of the view-invariant self-supervised paradigm that improves performance by +2% on ImageNet over contrastive implementations. Unlike contrastive methods, it works in both large and small batch settings without a large memory bank or a momentum encoder. -We introduce the multi-crop strategy to increase the number of views of an image with limited computational or memory overhead. Multi-crop encourages local-to-global matching of the representations. We observe a consistent improvement of between 2% and 4% on ImageNet with this strategy on several self-supervised methods. -Combining both technical contributions into a single model, SwAV, we improved the performance of self-supervised methods by +4.2% on ImageNet. Our features outperform supervised ImageNet pretraining on multiple downstream tasks. At the time of the publication, this was the first method to do so without finetuning the features, i.e., only with a linear classifier on top of frozen features. -We explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SwAV model trained with a RegNetY [START_REF] Radosavovic | Designing network design spaces[END_REF] with 1.3B parameters on 1B random images achieves 84.2% top-1 accuracy confirming that self-supervised learning works in a real world setting.

SwAV Methodology

Our goal is to learn visual features in an online fashion without supervision. To that effect, we propose a new, online implementation of the view-invariant paradigm (introduced in Section 2.3) which consists in learning representations that are invariant to distortions of the input sample. Our formulation aims primarily at overcoming both the limitations of the contrastive-based implementations, that rely on direct feature comparison, and of the clustering-based methods [START_REF] Asano | Self-labelling via simultaneous clustering and representation learning[END_REF], Caron et al., 2018] like DeepCluster (see Chapter 3), that are offline. This means that they alternate between a cluster assignment step where image features of the entire dataset are clustered, and a training step where the cluster assignments, i.e., "pseudo-labels" are predicted for different image views. Unfortunately, these methods are not suitable for online learning as they require multiple passes over the dataset to compute the image features necessary for clustering.

In this section, we describe an alternative implementation where we directly use the output of the network as a pseudo-label to predict from another view of the same image. We enforce explicit constraints on these pseudo-labels to prevent the network from learning a constant mapping. Indeed, if the network outputs a constant pseudo-label regardless of its input then it is of course trivial to predict.

Matching views through pseudo-labeling

SwAV works by computing a pseudo-label from an augmented version of the image and predicting it from other augmented versions of the same image. More precisely, given an input image, we take two different random augmentations of that same image, resulting in crops x t and x s . The augmented views are mapped to a K-dimensional vector representation by applying a non-linear mapping f θ parameterized by ◊. We denote by z = f θ (x) the output of the network which we interpret as a vector of scores for K latent pseudo-classes or "clusters". To transform this vector of scores into a soft pseudo-label over K dimensions, we adjust it based on constraints on the other mini-batch scores in order to prevent the collapse of the representations. We will explicit these constraints and describe the adjustment process in the following paragraph, but for now let us denote by q t the resulting adjusted and normalized pseudo-label obtained from z t . We train the network to predict this pseudo-label q t from the representation of another view z s . A probability distribution p s is obtained by normalizing z s with a standard softmax function:

p (i) s = exp(z (i) s /• ) q K k=1 exp(z (k) s /• ) , (4.1)
with •>0 a temperature parameter that controls the sharpness of the output distribution. We minimize the cross-entropy loss w.r.t. the parameters ◊ of the feature encoder and considered the pseudo-label q t fixed (i.e. we apply a stop-gradient operator):

min θ H(q t , p s ), (4.2)
where H(q, p)=≠ q K i=1 q (i) log p (i) .

Online pseudo-labels with optimal transport

In order to make our method online, we compute the pseudo-labels using only the image features within a batch. We enforce that all the examples in a batch are equally partitioned into the K different dimensions or "clusters". This equipartition constraint ensures that the pseudo-labels for different images in a batch are distinct, thus preventing the trivial solution where every image has the same pseudo-label. We denote by B the mini-batch of feature vectors Z =[z 1 ,...,z B ] and by Q =[q 1 ,...,q B ] the desired matrix of pseudo-labels. We establish the following constraints for Q:

Q = Ó Q oe R K◊B + | Q1 B = 1 K , Q € 1 K = 1 B Ô , (4.3)
where 1 K denotes the vector of ones in dimension K. These constraints mean that each q n should sum to 1 (i.e. it can be interpreted as an assignment over the K dimensions) and that, on average, each dimension "is used B K times in the batch". As we derive Q from Z, we want Q to stay as close to Z as possible while satisfying the constraints above. More precisely, we have the following problem:

max QoeQ Tr 1 Q € Z 2 + ÁH(Q), (4.4)
where H is the entropy function,

H(Q)=≠ q ij Q ij log Q ij and
Á is a parameter that controls the smoothness of the solution Q. We observe that a strong entropy regularization (i.e. using a high Á) generally leads to a trivial solution where all samples collapse into an unique uniform pseudo-label. Hence in practice we keep Á low.

Once a continuous solution Q ú to Prob. (4.4) is found, a discrete pseudo-label can be obtained by using a rounding procedure [START_REF] Asano | Self-labelling via simultaneous clustering and representation learning[END_REF]. However, in our setting we find that using the discrete pseudo-label performs worse than using the soft continuous ones (we lose 0.5% in the linear evaluation on ImageNet). An explanation is that the rounding needed to obtain discrete assignments is a more aggressive optimization step than gradient updates. While it makes the model converge rapidly, it leads to a worse solution. We thus preserve the soft pseudo-label Q ú instead of rounding it. These soft assignments Q ú are the solution of Prob. (4.4) over the set Q and takes the form of a normalized exponential matrix [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF]:

Q ú = Diag(u)exp(Z/Á) Diag(v), (4.5)
where u and v are re-normalization vectors in R K and R B respectively. The re-normalization vectors are computed using a small number of matrix multiplications using the iterative Sinkhorn-Knopp algorithm [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF]. In practice, we observe that using only 3 iterations is fast and sufficient to obtain good performance. Indeed, this algorithm can be efficiently implemented on GPU, and the alignment of 4K features to 3K codes takes 35ms

in our experiments. Interestingly, the parameter Á plays a similar role of controlling the smoothness of the exponential in Eq. (4.5) as the temperature • in Eq. (4.2). Overall, the optimal transport algorithm [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF] used to adjust the scores Z is implemented simply with the following PyTorch style code.

# Z is B-by-K # eps is Sinkhorn regularization parameter Q = exp(Z / eps) for _ in range(num_iters): # 1 iter of Sinkhorn # total weight per dimension (or cluster)

c = sum(Q, dim=0, keepdim=True) Q /= c # total weight per sample n = sum(Q, dim=1, keepdim=True) # Q sums to 1 for each sample (assignment) Q /= n return Q
As a matter of fact, when performing a single Sinkhorn iteration (num_iters=1) this is equivalent to taking a softmax function in the batch dimension and the implementation can be highly simplified:

Q = softmax(Z / eps, dim=0) Q /= sum(Q, dim=1, keepdim=True)
Intuitively, the softmax operation on the batch axis allows to select for each dimension (or "cluster") its best match within the batch.

Working with small batches

When the number B of batch features is too low, our constraint to equally partition the batch into the K prototypes does not really make sense anymore. Therefore, when working with small batches, we use features from the previous batches to augment the size of Z in Prob. (4.4). This is similar to the memory bank mechanism used in contrastive methods [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF]. Then, we only use the pseudo-labels of the batch features in our training loss. In practice, we store around 3K features, i.e., in the same range as the number of code vectors. This means that we only keep features from the last 15 batches with a batch size of 256, while contrastive methods typically need to store the last 65K instances obtained from the last 250 batches to maintain good performance [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF].

Multi-crop: local-to-global matching

As noted in prior works [Chen et al., 2020b, Misra and[START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF] and as observed in our previous experiments of Section 3.4, comparing random crops of an image plays a central role by capturing information in terms of relations between parts of a scene or an object. Instead of sampling a pair of matching views identically distributed as in Chen et al. [2020b], [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF], we design a cropping strategy that encourage local-to-global correspondences in the features. This way, the task of the network is to extrapolate a situation from a narrow restrained view. More precisely, from a given image, we generate a set V of different views. This set contains two global views, x g 1 and x g 2 and several local views of smaller resolution. Only the global views are used to compute a pseudo-label which then needs to be predicted from each of the other views of the same image. We minimize the per-example following loss:

min θ ÿ xtoe{x g 1 ,x g 2 } ÿ xsoeV xs" = xt H(q(x t ), p(x s )). (4.6)
This loss is general and can be used on any number of views, even only 2. However, we find that using 2 global views at resolution 224 2 covering a large (for example greater than 50%) area of the original image, and several local views of resolution 96 2 covering only small areas (for example less than 50%) of the original image brings considerable boosts of performance. We refer to this setting as the basic parametrization of SwAV, unless mentioned otherwise. As a matter of fact, using low resolution images ensures only a small increase in the compute cost.

Implementation details

We implement in SwAV the generic implementation improvements used in Sim-CLR [Chen et al., 2020b], i.e., LARS [START_REF] You | Large batch training of convolutional networks[END_REF], cosine learning rate [START_REF] Loshchilov | Sgdr: Stochastic gradient descent with warm restarts[END_REF] and the MLP projection head [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF]. We train SwAV with stochastic gradient descent using large batches of 4096 different instances. We distribute the batches over 64 V100 16Gb GPUs, resulting in each GPU treating 64 instances. The temperature parameter • is set to 0.1 and the Sinkhorn regularization parameter Á is set to 0.05 for all runs. We use a weight decay of 10 ≠6 and a learning rate of 4.8 which is linearly ramped up during the first 10 epochs. We find empirically that freezing the last layer of the network during the first epoch of training improves stability. We synchronize batch-normalization layers across GPUs using the optimized implementation with kernels through CUDA/C-v2 extension from apex 1 . We also use apex library for training with mixed precision [START_REF] Micikevicius | Mixed precision training[END_REF]. Overall, thanks to these training optimizations (mixed precision, kernel batch-normalization and use of large batches), 100 epochs of training for our best SwAV model take approximately 6 hours. Similarly to previous works [Chen et al., 2020b[START_REF] Sohn | Fixmatch: Simplifying semi-supervised learning with consistency and confidence[END_REF], we use a projection head consisting of a 2-layer MLP on top of the backbone features. Note that SwAV is more suitable for a multi-node distributed implementation compared to contrastive approaches SimCLR or MoCo. The latter methods require sharing the feature matrix across all GPUs at every batch which might become a bottleneck when distributing across many GPUs. On the contrary, SwAV requires sharing only matrix normalization statistics (sum of rows and columns) during the Sinkhorn algorithm. For multi-crop, we perform crops of random sizes, scales and aspect ratios. Specifically we use the RandomResizedCrop method from torchvision.transforms module of PyTorch with s=(0.14, 1) scaling parameters for the global views and s=(0.05, 0.14) parameters for the local ones. Then, we resize global views to 224 ◊ 224 pixels, unless specified otherwise (we use 160 ◊ 160 resolutions in some of our experiments). Local views are resized to 96 ◊ 96 resolution. Finally, we apply random horizontal flips, color distortion and Gaussian blur to each resulting crop, exactly following the SimCLR implementation [Chen et al., 2020b]. Our code is publicly available at https://github.com/facebookresearch/swav.

Main Results

We analyze the features learned by SwAV trained on ImageNet without labels by transfer learning on multiple datasets.

Evaluating the unsupervised features on ImageNet

We evaluate the features of a ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] trained with SwAV on ImageNet by two experiments: linear classification on frozen features and semi-supervised learning by finetuning with few labels. When using frozen features (Figure 4.1 (left)), SwAV outperforms the state of the art by +4.2% top-1 accuracy and is only 1.2% below the performance of a fully supervised model. Note that we train SwAV during 800 epochs with large batches (4096). We refer to Figure 4.2 (right) for results with shorter trainings and to Table 4.3 for experiments with small batches.

On semi-supervised learning (see Table 4.1), SwAV outperforms other self-supervised methods and is on par with state-of-the-art semi-supervised models [START_REF] Sohn | Fixmatch: Simplifying semi-supervised learning with consistency and confidence[END_REF], despite the fact that SwAV is not specifically designed for semi-supervised learning. Note that in [START_REF] Assran | Semisupervised learning of visual features by non-parametrically predicting view assignments with support samples[END_REF], we extend SwAV methodology to a semi-supervised setting, 1% labels 10% labels Method Top-1 Top-5 Top-1 Top-5 Supervised 25.4 48.4 56.4 80.4

Methods using label-propagation

UDA [Xie et al., 2020a] --68.8 88.5 FixMatch [START_REF] Sohn | Fixmatch: Simplifying semi-supervised learning with consistency and confidence[END_REF]]--71.5 89.1

Methods using self-supervision only

PIRL [START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF] 30.7 57.2 60.4 83.8 PCL [START_REF] Li | Prototypical contrastive learning of unsupervised representations[END_REF] -75.6 -86.2 SimCLR [Chen et al., 2020b] 48.3 75.5 65.6 87.8 SwAV 53.9 78.5 70.2 89.9

Table 4.1 -Semi-supervised learning on ImageNet with a ResNet-50. We finetune the model with 1% and 10% labels and report top-1 and top-5 accuracies. We use the same splits as Chen et al. [2020b].

compare the performance of SwAV features with ImageNet supervised pre-training and with self-supervised features available at the time of SwAV publication. First, we report the linear classification performance on the Places205 [START_REF] Zhou | Learning deep features for scene recognition using places database[END_REF], VOC07 [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF], and iNaturalist2018 [START_REF] Van Horn | The inaturalist species classification and detection dataset[END_REF] datasets. Our method outperforms supervised features on all three datasets. We observe that SwAV is the first self-supervised method to surpass ImageNet supervised features on these datasets. Second, we report network finetuning on object detection on VOC07+12 using Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] and on COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] with DETR [START_REF] Carion | End-to-end object detection with transformers[END_REF]. DETR is a recent object detection framework that reaches competitive performance with Faster R-CNN while being conceptually simpler and trainable end-to-end. We use DETR because, unlike Faster R-CNN [START_REF] He | Rethinking imagenet pre-training[END_REF], using a pretrained backbone in this framework is crucial to obtain good results compared to training from scratch [START_REF] Carion | End-to-end object detection with transformers[END_REF]. In Table 4.2, we show that SwAV outperforms the supervised pretrained model on both VOC07+12 and COCO datasets. Note that this is line with previous works that also show that self-supervision can outperform supervised pretraining on object detection [Gidaris et al., 2020a, He et al., 2020[START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF]. Overall, our SwAV ResNet-50 model surpasses supervised ImageNet pretraining on all the considered transfer tasks and datasets. We have released this model so other researchers might also benefit by replacing the ImageNet supervised network with our model. ResNet-50 trained on ImageNet with SwAV or supervised learning. We also report numbers from other self-supervised methods available at the time of SwAV publication ( † for numbers from other methods run by us). We consider two settings.

(1) Linear classification on top of frozen features. We report top-1 accuracy on all datasets except VOC07 where we report mAP.

(2) Object detection with finetuned features on VOC07+12 trainval using Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] and on COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] using Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] or DETR [START_REF] Carion | End-to-end object detection with transformers[END_REF]. We report the most standard detection metrics for these datasets: AP 50 on VOC07+12 and AP on COCO.

Training with small batches

We train SwAV with small batches of 256 images on 4 GPUs and compare with MoCov2 and SimCLR trained in the same setup. In Table 4.3, we see that SwAV maintains stateof-the-art performance even when trained in the small batch setting. Note that SwAV only stores a queue of 3, 840 features. In comparison, to obtain good performance, MoCov2 needs to store 65, 536 features while keeping an additional momentum encoder network. When SwAV is trained using 2 ◊ 160 + 4 ◊ 96 crops, SwAV has a running time 1.2◊ higher than SimCLR with 2◊224 crops and is around 1.4◊ slower than MoCov2 due to the additional back-propagation [Chen et al., 2020e]. Hence, one epoch of MoCov2 or SimCLR is faster in wall clock time than one of SwAV, but these methods need more epochs for good downstream performance. Indeed, as shown in We also report the number of stored features, the type of cropping used and the number of epochs.

(102 hours) while MoCov2 needs 800 epochs to achieve 71.1%. Increasing the resolution and the number of epochs, SwAV reaches 74.3% with a small batch size, a small number of stored features and no momentum encoder. Finally, note that SwAV can be combined with a momentum mechanism and we leave these explorations to the next chapter.

Ablation Study and Analyses

Improving deep clustering approaches. In this section, we re-implement and improve two clustering-based models in order to assess if they can compete with well-known contrastive methods such as SimCLR [Chen et al., 2020b]. In particular, we consider two clustering-based models: DeepCluster-v2 and SeLa-v2, which are obtained by applying various training improvements introduced in other self-supervised learning papers to DeepCluster of [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF] and SeLa of [START_REF] Asano | Self-labelling via simultaneous clustering and representation learning[END_REF]. Among these improvements are the use of stronger data augmentation [Chen et al., 2020b], MLP projection head [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF], cosine learning rate schedule [START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF], use of temperaturebased softmax [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF], memory bank [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF], multi-clustering [START_REF] Asano | Self-labelling via simultaneous clustering and representation learning[END_REF], etc. The implementation of DeepCluster-v2 is publicly available at https: //github.com/facebookresearch/swav/main_deepclusterv2.py. Besides, we also improve DeepCluster model by introducing explicit comparisons to k-means centroids, which increase stability and performance. Indeed, a main issue in DeepCluster is that there is no correspondence between two consecutive cluster assignments. Hence, the final classification layer learned for an assignment becomes irrelevant for the following one and thus needs to be re-initialized from scratch at each epoch. regularly update centroids and cluster assignments for each image.

As a matter of fact, DeepCluster-v2 can be interpreted as a special case of our proposed swapping mechanism: swapping is done across epochs rather than within a batch. Given a crop of an image DeepCluster-v2 predicts the assignment of another crop, which was obtained at the previous epoch. SwAV swaps assignments directly at the batch level and can thus work online.

Applying the multi-crop strategy to different methods. In Table 4.2 (left), we report the impact of applying our multi-crop strategy on the performance of a selection of other methods. We see that the multi-crop strategy consistently improves the performance for all the considered methods by a significant margin of 2≠4% top-1 accuracy. Interestingly, multi-crop seems to benefit more clustering-based methods than contrastive methods. We note that multi-crop does not improve the supervised model.

Impact of longer training. In Figure 4.2 (right), we show the impact of the number of training epochs on performance for SwAV. We train separate models for 100, 200, 400 and 800 epochs and report the top-1 accuracy on ImageNet using the linear classification evaluation. We train each ResNet-50 on 64 V100 16GB GPUs and a batch size of 4096. While SwAV benefits from longer training, it already achieves strong performance after 100 epochs, i.e., 72.1% in 6h15.

Running times. In Table 4.4, we report compute and GPU memory requirements based on our implementation for different settings. As described in Section 4.2.5, we train each method on 64 V100 16GB GPUs, with a batch size of 4096, using mixed precision and apex optimized version of synchronized batch-normalization layers. We report results with ResNet-50 for all methods. In Figure 4.3, we report SwAV performance for different training lengths measured in hours based on our implementation. We observe that after only 6 hours of training, SwAV outperforms SimCLR trained for 1000 epochs (40 hours

Method

Frozen Finetuned Random weights 15.0 76.5

SimCLR [Chen et al., 2020b] 60.4 77.2 SwAV 66.5 77.8

Table 4.6 -Pretraining ResNet-50 with SwAV on uncurated data from Instagram.

Top-1 accuracy on ImageNet for pretrained models on an uncurated set of 1B random Instagram images. We compare ResNet-50 pretrained with either SimCLR or SwAV on two downstream tasks: linear classification on frozen features or finetuned features.

Scaling self-supervised learning

Scaling the data. First, we pre-train SwAV on an uncurated dataset of 1 billion random public non-EU images from Instagram by keeping the same architecture as in our previous experiments, i.e., ResNet-50. This is to test if SwAV can serve as a pre-training method for supervised learning. In Table 4.6, we measure the performance of ResNet-50 models when transferring to ImageNet with frozen or finetuned features. We compare SwAV with a randomly initialized network and with a network pre-trained on the same data using SimCLR [Chen et al., 2020b]. First, we observe that SwAV maintains a similar gain of 6% over SimCLR as when pre-trained on ImageNet (see Figure 4.1), showing that our improvements do not depend on the data distribution. We also see that pre-training with SwAV on random images significantly improves over training from scratch on ImageNet (+1.3%). This result is in line with [START_REF] Caron | Unsupervised pre-training of image features on non-curated data[END_REF] and [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF]. This preliminary experiment motivates the exploration of the limits of pre-training as we increase not only the dataset size but also the model capacity. We consider the variants of the ResNeXt architecture [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF] as in [START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF] and also models from the RegNet family [START_REF] Radosavovic | Designing network design spaces[END_REF] which we describe in the following paragraph.

Scale efficient model family: RegNetY. Scaling data and model capacity jointly requires using architectures that are efficient in terms of both memory and runtime. RegNets are a family of models designed for this purpose and we briefly describe them in this paragraph. We refer to [START_REF] Radosavovic | Designing network design spaces[END_REF] for more details. RegNets are a family of architectures defined by a design space of convnets consisting of 4 stages, with each stage containing a series of identical blocks, while keeping the structure of their blocks fixed -namely the residual bottleneck block of [START_REF] He | Deep residual learning for image recognition[END_REF]. Here, we focus on the RegNetY architectures, that add a Squeeze-and-excitation op [START_REF] Hu | Squeeze-and-excitation networks[END_REF] We compare pretraining a ResNeXt101-32dx8d with self-supervision on random images with pretraining on filtered images labeled with hashtags that are similar to ImageNet classes [START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF]. We report top-1 accuracy on ImageNet with finetuning. For completeness, we also report the best performance reported with larger architectures.

improve their performance. The RegNetY model family is parameterized by 5 parameters, allowing the search of a good instance with a certain number of FLOPs with reasonable resources. The models we used were all searched on ImageNet using the same procedure as [START_REF] Radosavovic | Designing network design spaces[END_REF]. We believe our results can further be improved by searching for RegNetYs directly on our self-supervised pre-training task. Our model of focus is the RegNetY-256GF architecture. Its parametrization is given by the scaling rules of RegNets [START_REF] Radosavovic | Designing network design spaces[END_REF]:

w 0 =640,w a =230.

Comparing to weakly-supervised pre-training

Many online images have some metadata, e.g., hashtags or geo-localization, that can be leveraged during pre-training. In particular, [START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF] show that pre-training by predicting a curated set of hashtags can greatly improve the quality of the resulting visual features. Their approach requires to filter images and only works in the presence of textual metadata. In Table 4.7, we compare our SwAV pre-training on random images to theirs on the same architecture, a ResNeXt101-32x8d, with finetuning. For completeness, we also report their best number with their largest architecture. First, we observe that both pre-trainings improve top-1 accuracy over a model trained from scratch, showing in general the benefits of pre-training. Our approach is also in the same ballpark as theirs even though we do not rely on data curation nor supervision. Note that, when the features are frozen, their approach maintains high performance on ImageNet, with 81.6% top-1 accuracy while our model performance drops significantly. This result is not surprising: they pre-train on data that follows the same concepts as ImageNet classes and thus the learned features are more aligned with the target distribution. Since we pre-train our model on random images, we require a full-finetuning step of 35 epochs to adapt to the target distribution. This experiment shows that the benefits of pre-training with finetuning exist even if the features come from a different image distribution.

Chapter 5

Self-Supervised Vision Transformers

Like the contributions presented in this manuscript so far, most of the research in self-supervised learning has been conducted on convolutional networks (convnets). In this final chapter, we explore if self-supervised learning provides new properties to Vision Transformer (ViT) [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] that stand out compared to convnets. Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], multi-crop training [START_REF]1007 73.5 66[END_REF], and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base. This chapter is based on a work published at ICCV 2021 (see [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF]).

Introduction

Transformers [START_REF] Vaswani | Attention is all you need[END_REF] have recently emerged as an alternative to convnets for visual recognition [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF][START_REF] Zhao | Exploring self-attention for image recognition[END_REF]. Their adoption has been coupled with a training strategy inspired by natural language processing (NLP), that is, pretraining on large quantities of data and finetuning on the target dataset [START_REF] Devlin | Pre-training of deep bidirectional transformers for language understanding[END_REF], Radford et al., 2019]. The resulting Vision Transformers (ViT) [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] are competitive with convnets but, they have not yet delivered clear benefits over them: they are computationally more demanding, require more training data, and their features do not exhibit unique properties.

In this paper, we question whether the muted success of Transformers in vision can be explained by the use of supervision in their pre-training. Our motivation is that one of the main ingredients for the success of Transformers in NLP was the use of self-supervised pretraining, in the form of close procedure in BERT [START_REF] Devlin | Pre-training of deep bidirectional transformers for language understanding[END_REF] or language modeling in GPT [START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. These self-supervised pretraining objectives use the words in a sentence to create pretext tasks that provide a richer learning signal than the supervised objective of predicting a single label per sentence. Similarly, in images, imagelevel supervision often reduces the rich visual information contained in an image to a single concept selected from a predefined set of a few thousand categories of objects [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF].

While the self-supervised pretext tasks used in NLP are text specific, many existing self-supervised methods have shown their potential on images with convnets [START_REF]1007 73.5 66[END_REF], Chen et al., 2020b, Grill et al., 2020, He et al., 2020]. They typically share a similar structure but with different components designed to avoid trivial solutions (collapse) or to improve performance [Chen and He, 2020]. In this work, inspired from these methods, we study the impact of self-supervised pretraining on ViT features. Of particular interest, we have identified several interesting properties that do not emerge with supervised ViTs, nor with convnets:

-Self-supervised ViT features explicitly contain the scene layout and, in particular, object boundaries. This information is directly accessible in the self-attention modules of the last block. -Self-supervised ViT features perform particularly well with a basic nearest neighbors classifier (k-NN). For instance, a small ViT achieves 78.3% top-1 accuracy on Imagenet with a k-NN classifier on frozen features, without any finetuning, linear classifier nor data augmentation. This property is not as prominent when using convnets, nor when training ViT networks with other self-supervised components.

The emergence of segmentation masks seems to be a property shared across selfsupervised methods. However, the good performance with k-NN only emerge with DINO. Another finding from our study is the importance of using smaller patches with ViTs to improve the quality of the resulting features.

Overall, our findings about the importance of certain components for ViT lead us to design a simple self-supervised approach that can be interpreted as a form of knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] with no labels. The resulting framework, DINO, simplifies self-supervised training by directly predicting the output of a teacher network-built with a momentum encoder-by using a standard cross-entropy loss. Interestingly, our method can work with only a centering and sharpening of the teacher output to avoid .1 -Self-distillation with no labels. We illustrate DINO in the case of one single pair of views (x 1 , x 2 ) for simplicity. The model passes two different random transformations of an input image to the student and teacher networks. Both networks have the same architecture but different parameters. The output of the teacher network is centered with a mean computed over the batch. Each networks outputs a K dimensional feature that is normalized with a temperature softmax over the feature dimension. Their similarity is then measured with a cross-entropy loss. We apply a stop-gradient (sg) operator on the teacher to propagate gradients only through the student. The teacher parameters are updated with an exponential moving average (ema) of the student parameters. collapse, while other popular components such as predictor [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF], equipartition constraints [START_REF]1007 73.5 66[END_REF] or contrastive loss [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] add little benefits in terms of stability or performance. Of particular importance, our framework is flexible and works on both convnets and ViTs without the need to modify the architecture, nor adapt the internal normalizations [START_REF] Richemond | Byol works even without batch statistics[END_REF].

We further validate the synergy between DINO and ViT by outperforming previous self-supervised features on the ImageNet linear classification benchmark with 80.1% top-1 accuracy with a ViT-Base with small patches. We also confirm that DINO works with convnets by matching the state of the art with a ResNet-50 architecture. Finally, we discuss different scenarios to use DINO with ViTs in case of limited computation and memory capacity. In particular, training DINO with ViT takes just two 8-GPU servers over 3 days to achieve 76.1% on ImageNet linear benchmark, which outperforms selfsupervised systems based on convnets of comparable sizes with significantly reduced compute requirements [START_REF]1007 73.5 66[END_REF], Grill et al., 2020].

Related work

As we interpret our method as a kind of unsupervised self-distillation, we give a brief overview of related work about supervised knowledge distillation. We also quickly review the Vision Transformer architecture.

Self-training and knowledge distillation.

Self-training aims at improving the quality of features by propagating a small initial set of annotations to a large set of unlabeled instances. This propagation can either be done with hard assignments of labels [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF][START_REF] Xu | Maximum margin clustering[END_REF][START_REF] Yalniz | Billion-scale semi-supervised learning for image classification[END_REF] or with a soft assignment [Xie et al., 2020b]. When using soft labels, the approach is often referred to as knowledge distillation [START_REF] Buciluǎ | Model compression[END_REF][START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] and has been primarily designed to train a small network to mimic the output of a larger network to compress models. Xie et al. [2020b] have shown that distillation could be used to propagate soft pseudo-labels to unlabeled data in a self-training pipeline, drawing an essential connection between self-training and knowledge distillation. Our work builds on this relation and extends knowledge distillation to the case where no labels are available. Previous works have also combined self-supervised learning and knowledge distillation [Chen et al., 2020c[START_REF] Fang | Seed: Self-supervised distillation for visual representation[END_REF][START_REF] Noroozi | Boosting self-supervised learning via knowledge transfer[END_REF][START_REF] Shen | S2-bnn: Bridging the gap between self-supervised real and 1-bit neural networks via guided distribution calibration[END_REF], enabling self-supervised model compression and performance gains. However, these works rely on a pre-trained fixed teacher while our teacher is dynamically built during training. This way, in our work, knowledge distillation instead of being used as a post-processing step to self-supervised pre-training, is directly cast as a self-supervised objective. Finally, our work is also related to co-distillation [START_REF] Anil | Large scale distributed neural network training through online distillation[END_REF] where student and teacher have the same architecture and use distillation during training. However, the teacher in co-distillation is also distilling from the student, while it is updated with an average of the student in our work.

Vision transformers

The Transformer architecture has originally been proposed in the context of machine translation by [START_REF] Vaswani | Attention is all you need[END_REF]. Since its introduction, this architecture has been successfully applied to sentence representation [START_REF] Devlin | Pre-training of deep bidirectional transformers for language understanding[END_REF], language modeling [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] and more recently speech recognition [START_REF] Baevski | wav2vec 2.0: A framework for selfsupervised learning of speech representations[END_REF]. While there have been many attempts at adapting the Transformer architecture to images [START_REF] Child | Generating long sequences with sparse transformers[END_REF][START_REF] Cordonnier | On the relationship between self-attention and convolutional layers[END_REF], Parmar et al., 2018], it is only recently that standard Transformers have obtained competitive results on challenging image classification datasets [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] Zhao | Exploring self-attention for image recognition[END_REF]. In particular, [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] 5.3 DINO Methodology

SSL with knowledge distillation

The framework used for this work, DINO, shares the same overall structure as most recent self-supervised approaches [START_REF]1007 73.5 66[END_REF], Chen et al., 2020b, Chen and He, 2020, Grill et al., 2020, He et al., 2020] following the view-invariant paradigm described in Section 2.3. However, our method shares also similarities with knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] and we present it under this angle. We illustrate DINO in Figure 5.1 and propose a pseudo-code implementation in Algorithm 1.

Knowledge distillation is a learning paradigm where we train a student network g θs to match the output of a given teacher network g θt , parameterized by ◊ s and ◊ t respectively. Given an input image x, both networks output probability distributions over K dimensions denoted by P s and P t . The probability P is obtained by normalizing the output of the network g with a softmax function. More precisely, where H(a, b)=≠a log b.

P s (x) (i) = exp(g θs (x) (i) /• s ) q K k=1 exp(g θs (x) (k) /• s ) , ( 5
In the following, we detail how we adapt the problem in Eq. (5.2) to self-supervised learning. First, we construct different distorted views, or crops, of an image with multi-crop strategy (see Section 4.2.4). More precisely, from a given image, we generate a set V of different views. This set contains two global views, x g 1 and x g 2 and several local views of smaller resolution. All crops are passed through the student while only the global views are passed through the teacher, therefore encouraging "local-to-global" correspondences. We minimize the loss:

min θs ÿ xoe{x g 1 ,x g 2 } ÿ x Õ oeV x Õ " = x
H(P t (x),P s (x Õ )).

(5.3)

This loss is general and can be used on any number of views, even only 2. However, we follow the standard setting for multi-crop by using 2 global views at resolution 224 2 covering a large (for example greater than 50%) area of the original image, and several local views of resolution 96 2 covering only small areas (for example less than 50%) of the original image. Both networks share the same architecture g with different sets of parameters ◊ s and ◊ t . We learn the parameters ◊ s by minimizing Eq. ( 5.3) with stochastic gradient descent.

Teacher network. Unlike knowledge distillation, we do not have a teacher g θt given a priori and hence, we build it from past iterations of the student network. We study different update rules for the teacher in Section 5.6.2 and show that freezing the teacher network over an epoch works surprisingly well in our framework, while copying the student weight for the teacher fails to converge. Of particular interest, using an exponential moving average (EMA) on the student weights, i.e., a momentum encoder [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], is particularly well suited for our framework. The update rule is ◊ t Ω ⁄◊ t +(1≠ ⁄)◊ s , with ⁄ following a cosine schedule from 0.996 to 1 during training [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF]. Originally the momentum encoder has been introduced as a substitute for a queue in contrastive learning [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF]. However, in our framework, its role differs since we do not have a queue nor a contrastive loss, and may be closer to the role of the mean teacher used in self-training [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF]. Indeed, we observe that this teacher performs a form of model ensembling similar to Polyak-Ruppert averaging with an exponential decay [START_REF] Polyak | Acceleration of stochastic approximation by averaging[END_REF]Juditsky, 1992, Ruppert, 1988]. Using Polyak-Ruppert averaging for model ensembling is a standard practice to improve the performance of a model [START_REF] Jean | On using very large target vocabulary for neural machine translation[END_REF]. We observe that this teacher has better performance than the student throughout the training, and hence, guides the training of the student by providing target features of higher quality. This dynamic was not observed in previous works [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF], Richemond et al., 2020].

Network architecture. The neural network g is composed of a backbone f (ViT [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] or ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]), and of a projection head h: g = h ¶ f . The features used in downstream tasks are the backbone f output. The projection head consists of a 3-layer multi-layer perceptron (MLP) with hidden dimension 2048 followed by ¸2 normalization and a weight normalized fully connected layer [START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF] with K dimensions, which is similar to the design from SwAV [START_REF]1007 73.5 66[END_REF].

We have tested other projection heads and this particular design appears to work best for DINO. We do not use a predictor [Chen andHe, 2020, Grill et al., 2020], resulting in the exact same architecture in both student and teacher networks. Of particular interest, we note that unlike standard convnets, ViT architectures do not use batch normalizations (BN) by default. Therefore, when applying DINO to ViT we do not use any BN also in the projection heads, making the system entirely BN-free.

Avoiding collapse. Several self-supervised methods differ by the operation used to avoid collapse, either through contrastive loss [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF], clustering constraints [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF][START_REF]1007 73.5 66[END_REF], predictor [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF] or batch normalizations [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF], Richemond et al., 2020]. While our framework can be stabilized with multiple normalizations [START_REF]1007 73.5 66[END_REF], it can also work with only a centering and sharpening of the momentum teacher outputs to avoid model collapse. As shown experimentally in Section 5.6.3, centering prevents one dimension to dominate but encourages collapse to the uniform distribution, while the sharpening has the opposite effect. Applying both operations balances their effects which is sufficient to avoid collapse in presence of a momentum teacher. Choosing this method to avoid collapse trades stability for less dependence over the batch: the centering operation only depends on first-order batch statistics and can be interpreted as adding a bias term c to the teacher: g t (x) Ω g t (x)+c. The center c is updated with an exponential moving average, which allows the approach to work well across different batch sizes as shown in Section 5.6.5:

c Ω mc +(1≠ m) 1 B B ÿ i=1 g θt (x i ), (5.4)
where m>0 is a rate parameter and B is the batch size. Output sharpening is obtained by using a low value for the temperature • t in the teacher softmax normalization.

Implementation and evaluation protocols

In this section, we provide the implementation details to train with DINO and present the evaluation protocols used in our experiments.

Implementation details. We pretrain the models on the ImageNet dataset [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] without labels. We train with the adamw optimizer [START_REF] Loshchilov | Fixing weight decay regularization in adam[END_REF] and a batch size of 1024, distributed over 16 GPUs when using ViT-S/16. The learning rate is linearly ramped up during the first 10 epochs to its base value determined with the following linear scaling rule [START_REF] Goyal | Accurate, large minibatch sgd: Training imagenet in 1 hour[END_REF]: lr =0 .0005 ú batchsize/256. After this warm-up, we decay the learning rate with a cosine schedule [START_REF] Loshchilov | Sgdr: Stochastic gradient descent with warm restarts[END_REF]. The weight decay also follows a cosine schedule from 0.04 to 0.4. The temperature • s is set to 0.1 while we use a linear warm-up for • t from 0.04 to 0.07 during the first 30 epochs. We follow the data augmentations of BYOL [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF] (color jittering, Gaussian blur and solarization) and multi-crop [START_REF]1007 73.5 66[END_REF] with a bicubic interpolation to adapt the position embeddings to the scales [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF]. The code and models to reproduce our results is publicly available at https://github.com/facebookresearch/dino. Evaluation protocols. Standard protocols for self-supervised learning are to either learn a linear classifier on frozen features [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], Zhang et al., 2016] or to finetune the features on downstream tasks. For linear evaluations, we apply random resize crops and horizontal flips augmentation during training, and report accuracy on a central crop. For finetuning evaluations, we initialize networks with the pretrained weights and adapt them during training. However, both evaluations are sensitive to hyperparameters, and we observe a large variance in accuracy between runs when varying the learning rate for example. We thus also evaluate the quality of features with a simple weighted nearest neighbor classifier (k-NN) as in [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF]. We freeze the pretrain model to compute and store the features of the training data of the downstream task. The nearest neighbor classifier then matches the feature of an image to the k nearest stored features that votes for the label. We sweep over different number of nearest neighbors and find that 20 NN is consistently working the best for most of our runs. This evaluation protocol does not require any other hyperparameter tuning, nor data augmentation and can be run with only one pass over the downstream dataset, greatly simplifying the feature evaluation. We compare the performance in retrieval of off-the-shelf features pretrained with supervision or with DINO on ImageNet and Google Landmarks v2 (GLDv2) dataset. We report mAP on revisited Oxford and Paris. Pretraining with DINO on a landmark dataset performs particularly well. For reference, we also report the best retrieval method with off-the-shelf features [START_REF] Revaud | Learning with average precision: Training image retrieval with a listwise loss[END_REF].

Nearest neighbor retrieval with DINO ViT

The results on ImageNet classification have exposed the potential of our features for tasks relying on nearest neighbor retrieval. In this set of experiments, we further consolidate this finding on landmark retrieval and copy detection tasks.

Image Retrieval. We consider the revisited [Radenović et al., 2018a] Oxford and Paris image retrieval datasets [START_REF] Philbin | Lost in quantization: Improving particular object retrieval in large scale image databases[END_REF]. They contain 3 different splits of gradual difficulty with query/database pairs. We report the Mean Average Precision (mAP) for the Medium (M) and Hard (H) splits. In Table 5.3, we compare the performance of different off-the-shelf features obtained with either supervised or DINO training. We freeze the features and directly apply k-NN for retrieval. We observe that DINO features outperform those trained on ImageNet with labels.

An advantage of SSL approaches is that they can be trained on any dataset, without requiring any form of annotations. We train DINO on the 1.2M clean set from Google Landmarks v2 (GLDv2) [START_REF] Weyand | Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval[END_REF], a dataset of landmarks designed for retrieval purposes. DINO ViT features trained on GLDv2 are remarkably good, outperforming previously published methods based on off-the-shelf descriptors [START_REF] Revaud | Learning with average precision: Training image retrieval with a listwise loss[END_REF][START_REF] Tolias | Particular object retrieval with integral max-pooling of cnn activations[END_REF].

Copy detection. We also evaluate the performance of ViTs trained with DINO on a copy detection task. We report the mean average precision on the "strong" subset of the INRIA Method Arch.

Dim. Resolution mAP

Multigrain [START_REF] Berman | MultiGrain: a unified image embedding for classes and instances[END_REF] ResNet-50 2048 224 2 75.1 Multigrain [START_REF] Berman | MultiGrain: a unified image embedding for classes and instances[END_REF] ResNet-50 2048 largest side 800 82.5 Supervised [START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF] Table 5.4 -Copy detection. We report the mAP performance in copy detection on Copydays "strong" subset [START_REF] Douze | Evaluation of gist descriptors for web-scale image search[END_REF]. For reference, we also report the performance of the multigrain model [START_REF] Berman | MultiGrain: a unified image embedding for classes and instances[END_REF], trained specifically for particular object retrieval.

Copydays dataset [START_REF] Douze | Evaluation of gist descriptors for web-scale image search[END_REF]. The task is to recognize images that have been distorted by blur, insertions, print and scan, etc. Following prior work [START_REF] Berman | MultiGrain: a unified image embedding for classes and instances[END_REF], we add 10k distractor images randomly sampled from the YFCC100M dataset [START_REF] Thomee | Yfcc100m: The new data in multimedia research[END_REF]. We perform copy detection directly with cosine similarity on the features obtained from our pretrained network. The features are obtained as the concatenation of the output [CLS] token and of the GeM pooled [Radenović et al., 2018b] output patch tokens. This results in a 1536d descriptor for ViT-B. Following [START_REF] Berman | MultiGrain: a unified image embedding for classes and instances[END_REF], we apply whitening on the features. We learn this transformation on an extra 20K random images from YFCC100M, distinct from the distractors. Table 5.4 shows that ViT trained with DINO is very competitive on copy detection.

Image classification. In Table 5.5, we evaluate the frozen representations given by ResNet-50 or ViT-small pre-trained with DINO with two evaluation protocols: linear or k-NN. For both evaluations, we extract representations from a pre-trained network without using any data augmentation. Then, we perform classification either with weighted k-NN or with a linear regression learned with cyanure library [START_REF]Cyanure: An open-source toolbox for empirical risk minimization for python, c++, and soon more[END_REF]. In Table 5.5 we see that ViT-S accuracies are better than accuracies obtained with RN50 both with a linear or a k-NN classifier. However, the performance gap when using the k-NN evaluation is much more significant than when considering linear evaluation. Table 5.5 -k-NN and linear evaluation for ViT-S/16 and ResNet-50 pre-trained with DINO. We use ImageNet-1k [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] ("Inet"), Places205 [START_REF] Zhou | Learning deep features for scene recognition using places database[END_REF], PASCAL VOC [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] and Oxford-102 flowers ("FLOW-ERS") [START_REF] Nilsback | Automated flower classification over a large number of classes[END_REF]. ViT trained with DINO provides features that are particularly k-NN friendly.

Class Representation As a final study of the k-NN property, we propose to look at the distribution of ImageNet concepts in the feature space from DINO. We represent each ImageNet class with the average feature vector for its validation images. We reduce the dimension of these features to 30 with PCA, and run t-SNE with a perplexity of 20, a learning rate of 200 for 5000 iterations. We present the resulting class embeddings in Figure 5.2. Our model recovers structures between classes: similar animal species are grouped together, forming coherent clusters of birds (top) or dogs, and especially terriers (far right).

Discovering the semantic layout of scenes

As shown qualitatively in Figure 5.3, our self-attention maps contain information about the segmentation of an image. In this study, we measure this property on a standard benchmark as well as by directly probing the quality of masks generated from these attention maps.

Video instance segmentation. In Table 5.6, we evaluate the output patch tokens on the DAVIS-2017 video instance segmentation benchmark [START_REF] Pont-Tuset | The 2017 davis challenge on video object segmentation[END_REF]. We follow the experimental protocol in [START_REF] Jabri | Space-time correspondence as a contrastive random walk[END_REF] and segment scenes with a nearest-neighbor We study the impact of the smoothing parameters in the update rule for the center c used in the output of the teacher network (see Eq. ( 5.4)).

The convergence is robust to a wide range of smoothing, and the model only collapses when the update is too slow, i.e., m =0.999.

Compute requirements

In Table 5.14, we detail the time and GPU memory requirements when running ViT-S/16 DINO models on two 8-GPU machines. We report results with several variants of multi-crop training, each having a different level of compute requirement. We observe in Table 5.14 that using multi-crop improves the accuracy / running-time trade-off for DINO runs. For example, the performance is 72.5% after 46 hours of training without multi-crop (i.e. 2◊224 2 ) while DINO in 2◊224 2 +10◊96 2 crop setting reaches 74.6% in 24 hours only. This is an improvement of +2% while requiring 2◊ less time, though the memory usage is higher (15.4G versus 9.3G). We observe that the performance boost brought with multi-crop cannot be caught up by more training in the 2◊224 2 setting, which shows the value of the "local-to-global" augmentation. Finally, the gain from adding more views diminishes (+.2% form 6◊ to 10◊ 96 2 crops) for longer trainings.

Overall, training DINO with Vision Transformers achieves 76.1 top-1 accuracy using two 8-GPU servers for 3 days. This result outperforms state-of-the-art self-supervised systems based on convolutional networks of comparable sizes with a significant reduction of computational requirements [START_REF]1007 73.5 66[END_REF], Grill et al., 2020]. Our code is available to train self-supervised ViT on a limited number of GPUs.

Training with small batches

bs 128 256 512 1024 top-1 57.9 59.1 59.6 59.9

Table 5.15 -Effect of batch sizes.

Top-1 with k-NN for models trained for 100 epochs without multi-crop.

In Table 5.15, we study the impact of the batch size on the features obtained with DINO. We also study the impact of the smooth parameter m used in the centering update rule of Eq. (5.4) in Table 5.13. We scale the learning rate linearly with the batch size [START_REF] Goyal | Accurate, large minibatch sgd: Training imagenet in 1 hour[END_REF]: lr =0.0005 ú batchsize/256. Table 5.15 confirms that we can train models to high performance with small batches. Results with the smaller batch sizes (bs =128) are slightly below our default training setup of bs =1024, and would certainly require to re-tune hyperparameters like the momentum rates for example. Note that the experiment with batch size of 128 runs on only 1 GPU. We have explored training a model with a batch size of 8, reaching 35.2% after 50 epochs, showing the potential for training large models that barely fit an image per GPU.

Ablation study on the projection head

Similarly to other self-supervised frameworks, we observe that using a projection head [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF] improves greatly the accuracy of our method. The projection head starts with a n-layer multi-layer perceptron (MLP). The hidden layers are 2048d and are with Gaussian error linear units (GELU) activations. The last layer of the MLP is without GELU. Then we apply a ¸2 normalization and a weight normalized fully connected layer [Chen andHe, 2020, Salimans and[START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF] with K dimensions. This design is inspired from the projection head with a "prototype layer" used in SwAV [START_REF]1007 73.5 66[END_REF]. We do not apply batch normalizations. In this set of experiments, we study the effect of the l2-normalization bottleneck, the number of linear layers in the projection head, the output dimension K, the choice of activation unit and the impact of adding batch normalizations. We observe that adding BN in the projection heads has little impact, showing that BN is not important in our framework. Overall, when applying DINO to ViT, we do not use any BN anywhere, making the system entirely BN-free. This is a great advantage of DINO + ViT to work at state-of-the-art performance without requiring any BN. Indeed, training with BN typically slows down trainings considerably, especially when these BN modules need to be synchronized across processes [START_REF] Caron | Unsupervised pre-training of image features on non-curated data[END_REF], 2020, Grill et al., 2020, He et al., 2020].

BN

L2-normalization bottleneck in projection head. We illustrate the design of the projection head with or without l2-normalization bottleneck in Figure 5.9.

Ablation study on multi-crop

In this section, we study a core component of DINO: multi-crop training which we introduced in previous Chapter 4 (see Section 4.2.4).

Range of scales in multi-crop. We use the RandomResizedCrop in PyTorch for generating the different views. We sample two global views with scale range (s, 1) before (0.05, s), (s, 1), s: 0.08 0.16 0.24 0.32 0.48 k-NN top-1 65.6 68.0 69.7 69.8 69.5 resizing them to 224 2 and 6 local views with scale sampled in the range (0.05,s) resized to 96 2 pixels. Note that we arbitrarily choose to have non-overlapping scaling range for the global and local views following the original design of SwAV. However, the ranges could definitely be overlapping and experimenting with finer hyperparameters search could lead to a more optimal setting. In this table, we vary the parameter s that controls the range of scales used in multi-crop and find the optimum to be around 0.3 in our experiments. We note that this is higher than the parameter used in SwAV which was of 0.14.

Multi-crop in different self-supervised frameworks. Finally, we compare the impact of multi-crop with ViT trained with different recent self-supervised learning frameworks, namely MoCo-v2 [Chen et al., 2020e], BYOL [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF] and SwAV [START_REF]1007 73.5 66[END_REF]. For fair comparisons, all models are pretrained either with two 224 2 crops or with multi-crop [START_REF]1007 73.5 66[END_REF] training, i.e. two 224 2 crops and six 96 2 crops for each image. We report k-NN and linear probing evaluations after 300 epochs of training. In Table 5.16, we observe that multi-crop does not benefit all frameworks equally, which has been ignored in benchmarks considering only the two crops setting [Chen and He, 2020].

The effectiveness of multi-crop depends on the considered framework, which positions multi-crop as a core component of a model and not a simple "add-ons" that will boost any framework the same way. Without multi-crop, DINO has better accuracy than other frameworks, though by a moderate margin (1%). Remarkably, DINO benefits the most from multi-crop training (+3.4% in linear eval). Interestingly, we also observe that the ranking of the frameworks depends on the evaluation protocol considered.

Chapter 6

Conclusion

In conclusion, we summarize the contributions presented in this manuscript before giving an overview of some of the open problems and challenges in the field of selfsupervised learning.

Summary of Contributions

Deep clustering. In Chapter 3, we have introduced deep clustering and presented an extension to this framework: the DeeperCluster method. DeeperCluster iterates between clustering with k-means the features produced by the network and updating its weights by predicting the cluster assignments as pseudo-labels. If trained on large uncurated datasets like YFCC100M for example, it achieves high performance on several standard transfer tasks. Indeed, DeeperCluster outperforms the state of the art at the time of the publication and is getting very close to supervised methods, while not surpassing them. We have identified at the end of Chapter 3 the main limitations of deep clustering (see Section 3.4) that we would need to overcome with the goal of eventually outperforming supervised pre-training. These limitations include inefficiency, limited scalability, under-explored dependence in the data transformations and use of empirical tricks to avoid feature collapse.

SwAV. Subsequently, in Chapter 4, we have proposed a new and improved model for self-supervised learning, SwAV, that overcomes the limitations of deep clustering. As a matter of fact, SwAV also overcomes the problems posed by the popular contrastive implementations described in Section 2.3.2. SwAV is an implementation of the view-invariant feature learning paradigm (introduced in Section 2.3) that, unlike contrastive methods, does not rely on direct feature comparison. This makes it more practical as it does not need "negative pairs", large mini-batches or memory banks. Within our SwAV framework, we also introduce multi-crop training which encourages the emergence of holistic representations through a local-to-global matching. We show with extensive experiments that multi-crop allows important performance gains. At the time of its publication in [START_REF]1007 73.5 66[END_REF], SwAV is state-of-the-art on ImageNet among the self-supervised learning methods. Perhaps more importantly, we show that SwAV features transfer very well to several downstream tasks on different datasets, outperforming supervised features on all the considered transfer benchmarks. Finally, we have assessed SwAV in an uncontrolled setting by training on uncurated "in the wild" images. We have shown in this context the importance of scaling both dataset size and model capacity to make up for the lack of curation. Our final large-scale SwAV model, trained with a 1.3B parameters architecture on 1B random images from Instagram, achieves 84.2% top-1 accuracy on ImageNet. This confirms that self-supervised learning can work in a real world setting to some extent.

Self-supervised transformers. Finally, in Chapter 5, we have shown the potential of self-supervised pre-trained Vision Transformers, achieving performance that are comparable with the best convnets specifically designed for this setting while using much less parameters. We have also seen emerged two properties that can be leveraged in future applications: the quality of the features in k-NN classification has a potential for image retrieval, matching or similarity search. Second, the presence of information about the scene layout in the features can also benefit weakly supervised or fully unsupervised image segmentation. However, the main result of Chapter 5 is to show that we have evidence that self-supervised learning could be the key to developing a BERT-like model based on transformer architectures, which would allow to push the limits of visual features.

Open Problems in Self-Supervised Learning

The last decade has seen tremendous progress in self-supervised learning. At this point in time and at the end of this manuscript, one is entitled to ask if this is a solved field. Our (subjective) answer to that is a clear no because much remains to be done to tackle the open challenges raised by self-supervised learning. Also, we remark that most of the research so far has been conducted in very controlled settings, still far from real-world applications. In this final section, we give an overview of some of the open questions and remaining challenges in self-supervised learning.

Self-supervised versus supervised pre-training

First of all, in this manuscript (see Sections 4.3.2 and 5.5.3 for example), consistently with other works in the self-supervised community [He et al., 2020, Misra and[START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF], we have seen that self-supervised pre-training leads to better performance on downstream tasks compared to supervised methods. However, these results might not imply that we can replace supervised pre-training with self-supervised one just yet. This is because at the moment, the current self-supervised systems are much slower than the supervised ones to learn representations of equal quality. Self-supervised models require much more compute and epochs to reach convergence compared to supervised training on ImageNet. The performance improvements observed with self-supervised learning is probably not significant enough to justify the need for such an increase in the compute requirements. At this stage, if there are some labels available then it is likely better to use them. Hopefully, future improvements will make self-supervised pre-training more efficient and, as a result, more accessible to the community.

Actually, improving pre-training efficiency has been the focus of a collaboration conducted during the time of the PhD program but not presented in this manuscript (see [START_REF] Assran | Semisupervised learning of visual features by non-parametrically predicting view assignments with support samples[END_REF]). In this work, we propose a semi-supervised approach inspired by SwAV [START_REF]1007 73.5 66[END_REF], that achieves state-of-the-art performance on ImageNet with either 10% or 1% of the training instances labeled while using only 200 epochs of training, which is 4◊ less than previous works. These results suggest that, at least in the short term, the best way to go might not be purely self-supervised but to better leverage small amount of annotated data.

Towards real-world self-supervised learning

In this section, we mention several directions to make self-supervised learning work in more realistic scenario. Uncurated data. A major limitation of the current self-supervised systems is that they are typically trained in the highly controlled setting of using ImageNet without labels. However, as mentioned at length in this manuscript, this does not reflect a real-world situation of unsupervised learning since removing the labels only removes part of the human supervision. We have shown in Sections 3.3 and 4.5 some trainings on raw, "neverbefore-labeled" images but the conclusions of our study is arguably slightly underwhelming. Indeed, our results show that it is possible to learn good representations from uncurated data but they also exhibit a strong diminishing return with the dataset scale. It is possible that we would need to completely re-think our current self-supervised frameworks to make systems work better in uncontrolled settings with long tail distributions.

Actually, the current most successful methods are all declination of the same viewinvariant paradigm, which might have led recently to certain performance saturation in the benchmarks. Ways to move forward might be to seek progress in orthogonal directions (this is what we propose in Chapter 5 by considering self-supervised features with Vision Transformers) or to explore different and original self-supervised paradigms.

Unsupervised recognition tasks. The well-known recipe used in this manuscript to solve visual tasks is twofold: first a network is pre-trained without using any annotations before being transferred to a downstream task. While the first stage does not rely on annotations, the second one often still requires a lot of labels, especially when transferring to dense recognition tasks. Indeed, even if networks trained with this paradigm have shown good performance on dense-level recognition problems such as object detection, they need to be heavily adapted to these tasks, which is not practical in real applications. In contrast, we have seen in Chapter 5 that it is possible to obtain excellent performance for ImageNet classification task without any finetuning (see Section 5.4), which is akin to a zero-shot setting. This probably can be explained by the fact that methods developed in this manuscript work with holistic image representations, which are especially suited for global tasks like classification. This raises the question if we can obtain similar zero-shot results on dense tasks. Our preliminary results in Chapter 5 suggest that unsupervised dense segmentation, matching or localization might be possible, which can represent promising future directions.

Multi-modal learning. The models developed in this manuscript operate on unlabeled images only. However most real applications are multi-modal in the sense that they deal with data coming from different modalities. For example a video uploaded on a social media platform will usually come with one of several additional data stream like a soundtrack, a caption, some hashtags, geolocalization coordinates, emojis, etc. This motivates for developing better multi-modal systems capable of ingesting different media of data. Intuitively, these plural modalities should complement one another and could allow to learn richer and more generic representations.

Is self-supervised learning fully unsupervised?

Finally, we can question if the research in self-supervised learning is really fully unsupervised. In some aspects, we can argue that self-supervised learning still relies on some human labor and is not totally label-agnostic. For example, we still use a significant amount of labels during hyperparameter searches and model explorations to develop selfsupervised systems.

Handcrafted data transformations. In most modern and successful self-supervised approaches, the data augmentation scheme strongly drives the performance of the system. Indeed, the details of view generation are crucial and require a careful design. For example, Chen et al. [2020b] have conducted extensive ablations on the data augmentation pipeline which has boosted considerably the performance on ImageNet pre-training. These optimized pipelines are tuned on a certain kind of data and downstream tasks and are not likely to transfer to other domains (medical images for example). In addition, they are very handcrafted. We can criticize that the labor needed to annotate the data is being replaced by a different kind of labor which consists in manually tuning the data transformations to the considered task and application. It would be interesting to automate the search for an optimal data augmentation pipeline, in a similar manner as Auto-augment [START_REF] Cubuk | Autoaugment: Learning augmentation policies from data[END_REF] in supervised learning.

Model selection. In standard supervised learning, model selection is usually performed by looking at the loss value on a held-out validation set. In self-supervised learning though, the loss is not necessarily aligned with the final objective which is to learn representations useful for downstream tasks. Thus, the model is typically validated by looking at the transfer performance of the network to supervised downstream tasks. As a result, the current self-supervised methods still use some supervision in a sense, and might actually be over-fitted to the supervised task used for validation. A future and challenging research question could be to find ways to select self-supervised models without using supervision.

  Figure 3.3(a) shows the evolution of the NMI between the cluster assignments and the ImageNet labels during training.

  Figure3.7 shows the filters from the first layer of an AlexNet trained with DeepCluster on raw RGB images and images pre-processed with a Sobel filtering. As shown in the left panel of Figure3.7, most filters capture only color information that typically plays a little role for object classification[START_REF] Van De Sande | Evaluating color descriptors for object and scene recognition[END_REF]. Filters obtained with Sobel pre-processing act like edge detectors.

Figure 3 . 7 -

 37 Figure 3.7 -Visualizing first layer filters. Filters from the first layer of an AlexNet trained on unsupervised ImageNet on raw RGB input (left) or after a Sobel filtering (right).

Figure 3 . 8 -

 38 Figure 3.8 -Visualizing filters from deeper layers. Filter visualization and top 9 activated images (immediate right to the corresponding synthetic image) from a subset of 1 million images from YFCC100M for target filters in the last convolutional layer of a VGG-16 trained with DeepCluster.

Figure 3 . 13 -

 313 Figure 3.13 -Cluster visualizations. We randomly select 9 images per cluster and indicate the dominant cluster metadata. The bottom row depicts clusters pure for GPS coordinates but unpure for user IDs. As expected, they turn out to correlate with tourist landmarks. No metadata is used during training.

  Figure5.1 -Self-distillation with no labels. We illustrate DINO in the case of one single pair of views (x 1 , x 2 ) for simplicity. The model passes two different random transformations of an input image to the student and teacher networks. Both networks have the same architecture but different parameters. The output of the teacher network is centered with a mean computed over the batch. Each networks outputs a K dimensional feature that is normalized with a temperature softmax over the feature dimension. Their similarity is then measured with a cross-entropy loss. We apply a stop-gradient (sg) operator on the teacher to propagate gradients only through the student. The teacher parameters are updated with an exponential moving average (ema) of the student parameters.

Algorithm 1

 1 DINO PyTorch pseudo-code w/o multi-crop. ((t -C) / tpt, dim=1) # center + sharpen return -(t * log(s)).sum(dim=1).mean()

Table 3

 3

	Detection	Segmentation

.1 -Comparing DeepCluster with the state of the art. Comparison of Deep-Cluster to state-of-the-art (prior to 2018) self-supervised feature learning on classification, detection and segmentation on PASCAL VOC. Description of the concurrent self-supervised approaches are detailed in Chapter 2. ú indicates the use of the data-dependent initialization of Krähenbühl et al. [2015]. †: numbers for other methods produced by us.

  It measures the capability of the model to predict class level information. Note that we only use this measure for this analysis and not in any model selection process. The dependence between the clusters and the labels increases over time, showing that our features progressively capture information Number of reassignments between epochs. At each epoch, we reassign the images to a new set of clusters, with no guarantee of stability. Measuring the NMI between the clusters at epoch t≠1 and t gives an insight on the actual stability of our model.

		Clustering algorithm Classification	Detection	Segmentation
			FC6-8 ALL	FC6-8 ALL	FC6-8 ALL
	DeepCluster	k-means	72.0 73.7	51.4 55.4	43.2 45.1
	DeepCluster	PIC	71.0 73.0	53.6 54.4	42.4 43.8
	Table 3.2 -Impact of changing the clustering algorithm. Evaluation of PIC versus
	k-means for DeepCluster on PASCAL VOC transfer tasks.
	related to object classes.

  .3, we compare DeeperCluster with DeepCluster and RotNet when a linear classifier is trained on top of the last convolutional layer of a VGG-16 on several datasets. For reference, we also report previously Table3.3 -Comparison between DeeperCluster, RotNet and DeepCluster when pretrained on curated and non-curated dataset. We report the accuracy on several datasets of a linear classifier trained on top of features of the last convolutional layer. All the methods use the same architecture. DeepCluster does not scale to the full YFCC100M dataset, we thus train it on a random subset of 1.3M images.

	Method	Data	ImageNet Places VOC2007
	Supervised	ImageNet	70.24 5 .98 4 .8
	Wu et al. [2018]	ImageNet	39.23 6 .3	-
	RotNet [Gidaris et al., 2018] ImageNet	32.73 2 .66 0 .9
	DeepCluster	ImageNet	48.4	37.97 1 .9
	RotNet [Gidaris et al., 2018] YFCC100M	33.03 5 .56 2 .2
	DeepCluster	YFCC100M	34.13 5 .46 3 .9
	DeeperCluster	YFCC100M	45.6	42.1	73.0

  Impact of data augmentation (random flips and cropping). We report the performance of different models, measured as the proportion of intra-class edges into a nearest neighbor graph on the descriptors on ImageNet. DeepCluster models are trained for 50 epochs on ImageNet without labels with AlexNet. We observe that random cropping and scaling is crucial for the method to learn semantic representations.

			Cropping	Performance
	1 Default Horizontal	Random cropping and scaling	12.5%
	2	Vertical	Random cropping and scaling	11.1%
	3	7	Random cropping and scaling	12.4%
	4	Horizontal	Random cropping	3.5%
	5	Vertical	Random cropping	3.5%
	6	7	Random cropping	2.1%
	7	Horizontal	7	0.9%
	8	Vertical	7	0.9%
	9	77	0.6%
	Table 3.4 -

  Table 4.2 -Transfer learning on downstream tasks. Comparison between features from

		Linear Classification	Object Detection
		Places205 VOC07 iNat18	VOC07+12	COCO	COCO
					(Faster R-CNN R50-C4) (Mask R-CNN R50-FPN)	(DETR)
	Supervised	53.28 7 .54 6 .78 1 .33 9 .74 0 .8
	Self-supervised methods
	Gidaris et al. [2020a]	45.06 4 .6	--	-
	He et al. [2020]	46.9 †	79.8 † 31.5 †	81.5	-
	Misra and Maaten [2020] 49.88 1 .13 4 .18 0 .7	-
	Li et al. [2020]	49.88 4 .0	--	-
	Gidaris et al. [2020a]	51.17 9 .3	-	81.3	-
	Chen et al. [2020b]	53.3 †	86.4 † 36.2 †	--
	Chen et al. [2020e]	52.9 †	87.1 † 38.9 †	82.53 9 .84 2 .0 †
	SwAV	56.7	88.9 48.6	82.6	41.6	42.1

  Table 4.3, SwAV learns much faster and reaches higher performance in 4◊ fewer epochs: 72% after 200 epochs Table 4.3 -Training in small batch setting. Top-1 accuracy on ImageNet with a linear classifier trained on top of frozen features from a ResNet-50. All methods are trained with a batch size of 256.

	Method Mom. Encoder Stored Features	multi-crop	epoch batch Top-1
	SimCLR	02 ◊224	200	256	61.9
	MoCov2	65, 536	2◊224	200	256	67.5
	MoCov2	65, 536	2◊224	800	256	71.1
	SwAV	3, 840	2◊160 + 4◊96 200	256	72.0
	SwAV	3, 840	2◊224 + 6◊96 200	256	72.7
	SwAV	3, 840	2◊224 + 6◊96 400	256	74.3

  Computational cost. We report time and GPU memory requirements based on our implementation for different models trained during 100 epochs.

	Method	multi-crop	time / 100 epochs peak memory / GPU
	SimCLR	2 ◊ 224	4h00	8.6G
	SwAV	2 ◊ 224	4h09	8.6G
	SwAV	2 ◊ 160 + 4 ◊ 96	4h50	8.5G
	SwAV	2 ◊ 224 + 6 ◊ 96	6h15	12.8G
	Table 4.4 -
				This considerably disrupts

  to the standard RegNets to further Table 4.7 -Comparison with weakly-supervised pretraining on curated data.

	Pretraining	Arch.	#param Top-1
	Same architecture
	Scratch	RX101-32x8d	91M	79.6
	Hashtag pred. [Mahajan et al., 2018] RX101-32x8d	91M	82.6
	SwAV	RX101-32x8d	91M	81.6
	Different architectures
	Hashtag pred. [Mahajan et al., 2018] RX101-32x48d 831M	85.4
	SwAV	RG128	693M	83.8
	SwAV	RG256	1.3B	84.2

  83,w m =2.53, group width =373 It has 4 stages with stage depths (2, 7, 17, 1) and stage widths (528, 1056, 2904, 7392), leading to a total of 695.5M parameters. It takes 6125ms for a single training iteration over 8, 704 images on 512 V100 32GB NVIDIA GPUs. Training this model on 1 billion images requires 114, 890 training iterations for a batch size of 8, 704 images, summing to 8 days of training over 512 GPUs.

  .1) with • s > 0 a temperature parameter that controls the sharpness of the output distribution, and a similar formula holds for P t with temperature • t . Given a fixed teacher network g θt , we learn to match these distributions by minimizing the cross-entropy loss w.r.t. the parameters of the student network ◊ s : min

	θs	H(P t (x),P s (x)),	(5.2)

  For example on ImageNet 1%, ViT-S outperforms ResNet-50 by a large margin of +14.1% with k-NN evaluation. This suggests that transformers architectures trained with DINO might offer more model flexibility that benefits the k-NN evaluation. K-NN classifiers have the great advantage of being fast and light to deploy, without requiring any domain adaptation. Overall, ViT trained with DINO provides features that combine particularly well with k-NN classifiers.

			Logistic		k-NN
		RN50 ViT-S ∆	RN50 ViT-S ∆
	Inet 100%	72.1	75.7 3.6	67.5	74.5	7.0
	Inet 10%	67.8	72.2 4.4	59.3	69.1	9.8
	Inet 1%	55.1	64.5 9.4	47.2	61.3 14.1
	Pl. 10%	53.4	52.1 -1.3	46.9	48.6	1.7
	Pl. 1%	46.5	46.3 -0.2	39.2	41.3	2.1
	VOC07	88.9	89.2 0.3	84.9	88.0	3.1
	FLOWERS 95.6	96.4 0.8	87.9	89.1	1.2
	Average ∆		2.4			5.6

Table 5 .

 5 12 -Impact of teacher target sharpening. Sharpening plays a crucial role in preventing collapse. Hence, DINO is particularly sensitive to collapse at the beginning of training. Our experiments seem to suggest to seek for the maximum temperature that does not collapse.of training. Finally, note that • ae 0 (extreme sharpening) correspond to the argmax operation and leads to one-hot hard distributions.

	• t	00 .02 0.04 0.06 0.08 0.04 ae 0.07
	k-NN top-1 43.9 66.7 69.6 68.7 0.1	69.7
	m	0	0.9	0.99 0.999
	k-NN top-1 69.1 69.7 69.4 0.1

Table 5 .

 5 13 -Impact of online centering.

Table 5 .

 5 14 -Time and memory requirements. We show total running time and peak memory per GPU ("mem.") when running ViT-S/16 DINO models on two 8-GPU machines. We report top-1 ImageNet val acc with linear evaluation for several variants of multi-crop, each having a different level of compute requirement.

		100 epochs	300 epochs
	multi-crop	top-1 time	top-1 time mem.
	2◊224 2	67.8 15.3h	72.5 45.9h 9.3G
	2◊224 2 +2 ◊96 2	71.5 17.0h	74.5 51.0h 10.5G
	2◊224 2 +6 ◊96 2	73.8 20.3h	75.9 60.9h 12.9G
	2◊224 2 +10◊96 2	74.6 24.2h	76.1 72.6h 15.4G

  -free system. Unlike standard convnets, ViT architectures do not use batch normalizations (BN) by default. Therefore, when applying DINO to ViT we do not use any BN ViT-S, 100 epochs heads w/o BN heads w/ BN In this table we evaluate the impact of adding BN in the heads.

	k-NN top-1	69.7	68.6
	also in the projection heads.

  models trained with or without l2-normalization bottleneck and we vary the number of linear layers in the projection head. With l2 bottleneck, the total number of linear layers is n +1(n from the MLP and 1 from the weight normalized layer) while without bottleneck the total number of linear layers is n in the head. In this table, we report ImageNet top-1 k-NN evaluation accuracy after 100 epochs pre-training with ViT-S/16. The output dimensionality K is set to 4096 in this experiment. We observe that DINO training fails without the l2-normalization bottleneck when increasing the depth of the projection head. L2-normalization bottleneck stabilizes the training of DINO with deep projection head. We observe that increasing the depth of the projection head improves Projection head design w/ or w/o l2-norm bottleneck.accuracy. Our default is to use a total of 4 linear layers: 3 are in the MLP and one is after the l2 bottleneck.Output dimension. In this table, we evaluate the effect of varying the output dimensionality K. We observe that a large output dimensionality improves the performance. We note that the use of l2-normalization bottleneck permits to use a large output dimension with a moderate increase in the total number of parameters. Our default is to use K equals to 65536 and d =256for the bottleneck.GELU activations. By default, the activations used in ViT are Gaussian error linear units (GELU). Therefore, for consistency within the architecture, we choose to use GELU also ViT-S, 100 epochs heads w/ GELU heads w/ ReLU We evaluate the effect of using ReLU instead of GELU in this table and observe that changing the activation unit to ReLU has relatively little impact. It is likely that re-tuning the hyperparameters would allow to recover the 0.8% performance gap between our default and the heads with ReLU. Table 5.16 -Multi-crop applied to different self-supervised frameworks. We report top-1 accuracy on ImageNet for different methods using two different configurations of multi-crop: (i) 2 ◊ 224 2 refers to 2 global crops and no local crops, (ii) 2 ◊ 224 2 +6◊ 96 2 refers to 2 global crops and 6 local crops of resolution 96 by 96 pixels.

	# proj. head linear layers w/ l2-norm bottleneck w/o l2-norm bottleneck accuracy of DINO f n-layer MLP l2 normalization linear layer x B x 3 x 224 x 224 B x 384 B x 256 B x 256 B x K g(x) projection head h projection head h w/ l2-bottleneck Figure 5.9 -K 1024 4096 16384 65536 262144 123 -62.2 68.0 69.3 4 61.6 62.9 0.1 0.1 f n-layer MLP x B x 384 B x K g(x) B x 3 x 224 x 224 w/o l2-bottleneck k-NN top-1 67.8 69.3 69.2 69.7 69.1 k-NN top-1 69.7 68.9 2 ◊ 224 2 2 ◊ 224 2 +6◊ 96 2 eval k-NN linear k-NN linear BYOL 66.6 71.4 59.8 64.8 SwAV 60.5 68.5 64.7 71.8 MoCo-v2 62.0 71.6 65.4 73.4 in the projection head. crops DINO 67.9 72.5 72.7 75.9	We evaluate the

github.com/NVIDIA/apex

Acknowledgments

Table 5.1 -Networks configuration. "Blocks" is the number of Transformer blocks, "dim" is channel dimension and "heads" is the number of heads in multi-head attention. "# tokens" is the length of the token sequence when considering 224 2 resolution inputs, "# params" is the total number of parameters (without counting the projection head) and "im/s" is the inference time on a NVIDIA V100 GPU with 128 samples per forward.

show that a patch based Transformer is particularly suited for image classification. While these models originally required a lot of annotated data, [START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF] show that they also achieve competitive performance when trained on ImageNet alone by means of strong regularization and by guiding their training with a convnet through distillation.

Overall, we refer the reader to [START_REF] Vaswani | Attention is all you need[END_REF] for details about Transformers and to [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] for its adaptation to images. In this chapter, we follow the implementation used in DeiT [START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF]. We summarize the configuration of the different networks used in this chapter in Table 5.1. The ViT architecture takes as input a grid of non-overlapping contiguous image patches of resolution N ◊ N . In this paper we typically use N =1 6("/16") or N =8("/8"). The patches are then passed through a linear layer to form a set of embeddings. We add an extra learnable token to the sequence [START_REF] Devlin | Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF]. The role of this token is to aggregate information from the entire sequence. We refer to this token as the class token [CLS] for consistency with previous works [START_REF] Devlin | Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF], even though it is not attached to any label nor supervision in our case. The set of patch tokens and [CLS] token are fed to a standard Transformer network with a "pre-norm" layer normalization [START_REF] Chen | The best of both worlds: Combining recent advances in neural machine translation[END_REF][START_REF] Klein | Opennmt: Open-source toolkit for neural machine translation[END_REF]. The Transformer is a sequence of self-attention and feed-forward layers, paralleled with skip connections. The self-attention layers update the token representations by looking at the other token representations with an attention mechanism [Bahdanau et al., 2014].

Comparison across architectures SCLR [Chen et al., 2020b] RN50w4 375 117 76.8 69.3 SwAV [START_REF]1007 73.5 66[END_REF] RN50w2 93 384 77.3 67.3 BYOL [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF] [START_REF]1007 73.5 66[END_REF] RN50w5 586 76 78.5 67.1 BYOL [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF] RN50w4 375 117 78.6 -BYOL [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF]

Comparing with SSL Frameworks on ImageNet

We first validate the DINO framework used in this study with the standard selfsupervised benchmark on ImageNet. We consider two different settings: comparison with the same architecture and across architectures.

Comparing with the same architecture. In top panel of Table 5.2, we compare DINO with other self-supervised methods with the same architecture, either a ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] or a ViT-small (which follows the design of DeiT-S [START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF]). The choice of ViT-S is motivated by its similarity with ResNet-50 along several axes: number of parameters (21M vs 23M), throughput (1237/sec VS 1007 im/sec) and supervised performance on ImageNet with the training procedure of [START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF] (79.3% VS 79.8%). First, we observe that DINO performs on par with the state of the art on ResNet-50, validating that DINO works in the standard setting. When we switch to a ViT architecture, DINO outperforms BYOL, MoCov2 and SwAV by +3.5% with linear classification and by +7.9% with k-NN evaluation. More surprisingly, the performance with a simple k-NN classifier is almost on par with a linear classifier (74.5% versus 77.0%). This property emerges only when using DINO with ViT architectures, and does not appear with other existing self-supervised methods nor with a ResNet-50.

Comparing across architectures. On the bottom panel of Table 5.2, we compare the best performance obtained across architectures. The interest of this setting is not to compare methods directly, but to evaluate the limits of a ViT trained with DINO when moving to larger architectures. While training a larger ViT with DINO improves the performance, reducing the size of the patches ("/8" variants) has a bigger impact on the performance. While reducing the patch size do not add parameters, it still leads to a significant reduction of running time, and larger memory usage. Nonetheless, a base ViT with 8 ◊ 8 patches trained with DINO achieves 80.1% top-1 in linear classification and 77.4% with a k-NN classifier with 10◊ less parameters and 1.4◊ faster run time than previous state of the art [Chen et al., 2020c].

Properties of Self-Supervised ViT

We evaluate properties of the DINO features in terms of nearest neighbor search, retaining information about object location and transferability to downstream tasks. We report top-1 accuracy. Self-supervised pretraining with DINO transfers better than supervised pretraining.

Method

Transfer learning on downstream tasks

In Table 5.7, we evaluate the quality of the features pretrained with DINO on different downstream tasks. We compare with features from the same architectures trained with supervision on ImageNet. We follow the protocol used in [START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF] and finetune the features on each downstream task. We observe that for ViT architectures, self-supervised Pretraining method data res. tr. proc.

Top-1

Pretrain on additional data MMP JFT-300M 384 [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] 79.9 Supervised JFT-300M

384 [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] First, we observe that in the absence of momentum, our framework does not work (row 2) and more advanced operations, SK for example, are required to avoid collapse (row 9). However, with momentum, using SK has little impact (row 3). In addition, comparing rows 3 and 9 highlights the importance of the momentum encoder for performance. Second, in rows 4 and 5, we observe that multi-crop training and the cross-entropy loss in DINO are important components to obtain good features. We also observe that adding a predictor to the student network has little impact (row 6) while it is critical in BYOL to prevent collapse [Chen andHe, 2020, Grill et al., 2020].

Relation to SwAV. In Table 5.10, we further evaluate the differences between DINO and SwAV: the presence of the momentum encoder and the operation on top of the teacher output. In absence of the momentum, a copy of the student with a stop-gradient is used. We consider three operations on the teacher output: Centering, Sinkhorn-Knopp or a Softmax along the batch axis. The Softmax is similar to a single Sinkhorn-Knopp iteration as detailed in Section 4.2.2. First, these ablations show that using a momentum encoder significantly improves the performance for ViT (3 versus 6, and 2 versus 5). Second, the momentum encoder also avoids collapse when using only centering (row 1). 5.9 -Important component for self-supervised ViT pretraining. Models are trained for 300 epochs with ViT-S/16. We study the different components that matter for the k-NN and linear ("Lin.") evaluations. For the different variants, we highlight the differences from the default DINO setting. The best combination is the momentum encoder with the multi-crop augmentation and the cross-entropy loss. We also report results with BYOL [START_REF] Grill | Bootstrap your own latent: A new approach to self-supervised learning[END_REF], MoCo-v2 [Chen et al., 2020e] and SwAV [START_REF]1007 73.5 66[END_REF].

In the absence of momentum, centering the outputs does not work (4) and more advanced operations are required (5,6). Overall, these ablations highlight the importance of the momentum encoder, not only for performance but also to stabilize training, removing the need for normalization beyond centering.

Relation to MoCo-v2 and BYOL. In Table 5.11, we present in further details the impact of ablating components that differ between DINO, MoCo-v2 and BYOL: the choice of loss, the predictor in the student head, the centering operation, the batch normalization in the projection heads, and finally, the multi-crop augmentation. The loss in DINO is a cross-entropy on sharpened softmax outputs (CE) while MoCo-v2 uses the InfoNCE contrastive loss (INCE) and BYOL a mean squared error on l2-normalized outputs (MSE). No sharpening is applied with the MSE criterion. Though, DINO surprisingly still works when changing the loss function to MSE, but this significantly alters the performance (see rows (1, 2) and (4, 9)). We also observe that adding a predictor has little impact (1, 3). However, in the case of BYOL, the predictor is critical to prevent collapse (7, 8) which is consistent with previous studies [Chen andHe, 2020, Grill et al., 2020]. Interestingly, we observe that the teacher output centering avoids collapse without predictor nor batch epochs. We observe that the performance greatly improves as we decrease the size of the patch. It is interesting to see that performance can be greatly improved without adding additional parameters. However, the performance gain from using smaller patches comes at the expense of throughput: when using 5◊5 patches, the throughput falls to 44 im/s, vs 180 im/s for 8◊8 patches.

Impact of the choice of teacher network

In this ablation, we experiment with different teacher network to understand its role in DINO. We compare models trained for 300 epochs using the k-NN protocol.

Building different teachers from the student. In Figure 5.7 (right), we compare different strategies to build the teacher from previous instances of the student besides the momentum teacher. First we consider using the student network from a previous epoch as a teacher. This strategy has been used in a memory bank [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF] or as a form of clustering hard-distillation [START_REF] Asano | Self-labelling via simultaneous clustering and representation learning[END_REF], Caron et al., 2018, Chen et al., 2020d]. Second, we consider using the student network from the previous iteration, as well as a copy of the student for the teacher. In our setting, using a teacher based on a recent version of the