Jean-Bastien Grill

Florian Strub

Florent Altché
email: altche@google.com

Corentin Tallec
email: corentint@google.com

Pierre H Richemond
email: richemond]@google.com

Elena Buchatskaya

Carl Doersch

Bernardo Avila Pires

Zhaohan Daniel Guo

Mohammad Gheshlaghi Azar

Bilal Piot

Koray Kavukcuoglu

Rémi Munos

Michal Valko

Deepmind

Bootstrap Your Own Latent A New Approach to Self-Supervised Learning

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches 74.3% top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and 79.6% with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

Figure 1: Performance of BYOL on ImageNet (linear evaluation) using ResNet-50 and our best architecture ResNet-200 (2×), compared to other unsupervised and supervised (Sup.) baselines [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF].

Learning good image representations is a key challenge in computer vision [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[END_REF][START_REF] Wiskott | Slow feature analysis: Unsupervised learning of invariances[END_REF][START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF] as it allows for efficient training on downstream tasks [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF][START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF][START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]. Many different training approaches have been proposed to learn such representations, usually relying on visual pretext tasks. Among them, state-of-the-art contrastive methods [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF][START_REF] Tian | Contrastive multiview coding[END_REF][START_REF] Tian | What makes for good views for contrastive learning[END_REF] are trained by reducing the distance between representations of different augmented views of the same image ('positive pairs'), and increasing the distance between representations of augmented views from different images ('negative pairs'). These methods need careful treatment of negative pairs [START_REF] Saunshi | A theoretical analysis of contrastive unsupervised representation learning[END_REF] by either relying on large batch sizes [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Tian | What makes for good views for contrastive learning[END_REF], memory banks [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] or customized mining strategies [START_REF] Manmatha | Sampling matters in deep embedding learning[END_REF][START_REF] Harwood | Smart mining for deep metric learning[END_REF] to retrieve the negative pairs. In addition, their performance critically depends on the choice of image augmentations [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Tian | What makes for good views for contrastive learning[END_REF].

In this paper, we introduce Bootstrap Your Own Latent (BYOL), a new algorithm for self-supervised learning of image representations. BYOL achieves higher performance than state-of-the-art contrastive methods without using negative pairs. It iteratively bootstraps 4 the outputs of a network to serve as targets for an enhanced representation. Moreover, BYOL is more robust to the choice of image augmentations than contrastive methods; we suspect that not relying on negative pairs is one of the leading reasons for its improved robustness. While previous methods based on bootstrapping have used pseudo-labels [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF], cluster indices [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF] or a handful of labels [START_REF] Bachman | Learning with pseudo-ensembles[END_REF][START_REF] Laine | Temporal ensembling for semi-supervised learning[END_REF][START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF], we propose to directly bootstrap the representations. In particular, BYOL uses two neural networks, referred to as online and target networks, that interact and learn from each other. Starting from an augmented view of an image, BYOL trains its online network to predict the target network's representation of another augmented view of the same image. While this objective admits collapsed solutions, e.g., outputting the same vector for all images, we empirically show that BYOL does not converge to such solutions. We hypothesize (see Section 3.2) that the combination of (i) the addition of a predictor to the online network and (ii) the use of a slow-moving average of the online parameters as the target network encourages encoding more and more information within the online projection and avoids collapsed solutions.

We evaluate the representation learned by BYOL on ImageNet [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF] and other vision benchmarks using ResNet architectures [START_REF] He | Deep residual learning for image recognition[END_REF]. Under the linear evaluation protocol on ImageNet, consisting in training a linear classifier on top of the frozen representation, BYOL reaches 74.3% top-1 accuracy with a standard ResNet-50 and 79.6% top-1 accuracy with a larger ResNet (Figure 1). In the semi-supervised and transfer settings on ImageNet, we obtain results on par or superior to the current state of the art. Our contributions are: (i) We introduce BYOL, a self-supervised representation learning method (Section 3) which achieves state-of-the-art results under the linear evaluation protocol on ImageNet without using negative pairs. (ii) We show that our learned representation outperforms the state of the art on semi-supervised and transfer benchmarks (Section 4). (iii) We show that BYOL is more resilient to changes in the batch size and in the set of image augmentations compared to its contrastive counterparts (Section 5).

In particular, BYOL suffers a much smaller performance drop than SimCLR, a strong contrastive baseline, when only using random crops as image augmentations.

Related work

Most unsupervised methods for representation learning can be categorized as either generative or discriminative [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]. Generative approaches to representation learning build a distribution over data and latent embedding and use the learned embeddings as image representations. Many of these approaches rely either on auto-encoding of images [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF][START_REF] Diederik | Auto-encoding variational bayes[END_REF][START_REF] Jimenez Rezende | Stochastic back-propagation and variational inference in deep latent gaussian models[END_REF] or on adversarial learning [START_REF] Goodfellow | Generative adversarial nets[END_REF], jointly modelling data and representation [START_REF] Donahue | Adversarial feature learning[END_REF][START_REF] Vincent Dumoulin | Adversarially learned inference[END_REF][START_REF] Donahue | Large scale adversarial representation learning[END_REF][START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]. Generative methods typically operate directly in pixel space. This however is computationally expensive, and the high level of detail required for image generation may not be necessary for representation learning.

Among discriminative methods, contrastive methods [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF][START_REF] Olivier | Data-efficient image recognition with contrastive predictive coding[END_REF][START_REF] Hjelm | Learning deep representations by mutual information estimation and maximization[END_REF][START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF][START_REF] Tian | Contrastive multiview coding[END_REF][START_REF] Misra | Self-supervised learning of pretext-invariant representations[END_REF][START_REF] Li | Prototypical contrastive learning of unsupervised representations[END_REF] currently achieve state-of-the-art performance in self-supervised learning [START_REF] Jain | Improved baselines with momentum contrastive learning[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Chen | Big self-supervised models are strong semi-supervised learners[END_REF][START_REF] Tian | What makes for good views for contrastive learning[END_REF]. Contrastive approaches avoid a costly generation step in pixel space by bringing representation of different views of the same image closer ('positive pairs'), and spreading representations of views from different images ('negative pairs') apart [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF][START_REF] Doersch | Multi-task self-supervised visual learning[END_REF]. Contrastive methods often require comparing each example with many other examples to work well [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] prompting the question of whether using negative pairs is necessary.

DeepCluster [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF] partially answers this question. It uses bootstrapping on previous versions of its representation to produce targets for the next representation; it clusters data points using the prior representation, and uses the cluster index of each sample as a classification target for the new representation. While avoiding the use of negative pairs, this requires a costly clustering phase and specific precautions to avoid collapsing to trivial solutions. Some self-supervised methods are not contrastive but rely on using auxiliary handcrafted prediction tasks to learn their representation. In particular, relative patch prediction [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF][START_REF] Doersch | Multi-task self-supervised visual learning[END_REF], colorizing gray-scale images [START_REF] Zhang | Colorful image colorization[END_REF][START_REF] Larsson | Learning representations for automatic colorization[END_REF], image inpainting [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF], image jigsaw puzzle [START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF], image super-resolution [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF], and geometric transformations [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF][START_REF] Gidaris | Unsupervised representation learning by predicting image rotations[END_REF] have been shown to be useful. Yet, even with suitable architectures [START_REF] Kolesnikov | Revisiting self-supervised visual representation learning[END_REF], these methods are being outperformed by contrastive methods [START_REF] Jain | Improved baselines with momentum contrastive learning[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Tian | What makes for good views for contrastive learning[END_REF].

Our approach has some similarities with Predictions of Bootstrapped Latents (PBL, [START_REF] Guo | Bootstrap latent-predictive representations for multitask reinforcement learning[END_REF]), a self-supervised representation learning technique for reinforcement learning (RL). PBL jointly trains the agent's history representation and an encoding of future observations. The observation encoding is used as a target to train the agent's representation, and the agent's representation as a target to train the observation encoding. Unlike PBL, BYOL uses a slow-moving average of its representation to provide its targets, and does not require a second network.

The idea of using a slow-moving average target network to produce stable targets for the online network was inspired by deep RL [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF][START_REF] Hessel | Rainbow: Combining improvements in deep reinforcement learning[END_REF][START_REF] Van Hasselt | Deep reinforcement learning and the deadly triad[END_REF]. Target networks stabilize the bootstrapping updates provided by the Bellman equation, making them appealing to stabilize the bootstrap mechanism in BYOL. While most RL methods use fixed target networks, BYOL uses a weighted moving average of previous networks (as in [START_REF] Timothy P Lillicrap | Continuous control with deep reinforcement learning[END_REF]) in order to provide smoother changes in the target representation.

In the semi-supervised setting [START_REF] Chapelle | Semi-supervised learning[END_REF][START_REF] Zhu | Introduction to semi-supervised learning[END_REF], an unsupervised loss is combined with a classification loss over a handful of labels to ground the training [START_REF] Laine | Temporal ensembling for semi-supervised learning[END_REF][START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF][START_REF] Durk P Kingma | Semi-supervised learning with deep generative models[END_REF][START_REF] Rasmus | Semi-supervised learning with ladder networks[END_REF][START_REF] Berthelot | Mixmatch: A holistic approach to semi-supervised learning[END_REF][START_REF] Miyato | Virtual adversarial training: a regularization method for supervised and semi-supervised learning[END_REF][START_REF] Berthelot | Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring[END_REF][START_REF] Sohn | Fixmatch: Simplifying semi-supervised learning with consistency and confidence[END_REF]. Among these methods, mean teacher (MT) [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF] also uses a slow-moving average network, called teacher, to produce targets for an online network, called student. An 2 consistency loss between the softmax predictions of the teacher and the student is added to the classification loss. While [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF] demonstrates the effectiveness of MT in the semi-supervised learning case, in Section 5 we show that a similar approach collapses when removing the classification loss. In contrast, BYOL introduces an additional predictor on top of the online network, which prevents collapse.

Finally, in self-supervised learning, MoCo [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] uses a slow-moving average network (momentum encoder) to maintain consistent representations of negative pairs drawn from a memory bank. Instead, BYOL uses a moving average network to produce prediction targets as a means of stabilizing the bootstrap step. We show in Section 5 that this mere stabilizing effect can also improve existing contrastive methods.

Method

We start by motivating our method before explaining its details in Section 3.1. Many successful self-supervised learning approaches build upon the cross-view prediction framework introduced in [START_REF] Becker | Self-organizing neural network that discovers surfaces in random-dot stereograms[END_REF]. Typically, these approaches learn representations by predicting different views (e.g., different random crops) of the same image from one another. Many such approaches cast the prediction problem directly in representation space: the representation of an augmented view of an image should be predictive of the representation of another augmented view of the same image. However, predicting directly in representation space can lead to collapsed representations: for instance, a representation that is constant across views is always fully predictive of itself. Contrastive methods circumvent this problem by reformulating the prediction problem into one of discrimination: from the representation of an augmented view, they learn to discriminate between the representation of another augmented view of the same image, and the representations of augmented views of different images. In the vast majority of cases, this prevents the training from finding collapsed representations. Yet, this discriminative approach typically requires comparing each representation of an augmented view with many negative examples, to find ones sufficiently close to make the discrimination task challenging. In this work, we thus tasked ourselves to find out whether these negative examples are indispensable to prevent collapsing while preserving high performance.

To prevent collapse, a straightforward solution is to use a fixed randomly initialized network to produce the targets for our predictions. While avoiding collapse, it empirically does not result in very good representations. Nonetheless, it is interesting to note that the representation obtained using this procedure can already be much better than the initial fixed representation. In our ablation study (Section 5), we apply this procedure by predicting a fixed randomly initialized network and achieve 18.8% top-1 accuracy (Table 5a) on the linear evaluation protocol on ImageNet, whereas the randomly initialized network only achieves 1.4% by itself. This experimental finding is the core motivation for BYOL: from a given representation, referred to as target, we can train a new, potentially enhanced representation, referred to as online, by predicting the target representation. From there, we can expect to build a sequence of representations of increasing quality by iterating this procedure, using subsequent online networks as new target networks for further training. In practice, BYOL generalizes this bootstrapping procedure by iteratively refining its representation, but using a slowly moving exponential average of the online network as the target network instead of fixed checkpoints.

Description of BYOL

BYOL's goal is to learn a representation y θ which can then be used for downstream tasks. As described previously, BYOL uses two neural networks to learn: the online and target networks. The online network is defined by a set of weights θ and is comprised of three stages: an encoder f θ , a projector g θ and a predictor q θ , as shown in Figure 2 and Figure 8. The target network has the same architecture as the online network, but uses a different set of weights ξ. The target network provides the regression targets to train the online network, and its parameters ξ are an exponential moving average of the online parameters θ [START_REF] Timothy P Lillicrap | Continuous control with deep reinforcement learning[END_REF]. More precisely, given a target decay rate τ ∈ [0, 1], after each training step we perform the following update,

ξ ← τ ξ + (1 -τ )θ.
(1)

x v y θ z θ q θ (z θ ) v ′ y ′ ξ z ′ ξ sg(z ′ ξ ) view input image representation projection prediction t f θ g θ q θ t ′ f ξ g ξ sg loss online target Figure 2: BYOL's architecture.
BYOL minimizes a similarity loss between q θ (z θ ) and sg(z ξ ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but f θ is discarded, and y θ is used as the image representation.

Given a set of images D, an image x ∼ D sampled uniformly from D, and two distributions of image augmentations T and T , BYOL produces two augmented views v = ∆ t(x) and v = ∆ t (x) from x by applying respectively image augmentations t ∼ T and t ∼ T . From the first augmented view v, the online network outputs a representation y θ = ∆ f θ (v) and a projection z θ = ∆ g θ (y). The target network outputs y ξ = ∆ f ξ (v ) and the target projection z ξ = ∆ g ξ (y ) from the second augmented view v . We then output a prediction q θ (z θ ) of z ξ and 2 -normalize both q θ (z θ ) and z ξ to q θ (z θ ) = ∆ q θ (z θ )/ q θ (z θ ) 2 and z ξ = ∆ z ξ / z ξ 2 . Note that this predictor is only applied to the online branch, making the architecture asymmetric between the online and target pipeline. Finally we define the following mean squared error between the normalized predictions and target projections,5

L θ,ξ = ∆ q θ (z θ ) -z ξ 2 2 = 2 -2 • q θ (z θ ), z ξ q θ (z θ ) 2 • z ξ 2 • (2)
We symmetrize the loss L θ,ξ in Eq. 2 by separately feeding v to the online network and v to the target network to compute L θ,ξ . At each training step, we perform a stochastic optimization step to minimize L BYOL θ,ξ = L θ,ξ + L θ,ξ with respect to θ only, but not ξ, as depicted by the stop-gradient in Figure 2. BYOL's dynamics are summarized as

θ ← optimizer θ, ∇ θ L BYOL θ,ξ , η , (3)
ξ ← τ ξ + (1 -τ )θ, (1)
where optimizer is an optimizer and η is a learning rate.

At the end of training, we only keep the encoder f θ ; as in [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF]. When comparing to other methods, we consider the number of inference-time weights only in the final representation f θ . The full training procedure is summarized in Appendix A, and python pseudo-code based on the libraries JAX [START_REF] Bradbury | JAX: composable transformations of Python+NumPy programs[END_REF] and Haiku [START_REF] Hennigan | Haiku: Sonnet for JAX[END_REF] is provided in in Appendix J.

Intuitions on BYOL's behavior

As BYOL does not use an explicit term to prevent collapse (such as negative examples [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF]) while minimizing L BYOL θ,ξ with respect to θ, it may seem that BYOL should converge to a minimum of this loss with respect to (θ, ξ) (e.g., a collapsed constant representation). However BYOL's target parameters ξ updates are not in the direction of ∇ ξ L BYOL θ,ξ . More generally, we hypothesize that there is no loss L θ,ξ such that BYOL's dynamics is a gradient descent on L jointly over θ, ξ. This is similar to GANs [START_REF] Goodfellow | Generative adversarial nets[END_REF], where there is no loss that is jointly minimized w.r.t. both the discriminator and generator parameters. There is therefore no a priori reason why BYOL's parameters would converge to a minimum of L BYOL θ,ξ . While BYOL's dynamics still admit undesirable equilibria, we did not observe convergence to such equilibria in our experiments. In addition, when assuming BYOL's predictor to be optimal6 i.e., q θ = q with q = ∆ arg min

q E q(z θ ) -z ξ 2 2 , where q (z θ ) = E z ξ |z θ , (4)
we hypothesize that the undesirable equilibria are unstable. Indeed, in this optimal predictor case, BYOL's updates on θ follow in expectation the gradient of the expected conditional variance (see Appendix H for details),

∇ θ E q (z θ ) -z ξ 2 2 = ∇ θ E E z ξ |z θ -z ξ 2 2 = ∇ θ E i Var(z ξ,i |z θ ) , (5)
where z ξ,i is the i-th feature of z ξ .

Note that for any random variables X, Y, and Z, Var(X|Y, Z) ≤ Var(X|Y ). Let X be the target projection, Y the current online projection, and Z an additional variability on top of the online projection induced by stochasticities in the training dynamics: purely discarding information from the online projection cannot decrease the conditional variance.

In particular, BYOL avoids constant features in z θ as, for any constant c and random variables z θ and z ξ , Var(z ξ |z θ ) ≤ Var(z ξ |c); hence our hypothesis on these collapsed constant equilibria being unstable. Interestingly, if we were to minimize E[ i Var(z ξ,i |z θ )] with respect to ξ, we would get a collapsed z ξ as the variance is minimized for a constant z ξ . Instead, BYOL makes ξ closer to θ, incorporating sources of variability captured by the online projection into the target projection.

Furthemore, notice that performing a hard-copy of the online parameters θ into the target parameters ξ would be enough to propagate new sources of variability. However, sudden changes in the target network might break the assumption of an optimal predictor, in which case BYOL's loss is not guaranteed to be close to the conditional variance. We hypothesize that the main role of BYOL's moving-averaged target network is to ensure the near-optimality of the predictor over training; Section 5 and Appendix I provide some empirical support of this interpretation.

Implementation details

Image augmentations BYOL uses the same set of image augmentations as in SimCLR [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]. First, a random patch of the image is selected and resized to 224 × 224 with a random horizontal flip, followed by a color distortion, consisting of a random sequence of brightness, contrast, saturation, hue adjustments, and an optional grayscale conversion. Finally Gaussian blur and solarization are applied to the patches. Additional details on the image augmentations are in Appendix B.

Architecture We use a convolutional residual network [START_REF] He | Deep residual learning for image recognition[END_REF] with 50 layers and post-activation (ResNet-50(1×) v1) as our base parametric encoders f θ and f ξ . We also use deeper (50, 101, 152 and 200 layers) and wider (from 1× to 4×) ResNets, as in [START_REF] Zagoruyko | Wide residual networks[END_REF][START_REF] Kolesnikov | Revisiting self-supervised visual representation learning[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]. Specifically, the representation y corresponds to the output of the final average pooling layer, which has a feature dimension of 2048 (for a width multiplier of 1×). As in SimCLR [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], the representation y is projected to a smaller space by a multi-layer perceptron (MLP) g θ , and similarly for the target projection g ξ . This MLP consists in a linear layer with output size 4096 followed by batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], rectified linear units (ReLU) [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF], and a final linear layer with output dimension 256. Contrary to SimCLR, the output of this MLP is not batch normalized. The predictor q θ uses the same architecture as g θ .

Optimization We use the LARS optimizer [START_REF] You | Scaling SGD batch size to 32k for imagenet training[END_REF] with a cosine decay learning rate schedule [START_REF] Loshchilov | SGDR: stochastic gradient descent with warm restarts[END_REF], without restarts, over 1000 epochs, with a warm-up period of 10 epochs. We set the base learning rate to 0.2, scaled linearly [START_REF] Goyal | Accurate, large minibatch sgd: Training imagenet in 1 hour[END_REF] with the batch size (LearningRate = 0.2 × BatchSize/256). In addition, we use a global weight decay parameter of 1.5

Experimental evaluation

We assess the performance of BYOL's representation after self-supervised pretraining on the training set of the ImageNet ILSVRC-2012 dataset [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF]. We first evaluate it on ImageNet (IN) in both linear evaluation and semisupervised setups. We then measure its transfer capabilities on other datasets and tasks, including classification, segmentation, object detection and depth estimation. For comparison, we also report scores for a representation trained using labels from the train ImageNet subset, referred to as Supervised-IN. In Appendix E, we assess the generality of BYOL by pretraining a representation on the Places365-Standard dataset [START_REF] Zhou | Places: A 10 million image database for scene recognition[END_REF] before reproducing this evaluation protocol.

Linear evaluation on ImageNet

We first evaluate BYOL's representation by training a linear classifier on top of the frozen representation, following the procedure described in [START_REF] Kolesnikov | Revisiting self-supervised visual representation learning[END_REF][START_REF] Kornblith | Do better ImageNet models transfer better?[END_REF][START_REF] Zhang | Colorful image colorization[END_REF][START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], and appendix C.1; we report top-1 and top-5 accuracies in % on the test set in Table 1. With a standard ResNet-50 (×1) BYOL obtains 74.3% top-1 accuracy (91.6% top-5 accuracy), which is a 1.3% (resp. 0.5%) improvement over the previous self-supervised state of the art [START_REF] Tian | What makes for good views for contrastive learning[END_REF]. This tightens the gap with respect to the supervised baseline of [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], 76.5%, but is still significantly below the stronger supervised baseline of [START_REF] Gong | Maxup: A simple way to improve generalization of neural network training[END_REF], 78.9%. With deeper and wider architectures, BYOL consistently outperforms the previous state of the art (Appendix C.2), and obtains a best performance of 79.6% top-1 accuracy, ranking higher than previous self-supervised approaches. On a ResNet-50 (4×) BYOL achieves 78.6%, similar to the 78.9% of the best supervised baseline in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] for the same architecture. Semi-supervised training on ImageNet Next, we evaluate the performance obtained when fine-tuning BYOL's representation on a classification task with a small subset of ImageNet's train set, this time using label information. We follow the semi-supervised protocol of [START_REF] Kornblith | Do better ImageNet models transfer better?[END_REF][START_REF] Zhai | A large-scale study of representation learning with the visual task adaptation benchmark[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Olivier | Data-efficient image recognition with contrastive predictive coding[END_REF] detailed in Appendix C.1, and use the same fixed splits of respectively 1% and 10% of ImageNet labeled training data as in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]. We report both top-1 and top-5 accuracies on the test set in Transfer to other classification tasks We evaluate our representation on other classification datasets to assess whether the features learned on ImageNet (IN) are generic and thus useful across image domains, or if they are ImageNet-specific. We perform linear evaluation and fine-tuning on the same set of classification tasks used in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Kornblith | Do better ImageNet models transfer better?[END_REF], and carefully follow their evaluation protocol, as detailed in Appendix D. Performance is reported using standard metrics for each benchmark, and results are provided on a held-out test set after hyperparameter selection on a validation set. We report results in Table 3, both for linear evaluation and fine-tuning. BYOL outperforms SimCLR on all benchmarks and the Supervised-IN baseline on 7 of the 12 benchmarks, providing only slightly worse performance on the 5 remaining benchmarks. BYOL's representation can be transferred over to small images, e.g., CIFAR [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], landscapes, e.g., SUN397 [START_REF] Xiao | Sun database: Large-scale scene recognition from abbey to zoo[END_REF] or VOC2007 [START_REF] Everingham | The Pascal visual object classes (VOC) challenge[END_REF], and textures, e.g., DTD [START_REF] Cimpoi | Describing textures in the wild[END_REF]. Transfer to other vision tasks We evaluate our representation on different tasks relevant to computer vision practitioners, namely semantic segmentation, object detection and depth estimation. With this evaluation, we assess whether BYOL's representation generalizes beyond classification tasks.

We first evaluate BYOL on the VOC2012 semantic segmentation task as detailed in Appendix D. [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF], where the goal is to classify each pixel in the image [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]. We report the results in Table 4a. BYOL outperforms both the Supervised-IN baseline (+1.9 mIoU) and SimCLR (+1.1 mIoU).

Similarly, we evaluate on object detection by reproducing the setup in [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] using a Faster R-CNN architecture [START_REF] Shaoqing Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF], as detailed in Appendix D.5. We fine-tune on trainval2007 and report results on test2007 using the standard AP 50 metric; BYOL is significantly better than the Supervised-IN baseline (+3.1 AP 50 ) and SimCLR (+2.3 AP 50 ).

Finally, we evaluate on depth estimation on the NYU v2 dataset, where the depth map of a scene is estimated given a single RGB image. Depth prediction measures how well a network represents geometry, and how well that information can be localized to pixel accuracy [START_REF] Doersch | Multi-task self-supervised visual learning[END_REF]. The setup is based on [START_REF] Laina | Deeper depth prediction with fully convolutional residual networks[END_REF] and detailed in Appendix D.6. We evaluate on the commonly used test subset of 654 images and report results using several common metrics in Table 4b: relative (rel) error, root mean squared (rms) error, and the percent of pixels (pct) where the error, max(d gt /d p , d p /d gt ), is below 1.25 n thresholds where d p is the predicted depth and d gt is the ground truth depth [START_REF] Doersch | Multi-task self-supervised visual learning[END_REF]. BYOL is better or on par with other methods for each metric. For instance, the challenging pct.<1.25 measure is respectively improved by +3.5 points and +1.

Building intuitions with ablations

We present ablations on BYOL to give an intuition of its behavior and performance. For reproducibility, we run each configuration of parameters over three seeds, and report the average performance. We also report the half difference between the best and worst runs when it is larger than 0.25. Although previous works perform ablations at 100 epochs [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Tian | What makes for good views for contrastive learning[END_REF], we notice that relative improvements at 100 epochs do not always hold over longer training. For this reason, we run ablations over 300 epochs on 64 TPU v3 cores, which yields consistent results compared to our baseline training of 1000 epochs. For all the experiments in this section, we set the initial learning rate to 0.3 with batch size 4096, the weight decay to 10 -6 as in SimCLR [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] and the base target decay rate τ base to 0.99. In this section we report results in top-1 accuracy on ImageNet under the linear evaluation protocol as in Appendix C.1.

Batch size Among contrastive methods, the ones that draw negative examples from the minibatch suffer performance drops when their batch size is reduced. BYOL does not use negative examples and we expect it to be more robust to smaller batch sizes. To empirically verify this hypothesis, we train both BYOL and SimCLR using different batch sizes from 128 to 4096. To avoid re-tuning other hyperparameters, we average gradients over N consecutive steps before updating the online network when reducing the batch size by a factor N . The target network is updated once every N steps, after the update of the online network; we accumulate the N -steps in parallel in our runs.

As shown in Figure 3a, the performance of SimCLR rapidly deteriorates with batch size, likely due to the decrease in the number of negative examples. In contrast, the performance of BYOL remains stable over a wide range of batch sizes from 256 to 4096, and only drops for smaller values due to batch normalization layers in the encoder. Image augmentations Contrastive methods are sensitive to the choice of image augmentations. For instance, SimCLR does not work well when removing color distortion from its image augmentations. As an explanation, SimCLR shows that crops of the same image mostly share their color histograms. At the same time, color histograms vary across images. Therefore, when a contrastive task only relies on random crops as image augmentations, it can be mostly solved by focusing on color histograms alone. As a result the representation is not incentivized to retain information beyond color histograms. To prevent that, SimCLR adds color distortion to its set of image augmentations. Instead, BYOL is incentivized to keep any information captured by the target representation into its online network, to improve its predictions. Therefore, even if augmented views of a same image share the same color histogram, BYOL is still incentivized to retain additional features in its representation. For that reason, we believe that BYOL is more robust to the choice of image augmentations than contrastive methods.

Results presented in Figure 3b support this hypothesis: the performance of BYOL is much less affected than the performance of SimCLR when removing color distortions from the set of image augmentations (-9.1 accuracy points for BYOL, -22.2 accuracy points for SimCLR). When image augmentations are reduced to mere random crops, BYOL still displays good performance (59.4%, i.e. -13.1 points from 72.5% ), while SimCLR loses more than a third of its performance (40.3%, i.e. -27.6 points from 67.9%). We report additional ablations in Appendix F.3.

Bootstrapping BYOL uses the projected representation of a target network, whose weights are an exponential moving average of the weights of the online network, as target for its predictions. This way, the weights of the target network represent a delayed and more stable version of the weights of the online network. When the target decay rate is 1, the target network is never updated, and remains at a constant value corresponding to its initialization. When the target decay rate is 0, the target network is instantaneously updated to the online network at each step. There is a trade-off between updating the targets too often and updating them too slowly, as illustrated in Table 5a. Instantaneously updating the target network (τ = 0) destabilizes training, yielding very poor performance while never updating the target (τ = 1) makes the training stable but prevents iterative improvement, ending with low-quality final representation. All values of the decay rate between 0.9 and 0.999 yield performance above 68.4% top-1 accuracy at 300 epochs. Ablation to contrastive methods In this subsection, we recast SimCLR and BYOL using the same formalism to better understand where the improvement of BYOL over SimCLR comes from. Let us consider the following objective that extends the InfoNCE objective [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF][START_REF] Poole | On variational bounds of mutual information[END_REF] (see Appendix F.4),

InfoNCE α,β θ = ∆ 2 B B i=1 S θ (v i , v i )-β • 2α B B i=1 ln   j =i exp S θ (v i , v j ) α + j exp S θ (v i , v j ) α  , (6)
where α > 0 is a fixed temperature, β ∈ [0, 1] a weighting coefficient, B the batch size, v and v are batches of augmented views where for any batch index i, v i and v i are augmented views from the same image; the realvalued function S θ quantifies pairwise similarity between augmented views. For any augmented view u we denote z θ (u) f θ (g θ (u)) and z ξ (u) f ξ (g ξ (u)). For given φ and ψ, we consider the normalized dot product

S θ (u 1 , u 2 ) = ∆ φ(u 1 ), ψ(u 2 ) φ(u 1 ) 2 • ψ(u 2 ) 2 • (7)
Up to minor details (cf. Appendix F.5), we recover the SimCLR loss with φ(u 1 ) = z θ (u 1 ) (no predictor), ψ(u 2 ) = z θ (u 2 ) (no target network) and β = 1. We recover the BYOL loss when using a predictor and a target network, i.e., φ(u 1 ) = p θ (z θ (u 1 )) and ψ(u 2 ) = z ξ (u 2 ) with β = 0. To evaluate the influence of the target network, the predictor and the coefficient β, we perform an ablation over them. Results are presented in Table 5b and more details are given in Appendix F.4.

The only variant that performs well without negative examples (i.e., with β = 0) is BYOL, using both a bootstrap target network and a predictor. Adding the negative pairs to BYOL's loss without re-tuning the temperature parameter hurts its performance. In Appendix F.4, we show that we can add back negative pairs and still match the performance of BYOL with proper tuning of the temperature.

Simply adding a target network to SimCLR already improves performance (+1.6 points). This sheds new light on the use of the target network in MoCo [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], where the target network is used to provide more negative examples. Here, we show that by mere stabilization effect, even when using the same number of negative examples, using a target network is beneficial. Finally, we observe that modifying the architecture of S θ to include a predictor only mildly affects the performance of SimCLR.

Network hyperparameters In Appendix F, we explore how other network parameters may impact BYOL's performance. We iterate over multiple weight decays, learning rates, and projector/encoder architectures to observe that small hyperparameter changes do not drastically alter the final score. We note that removing the weight decay in either BYOL or SimCLR leads to network divergence, emphasizing the need for weight regularization in the self-supervised setting. Furthermore, we observe that changing the scaling factor in the network initialization [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF] did not impact the performance (higher than 72% top-1 accuracy).

Relationship with Mean Teacher Another semi-supervised approach, Mean Teacher (MT) [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF], complements a supervised loss on few labels with an additional consistency loss. In [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF], this consistency loss is the 2 distance between the logits from a student network, and those of a temporally averaged version of the student network, called teacher. Removing the predictor in BYOL results in an unsupervised version of MT with no classification loss that uses image augmentations instead of the original architectural noise (e.g., dropout). This variant of BYOL collapses (Row 7 of Table 5) which suggests that the additional predictor is critical to prevent collapse in an unsupervised scenario.

Importance of a near-optimal predictor Table 5b already shows the importance of combining a predictor and a target network: the representation does collapse when either is removed. We further found that we can remove the target network without collapse by making the predictor near-optimal, either by (i) using an optimal linear predictor (obtained by linear regression on the current batch) before back-propagating the error through the network (52.5% top-1 accuracy), or (ii) increasing the learning rate of the predictor (66.5% top-1). By contrast, increasing the learning rates of both projector and predictor (without target network) yields poor results (≈ 25% top-1). See Appendix I for more details. This seems to indicate that keeping the predictor near-optimal at all times is important to preventing collapse, which may be one of the roles of BYOL's target network.

Conclusion

We introduced BYOL, a new algorithm for self-supervised learning of image representations. BYOL learns its representation by predicting previous versions of its outputs, without using negative pairs. We show that BYOL achieves state-of-the-art results on various benchmarks. In particular, under the linear evaluation protocol on ImageNet with a ResNet-50 (1×), BYOL achieves a new state of the art and bridges most of the remaining gap between self-supervised methods and the supervised learning baseline of [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]. Using a ResNet-200 (2×), BYOL reaches a top-1 accuracy of 79.6% which improves over the previous state of the art (76.8%) while using 30% fewer parameters.

Nevertheless, BYOL remains dependent on existing sets of augmentations that are specific to vision applications.

To generalize BYOL to other modalities (e.g., audio, video, text, . . . ) it is necessary to obtain similarly suitable augmentations for each of them. Designing such augmentations may require significant effort and expertise. Therefore, automating the search for these augmentations would be an important next step to generalize BYOL to other modalities.

Broader impact

The presented research should be categorized as research in the field of unsupervised learning. This work may inspire new algorithms, theoretical, and experimental investigation. The algorithm presented here can be used for many different vision applications and a particular use may have both positive or negative impacts, which is known as the dual use problem. Besides, as vision datasets could be biased, the representation learned by BYOL could be susceptible to replicate these biases.

A Algorithm

Algorithm 1: BYOL: Bootstrap Your Own Latent Inputs : D, T , and T set of images and distributions of transformations θ, f θ , g θ , and q θ initial online parameters, encoder, projector, and predictor ξ, f ξ , g ξ initial target parameters, target encoder, and target projector optimizer optimizer, updates online parameters using the loss gradient K and N total number of optimization steps and batch size {τ k } K k=1 and {η k } K k=1 target network update schedule and learning rate schedule

for k = 1 to K do B ← {x i ∼ D} N i=1 // sample a batch of N images for x i ∈ B do t ∼ T and t ∼ T // sample image transformations z 1 ← g θ (f θ (t(x i ))) and z 2 ← g θ (f θ (t (x i ))) // compute projections z 1 ← g ξ (f ξ (t (x i ))) and z 2 ← g ξ (f ξ (t(x i )))
// compute target projections

l i ← -2 • q θ (z1),z 1 q θ (z1) 2 • z 1 2 + q θ (z2),z 2 q θ (z2) 2 • z 2 2 // compute the loss for x i end δθ ← 1 N N i=1 ∂ θ l i // compute

B Image augmentations

During self-supervised training, BYOL uses the following image augmentations (which are a subset of the ones presented in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]):

• random cropping: a random patch of the image is selected, with an area uniformly sampled between 8% and 100% of that of the original image, and an aspect ratio logarithmically sampled between 3/4 and 4/3. This patch is then resized to the target size of 224 × 224 using bicubic interpolation;

• optional left-right flip;

• color jittering: the brightness, contrast, saturation and hue of the image are shifted by a uniformly random offset applied on all the pixels of the same image. The order in which these shifts are performed is randomly selected for each patch;

• color dropping: an optional conversion to grayscale. When applied, output intensity for a pixel (r, g, b) corresponds to its luma component, computed as 0.2989r + 0.5870g + 0.1140b; Augmentations from the sets T and T (introduced in Section 3) are compositions of the above image augmentations in the listed order, each applied with a predetermined probability. The image augmentations parameters are listed in Table 6.

•
During evaluation, we use a center crop similar to [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]: images are resized to 256 pixels along the shorter side using bicubic resampling, after which a 224 × 224 center crop is applied. In both training and evaluation, we normalize color channels by subtracting the average color and dividing by the standard deviation, computed on ImageNet, after applying the augmentations. In both cases, we normalize the color channels by subtracting the average color and dividing by the standard deviation (computed on ImageNet), after applying the augmentations. We optimize the cross-entropy loss using SGD with Nesterov momentum over 80 epochs, using a batch size of 1024 and a momentum of 0.9. We do not use any regularization methods such as weight decay, gradient clipping [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF], tclip [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF], or logits regularization. We finally sweep over 5 learning rates {0.4, 0.3, 0.2, 0.1, 0.05} on a local validation set (10009 images from ImageNet train set), and report the accuracy of the best validation hyperparameter on the test set (which is the public validation set of the original ILSVRC2012 ImageNet dataset).

Variant on linear evaluation on ImageNet

In this paragraph only, we deviate from the protocol of [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Jain | Improved baselines with momentum contrastive learning[END_REF] and propose another way of performing linear evaluation on top of a frozen representation. This method achieves better performance both in top-1 and top-5 accuracy.

• We replace the spatial augmentations (random crops with resize to 224 × 224 pixels and random flips) with the pre-train augmentations of Appendix B. This method was already used in [START_REF] Olivier | Data-efficient image recognition with contrastive predictive coding[END_REF] with a different subset of pre-train augmentations.

• We regularize the linear classifier as in [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF] 8 by clipping the logits using a hyperbolic tangent function

tclip(x) α • tanh(x/α),
where α is a positive scalar, and by adding a logit-regularization penalty term in the loss Loss(x, y) cross_entropy(tclip(x), y)

+ β • average(tclip(x) 2 ),
where x are the logits, y are the target labels, and β is the regularization parameter. We set α = 20 and β = 1e-2.

We report in Table 7 the top-1 and top-5 accuracy on ImageNet using this modified protocol. These modifications in the evaluation protocol increase the BYOL's top-1 accuracy from 74.3% to 74.8% with a ResNet-50 (1×).

Semi-supervised learning on ImageNet We follow the semi-supervised learning protocol of [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Zhai | S4L: Self-supervised semi-supervised learning[END_REF]. We first initialize the network with the parameters of the pretrained representation, and fine-tune it with a subset of ImageNet labels. At training time, we apply spatial augmentations, i.e., random crops with resize to 224 × 224 pixels and random flips. At test time, images are resized to 256 pixels along the shorter side using bicubic resampling, after which a 224 × 224 center crop is applied. In both cases, we normalize the color channels by subtracting the average color and dividing by the standard deviation (computed on ImageNet), after applying the augmentations. We optimize the cross-entropy loss using SGD with Nesterov momentum. We used a batch size of 1024, a momentum of No pre-train augmentations and no logits regularization correspond to the evaluation protocol of the main paper, which is the same as in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Jain | Improved baselines with momentum contrastive learning[END_REF].

0.9. We do not use any regularization methods such as weight decay, gradient clipping [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF], tclip [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF], or logits rescaling. We sweep over the learning rate {0.01, 0.02, 0.05, 0.1, 0.005} and the number of epochs {30, 50} and select the hyperparameters achieving the best performance on our local validation set to report test performance. In Table 2 presented in the main text, we fine-tune the representation over the 1% and 10% ImageNet splits from [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] with various ResNet architectures.

In Figure 4, we fine-tune the representation over 1%, 2%, 5%, 10%, 20%, 50%, and 100% of the ImageNet dataset as in [START_REF] Olivier | Data-efficient image recognition with contrastive predictive coding[END_REF] with a ResNet-50 (1×) architecture, and compare them with a supervised baseline and a fine-tuned SimCLR representation. In this case and contrary to Table 2 we don't reuse the splits from SimCLR but we create our own via a balanced selection. In this setting, we observed that tuning a BYOL representation always outperforms a supervised baseline trained from scratch. In Figure 5, we then fine-tune the representation over multiple ResNet architectures. We observe that the largest networks are prone to overfitting as they are outperformed by ResNets with identical depth but smaller scaling factor. This overfitting is further confirmed when looking at the training and evaluation loss: large networks have lower training losses, but higher validation losses than some of their slimmer counterparts. Regularization methods are thus recommended when tuning on large architectures. We also report other fully supervised methods for extensive comparisons.

Finally, we fine-tune the representation over the full ImageNet dataset. We report the results in Table 8 along with supervised baselines trained on ImageNet. We observe that fine-tuning the SimCLR checkpoint does not yield better results (in our reproduction, which matches the results reported in the original paper [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]) than using a random initialization (76.5 top-1). Instead, BYOL's initialization checkpoint leads to a high final score (77.7 top-1), higher than the vanilla supervised baseline of [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], matching the strong supervised baseline of AutoAugment [START_REF] Ekin D Cubuk | Randaugment: Practical automated data augmentation with a reduced search space[END_REF] but still 1.2 points below the stronger supervised baseline [START_REF] Gong | Maxup: A simple way to improve generalization of neural network training[END_REF], which uses advanced supervised learning techniques.

C.2 Linear evaluation on larger architectures and supervised baselines

Here we investigate the performance of BYOL with deeper and wider ResNet architectures. We compare ourselves to the best supervised baselines from [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] when available (rightmost column in table 9), which are also presented in Figure 1. Importantly, we close in on those baselines using the ResNet-50 (2×) and the ResNet-50 (4×) architectures, where we are within 0.4 accuracy points of the supervised performance. To the best of our knowledge, this is the first time that the gap to supervised has been closed to such an extent using a self-supervised method under the linear evaluation protocol. Therefore, in order to ensure fair comparison, and suspecting that the supervised baselines' performance in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] could be even further improved with appropriate data augmentations, we also report on our own reproduction of strong supervised baselines. We use RandAugment [START_REF] Ekin D Cubuk | Randaugment: Practical automated data augmentation with a reduced search space[END_REF] data augmentation for all large ResNet architectures (which are all version 1, as per [START_REF] He | Deep residual learning for image recognition[END_REF]). We train our supervised baselines for up to 200 epochs, using SGD with a Nesterov momentum value of 0.9, a cosine-annealed learning rate after a 5 epochs linear warmup period, weight decay with a value of 1e -4, and a label smoothing [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF] We perform transfer via linear classification and fine-tuning on the same set of datasets as in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], namely Food-101 dataset [START_REF] Bossard | Food-101 -mining discriminative components with random forests[END_REF], CIFAR-10 [78] and CIFAR-100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], Birdsnap [START_REF] Berg | Birdsnap: Large-scale fine-grained visual categorization of birds[END_REF], the SUN397 scene dataset [START_REF] Xiao | Sun database: Large-scale scene recognition from abbey to zoo[END_REF], Stanford Cars [START_REF] Krause | 3D object representations for fine-grained categorization[END_REF], FGVC Aircraft [START_REF] Maji | Fine-grained visual classification of aircraft[END_REF], the PASCAL VOC 2007 classification task [START_REF] Everingham | The Pascal visual object classes (VOC) challenge[END_REF], the Describable Textures Dataset (DTD) [START_REF] Cimpoi | Describing textures in the wild[END_REF], Oxford-IIIT Pets [START_REF] Parkhi | Cats and dogs[END_REF], Caltech-101 [START_REF] Fei-Fei | Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories[END_REF], and Oxford 102 Flowers [START_REF] Nilsback | Automated flower classification over a large number of classes[END_REF]. As in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], we used the validation sets specified by the dataset creators to select hyperparameters for FGVC Aircraft, PASCAL VOC 2007, DTD, and Oxford 102 Flowers. On other datasets, we use the validation examples as test set, and hold out a subset of the training examples that we use as validation set. We use standard metrics for each datasets:

• Top-1: We compute the proportion of correctly classified examples.

• Mean per class: We compute the top-1 accuracy for each class separately and then compute the empirical mean over the classes.

• Point 11-mAP: We compute the empirical mean average precision as defined in [START_REF] Everingham | The Pascal visual object classes (VOC) challenge[END_REF].

• Mean IoU: We compute the empirical mean Intersection-Over-Union as defined in [START_REF] Everingham | The Pascal visual object classes (VOC) challenge[END_REF].

• AP50: We compute the Average Precision as defined in [START_REF] Everingham | The Pascal visual object classes (VOC) challenge[END_REF].

We detail the validation procedures for some specific datasets:

• For Sun397 [START_REF] Xiao | Sun database: Large-scale scene recognition from abbey to zoo[END_REF], the original dataset specifies 10 train/test splits, all of which contain 50 examples/images of 397 different classes. We use the first train/test split. The original dataset specifies no validation split and therefore, the training images have been further subdivided into 40 images per class for the train split and 10 images per class for the valid split.

• For Birdsnap [START_REF] Berg | Birdsnap: Large-scale fine-grained visual categorization of birds[END_REF], we use a random selection of valid images with the same number of images per category as the test split.

• For DTD [START_REF] Cimpoi | Describing textures in the wild[END_REF], the original dataset specifies 10 train/validation/test splits, we only use the first split.

• For Caltech-101 [START_REF] Fei-Fei | Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories[END_REF], the original does not dataset specifies any train/test splits. We have followed the approach used in [START_REF] Donahue | Decaf: A deep convolutional activation feature for generic visual recognition[END_REF]: This file defines datasets for 5 random splits of 25 training images per category, with 5 validation images per category and the remaining images used for testing.

• For ImageNet, we took the last 10009 last images of the official tensorflow ImageNet split.

• For Oxford-IIIT Pets, the valid set consists of 20 randomly selected images per class.

Information about the dataset are summarized in Table 10.

D.2 Transfer via linear classification

We follow the linear evaluation protocol of [START_REF] Kolesnikov | Revisiting self-supervised visual representation learning[END_REF][START_REF] Kornblith | Do better ImageNet models transfer better?[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] that we detail next for completeness. We train a regularized multinomial logistic regression classifier on top of the frozen representation, i.e., with frozen pretrained parameters and without re-computing batch-normalization statistics. In training and testing, we do not perform any image augmentations; images are resized to 224 pixels along the shorter side using bicubic resampling and then normalized with ImageNet statistics. Finally, we minimize the cross-entropy objective using LBFGS with 2 -regularization, where we select the regularization parameters from a range of 45 logarithmically-spaced values between 10 -6 and 10 5 . After choosing the best-performing hyperparameters on the validation set, the model is retrained on combined training and validation images together, using the chosen parameters. The final accuracy is reported on the test set.

D.3 Transfer via fine-tuning

We follow the same fine-tuning protocol as in [START_REF] Olivier | Data-efficient image recognition with contrastive predictive coding[END_REF][START_REF] Kolesnikov | Revisiting self-supervised visual representation learning[END_REF][START_REF] Zhai | A large-scale study of representation learning with the visual task adaptation benchmark[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] that we also detail for completeness. Specifically, we initialize the network with the parameters of the pretrained representation. At training time, we apply spatial transformation, i.e., random crops with resize to 224 × 224 pixels and random flips. At test time, images are resized to 256 pixels along the shorter side using bicubic resampling, after which a 224 × 224 center crop is extracted. In both cases, we normalize the color channels by subtracting the average color and dividing by the standard deviation (computed on ImageNet), after applying the augmentations. We optimize the loss using SGD with Nesterov momentum for 20000 steps with a batch size of 256 and with a momentum of 0.9. We set the momentum parameter for the batch normalization statistics to max(1 -10/s, 0.9) where s is the number of steps per epoch. The learning rate and weight decay are selected respectively with a grid of seven logarithmically spaced learning rates between 0.0001 and 0.1, and 7 logarithmically-spaced values of weight decay between 10 -6 and 10 -3 , as well as no weight decay. These values of weight decay are divided by the learning rate. After choosing the best-performing hyperparameters on the validation set, the model is retrained on combined training and validation images together, using the chosen parameters. The final accuracy is reported on the test set.

D.4 Implementation details for semantic segmentation

We use the same fully-convolutional network (FCN)-based [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] architecture as [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF]. The backbone consists of the convolutional layers in ResNet-50. The 3 × 3 convolutions in the conv5 blocks use dilation 2 and stride 1. This is followed by two extra 3 × 3 convolutions with 256 channels, each followed by batch normalization and ReLU activations, and a 1 × 1 convolution for per-pixel classification. The dilation is set to 6 in the two extra 3 × 3 convolutions. The total stride is 16 (FCN-16s [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]).

We train on the train_aug2012 set and report results on val2012. Hyperparameters are selected on a 2119 images held-out validation set. We use a standard per-pixel softmax cross-entropy loss to train the FCN. Training is done with random scaling (by a ratio in [0.5, 2.0]), cropping, and horizontal flipping. The crop size is 513. Inference is performed on the [513, 513] central crop. For training we use a batch size of 16 and weight decay of 0.0001.

We select the base learning rate by sweeping across 5 logarithmically spaced values between 10 -3 and 10 -1 . The learning rate is multiplied by 0.1 at the 70-th and 90-th percentile of training. We train for 30000 iterations, and average the results on 5 seeds.

D.5 Implementation details for object detection

For object detection, we follow prior work on Pascal detection transfer [START_REF] Doersch | Multi-task self-supervised visual learning[END_REF][START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF] wherever possible. We use a Faster R-CNN [START_REF] Shaoqing Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] detector with a R50-C4 backbone with a frozen representation. The R50-C4 backbone ends with the conv4 stage of a ResNet-50, and the box prediction head consists of the conv5 stage (including global pooling). We preprocess the images by applying multi-scale augmentation (rescaling the image so its longest edge is between 480 and 1024 pixels) but no other augmentation. We use an asynchronous SGD optimizer with 9 workers and train for 1.5M steps. We used an initial learning rate of 10 -3 , which is reduced to 10 -4 at 1M steps and to 10 -5 at 1.2M steps.

D.6 Implementation details for depth estimation

For depth estimation, we follow the same protocol as in [START_REF] Laina | Deeper depth prediction with fully convolutional residual networks[END_REF], and report its core components for completeness. We use a standard ResNet-50 backbone and feed the conv5 features into 4 fast up-projection blocks with respective filter sizes 512, 256, 128, and 64. We use a reverse Huber loss function for training [START_REF] Laina | Deeper depth prediction with fully convolutional residual networks[END_REF][START_REF] Owen | A robust hybrid of lasso and ridge regression[END_REF].

The original NYU Depth v2 frames of size [640, 480] are down-sampled by a factor 0.5 and center-cropped to [304, 228] pixels. Input images are randomly horizontally flipped and the following color transformations are applied:

• Grayscale with an application probability of 0.3.

• Brightness with a maximum brightness difference of 0.1255.

• Saturation with a saturation factor randomly picked in the interval [0.5, 1.5].

• Hue with a hue adjustment factor randomly picked in the interval [-0.2, 0.2].

We train for 7500 steps with batch size 256, weight decay 0.0005, and learning rate 0.16 (scaled linearly from the setup of [START_REF] Laina | Deeper depth prediction with fully convolutional residual networks[END_REF] to account for the bigger batch size).

D.7 Further comparisons on PASCAL and NYU v2 Depth

For completeness, Table 11

E Pretraining on Places 365

To ascertain that BYOL learns good representations on other datasets, we applied our representation learning protocol on the scene recognition dataset Places365-Standard [START_REF] Zhou | Places: A 10 million image database for scene recognition[END_REF] before performing linear evaluation. This dataset contains 1.80 million training images and 36500 validation images with labels, making it roughly similar to ImageNet in scale. We reuse the exact same parameters as in Section 4 and train the representation for 1000 epochs, using BYOL and our SimCLR reproduction. Results for the linear evaluation setup (using the protocol of Appendix C.1 for ImageNet and Places365, and that of Appendix D on other datasets) are reported in Table 13.

Interestingly, the representation trained by using BYOL on Places365 (BYOL-PL) consistently outperforms that of SimCLR on the same dataset, but underperforms the BYOL representation trained on ImageNet (BYOL-IN) on all tasks except Places365 and SUN397 [START_REF] Xiao | Sun database: Large-scale scene recognition from abbey to zoo[END_REF], another scene understanding dataset. Interestingly, all three unsupervised representation learning methods achieve a relatively high performance on the Places365 task; for comparison, reference [START_REF] Zhou | Places: A 10 million image database for scene recognition[END_REF] (in its linked repository) reports a top-1 accuracy of 55.2% for a ResNet-50v2 trained from scratch using labels on this dataset.

F Additional ablation results

To extend on the above results, we provide additional ablations obtained using the same experimental setup as in Section 5, i.e., 300 epochs, averaged over 3 seeds with the initial learning rate set to 0.3, the batch size to 4096, the weight decay to 10 -6 and the base target decay rate τ base to 0.99 unless specified otherwise. Confidence intervals correspond to the half-difference between the maximum and minimum score of these seeds; we omit them for half-differences lower than 0.25 accuracy points.

F.1 Architecture settings

Table 14 shows the influence of projector and predictor architecture on BYOL. We examine the effect of different depths for both the projector and predictor, as well as the effect of the projection size. We do not apply a ReLU activation nor a batch normalization on the final linear layer of our MLPs such that a depth of 1 corresponds to a linear layer. Using the default projector and predictor of depth 2 yields the best performance. Table 15a shows the influence of the initial learning rate on BYOL. Note that the optimal value depends on the number of training epochs. Table 15b displays the influence of the weight decay on BYOL.

F.2 Batch size

We run a sweep over the batch size for both BYOL and our reproduction of SimCLR. As explained in Section 5, when reducing the batch size by a factor N , we average gradients over N consecutive steps and update the target network once every N steps. We report in Table 16, the performance of both our reproduction of SimCLR and BYOL for batch sizes between 4096 (BYOL and SimCLR default) down to 64. We observe that the performance of SimCLR deteriorates faster than the one of BYOL which stays mostly constant for batch sizes larger than 256. We believe that the performance at batch size 256 could match the performance of the large 4096 batch size with proper parameter tuning when accumulating the gradient. We think that the drop in performance at batch size 64 in table [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF] is mainly related to the ill behaviour of batch normalization at low batch sizes [START_REF] Wu | Group normalization[END_REF]. Table 18: Top-1 accuracy in % under linear evaluation protocol at 300 epochs of sweep over the temperature α and the dispersion term weight β when using a predictor and a target network.

F.4 Details on the relation to contrastive methods

As mentioned in Section 5, the BYOL loss Eq. 2 can be derived from the InfoNCE loss

InfoNCE α,β θ = ∆ 2 B B i=1 S θ (v i , v i )- 2α • β B B i=1 ln   j =i exp S θ (v i , v j ) α + j exp S θ (v i , v j ) α  , (8)
with

S θ (u 1 , u 2 ) = ∆ φ(u 1 ), ψ(u 2 ) φ(u 1 ) 2 • ψ(u 2 ) 2 • (9)
The InfoNCE loss, introduced in [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF], can be found in factored form in [START_REF] Poole | On variational bounds of mutual information[END_REF] as

InfoNCE θ = ∆ 1 B B i=1 ln f (v i , v i ) 1 B j exp f (v i , v j ) • (10)
As in SimCLR [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] we also use negative examples given by (v i , v j ) j =i to get

1 B B i=1 ln exp f (v i , v i ) 1 B j =i exp f (v i , v j ) + 1 B j exp f (v i , v j ) (11)
= ln B + 1 B B i=1 f (v i , v i ) - 1 B B i=1 ln   j =i exp f (v i , v j ) + j exp f (v i , v j )   . (12)
To obtain Eq. 8 from Eq. 12, we subtract ln B (which is independent of θ), multiply by 2α, take f (x, y) = S θ (x, y)/α and finally multiply the second (negative examples) term by β. Using β = 1 and dividing by 2α gets us back to the usual InfoNCE loss as used by SimCLR. Using a target network in the loss has two effects: stopping the gradient through the prediction targets and stabilizing the targets with averaging. Stopping the gradient through the target change the objective while averaging makes the target stable and stale. In Table 5b we only shows results of the ablation when either using the online network as the prediction target (and flowing the gradient through it) or with a target network (both stopping the gradient into the prediction targets and computing the prediction targets with a moving average of the online network). We shown in Table 5b that using a target network is beneficial but it has two distinct effects we would like to understand from which effect the improvement comes from. We report in Table 19 the results already in Table 5b but also when the prediction target is computed with a stop gradient of the online network (the gradient does not flow into the prediction targets). This shows that making the prediction targets stable and stale is the main cause of the improvement rather than the change in the objective due to the stop gradient.

In our ablation in

F.5 SimCLR baseline of Section 5

The SimCLR baseline in Section 5 (β = 1, without predictor nor target network) is slightly different from the original one in [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF]. First we multiply the original loss by 2α. For comparaison here is the original SimCLR loss,

InfoNCE θ = ∆ 1 B B i=1 S θ (v i , v i ) α - 1 B B i=1 ln   j =i exp S θ (v i , v j ) α + j exp S θ (v i , v j ) α   • (13)
Note that this multiplication by 2α matters as the LARS optimizer is not completely invariant with respect to the scale of the loss. Indeed, LARS applies a preconditioning to gradient updates on all weights, except for biases and batch normalization parameters. Updates on preconditioned weights are invariant by multiplicative scaling of the loss. However, the bias and batch normalization parameter updates remain sensitive to multiplicative scaling of the loss.

We also increase the original SimCLR hidden and output size of the projector to respectively 4096 and 256. In our reproduction of SimCLR, these three combined changes improves the top-1 accuracy at 300 epochs from 67.9% (without the changes) to 69.2% (with the changes).

F.6 Ablation on the normalization in the loss function BYOL minimizes a squared error between the 2 -normalized prediction and target. We report results of BYOL at 300 epochs using different normalization function and no normalization at all. More precisely, given batch of prediction and targets in R d , (p i , t i ) i≤B with B the batch size, BYOL uses the loss function

n j BNi : x → x j i -µ j BN (x) σ j BN (x) • √ d , n j LNi : x → x j i -µ LNi (x) σ LNi (x) • √ d , n ID : x → x, µ j BN : x → 1 B B i=1 x j i , σ j BN : x → 1 B B i=1 x j i 2 -µ j BN (x) 2 , µ LNi : x → 1 d d j=1 x j i , σ LNi : x → x i -µ LNi (x) 2 √ d
When using no normalization at all, the projection 2 norm rapidly increases during the first 100 epochs and stabilizes at around 3 • 10 6 as shown in Figure 7. Despite this behaviour, using no normalization still performs reasonably well (67.4%). The 2 normalization performs the best.

G Training with smaller batch sizes

The results described in Section 4 were obtained using a batch size of 4096 split over 512 TPU cores. Due to its increased robustness, BYOL can also be trained using smaller batch sizes without significantly decreasing performance. Using the same linear evaluation setup, BYOL achieves 73.7% top-1 accuracy when trained over 1000 epochs with a batch size of 512 split over 64 TPU cores (approximately 4 days of training). For this setup, we reuse the same setting as in Section 3, but use a base learning rate of 0.4 (appropriately scaled by the batch size) and τ base = 0.9995 with the same weight decay coefficient of 1.5 • 10 -6 .

H Details on Equation 5 in Section 3.2

In this section we clarify why BYOL's update is related to Eq. 5 from Section 3.2,

∇ θ E q (z θ ) -z ξ 2 2 = ∇ θ E E z ξ |z θ -z ξ 2 2 = ∇ θ E i Var(z ξ,i |z θ ) . (5)
Recall that q is defined as q = ∆ arg min q E q(z θ )z ξ 2 2 , where q (z θ ) = E z ξ |z θ ,

and implicitly depends on θ and ξ; therefore, it should be denoted as q (θ, ξ) instead of just q . For simplicity we write q (θ, ξ)(z θ ) as q (θ, ξ, z θ ) the output of the optimal predictor for any parameters θ and ξ and input z θ .

BYOL updates its online parameters following the gradient of Eq. 5, but considering only the gradients of q with respect to its third argument z when applying the chain rule. If we rewrite E q (θ, ξ, z θ )z ξ 2 2 = E L(q (θ, ξ, z θ ), z ξ ) ,

the gradient of this quantity w.r.t. θ is

∂ ∂θ E L(q (θ, ξ, z θ ), z ξ ) = E ∂L ∂q • ∂q ∂θ + ∂L ∂q • ∂q ∂z • ∂z θ ∂θ , (15)
where ∂q ∂θ and ∂q ∂z are the gradients of q with respect to its first and last argument. Using the envelope theorem, and thanks to the optimality condition of the predictor, the term E ∂L ∂q • ∂q ∂θ = 0. Therefore, the remaining term

E ∂L ∂q • ∂q ∂z • ∂z θ ∂θ
where gradients are only back-propagated through the predictor's input is exactly the direction followed by BYOL.

I Importance of a near-optimal predictor

In this part we build upon the intuitions of Section 3.2 on the importance of keeping the predictor near-optimal. Specifically, we show that it is possible to remove the exponential moving average in BYOL's target network (i.e., simply copy weights of the online network into the target) without causing the representation to collapse, provided the predictor remains sufficiently good.

I.1 Predictor learning rate

In this setup, we remove the exponential moving average (i.e., set τ = 0 over the full training in Eq. 1), and multiply the learning rate of the predictor by a constant λ compared to the learning rate used for the rest of the network; all other hyperparameters are unchanged. As shown in Table 21, using sufficiently large values of λ provides a reasonably good level of performance and the performance sharply decreases with λ to 0.01% top-1 accuracy (no better than random) for λ = 0.

To show that this effect is directly related to a change of behavior in the predictor, and not only to a change of learning rate in any subpart of the network, we perform a similar experiment by using a multiplier λ on the predictor's learning rate, and a different multiplier µ for the projector. In Table 22, we show that the representation typically collapses or performs poorly when the predictor learning rate is lower or equal to that of the projector. As mentioned in Section 3.2, we further hypothesize that one of the contributions of the target network is to maintain a near optimal predictor at all times.

I.2 Optimal linear predictor in closed form

Similarly, we can get rid of the slowly moving target network if we use a closed form optimal predictor on the batch, instead of a learned, non-optimal one. In this case we restrict ourselves to a linear predictor,

q = arg min Q Z θ Q -Z ξ 2 2 = (Z T θ Z θ ) -1 Z T θ Z ξ
with • 2 being the Frobenius norm, Z θ and Z ξ of shape (B, F ) respectively the online and target projections, where B is the batch size and F the number of features; and q of shape (F, F ), the optimal linear predictor for the batch.

At 300 epochs, when using the closed form optimal predictor, and directly hard copying the weights of the online network into the target, we obtain a top-1 accuracy of f ill. Table 22: Top-1 accuracy at 300 epochs when removing the slowly moving target network, directly hard copying the weights of the online network in the target network, and applying a multiplier µ to the projector and λ to the predictor learning rate. The predictor learning rate needs to be higher than the projector learning rate in order to successfully remove the target network. This further suggests that the learning dynamic of predictor is central to BYOL's stability.

7

 7

0

  Decrease of accuracy from baselineBYOL SimCLR (repro) (b) Impact of progressively removing transformations

Figure 3 :

 3 Figure 3: Decrease in top-1 accuracy (in % points) of BYOL and our own reproduction of SimCLR at 300 epochs, under linear evaluation on ImageNet.

  Gaussian blurring: for a 224 × 224 image, a square Gaussian kernel of size 23 × 23 is used, with a standard deviation uniformly sampled over [0.1, 2.0]; • solarization: an optional color transformation x → x • 1 {x<0.5} + (1x) • 1 {x≥0.5} for pixels with values in [0, 1].

Figure 4 :

 4 Figure 4: Semi-supervised training with a fraction of ImageNet labels on a ResNet-50 (×1).

Figure 5 :

 5 Figure 5: Semi-supervised training with a fraction of ImageNet labels on multiple ResNets architecture pretrained with BYOL. Note that large networks are facing overfitting problems.

Figure 8 :

 8 Figure 8: BYOL sketch summarizing the method by emphasizing the neural architecture.

  • 10 -6 while excluding the biases and batch normalization parameters from both LARS adaptation and weight decay. For the target network, the exponential moving average parameter τ starts from τ base = 0.

	996 and is increased
	to one during training. Specifically, we set τ step and K the maximum number of training steps. We use a batch size of 4096 split over 512 Cloud TPU v3 cores. 1 -(1 -τ base ) • (cos(πk/K) + 1)/2 with k the current training With this setup, training takes approximately 8 hours for a ResNet-50(×1). All hyperparameters are summarized in Appendix J; an additional set of hyperparameters for a smaller batch size of 512 is provided in Appendix G.

Table 2

 2

	Method	Top-1 1% 10%	Top-5 1% 10%	Method	Architecture	Param.	Top-1 1% 10%	Top-5 1% 10%
	Supervised [77] 25.4 56.4 48.4 80.4 InstDisc --39.2 77.4 PIRL [35] --57.2 83.8 SimCLR [8] 48.3 65.6 75.5 87.8 BYOL (ours) 53.2 68.8 78.4 89.0	CPC v2 [32] ResNet-161 SimCLR [8] ResNet-50 (2×) BYOL (ours) ResNet-50 (2×) SimCLR [8] ResNet-50 (4×) BYOL (ours) ResNet-50 (4×) BYOL (ours) ResNet-200 (2×) 250M 71.2 77.7 89.5 93.7 305M --77.9 91.2 94M 58.5 71.7 83.0 91.2 94M 62.2 73.5 84.1 91.7 375M 63.0 74.4 85.8 92.6 375M 69.1 75.7 87.9 92.5
		(a) ResNet-50 encoder.			(b) Other ResNet encoder architectures.

. BYOL consistently outperforms previous approaches across a wide range of architectures. Additionally, as detailed in Appendix C.1, BYOL reaches 77.7% top-1 accuracy with ResNet-50 when fine-tuning over 100% of ImageNet labels.

Table 2 :

 2 Semi-supervised training with a fraction of ImageNet labels.

Table 3 :

 3 Transfer learning results from ImageNet (IN) with the standard ResNet-50 architecture.

	Method	Food101 CIFAR10 CIFAR100 Birdsnap SUN397 Cars Aircraft VOC2007 DTD	Pets	Caltech-101 Flowers
	Linear evaluation:
	BYOL (ours) SimCLR (repro) SimCLR [8] Supervised-IN [8]	75.3 72.8 68.4 72.3	91.3 90.5 90.6 93.6	78.4 74.4 71.6 78.3	57.2 42.4 37.4 53.7	62.2 60.6 58.8 61.9	67.8 49.3 50.3 66.7	60.6 49.8 50.3 61.0	82.5 81.4 80.5 82.8	75.5 90.4 75.7 84.6 74.5 83.6 74.9 91.5	94.2 89.3 90.3 94.5	96.1 92.6 91.2 94.7
	Fine-tuned:
	BYOL (ours) SimCLR (repro) SimCLR [8] Supervised-IN [8] Random init [8]	88.5 87.5 88.2 88.3 86.9	97.8 97.4 97.7 97.5 95.9	86.1 85.3 85.9 86.4 80.2	76.3 75.0 75.9 75.8 76.1	63.7 63.9 63.5 64.3 53.6	91.6 91.4 91.3 92.1 91.4	88.1 87.6 88.1 86.0 85.9	85.4 84.5 84.1 85.0 67.3	76.2 91.7 75.4 89.4 73.2 89.2 74.6 92.1 64.8 81.5	93.8 91.7 92.1 93.3 72.6	97.0 96.6 97.0 97.6 92.0

Table 4 :

 4 Results on transferring BYOL's representation to other vision tasks.

Table 5 :

 5 Ablations with top-1 accuracy (in %) at 300 epochs under linear evaluation on ImageNet.

Table 6 :

 6 Parameters used to generate image augmentations. As in[START_REF] Kolesnikov | Revisiting self-supervised visual representation learning[END_REF][START_REF] Kornblith | Do better ImageNet models transfer better?[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Jain | Improved baselines with momentum contrastive learning[END_REF], we use the standard linear evaluation protocol on ImageNet, which consists in training a linear classifier on top of the frozen representation, i.e., without updating the network parameters nor the batch statistics. At training time, we apply spatial augmentations, i.e., random crops with resize to 224 × 224 pixels, and random flips. At test time, images are resized to 256 pixels along the shorter side using bicubic resampling, after which a 224 × 224 center crop is applied.

	Parameter	T	T
	Random crop probability Flip probability Color jittering probability Brightness adjustment max intensity 0.4 0.4 1.0 1.0 0.5 0.5 0.8 0.8 Contrast adjustment max intensity 0.4 0.4 Saturation adjustment max intensity 0.2 0.2 Hue adjustment max intensity 0.1 0.1 Color dropping probability 0.2 0.2 Gaussian blurring probability 1.0 0.1 Solarization probability 0.0 0.2
	C Evaluation on ImageNet training
	C.1 Self-supervised learning evaluation on ImageNet
	Linear evaluation protocol on ImageNet

Table 7 :

 7 Different linear evaluation protocols on ResNet architectures by either replacing the spatial augmentations with pre-train augmentations, or regularizing the linear classifier.

	Architecture	Pre-train augmentations Logits regularization Top-1 Top-5
		74.3	91.6
	ResNet-50 (1×)	74.4 74.7	91.8 91.8
		74.8	91.8
		78.6	94.2
	ResNet-50 (4×)	78.6 78.9	94.3 94.3
		79.0	94.5
		79.6	94.8
	ResNet-200 (2×)	79.6 79.8	94.8 95.0
		80.0	95.0

Table 8 :

 8 Semi-supervised training with the full ImageNet on a ResNet-50 (×1).

	Supervised: Method	Top-1	Top-5	Semi-supervised (100%): Method Top-1	Top-5
	Supervised[8] AutoAugment [87] MaxUp [75]	76.5 77.6 78.9	-93.8 94.2	SimCLR [8] SimCLR (repro) BYOL	76.0 76.5 77.7	93.1 93.5 93.9

Table 9 :

 9 Linear evaluation of BYOL on ImageNet using larger encoders. Top-1 and top-5 accuracies are reported in %.

	BYOL Architecture Multiplier Weights Top-1 Top-5 Top-1 Supervised (ours) Supervised [8] Top-5 Top-1
	ResNet-50 ResNet-101 ResNet-152 ResNet-200 ResNet-50 ResNet-101 ResNet-50 ResNet-152 ResNet-200 ResNet-50 ResNet-101 ResNet-152	1× 1× 1× 1× 2× 2× 3× 2× 2× 4× 3× 3×	24M 43M 58M 63M 94M 170M 211M 232M 250M 375M 382M 522M	74.3 76.4 77.3 77.8 77.4 78.7 78.2 79.0 79.6 78.6 78.4 79.5	91.6 93.0 93.7 93.9 93.6 94.3 93.9 94.6 94.9 94.2 94.2 94.6	76.4 78.0 79.1 79.3 79.9 80.3 80.2 80.6 80.1 80.7 80.7 80.9	92.9 94.0 94.5 94.6 95.0 95.0 95.0 95.3 95.2 95.3 95.3 95.2	76.5 ---77.8 ----78.9 --

Table 10 :

 10 value of 0.1. Results are presented in Figure 6. Results for linear evaluation of BYOL compared to fully supervised baselines with various ResNet architectures. Our supervised baselines are ran with RandAugment [87] augmentations. Characteristics of image datasets used in transfer learning. When an official test split with labels is not publicly available, we use the official validation split as test set, and create a held-out validation set from the training examples.

	20

  and 12 extends Table4with other published baselines which use comparable networks. We see that in almost all settings, BYOL outperforms these baselines, even when those baselines use more data or deeper models. One notable exception is RMS error for NYU Depth prediction, which is a metric that's sensitive to outliers. The reason for this is unclear, but one possibility is that the network is producing higher-variance predictions due to being more confident about a test-set scene's similarities with those in the training set.

	Method	AP50 mIoU
	Supervised-IN [9]	74.4	74.4
	RelPos [23], by [40] * Multi-task [40] * LocalAgg [98] MoCo [9] MoCo + IG-1B [9] CPC[32] * * SimCLR (repro) BYOL (ours)	66.8 70.5 69.1 74.9 75.6 76.6 75.2 77.5	---72.5 73.6 -75.2 76.3

Table 11 :

 11 Transfer results in semantic segmentation and object detection. * uses a larger model (ResNet-101). * * uses an even larger model (ResNet-161).

	Method	pct.< 1.25	Higher better pct.< 1.25 2	pct.< 1.25 3	Lower better rms rel
	Supervised-IN [83]	81.1	95.3	98.8	0.573 0.127
	RelPos [23], by [40] * Color [41], by [40] * Exemplar [46, 40] * Mot. Seg. [99], by [40] * Multi-task [40] * SimCLR (repro) BYOL (ours)	80.6 76.8 71.3 74.2 79.3 83.3 84.6	94.7 93.5 90.6 92.4 94.2 96.5 96.7	98.3 97.7 96.5 97.4 98.1 99.1 99.1	0.399 0.146 0.444 0.164 0.513 0.191 0.473 0.177 0.422 0.152 0.557 0.134 0.541 0.129

Table 12 :

 12 Transfer results on NYU v2 depth estimation.

Table 13 :

 13 Transfer learning results (linear evaluation, ResNet-50) from Places365 (PL). For comparison purposes, we also report the results from BYOL trained on ImageNet (BYOL-IN).

Table 14 :

 14 Effect of architectural settings where top-1 and top-5 accuracies are reported in %.

Table 5b

 5b

	Method Predictor Target parameters β Top-1
	BYOL SimCLR	ξ ξ ξ sg(θ) θ sg(θ) sg(θ) θ sg(θ) θ ξ sg(θ) θ	0 1 1 1 1 1 1 1 0 0 0 0 0	72.5 70.9 70.7 70.2 69.4 70.1 69.2 69.0 5.5 0.3 0.2 0.1 0.1

, we set the temperature α to its best value in the SimCLR setting (i.e., α = 0.1). With this value, setting β to 1 (which adds negative examples), in the BYOL setting (i.e., with both a predictor and a target network) hurts the performances. In Table

18

, we report results of a sweep over both the temperature α and the

Table 19 :

 19 Top-1 accuracy in %, under linear evaluation protocol at 300 epochs, of intermediate variants between BYOL and SimCLR (with caveats discussed in Appendix F.5). sg means stop gradient.weight parameter β with a predictor and a target network where BYOL corresponds to β = 0. No run significantly outperforms BYOL and some values of α and β hurt the performance. While the best temperature for SimCLR (without the target network and a predictor) is 0.1, after adding a predictor and a target network the best temperature α is higher than 0.3.

  Figure 7: Effect of normalization on the 2 norm of network outputs.

	2 norm
	Representation			BatchNorm LayerNorm None
	0	100 Training epochs 200	300
		(a) Representation 2-norm
	2 norm
	Projection			BatchNorm LayerNorm None
	0	100 Training epochs 200	300
		(b) Projection 2-norm
		Normalization	Top-1	Top-5
		2-norm LAYERNORM No normalization BATCHNORM	72.5 72.5±0.4 67.4 65.3	90.8 90.1 87.1 85.3

1 B B i=1 n 2 (p i )n 2 (z i ) 2 2

with n 2 : x → x/ x 2 . We run BYOL with other normalization functions: non-trainable batch-normalization

Table 20 :

 20 Top-1 accuracy in % under linear evaluation protocol at 300 epochs for different normalizations in the loss. and layer-normalization and no normalization. We divide the batch normalization and layer normalization by √ d to have a consistent scale with the 2 -normalization. We report results in Table20where 2 , LAYERNORM, no normalization and BATCHNORM respectively denote using n 2 , n BN , n LN and n ID with

Table 21 :

 21 Top-1 accuracy at 300 epochs when removing the slowly moving target network, directly hard copying the weights of the online network into the target network, and applying a multiplier to the predictor learning rate.

					λ pred
			1	1.5	2	5	10
		1	3.2±2.9 25.7±6.6	60.8±2.9 66.7±0.4	66.9
		1.5 1.4±2.0	9.2±7.0	55.2±5.8 61.5±0.6 66.0±0.3
	µproj	2	2.0±2.8	5.3±1.9 15.8±13.4 60.9±0.8	66.3
		5	1.5±0.9	2.5±1.5	2.5±1.4 20.5±2.0 60.5±0.6
		10	0.1	2.1±0.3	1.9±0.8	2.8±0.4	8.3±6.8

Throughout this paper, the term bootstrap is used in its idiomatic sense rather than the statistical sense.

While we could directly predict the representation y and not a projection z, previous work[START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] have empirically shown that using this projection improves performance.

For simplicity we also consider BYOL without normalization (which performs reasonably close to BYOL, see Appendix F.6) nor symmetrization

The only dependency on batch size in our training pipeline sits within the batch normalization layers.

https://github.com/Philip-Bachman/amdim-public/blob/master/costs.py

Acknowledgements

The authors would like to thank the following people for their help throughout the process of writing this paper, in alphabetical order: Aaron van den Oord, Andrew Brock, Jason Ramapuram, Jeffrey De Fauw, Karen Simonyan, Katrina McKinney, Nathalie Beauguerlange, Olivier Henaff, Oriol Vinyals, Pauline Luc, Razvan Pascanu, Sander Dieleman, and the DeepMind team. We especially thank Jason Ramapuram and Jeffrey De Fauw, who provided the JAX SimCLR reproduction used throughout the paper.

//github.com/deepmind/deepmind-research/tree/master/byol

Table 15: Effect of learning rate and weight decay. We note that BYOL's performance is quite robust within a range of hyperparameters. We also observe that setting the weight decay to zero may lead to unstable results (as in SimCLR).

F.3 Image augmentations