Mourad Korichi

Vincent Gerbaud

Pascal Floquet

A.-Hassan Meniai

Saci Nacef

Xavier Joulia

Quantitative Structure -Odor Relationship: Using of Multidimensional Data Analysis and Neutral Network Approaches

Keywords: Molecular descriptors, Neural network, SOR

are key issues for the synthesis of new odorant molecules. But, this relation is hard to model, due to limited understanding of olfaction phenomena and the subjectivity of odor quantity and quality as stated in Rossitier's review (1996). Many molecular descriptors are used to correlate molecule's odor, but no universal rules emerge in this field. In this paper, we focus on the use of molecular descriptors as an alternative approach in the prediction of odors, by the mean of regression techniques. Principal Component Analysis (PCA) and Stepwise Collinearity Diagnosis (SCD) techniques are used to reduce the dimensionality of data, by the identification of significant molecular descriptors. Then, the chosen molecular descriptors are used with a neural networks algorithm to correlate the structure to molecular odor quality. The results are validated on balsamic flavor.

Introduction

Odorant compounds are found in a wide variety of products ranging from foods, perfumes, health care products and medicines. Either combined or alone, flavor and fragrance compounds are used to induce consumers to associate favorable impressions with a given product. In some cases, products have one predominant component which provides the characteristic odor. However, in most cases, products containing odors include a complex mixture of fragrant compounds. Some of them are classified within REACH, a European Community document, regulating the use of chemicals in terms of environment and toxicity. Structure -Odor relationships (SOR) are very important for the synthesis of new odorant molecules. This relation is difficult to model due to the subjectivity of the odor quantity and quality. Olfaction phenomenon is not yet completely understood and odor measurements are often inaccurate [START_REF] Amboni | [END_REF]. Research has been oriented to the use of structural, topological, geometrical, electronic, and physicochemical parameters as descriptors, to generate odor predictive equations. Therefore, a number of computational techniques have been used successfully. Artificial Neural Networks (ANN's) are one of these promising techniques readily suited for the assessment of poorly understood properties like odor. In this paper, we aim to use molecular descriptors as an alternative approach in the prediction of molecule's odor by the mean of regression techniques. Principal Component Analysis (PCA) and Pairwise Collinearity Diagnosis techniques are used to reduce the dimensionality of data, by the identification of significant molecular descriptors. Then, the chosen molecular descriptors are used with a neural networks algorithm to correlate the structure to molecular odor quality. Figure 1 summarizes the methodology.

Molecular Descriptor Selection

Molecular descriptors accounts for a particular aspect of the molecule structure. As examples, simply count of atoms, functional groups and characteristic fragments are some of the constitutional descriptors family of the studied structure. Topological descriptors are related to the two-dimensional representation of the molecular structure. Molecular descriptors are the most significant common features of molecular structure that can be used to develop Structure -property relationships. In our case, the property is the odor of a molecule. Our input data set contains 121 molecules of balsamic odor splited in 5 sub-notes of typical odors (see Table 1): anise, balsam, honey, vanilla and sweet (Aldrich Flavors and Fragrances catalog, 2005). The dragon software (TALETE, 2005)

Complete Correlation Analysis

The complete correlation analysis is used to select a subset of linearly independent descriptors. Descriptor dependency is evaluated using the Dragon software by setting a predefined value Rmax (In this work, Rmax = 0.97) below which descriptors are considered linearly independent.

Principal component analysis

Principal component analysis is one of the oldest, but still most widely used techniques of multivariate analysis. The basic idea of the method is to try to describe the variation of the variables in a set of multivariate data, as parsimoniously as possible using a set of uncorrelated variables, each of which is a particular linear combination of those in the original data. This enables us to reduce the molecular descriptors dimensionality, by the identification of the principal components that can be used in the structure -odor relationship. All eigenvalues greater than 1 are retained to describe the principal axes. In the first case, four principal components are kept to describe the 16 molecular descriptors, widely used in the correlation of the structure -odor. In the second case, ten principal components are retained to represent the 140 molecular descriptors.

Artificial Neural Networks (ANN) Approach

The ANN [START_REF] Dreyfus | Réseaux de neurones :: méthodologie et applications[END_REF] trained by back-propagation (BP) network algorithm has a two layers architecture: the first layer is called the hidden layer. The number of hidden neuron is a variable X between 7 and 9. The output layer consists of one neuron, namely the odor quality. The network configuration is m-X-1, where m represents the number of principal components encapsulating the maximum of information of the linearly independent molecular descriptors ( § 2.2). The ANN has a feed forward layered structure with connections allowed only between adjacent layers. The balsamic odor sub-notes as output, are represented by arbitrary continuous codification described in table 1. Input and output data are normalized, and hyperbolic tangent sigmoid transfer function is used in the hidden layer. The output layer is a linear function. In this work, the training and the validation sets are generated randomly, corresponding respectively to 70% and 30% of the input data set of 121 molecules. After several training sessions, an optimal number of hidden neurons X equal 8 and 7 is retained for case one and two respectively. The network is trained for 500 epochs with a gradient algorithm. The performance goal is 0.001.

Discriminant Analysis

Discriminant analysis is an analytical technique, whereby a multivariate data set containing m variables is separated into a number (k) of pre-defined groups, using discriminant functions (Z) which are linear combinations of the variables. Two cases are studied to discriminate the molecules into different odors based on the results of PCA described on the paragraph ( §2.2).

Discriminant Analysis based on the first PCA study (four principal components)

In the first case there are four principal components. Results are presented in the table 3 with an overall 69.4% of the molecules in the data set, are well classified. The molecules of vanilla odor have the highest correctly classification per cent.

Results and discussions

Artificial Neural Networks Approach

It is well-known that ANN performance depends on many variables, as the number of hidden neurons, the degree of homology between the training and the validation sets and the input variables (Principal components in this case). In figure 2, results are represented as the variation of the odor quality versus the molecule identification code, (a) for the first case and (b) for the second case. In the first case (a), the ANN does not converge as shown by the similarity of the response for all molecules despite their initial differences. In the second case (b), the training set is well represented, but almost all the validation set is not. This clearly shows the non predictive capacity of the ANN approach. Kovatcheva et al. (2004) on a kNN approach for modeling structureambergris odor relationship suggest to use division procedures based on sphereexclusion algorithms and demonstrate a predictive capacity. But the ambergris odor is due to well known chemical structures, unlike the balsamic odor where molecule structure is more heterogeneous with several odor sub-notes. Also, Chastrette et al., (1995Chastrette et al., ( & 1996)), Cherqaoui et al. (1998) and [START_REF] Zakarya | [END_REF] do not consider subnotes, only, requesting a discrete response.

Discriminant Analysis Approach

69.4% and 83.4% of molecules are well discriminated in the two cases respectively, especially anise and vanilla molecules groups. This is better than the ANN approach. Case 2 has more odor discriminant, because it incorporates more numerous and diverse molecular descriptors. Among the molecules that are not discriminated, the two molecules from anise, classified in the vanilla group bear similar molecular structure with vanilla type molecules, which have three oxygen atoms, high Kier symmetry index and n-path Kier alpha-modified shape index. For case 2, in balsam and honey sub-note odors, the molecule wrongly classified is considered differently depending on the referential nomenclature (we used Aldrich's) that both belong to balsamic and/or rose main odors. In the group of vanilla, one molecule is discriminated as balsam sub-note.

From the referential nomenclature, we can say that the molecule has different odor types. In the sweet sub-note, fourteen molecules are distributed into other sub-notes. The low discrimination of the sweet odor may be attributed to the subjectivity of this sub-note, unlike vanilla or anise. Indeed, sweet is not considered as a typical odor type in the reputed referential chart "the field of odors" of [START_REF] Jaubert | [END_REF].

Conclusion and perspectives

In this work we present different ways to estimate and discriminate odors of molecules, based on molecular descriptors using multidimensional data analysis, and neural network applied to balsamic odors. The multidimensional data analysis is a powerful tool to reduce data sets and encapsulate the maximum of molecule's structure information. Discriminant analysis results using only 2D molecular representation are encouraging. Further work using 3D representation molecular descriptors may improve the results. The neural network satisfactorily correlates the molecules with their assigned odor, based on sufficiently numerous and diverse molecular descriptors. But it is unable to predict balsamic odor and its sub-notes. Compared with literature, successful results in ANN approach are due to the well known families of odor. The heterogeneous nature of the molecules assigned to balsamic odor and the absence of evident structure -odor relationship, forces us to request a continuous discrimination between sub-notes.

Figure 2 .

 2 Figure 2. Reference odor (Aldrich, 2005) and calculated odor versus molecular identification.

Molecular Descriptors Selection 16 linearly independent descriptors 140 linearly independent descriptors 4 Principal components 10 Principal components 28 Molecular Descriptors:

  Constitutional, topological and connectivity. Mw, Aromaticity, nBz, nR5, nR6, SiK, XiA,

	is used to calculate up to 582 molecular descriptors of the input data set.
	According to figure 1, two cases are explored, first, 34 simple descriptors (molecular

582 Molecular Descriptors: Constitutional, topological, walk and path

	Complete

counts, connectivity , information, 2D autocorrelations, edge adjacency, burden eigenvalues and topological charge.

Correlation Analysis Principal Component Analysis Discriminant Analysis Neural Network Figure

  Kier symmetry index and n-path Kier alpha-modified shape index) used in the different relations described in Rossitier's review on structure -odor works since 1930(1996) are calculated. In the second case, 582 molecular descriptors are considered: constitutional descriptors, topological indices, walk and path counts, connectivity and information indices, 2D autocorrelations, edge adjacency indices, burden eigenvalues and topological charge indices. All descriptors are calculated from the 2D molecular representation.

1. Schematic representation of methodology on structure -odor relationships.

Classification & regression

weight, aromatic ratio, number of benzene-like rings, connectivity indices,

Table 1 .

 1 Input data set of molecular structure

	Odor type	Number of compounds	Arbitrary continuous Odor codification	Arbitrary discontinuous Odor codification
	Anise	10	0 to 0.15	0.15
	Balsam	18	0.25 to 0.35	0.35
	Honey	21	0.45 to 0.55	0.55
	Vanilla	15	0.60 to 0.75	0.75
	Sweet	58	0.85 to 0.95	0.95

Table 3 .

 3 Discriminant analysis based on the first PCA study. In the second case, there are ten principal components. 83.4% of 121 molecules in the data set are well classified, with the honey molecules having the highest classification, and the sweet molecules having the lowest, like in the first case.

	Groups	Anise	Predicted groups Balsam Honey Vanilla	Sweet	molecules	Correctly classified
	Anise	8	0	0	2	0	10	0.800
	Balsam	1	14	1	1	1	18	0.778
	Honey	0	3	15	0	3	21	0.714
	Vanilla	0	2	0	12	0	14	0.857
	Sweet	3	8	8	4	35	58	0.603
	4.2. Discriminant Analysis based on the second PCA study (ten principal components)

Table 4 .

 4 Discriminant analysis based on the second PCA study

	Groups	Anise	Predicted groups Balsam Honey Vanilla	Sweet	molecules	Correctly classified
	Anise	8	0	0	2	0	10	0.800
	Balsam	0	17	0	0	1	18	0.944
	Honey	0	0	20	0	1	21	0.952
	Vanilla	0	1	0	13	0	14	0.929
	Sweet	3	2	5	4	44	58	0.759


Timothée Darcet
email: timdarcet@meta.com

Maxime Oquab

Julien Mairal
email: julien.mairal@inria.fr

Piotr Bojanowski
email: bojanowski@meta.com

Meta Fair

VISION TRANSFORMERS NEED REGISTERS

Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing. Without registers With registers Input DeiT-III CLIP DINOv2 DeiT-III CLIP DINOv2

Figure 1: Register tokens enable interpretable attention maps in all vision transformers, similar to the original DINO method [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF]. Attention maps are calculated in high resolution for better visualisation. More qualitative results are available in appendix D.

INTRODUCTION

Embedding images into generic features that can serve multiple purposes in computer vision has been a long-standing problem. First methods relied on handcrafted principles, such as SIFT [START_REF] David G Lowe | Distinctive image features from scale-invariant keypoints[END_REF], before the scale of data and deep learning techniques allowed for end-to-end training. Pursuing generic feature embeddings is still relevant today, as collecting valuable annotated data for many specific tasks remains difficult. This difficulty arises because of the required expertise (e.g., medical data, or remote sensing) or the cost at scale. Today, it is common to pretrain a model for a task for which plenty of data is available and extract a subset of the model to use as a feature extractor. Multiple approaches offer this possibility; supervised methods, building on classification We consider ViTs trained with label supervision (DeiT-III), text-supervision (OpenCLIP) or selfsupervision (DINO and DINOv2). Interestingly, all models but DINO exhibit peaky outlier values in the attention maps. The goal of this work is to understand and mitigate this phenomenon.

or text-image alignment, allow training strong feature models to unlock downstream tasks. Alternatively, self-supervised methods building on the Transformer architecture have attracted significant attention due to their high prediction performance on downstream tasks and the intriguing ability of some models to provide unsupervised segmentations [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF] In particular, the DINO algorithm is shown to produce models that contain explicit information about the semantic layout of an image. Indeed, qualitative results show that the last attention layer naturally focuses on semantically consistent parts of images and often produces interpretable attention maps. Exploiting these properties, object discovery algorithms such as LOST [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF] build on top of DINO. Such algorithms can detect objects without supervision by gathering information in attention maps. They are effectively unlocking a new frontier in computer vision.

DINOv2 [START_REF] Oquab | Dinov2: Learning robust visual features without supervision[END_REF], a follow-up to DINO, provides features that allow tackling dense prediction tasks. DINOv2 features lead to successful monocular depth estimation and semantic segmentation with a frozen backbone and linear models. Despite the strong performance on dense tasks, we observed that DINOv2 is surprisingly incompatible with LOST. When used to extract features, it delivers disappointing performance, only on par with supervised alternative backbones in this scenario. This suggests that DINOv2 behaves differently than DINO. The investigation described in this work notably exposes the presence of artefacts in the feature maps of DINOv2 that were not present in the first version of this model. These are observable qualitatively using straightforward methods. Also surprisingly, applying the same observations to supervised vision transformers exposes similar artifacts, as shown in Fig. 2. This suggests that DINO is, in fact, an exception, while DINOv2 models match the baseline behavior of vision transformers.

In this work, we set out to better understand this phenomenon and develop methods to detect these artifacts. We observe that they are tokens with roughly 10x higher norm at the output and correspond to a small fraction of the total sequence (around 2%). We also show that these tokens appear around the middle layers of the vision transformer, and that they only appear after a sufficiently long training of a sufficiently big transformer. In particular, we show that these outlier tokens appear in patches similar to their neighbors, meaning patches that convey little additional information.

As part of our investigation, we evaluate the outlier tokens with simple linear models to understand the information they contain. We observe that, compared to non-outlier tokens, they hold less information about their original position in the image or the original pixels in their patch. This ob- We observe that DINOv2 has a few outlier patches, whereas DINO does not present these artifacts. For DINOv2, although most patch tokens have a norm between 0 and 100, a small proportion of tokens have a very high norm. We measure the proportion of tokens with norm larger than 150 at 2.37%.

servation suggests that the model discards the local information contained in these patches during inference. On the other hand, learning an image classifier on outlier patches yields significantly stronger accuracy than doing so on the other patches, suggesting that they contain global information about the image. We propose the following interpretation to these elements: the model learns to recognize patches containing little useful information, and recycle the corresponding tokens to aggregate global image information while discarding spatial information.

This interpretation is consistent with an inner mechanism in transformer models that allows performing computations within a restricted set of tokens. In order to test this hypothesis, we append additional tokens -that we call registers -to the token sequence, independent of the input image. We train several models with and without this modification and observe that the outlier tokens disappear from the sequence entirely. As a result, the performance of the models increases in dense prediction tasks, and the resulting feature maps are significantly smoother. These smooth feature maps enable object discovery methods like LOST mentioned above with the updated models.

PROBLEM FORMULATION

As shown in Fig. 2, most modern vision transformers exhibit artifacts in the attention maps. The unsupervised DINO backbone [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF] has been previously praised for the quality of local features and interpretability of attention maps. Surprisingly, the outputs of the subsequent DINOv2 models have been shown to hold good local information but exhibit undesirable artifacts in attention maps. In this section, we propose to study why and when these artifacts appear. While this work focuses on alleviating artefacts in all vision transformers, we focus our analysis on DINOv2.

ARTIFACTS IN THE LOCAL FEATURES OF DINOV2

Artifacts are high-norm outlier tokens. We want to find a quantitative way of characterizing artefacts that appear in the local features. We observe that an important difference between "artifact" patches and other patches is the norm of their token embedding at the output of the model. In Fig. 3 (left), we compare the norm of local features for a DINO and DINOv2 model given a reference image. We clearly see that the norm of artifact patches is much higher than the norm of other patches. We also plot the distribution of feature norms over a small dataset of images in Fig. 3 (right), which is clearly bimodal, allowing us to choose a simple criterion for the rest of this section: tokens with norm higher than 150 will be considered as "high-norm" tokens, and we will study their properties relative to regular tokens. This hand-picked cutoff value can vary across models. In the rest of this work, we use "high-norm" and "outlier" interchangeably.

Outliers appear during the training of large models. We make several additional observations about the conditions in which these outlier patches appear during the training of DINOv2. This analysis is illustrated in Fig. 4. First, these high-norm patches seem to differentiate themselves from other patches around layer 15 of this 40-layer ViT (Fig. 4a). Second, when looking at the distribution of norms along training of DINOv2, we see that these outliers only appear after one third of training (Fig. 4b). Finally, when analyzing more closely models of different size (Tiny, Small, Base, Large, Huge and giant), we see that only the three largest models exhibit outliers (Fig. 4c). High-norm tokens appear where patch information is redundant. To verify this, we measure the cosine similarity between high-norm tokens and their 4 neighbors right after the patch embedding layer (at the beginning of the vision transformer). We illustrate the density plot in Fig. 5a. We observe that high-norm tokens appear on patches that are very similar to their neighbors. This suggests that these patches contrain redundant information and that the model could discard their information without hurting the quality of the image representation. This matches qualitative observations (see Fig. 2) that they often appear in uniform, background areas.

High-norm tokens hold little local information. In order to better understand the nature of these tokens, we propose to probe the patch embeddings for different types of information. For that we consider two different tasks: position prediction and pixel reconstruction. For each of these tasks, we train a linear model on top of the patch embeddings, and measure the performance of this model. We compare the performance achieved with high-norm tokens and with other tokens, to see if high-norm tokens contain different information than "normal" tokens.

• Position prediction. We train a linear model to predict the position of each patch token in the image, and measure its accuracy. We note that this position information was injected in the tokens before the first ViT layer in the form of absolute position embeddings. We observe that high-norm tokens have much lower accuracy than the other tokens (Fig. 5b), suggesting they contain less information about their position in the image. • Pixel reconstruction. We train a linear model to predict the pixel values of the image from the patch embeddings, and measure the accuracy of this model. We observe again that high-norm tokens achieve much lower accuracy than other tokens (Fig. 5b). This suggests that high-norm tokens contain less information to reconstruct the image than the others. 1: Image classification via linear probing on normal and outlier patch tokens. We also report the accuracy of classifiers learnt on the class token. We see that outlier tokens have a much higher accuracy than regular ones, suggesting they are effectively storing global image information. benchmarks. For each image in a classification dataset, we forward it through DINOv2-g and extract the patch embeddings. From those, we choose a single token at random, either high-norm or normal. This token is then considered as the image representation. We then train a logistic regression classifier to predict the image class from this representation, and measure the accuracy. We observe that the high-norm tokens have a much higher accuracy than the other tokens (Table 1). This suggests that outlier tokens contain more global information than other patch tokens.

HYPOTHESIS AND REMEDIATION

Having made these observations, we make the following hypothesis: large, sufficiently trained models learn to recognize redundant tokens, and to use them as places to store, process and retrieve global information. Furthermore, we posit that while this behavior is not bad in itself, the fact that it happens inside the patch tokens is undesirable. Indeed, it leads the model to discard local patch information (Tab. 5b), possibly incurring decreased performance on dense prediction tasks.

We therefore propose a simple fix to this issue: we explicitly add new tokens to the sequence, that the model can learn to use as registers. We add these tokens after the patch embedding layer, with a learnable value, similarly to the [CLS] token. At the end of the vision transformer, these tokens are discarded, and the [CLS] token and patch tokens are used as image representations, as usual. This mechanism was first proposed in Memory Transformers [START_REF] Mikhail S Burtsev | Memory transformer[END_REF], improving translation tasks in NLP. Interestingly, we show here that this mechanism admits a natural justification for vision transformers, fixing an interpretability and performance issue that was present otherwise.

We note that we have not been able to fully determine which aspects of the training led to the appearance of artifacts in DINOv2 but not in DINO, but Fig. 4 suggests that scaling the model size beyond ViT-L, and longer training length may be possible causes.

EXPERIMENTS

In this section, we validate the proposed solution by training vision transformers with additional [reg] register tokens. We evaluate the effectiveness of our approach by a quantitative and qualitative analysis. We then ablate the number of registers used for training, to check that they do not cause a performance regression, evaluate an unsupervised object discovery method atop our features and finally provide a qualitative analysis of the patterns learnt by the registers.

TRAINING ALGORITHMS AND DATA

As the proposed solution is a simple architectural change, we can easily apply it to any training procedure. We try it on three different state-of-the-art training methods for supervised, text-supervised, and unsupervised learning, shortly described below.

DEIT-III [START_REF] Touvron | Deit iii: Revenge of the vit[END_REF]) is a simple and robust supervised training recipe for classification with ViTs on ImageNet-1k and ImageNet-22k. We choose this method as an example of labelsupervised training as it is simple, uses the base ViT architecture, achieves strong classification results, and is easy to reproduce and modify with our improvements. We run this method on the ImageNet-22k dataset, using the ViT-B settings, as provided in the official repository1 .

OpenCLIP [START_REF] Ilharco | Openclip[END_REF]) is a strong training method for producing text-image aligned models, following the original CLIP work. We chose this method as an example of text-supervised training because it is open-source, uses the base ViT architecture, and is easy to reproduce and modify with our improvements. We run the OpenCLIP method on a text-image-aligned corpus based on Shutterstock that includes only licensed image and text data. We use a ViT-B/16 image encoder, as proposed in the official repository2 .

DINOv2 [START_REF] Oquab | Dinov2: Learning robust visual features without supervision[END_REF]) is a self-supervised method for learning visual features, following the DINO work mentioned previously. We apply our changes to this method as it is the main focus of our study. We run this method on ImageNet-22k with the ViT-L configuration. We use the code from the official repository3 .

EVALUATION OF THE PROPOSED SOLUTION

As shown in Fig. 1, we get rid of the artifacts by training models with additional register tokens.

In the appendix, we provide additional qualitative results for more images in Fig. 14. In order to quantitatively measure this effect, for each model, we probe the norm of features at the output of the model. We report these norms for all three algorithms with and without registers in Fig. 7. We see that when training with registers, models do not exhibit large-norm tokens at the output, which confirms the initial qualitative assessment.

Performance regression. In the previous section, we have shown that the proposed approach removes artifacts from local feature maps. In this experiment, we want to check that the use of register tokens does not affect the representation quality of those features. We run linear probing on Im-ageNet classification, ADE20k Segmentation, and NYUd monocular depth estimation. We follow the experimental protocol outlined in [START_REF] Oquab | Dinov2: Learning robust visual features without supervision[END_REF]. We summarize the performance of the models described in Sec. 3.1 with and without register tokens in Table 2a. We see that when using registers, models do not lose performance and sometimes even work better. For completeness, we also provided the zero-shot classification performance on ImageNet for OpenCLIP ( Table 2: Evaluation of downstream performance of the models that we trained, with and without registers. We consider linear probing of frozen features for all three models, and zero-shot evaluation for the OpenCLIP model. We see that using register not only does not degrade performance, but even improves it by a slight margin in some cases. (bottom): performance on three tasks (ImageNet, ADE-20k and NYUd) as a function of number of registers used. While one register is sufficient to remove artefacts, using more leads to improved downstream performance. which remains unchanged. Please note that the absolute performance of our OpenCLIP reproduction is lower due to the data source we used.

Number of register tokens. As described in Sec. 2.2, we propose alleviating the feature maps' artifacts by adding register tokens. In this experiment, we study the influence of the number of such tokens on local features and downstream performance. We train DINOv2 ViT-L/14 models with 0, 1, 2, 4, 8 or 16 registers. In Fig. 8, we report the results of this analysis. In Fig. 8(top), we qualitatively study the attention maps and observe that the visible artifacts disappear when adding at least one register. We then examine in Fig. 8(bottom) performance on downstream evaluation benchmarks, following the protocol from [START_REF] Oquab | Dinov2: Learning robust visual features without supervision[END_REF]. There seems to be an optimal number of registers for dense tasks, and adding one brings most of the benefit. This optimum is likely explained by the disappearance of artifacts, leading to better local features. On ImageNet, however, performance improves when using more registers. In all our experiments, we kept 4 register tokens.

OBJECT DISCOVERY

Recent unsupervised object discovery methods rely on the quality and smoothness of local feature maps [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF][START_REF] Wang | Cut and learn for unsupervised object detection and instance segmentation[END_REF]. By leveraging DINO Caron et al. (2021), these methods have significantly surpassed the previous state of the art. However, the algorithm leads to poor performance when applied to modern backbones such as DINOv2 [START_REF] Oquab | Dinov2: Learning robust visual features without supervision[END_REF] or supervised ones [START_REF] Touvron | Deit iii: Revenge of the vit[END_REF]. We posit that this can be alleviated by the method proposed [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF] on models with and without registers. We evaluated three types of models trained with various amounts of supervision on VOC 2007, 2012 and COCO. We measure performance using corloc. We observe that adding register tokens makes all models significantly more viable for usage in object discovery.

Input [CLS] [reg 0 ] [reg 6 ] [reg 8 ] [reg 12 ]
Figure 9: Comparison of the attention maps of the [CLS] and register tokens. Register tokens sometimes attend to different parts of the feature map, in a way similar to slot attention [START_REF] Locatello | Object-centric learning with slot attention[END_REF]. Note that this behaviour was never required from the model, and emerged naturally from training.

in this work. We run LOST [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF] on features extracted from backbones trained using the algorithms described in Sec.3.1 with and without registers. We run object discovery on PASCAL VOC 2007 and 2012 and COCO 20k. We use values for DeiT and OpenCLIP, and for DINOv2, we use keys. Because the output features may have different conditionning, we manually add a bias to the gram matrix of features. The results of this experiment are presented in Table 3. For all models and on all datasets, adding registers for training improves the unsupervised object discovery performance. The performance of DINOv2 on VOC2007 still does not match that of DINO as reported in the work of [START_REF] Siméoni | Localizing objects with self-supervised transformers and no labels[END_REF] (61.9 corloc). However, the model with registers gets an improvement of 20.1 corloc (55.4 versus 35.3).

QUALITATIVE EVALUATION OF REGISTERS

In this final experiment, we qualitatively probe for the behavior of register tokens. We want to verify if they all exhibit similar attention patterns or whether a differentiation automatically emerges. To this end, we plot the attention maps of the class and register tokens to patch tokens. The result of this visualization is shown in Fig. 9. We see that registers do not have a completely aligned behavior. Some selected registers exhibit interesting attention patterns, attending to the different objects in the scene. While nothing enforced this behavior, their activations had some natural diversity. We leave the study of the regularization of registers for future work.

RELATED WORK

FEATURE EXTRACTION WITH PRETRAINED MODELS

Using pretrained neural network models for extracting visual features has stood the test of time since the AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] CNN model pretrained on ImageNet-1k [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. More recent models have upgraded the same setup with modern architectures, such as ResNets (used in, e.g., DETR, [START_REF] Carion | End-to-end object detection with transformers[END_REF] or even Vision Transformers. As Transformers are easily able to handle different modalities during training, off-the-shelf backbones are now commonly trained on label supervision (e.g., DeiT-III on ImageNet-22k, [START_REF] Touvron | Deit iii: Revenge of the vit[END_REF] or text supervision (e.g., CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF]), providing strong visual foundation models, scaling well with model sizes, and enabling excellent performance on a variety of tasks including detection [START_REF] Carion | End-to-end object detection with transformers[END_REF] and segmentation [START_REF] Zheng | Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[END_REF][START_REF] Kirillov | Segment anything[END_REF].

In this context, supervision relies on annotations in the form of labels or text alignment; the dataset biases [START_REF] Torralba | Unbiased look at dataset bias[END_REF] are not well characterized, yet they drive learning and shape the learned models. An alternative approach consists of not using supervision and letting the models learn from the data via a pretext task that is designed to require understanding the content of images [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF]. This self-supervised learning paradigm was explored in multiple methods using Vision Transformers: MAE [START_REF] He | Masked autoencoders are scalable vision learners[END_REF] trains a model at reconstructing pixel values of hidden areas of an image and then applies fine-tuning to address a new task. With a different approach, the self-distillation family of methods [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF][START_REF] Zhou | ibot: Image bert pre-training with online tokenizer[END_REF] showcase strong performance using frozen backbones, allowing for more robustness to domain shifts for task-specific downstream models.

In this work, we focused the analysis on self-supervised learning, and more specifically on the DI-NOv2 approach [START_REF] Oquab | Dinov2: Learning robust visual features without supervision[END_REF], which has shown to be particularly effective for learning local features. We showed that despite excellent benchmark scores, DINOv2 features exhibit undesirable artifacts and that correcting these artifacts in the learning process allows for further improvements in the benchmark performances. These phenomenon is even more surprising as DINOv2 builds upon DINO [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF], which does not show signs of artifacts. We then further showed that the correction techniques also hold for supervised training paradigms by testing on DeiT-III and CLIP.

ADDITIONAL TOKENS IN TRANSFORMERS

Extending the transformer sequence with special tokens was popularized in BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]. However, most approaches add new tokens either to provide the network with new information as for example [SEP] tokens in BERT and tape tokens in AdaTape [START_REF] Xue | Adaptive computation with elastic input sequence[END_REF], or to gather information in these tokens, and use their output value as an output of the model:

• for classification: as [CLS] tokens in BERT and ViT [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] • for generative learning: as [MASK] in BERT and BEiT [START_REF] Bao | Beit: Bert pre-training of image transformers[END_REF] • for detection: as object queries in DETR [START_REF] Carion | End-to-end object detection with transformers[END_REF], detection tokens in YO-LOS [START_REF] Fang | You only look at one sequence: Rethinking transformer in vision through object detection[END_REF], and ViDT [START_REF] Song | Vidt: An efficient and effective fully transformer-based object detector[END_REF] • for accumulating information from possibly multiple modalities before decoding, as latent token arrays in Perceivers [START_REF] Jaegle | Perceiver: General perception with iterative attention[END_REF]2022).

Different to these works, the tokens we add to the sequence add no information, and their output value is not used for any purpose. They are simply registers where the model can learn to store and retrieve information during the forward pass. The Memory Transformer [START_REF] Mikhail S Burtsev | Memory transformer[END_REF], closer to our work, presents a simple approach to improve transformer models using memory tokens added to the token sequence, improving translation performance. In follow-up work, [START_REF] Bulatov | Recurrent memory transformer[END_REF] address complex copy-repeat-reverse tasks. [START_REF] Sandler | Fine-tuning image transformers using learnable memory[END_REF] extend this line to the vision domain for fine-tuning but observe that such tokens do not transfer well across tasks.

In contrast, we do not perform fine-tuning and employ additional tokens during the pretraining phase to improve the features obtained for all tasks downstream. More importantly, our study contributes the following new insight in Sec. 2: the mechanism implemented through memory tokens already appears naturally in Vision Transformers; our study shows that such tokens allow us not to create but to isolate this existing behavior, and thus avoid collateral side-effects.

CONCLUSION

In this work, we exposed artifacts in the feature maps of DINOv2 models, and found this phenomenon to be present in multiple existing popular models. We have described a simple method to detect these artifacts by observing that they correspond to tokens with an outlier norm value at the output of the Transformer model. Studying their location, we have proposed an interpretation that models naturally recycle tokens from low-informative areas and repurpose them into a different role for inference. Following this interpretation, we have proposed a simple fix, consisting of appending

A INTERPOLATION ARTIFACTS AND OUTLIER POSITION DISTRIBUTION

We plot in Figure 10 (left) the proportion of outlier tokens, characterized by a norm larger than the cutoff value defined manually, following the distribution of norms shown in Fig. 3 (main text). We make two observations:

First, the distribution has a vertical-striped pattern. We investigate this phenomenon and notice that in the original DINOv2 implementation, during training the position embeddings are interpolated from a 16 × 16 map into a 7 × 7 map, without antialiasing. Propagating unit gradients through such an interpolation function (bicubic resize) leads to the following gradients, shown in Fig. 11. In this work, when producing results with DINOv2 (especially for the results in Tables 2a,3), we always apply antialiasing in the interpolation operator, removing the striping pattern, which gives an updated distribution of outlier positions as shown in Fig. 10 (right).

Second, the outliers tend to appear in areas closer to the border of the feature map rather than in the center. Our interpretation is that the base model tends to recycle tokens in low-informative areas to use as registers: pictures produced by people tend to be object-centric, and in this case the border areas often correspond to background, which contains less information than the center.

B COMPLEXITY ANALYSIS

Since our proposed fix introduces new tokens, it also increases the number of learnable parameters and the FLOP count of the model. We show in Fig. 12 the relationship between number of registers and increase in model FLOP count and parameter count. We observe that adding registers induces a negligible change in number of parameters, and a slight change in FLOP count. Still, for n = 4 registers, the increase in FLOPs stays below 2%. Adding registers can increase model FLOP count by up to 6% for 16 registers. However, in the more common case of using 4 registers, that we use in most of our experiments, this increase is below 2%. In all cases, the increase in model parameters is negligible.

C ANALYSIS OF LOST PERFORMANCE

The results presented in Sec. 3.3 show that adding registers allows us to obtain better object discovery performance with DINOv2 models. The conclusions for the two other models studied in this work could be more crisp. In order to understand why this is so, we qualitatively study the impact of removing artifacts on the intermediate computations in the LOST algorithm. We show the intermediate outputs of LOST for all models on a given input image in Fig. 13.

Adding registers improves the scores and the resulting seed expansion for DeiT-III and DINOv2. This observation is coherent with the improved numbers reported in Table 3. For CLIP, however, the LOST algorithm seems robust to the type of outliers observed in the local features. Adding registers does remove artifacts (as clearly shown in Fig. 15) but does not have much impact on the LOST score. It is also worth noting that CLIP, with or without registers, provides comparable performance to DINOv2 without registers and DeiT-III with registers. The qualitative assessment is coherent with the numbers reported in Table 3.

D QUALITATIVE RESULTS

We trained three popular models: DeiT-III, CLIP, DINOv2 with and without the introduction of register tokens. We observe in Fig. 14 the attention maps in the last layer of the Vision Transformer, for all three cases. We see that our approach provides much cleaner attention maps, with considerably fewer artifacts, explaining the improvement on the downstream object discovery task mentioned in Sec. 3.3. The feature maps are also visibly improved, as shown in Fig. 15. Finally, we also show the norm of the patch tokens in Fig. 16, and confirm that in all three models, artifact patches correspond to norm outliers.

Figure 2 :

 2 Figure 2: Illustration of artifacts observed in the attention maps of modern vision transformers. We consider ViTs trained with label supervision (DeiT-III), text-supervision (OpenCLIP) or selfsupervision (DINO and DINOv2). Interestingly, all models but DINO exhibit peaky outlier values in the attention maps. The goal of this work is to understand and mitigate this phenomenon.

Figure 3 :

 3 Figure 3: Comparison of local feature norms for DINO ViT-B/16 and DINOv2 ViT-g/14.We observe that DINOv2 has a few outlier patches, whereas DINO does not present these artifacts. For DINOv2, although most patch tokens have a norm between 0 and 100, a small proportion of tokens have a very high norm. We measure the proportion of tokens with norm larger than 150 at 2.37%.

  Norms along iterations. (c) Norms across model size.

Figure 4 :

 4 Figure 4: Illustration of several properties of outlier tokens in the 40-layer DINOv2 ViT-g model. (a): Distribution of output token norms along layers. (b): Distribution of norms along training iterations. (c): Distribution of norms for different model sizes. The outliers appear around the middle of the model during training; they appear with models larger than and including ViT-Large.

  Linear probing for local information.

Figure 5 :

 5 Figure 5: (a): Distribution of cosine similarity between input patches and their 4 neighbors. We plot separately artifact patches (norm of the output token over 150) and normal patches. (b): Local information probing on normal and outlier patch tokens. We train two models: one for predicting position, and one for reconstructing the input patch. Outlier tokens have much lower scores than the other tokens, suggesting they are storing less local patch information.

Figure 6 :

 6 Figure 6: Illustration of the proposed remediation and resulting model. We add N additional learnable input tokens (depicted in yellow), that the model can use as registers. At the output of the model, only the patch tokens and CLS tokens are used, both during training and inference.

Figure 7 :

 7 Figure 7: Effect of register tokens on the distribution of output norms on DINOv2, CLIP and DeiT-III. Using register tokens effectively removes the norm outliers that were present previously.

Figure 8 :

 8 Figure 8: Ablation of the the number of register tokens used with a DINOv2 model. (top): qualitative visualization of artifacts appearing as a function of number of registers.(bottom): performance on three tasks (ImageNet, ADE-20k and NYUd) as a function of number of registers used. While one register is sufficient to remove artefacts, using more leads to improved downstream performance.

Figure 10 :

 10 Figure 10: Feature norms along locations: proportion of tokens with norm larger than the cutoff value at a given location. Left: official DINOv2 model (no antialiasing), right: our models (with antialiasing). At some positions, more than 20% of tokens have a high norm.

Figure 11 :

 11 Figure 11: Propagating unit gradients through a bicubic interpolation (16 × 16 → 7 × 7) without antialiasing. We observe a striping pattern similar to the one of Fig. 10 (left).

Figure 12 :

 12 Figure 12: Increase in model parameter and FLOP count when adding different numbers of registers.Adding registers can increase model FLOP count by up to 6% for 16 registers. However, in the more common case of using 4 registers, that we use in most of our experiments, this increase is below 2%. In all cases, the increase in model parameters is negligible.

Figure 13 :

 13 Figure 13: Illustration of the intermediate computations in the LOST algorithm for all models. Adding registers drastically improves the look of all intermediate steps for DeiT-III and DINOv2. The difference is less striking for the CLIP model.

  86.0 66.4 87.3 99.4 94.5 91.3 96.9 91.5 85.2 99.7 94.7 96.9 78.6 89.1 normal 65.8 53.1 17.1 97.1 81.3 18.6 73.2 10.8 63.1 59.5 74.2 47.8 37.7 70.8 outlier 69.0 55.1 79.1 99.3 93.7 84.9 97.6 85.2 84.9 99.6 93.5 94.1 78.5 89.7

	Table

Artifacts hold global information. In order to evaluate how much global information is gathered in the high-norm tokens, we propose to evaluate them on standard image representation learning IN1k P205 Airc. CF10 CF100 CUB Cal101 Cars DTD Flow. Food Pets SUN VOC

[CLS]

Table 2b

 2b

		ImageNet ADE20k NYUd
		Top-1	mIoU	rmse ↓
	DeiT-III	84.7	38.9	0.511
	DeiT-III+reg	84.7	39.1	0.512
	OpenCLIP	78.2	26.6	0.702
	OpenCLIP+reg	78.1	26.7	0.661
	DINOv2	84.3	46.6	0.378
	DINOv2+reg	84.8	47.9	0.366
				),

Table 3 :

 3 Unsupervised Object Discovery using LOST

		VOC 2007 VOC 2012 COCO 20k
	DeiT-III	11.7	13.1	10.7
	DeiT-III+reg	27.1	32.7	25.1
	OpenCLIP	38.8	44.3	31.0
	OpenCLIP+reg	37.1	42.0	27.9
	DINOv2	35.3	40.2	26.9
	DINOv2+reg	55.4	60.0	42.0

https://github.com/facebookresearch/deit

https://github.com/mlfoundations/open_clip

https://github.com/facebookresearch/dinov2

additional tokens to the input sequence that are not used as outputs, and have found that this entirely removes the artifacts, improving the performance in dense prediction and object discovery. Moreover, we have shown that the proposed solution also removes the same artifacts present in supervised models such as DeiT-III and OpenCLIP, confirming the generality of our solution.